feat: add paged findKeys (memory-bounded iteration)

[JohnMcLear]

Summary

• Add Database#findKeysPaged(key, notKey, {limit, after}) returning up to limit keys sorted ascending; pages by passing the last returned key as after
• Native implementations on mysql (BINARY \key` > ? ORDER BY BINARY `key` LIMIT ?) and **postgres** (key > $n ORDER BY key LIMIT $m`)
• Other backends fall back to findKeys + JS slicing via CacheAndBufferLayer — correct, but defeats the OOM-mitigation benefit for very large keyspaces
• Bumps to 6.1.0

Why

Drives the fix for ether/etherpad#7830, where SessionStore._cleanup() loaded every sessionstorage:* key into memory at once and OOMed on decade-old MariaDB installs with millions of stale sessions. The reporter's heap snapshot pinned retention on _pool → _allConnections → … → _command._rows → keys — the unbounded result set dragged the process out of heap.

Test plan

• pnpm exec vitest run test/memory test/dirty — 200 passed (exercises the layer fallback)
• pnpm exec vitest run test/mysql — 100 passed (native paging path, all 4 new findKeysPaged cases green)
• pnpm exec vitest run test/postgres — 101 passed (native paging path)
• pnpm run ts-check clean
• pnpm run lint clean

🤖 Generated with Claude Code

[qodo-code-review]

Qodo reviews are paused for this user.

Troubleshooting steps vary by plan Learn more →

On a Teams plan?
Reviews resume once this user has a paid seat and their Git account is linked in Qodo.
Link Git account →

Using GitHub Enterprise Server, GitLab Self-Managed, or Bitbucket Data Center?
These require an Enterprise plan - Contact us
Contact us →

[qodo-free-for-open-source-projects]

Review Summary by Qodo

Add paged findKeys API for memory-bounded iteration

✨ Enhancement

Walkthroughs

Description

• Add memory-bounded findKeysPaged() API for paginated key iteration
• Native implementations on MySQL and PostgreSQL with cursor-based paging
• Fallback to in-memory slicing for backends without native support
• Resolves OOM issue in SessionStore cleanup with millions of stale keys

Diagram

flowchart LR
  A["Database#findKeysPaged<br/>key, notKey, limit, after"] --> B{Backend Type}
  B -->|MySQL| C["BINARY key > after<br/>ORDER BY BINARY key LIMIT"]
  B -->|PostgreSQL| D["key > after<br/>ORDER BY key LIMIT"]
  B -->|Other| E["findKeys + JS slice<br/>via CacheAndBufferLayer"]
  C --> F["Paginated Results"]
  D --> F
  E --> F

Loading

File Changes

1. databases/mysql_db.ts ✨ Enhancement +26/-0

Add MySQL native paged key iteration

• Implements native findKeysPaged() with BINARY-safe cursor-based pagination
• Uses BINARY key > ? comparison for deterministic paging across case-insensitive collations
• Validates limit parameter and constructs parameterized SQL query
• Returns sorted keys up to the specified limit

databases/mysql_db.ts

---

2. databases/postgres_db.ts ✨ Enhancement +31/-0

Add PostgreSQL native paged key iteration

• Implements callback-based findKeysPaged() for PostgreSQL
• Uses parameterized queries with dynamic parameter numbering
• Supports cursor-based pagination via key > $n comparison
• Maintains consistency with existing callback pattern

databases/postgres_db.ts

---

3. index.ts ✨ Enhancement +14/-0

Expose findKeysPaged in public API

• Exposes findKeysPaged() method on public Database class
• Includes JSDoc documentation explaining pagination semantics
• Delegates to underlying database implementation

index.ts

---

View more (4)

4. lib/AbstractDatabase.ts 📝 Documentation +6/-0

Document findKeysPaged override expectations

• Adds documentation comment explaining backend override expectations
• Notes that backends with large keyspaces should implement native findKeysPaged()
• Clarifies that CacheAndBufferLayer fallback defeats OOM-mitigation purpose

lib/AbstractDatabase.ts

---

5. lib/CacheAndBufferLayer.ts ✨ Enhancement +43/-1

Add CacheAndBufferLayer paging support

• Adds findKeysPaged() to InternalDB type definition
• Updates promisification loop to include findKeysPaged for legacy callback-based backends
• Implements fallback findKeysPaged() using binary search to find cursor position
• Falls back to in-memory findKeys() + slicing when backend lacks native implementation

lib/CacheAndBufferLayer.ts

---

6. test/lib/test_lib.ts 🧪 Tests +68/-0

Add findKeysPaged test coverage

• Adds comprehensive test suite for findKeysPaged() with 5 test cases
• Tests empty results, single page, limit enforcement, cursor-based pagination
• Tests notKey exclusion across multiple pages
• Uses isolated prefixes to prevent test interference

test/lib/test_lib.ts

---

7. package.json ⚙️ Configuration changes +1/-1

Bump version to 6.1.0

• Bumps version from 6.0.4 to 6.1.0 (minor version increment)

package.json

---

[qodo-free-for-open-source-projects]

Code Review by Qodo

🐞 Bugs (0) 📘 Rule violations (0) 📎 Requirement gaps (0)

1. Missing limit validation ✓ Resolved 🐞 Bug ≡ Correctness

Description

CacheAndBufferLayer.findKeysPaged() does not validate options.limit before slicing or
delegating, so invalid limits (0/negative/NaN/floats) can silently return empty pages and behave
differently than MySQL/Postgres which reject them. This can break typical paging loops (e.g., `while
(page.length === limit)`) and lead to infinite loops or silent data loss.

Code

lib/CacheAndBufferLayer.ts[R341-365]

Evidence

The wrapper implements a JS-slicing fallback that uses options.limit without validation, while
both native implementations explicitly reject invalid limits. This creates observable behavior
differences across backends and makes invalid inputs silently succeed in the fallback path.

lib/CacheAndBufferLayer.ts[341-365]
databases/mysql_db.ts[158-165]
databases/postgres_db.ts[154-162]

Agent prompt

The issue below was found during a code review. Follow the provided context and guidance below and implement a solution

## Issue description
`CacheAndBufferLayer.findKeysPaged()` lacks input validation for `options.limit`. In the fallback path it uses `options.limit` directly in `slice()`, and in the non-fallback path it delegates to the backend without guaranteeing consistent validation.
MySQL/Postgres implementations explicitly reject non-positive/non-integer limits, so callers get inconsistent behavior depending on backend and/or whether the fallback is used.
## Issue Context
This API is intended for deterministic, safe iteration over large keyspaces. Returning an empty page for an invalid limit (instead of throwing) can break pagination loops and can cause silent failures.
## Fix Focus Areas
- lib/CacheAndBufferLayer.ts[341-366]
## Suggested fix
Add the same guard used by MySQL/Postgres at the start of `CacheAndBufferLayer.findKeysPaged()` (before doing any work), e.g.:
- if `!options || !Number.isInteger(options.limit) || options.limit <= 0`, throw `new Error('findKeysPaged requires a positive integer limit')`.
Optionally also add the same validation in `index.ts` `Database.findKeysPaged()` to enforce the contract at the public API boundary.

ⓘ Copy this prompt and use it to remediate the issue with your preferred AI generation tools

---

[JohnMcLear]

Addressed the missing-limit-validation bug in 6e80e77 — now throws on non-positive/non-integer limits, matching the mysql/postgres native paths so behaviour is uniform. Backend tests (memory + dirty + mysql, 300 cases) still green locally.