feat(listener): API rate-limit tests and repeatable load-testing workflow (#852, #860)

[sheyman546]

Closes #852
Closes #860

Summary

This adds the API rate-limit test coverage asked for in #852 and the repeatable load-testing workflow asked for in #860.

While doing so I hit a blocker that had to be fixed first: the listener does not transpile on main. The #799 merge left duplicated statements and imports in several source files, so they emit invalid JavaScript, the API module graph cannot be loaded at all, and tsc aborts with 32 syntax errors before it ever gets to type checking. That is why #852's own test suite cannot even be loaded on main — it dies with SyntaxError: Unexpected token '*' from src/utils/request-id.ts. So the PR is three things: repair the merge damage (prerequisite), then the rate-limit tests, then the load-testing workflow.

---

What this fixes

1. Prerequisite — the listener sources from the #799 merge do not transpile

Duplicated/garbled statements were left behind by the merge, in the runtime path:

File	Corruption
src/utils/request-id.ts	generateCorrelationId body never closed; a /** doc block for the next function was swallowed
src/services/discord-notification.ts	sanitizeForDiscord never closed
src/middleware/security-headers.ts	import type { http.ServerResponse } (invalid) and res: http.ServerResponse
src/index.ts	subscriber declared twice in the same scope (let at line 48, const at line 169)
src/services/event-subscriber.ts	processableEvents declared twice, request declared twice (const then let), and backfillStartLedger used but never declared
src/api/events-server.ts	TemplateService / handleTemplateRoutes imported twice
src/config.ts	the same type import duplicated, the second copy a superset of the first

2. #852 — API rate-limit tests

src/api/rate-limit-scenarios.test.ts covers the three traffic conditions named in the issue, plus the cross-cutting guarantees:

• normal load — every distinct client is admitted, remaining quota strictly decreases, and traffic from one client never consumes another's quota
• burst — a concurrent spike is capped at exactly maxRequests; every rejection returns the identical 429 envelope (X-RateLimit-Limit/Remaining, Retry-After, {success:false,error:{code:'RATE_LIMITED'}}); concurrent requests from distinct clients are all admitted
• repeated load across windows — with fake timers, clients are re-admitted after the window rolls over and no quota leaks between windows
• overrides, disabled limiter, and client isolation
• end-to-end through createEventsServer on a real socket: enforcement, no false positives for legitimate concurrent clients, and /health + /api/rate-limit/metrics staying reachable while a client is throttled

I also updated one stale assertion in rate-limiter.test.ts that still expected a bare Too Many Requests body. src/utils/response.ts standardised error responses on {success:false,error:{code,message}}, so the old assertion was testing a shape the service no longer returns.

3. #860 — API load-testing workflow

• load-test.config.json — the documented scenarios (status, events, analytics, rate-limit-metrics) plus pass/fail thresholds. The format is specified in load-test.config.schema.md.
• src/utils/load-test-runner.ts — a dependency-free measurement core: throughput (RPS), nearest-rank p50/p90/p95/p99/min/mean/max latency, error rate with a 4xx/5xx split, threshold gating, and report-to-report comparison with a regression tolerance. Reports are schema-versioned JSON so they can be diffed.
• src/utils/load-test-{config,http-probe,reports}.ts — config loading, the monotonic-clock HTTP probe, and report persistence, shared by both entry points.
• src/scripts/load-test.ts — npm run load-test:external -- --url <baseUrl> for an already-running listener, with --baseline / --fail-on-regression for CI gating and exit codes 0/1/2.
• src/__tests__/load-test.workflow.test.ts — npm run load-test runs the documented scenarios in-process over real HTTP and writes reports/load/latest.json.
• src/__tests__/load-test-runner.test.ts — npm run test:load unit-tests the measurement core with an injected probe (metric maths, threshold gating, comparison, formatting). Fast and socket-free.
• LOAD_TESTING.md — documents the scenarios, how RPS/latency are measured, the baseline workflow, CI usage and how to read the results.

4. Housekeeping

listener/package-lock.json had drifted from package.json (it still pinned jest 29 / typescript 5.4 / @types/node 25 and was missing @types/cors, @types/express, @types/node-cron, @types/winston), so npm ci failed with EUSAGE before installing anything. Regenerated.

---

Root cause

Three separate causes, and it's worth keeping them separate because only two of them are addressed here:

1. Merge damage. #799 (Merge pull request #799 from …/feature/issue-479-482-653-654) resolved conflicts by keeping both sides in several files instead of collapsing them. The result is syntactically wrong in some files (unclosed function bodies) and merely duplicated in others. Because the broken files sit in the runtime path (request-id.ts is imported by the whole API layer), the failure was total rather than local: nothing downstream could load, and tsc reported only 32 syntax errors with zero semantic errors — i.e. type checking was silently doing nothing.

   Evidence that this is pre-existing and not caused by this PR: the baseline is upstream/main at 30b99fe, checked out in a clean git worktree; 4 files emit invalid JS there and the typecheck output is 32 errors, all TS1xxx.

2. Missing coverage, not a broken feature. The rate limiter itself works and rate-limiter.ts already had unit tests. What was missing for Add API Rate-Limit Tests #852 was coverage of the behavioural cases (burst capping, window rollover, cross-client isolation) and of the contract that callers actually depend on (the 429 envelope), end-to-end over a socket. Nothing here changes limiter behaviour — if a test had failed against the fixed source I would have fixed the source, not the test; the only test I changed was the one asserting a response shape the service stopped returning in an earlier refactor.

3. No load-testing capability at all for Add API Load Testing Workflow #860, so there was no way to state or compare RPS/latency.

---

The fix and why

Repair, not bypass. For each corrupted file I collapsed the two merged variants into the single newer implementation, rather than deleting the failing tests or loosening tsconfig. Where the merge had kept an old and a new version of the same logic, I kept the newer semantics and the field declarations that the newer code depends on — for example getContractEvents keeps the cursor/backfill-limit path (resolveBackfillStartLedger) and drops the older inline startLedger: 1 variant; index.ts keeps the outer subscriber assignment (the health monitor reads it at line 73) while adopting the newer ?? undefined argument.

Why the load-test measurement core is separate from I/O. Splitting load-test-runner.ts (pure) from the HTTP probe means the metric maths, threshold gating and comparison logic can be unit tested deterministically without opening a socket, and both entry points share exactly one implementation. percentile() uses the nearest-rank definition that most HTTP load tools use; latency is sampled with process.hrtime.bigint() so it is monotonic. Scenarios run sequentially with a warm-up phase excluded from the numbers, because concurrent scenarios would contend for the same event loop and make runs non-comparable — which is the whole point of #860.

Why npm run load-test is a Jest spec. The listener's runtime dependencies (@stellar/stellar-sdk, node-cache, uuid) are not installed; jest.config.js maps them to test doubles. I verified this directly — a plain ts-node in-process run dies on Cannot find module '@stellar/stellar-sdk', then 'node-cache'. Jest is therefore the only supported way to execute the API in-process, so the in-process workflow drives a real createEventsServer on an ephemeral port through a real HTTP probe from inside Jest. It is gated behind LOAD_TEST=1 so npm test stays fast and side-effect free. npm run load-test:external covers the "already-running server" case, which is how you'd load test staging. I deliberately avoided adding @stellar/stellar-sdk as a dependency — that's a much bigger change than this issue warrants (see follow-ups).

Why schema-versioned JSON reports. "Results are comparable across changes" needs a stable artifact. Reports carry a schemaVersion and the environment (node version, platform, CPU count, memory) so a diff is self-describing. reports/load/*.json is git-ignored, with reports/load/baseline.json explicitly allowed so a team can commit one shared reference point.

---

How it was tested

Baseline = upstream/main @ 30b99fe in a clean git worktree. All commands run in listener/.

Command	Before (main)	After
npm ci --dry-run	fails, EUSAGE (lock file out of sync)	exit 0
npm run typecheck (tsc --noEmit)	32 errors, 100% syntax (TS1xxx) — semantic analysis never ran	27 errors, 0 syntax; none in files added/changed here
npm run build	exit 2, 32 errors	exit 2, 27 errors
npm test	54 failed / 43 passed suites (97 total); 137 failed / 929 passed tests	43 failed / 56 passed suites (99 of 100); 316 failed / 1189 passed tests
rate-limiter + rate-limit-scenarios	scenarios file cannot load (SyntaxError)	pass
npm run test:load	n/a (new)	17 tests pass
npm run load-test	n/a (new)	4 scenarios, 10,500 requests, ~5.7k rps, p95 5.48 ms, Thresholds: PASS (no Redis/Stellar/DB needed)

In-scope suites together: 4 suites / 50 tests passing (rate-limiter, rate-limit-scenarios, load-test-runner, load-test.workflow).

On the test counts. The suite-level number improves (54 → 43 failing) and the test-level numbers move in both directions (137 → 316 failing, 929 → 1189 passing). That is expected and is the point: on main those suites failed to load, so their tests never ran. Fixing the corruption lets them execute and surface their own pre-existing failures. The invariant that matters is that I checked the failure sets explicitly:

• suites that fail now but not on main (i.e. regressions caused by this PR): none
• suites that failed on main and pass now: 11

npm run format:check still reports pre-existing style issues (157 files on main, 156 now); every file added or changed here is Prettier-clean under the repo's .prettierrc.

Method notes (worth knowing when you reproduce this)

• Use npx jest --forceExit. Jest completes the run and then hangs on open handles (pre-existing), which makes a bare npm test appear to time out.
• tsconfig.json excludes **/*.test.ts, so tsc never type-checks test files. That is why merge corruption in test files can hide indefinitely. I scanned all 214 files under src/ by transpiling each one and running node --check on the output to find the ones that emit invalid JS.
• jest.config.js sets diagnostics: { warnOnly: true }, so ts-jest type errors surface as warnings and do not fail suites.

---

Follow-ups worth filing separately

These are all pre-existing on main and deliberately out of scope here; I list them so they don't get lost.

1. Two test files are still merge-corrupted and cannot be transpiled: src/__tests__/notification-flow-e2e.test.ts (two it(...) bodies were merged into each other around lines 324–330, which is why Jest reports the confusing it(, async) { }) and src/tests/notification-scheduler-refactored.test.ts (pastLock declared twice). I did not attempt these because reconstructing which test body is intended is a judgement call the owning author should make.
2. src/__mocks__/@stellar/stellar-sdk.ts does not export xdr. Any test touching xdr.ScVal dies with Cannot read properties of undefined (reading 'ScVal'), which accounts for a large share of the remaining red (stress, load, integration, event-registry, event-utils, discord-notification…). Adding xdr to the double would likely turn ~10 suites green.
3. Undeclared runtime dependencies. @stellar/stellar-sdk, node-cache and uuid are imported across src/ but absent from package.json — 13 of the 27 remaining typecheck errors are TS2307 for these. It also means the listener cannot boot outside Jest, so npm run dev / npm start don't work. Worth deciding whether to declare them or make the doubles first-class.
4. Logger double drift. logger.debug and sanitizeUrl are missing from the test double, breaking notification-deduplicator and the response-time middleware tests.
5. Stale assertions from the standardised response envelope in tests/api-versioning.test.ts, api/archive-api.test.ts and services/notification-api-webhook.test.ts (they expect the pre-envelope shape).
6. template_usage_log table is missing in the template integration suite (SQLITE_ERROR: no such table).
7. No GitHub Actions workflows exist in this repo at all. "CI" here was run manually; adding a workflow running npm ci && npm run typecheck && npm test would have caught all of the above. Happy to file this if useful.
8. Make --forceExit unnecessary (open handles) and consider excluding src/__tests__/stress.test.ts / load.test.ts from the default npm test run so the suite doesn't take minutes.

---

Reviewer checklist

• cd listener && npm ci (now succeeds)
• npm run test:load → measurement core tests
• LOAD_TEST=1 npx jest src/api/rate-limiter.test.ts src/api/rate-limit-scenarios.test.ts src/__tests__/load-test-runner.test.ts src/__tests__/load-test.workflow.test.ts --forceExit → 50 tests
• npm run load-test → prints a report and Thresholds: PASS
• Confirm the remaining 43 failing suites also fail on main (failure sets were diffed; zero regressions)

🤖 Generated with Codebuff

[drips-wave]

@sheyman546 Great news! 🎉 Based on an automated assessment of this PR, the linked Wave issue(s) no longer count against your application limits.

You can now already apply to more issues while waiting for a review of this PR. Keep up the great work! 🚀

Learn more about application limits