Skip to content

Rate-limit faceted listings under every locale prefix - #192

Merged
HarryCordewener merged 1 commit into
mainfrom
claude/mu-index-outage-0e3dbc
Sep 21, 2026
Merged

HarryCordewener merged 1 commit into
mainfrom
claude/mu-index-outage-0e3dbc

Conversation

@HarryCordewener

Copy link
Copy Markdown
Member

What happened (2026-09-21, ~03:00Z onward)

mu-index.com returns Cloudflare 522. The web container is fine (/health 2 ms from inside it); Traefik is the failure.

  • A crawler is walking combinatorial facet URLs, ~1,250 req/min through Cloudflare.
  • 44,825 of ~75,600 requests in the last hour were /zh-Hans/games?…, and every one went to mui-site@docker, not mui-games-protection@file.
  • The protection router's path regex ^/([a-z]{2}(-[A-Za-z]{2})?/)?games/?$ only admits a two-letter subtag. zh-Hans has four, so the Chinese-locale flood skipped the per-client, per-host and in-flight limits entirely.
  • Traefik sits at 511.7 / 512 MiB, thrashing (1.19 TB block read), was OOM-killed by its cgroup on 09-15 and 09-17, and its active healthcheck against web-2 times out, so the only backend is marked down → 503 → 522.

Change

Prefix is now ^/([a-z]{2,3}(-[A-Za-z0-9]{1,8})*/)?games/?$, which matches /games, /de/games, /zh-Hans/games, /qps-ploc/games, /ru-x-canary/games and still refuses /games/<slug> and /account/games.

Deploying

The file provider watches deploy/traefik, so the router changes on git pull --ff-only in /opt/muindex. Traefik also needs docker restart muindex-traefik-1 to shed the heap it is currently thrashing in.

🤖 Generated with Claude Code

…etter ones

The flood-protection router matched `^/([a-z]{2}(-[A-Za-z]{2})?/)?games/?$`,
so `/zh-Hans/games?...` (a four-letter script subtag) never reached it and fell
through to the unprotected `mui-site` router. On 2026-09-21 that was 44,825 of
~75,600 requests in an hour: a crawler walking facet combinations under the
Chinese locale, with no per-client, per-host or in-flight bound in front of it.

Traefik sat at its 512 MiB cgroup ceiling (OOM-killed on 09-15 and 09-17 as
well), its active healthcheck against web-2 timed out while web-2 itself
answered /health in 2 ms, the only backend was marked down, and the site
served 503 and then Cloudflare 522.

The prefix now admits any BCP 47-shaped tag (`zh-Hans`, `qps-ploc`,
`ru-x-canary`) and still refuses `/account/games` and `/games/<slug>`.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
@coderabbitai

coderabbitai Bot commented Sep 21, 2026

Copy link
Copy Markdown

Review Change StackReview Change Stack

Note

Currently processing new changes in this PR. This may take a few minutes, please wait...

⚙️ Run configuration

Configuration used: Organization UI

Review profile: ASSERTIVE

Plan: Essentials

Run ID: 207e46a9-3e26-4f8f-885f-9dc6e477947d

📥 Commits

Reviewing files that changed from the base of the PR and between 1d62f79 and b60ba84.

📒 Files selected for processing (1)
  • deploy/traefik/incident-limits.yml

Comment @coderabbitai help to get the list of available commands.

@HarryCordewener
HarryCordewener merged commit 4c75764 into main Sep 21, 2026
2 of 3 checks passed
@HarryCordewener
HarryCordewener deleted the claude/mu-index-outage-0e3dbc branch September 21, 2026 04:07
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant