Record whether a problem has a forum thread

[Chessing234]

Closes #370.

data/problems.yaml carried no signal that a problem has discussion on erdosproblems.com. The export is the natural entry point for bulk or automated work, so anyone starting there is routed away from where the subject-matter discussion actually is — which is what the issue was filed about.

scripts/update_forum_status.py parses the forum index, which lists every thread on a single page, and records per problem:

- number: "12"
  forum:
    posts: 13
    proof_claims: 2
    proof_claim_posts: 2

forum is derived, like formalized, and a problem with no discussion carries no key at all — so the field's presence is itself the signal. The script follows update_formalization_status.py: same YAML round-trip, same --check mode as derive_status.py.

What the index contains, and what is counted

The index has 1153 thread-item blocks, of three kinds:

• 901 problem threads, coded by the bare problem number → posts.
• 235 proof claims, coded <number>/proof-claims#proof-claim-<id>, spread over 123 problems. These are tied to a problem but are a distinct object — a claimed proof, plus whatever discussion it drew — so they are counted separately rather than folded into posts: proof_claims is how many claims the problem has, proof_claim_posts how many posts were made in reply to them. The index badges a claim only once it has replies (78 of the 235 carry a badge), so an unanswered claim counts in proof_claims and contributes nothing to proof_claim_posts; the claim itself is never counted as a post.
• 17 threads belonging to no problem: 7 blog threads (blog:N) and 10 named topic threads (Site suggestions, Formalisation, AI Contributions, and so on). Both kinds are skipped.

Counting the proof claims matters for the issue: 13 problems have a proof claim and no thread of their own — 117, 119, 131, 319, 416, 424, 450, 538, 634, 653, 882, 959, 1059 — and under a problem-threads-only reading the export would still report no discussion for them. A further 110 have both.

Only counts are recorded. The issue suggested last_post too, but the index dates each thread relatively ("a month ago"), so an exact date would mean fetching all ~900 threads individually. That is a much heavier job for a much smaller gain, and I would rather leave it out than approximate it. Easy to add later if you want it.

The workflow is workflow_dispatch only. Adding a schedule is your call, not something a contributor should switch on for you.

Evidence

Run against the live index today:

Found 901 problem threads and 235 proof claims across 914 problems, carrying 5435 posts.
✅ Updated forum thread counts on 125 entries in problems.yaml.

(5191 posts in problem threads, 244 in reply to proof claims.) The issue measured 894 threads / 5121 posts on 2026-07-30, three weeks earlier — consistent growth, which is the cross-check I wanted before trusting the parse. Every thread and proof-claim code maps to a problem in the export, with no orphans, and the export's 1217 problems match the issue's count exactly.

After the update: scripts/validate.py is ✅ Validation OK., derive_status.py --check is up to date, generate_readme.py reports the README already up to date (no column was added), and a second update_forum_status.py --check reports no drift, so the script is idempotent.

The data change is purely additive apart from two post counts that moved with the live index since the first commit.

The schema gains a forum entry, needed because the schema is additionalProperties: false. Each of the three keys is an integer >= 1 and is omitted when its count is zero, with minProperties: 1 so an empty forum is rejected; additionalProperties: false still rejects an unexpected last_post.

Uncertainty

The script parses HTML with a regex over <li class="thread-item"> blocks. That is fine for a page this regular and avoids a new dependency, but it will need a touch if the forum markup changes. It fails loudly rather than quietly if it parses nothing — a zero-thread result is treated as "the markup moved", not as "no problem has a thread", so a rendering change can never strip the field from every entry. That guard does not catch a partial failure, though: the script deletes forum for any problem missing from the listing, which assumes the index really is one page. There is now a comment on the delete path saying so.

This was AI-assisted.

[teorth]

Reviewed. The engineering here is in good shape: the data change is purely additive (exactly 901 forum:/posts: pairs, placed between formalized and tags), validate.py passes, derive_status.py --check is clean, the README is untouched, the workflow is workflow_dispatch only, and the ruamel round-trip settings match the rest of the repo. I re-ran your parser against the live index and reproduced your figures exactly — 901 threads, 5189 posts. Thanks also for disclosing the AI assistance.

One thing to resolve before this goes in, about coverage rather than correctness.

The PR says non-problem threads are "the site's own blog:N codes". That undercounts what is being skipped. Of the 1152 <li class="thread-item"> blocks on the index, 251 are skipped, and only 7 of those are blog:. 234 are proof-claim threads, coded like:

411/proof-claims#proof-claim-217

Those are tied to a specific problem, so code.isdigit() drops them. The remaining ~10 are named topic threads (AI Contributions, Site feedback, and so on), which are correctly excluded.

What that costs, measured against today's index:

• 122 problems have proof-claim threads.
• 13 of them have no plain thread at all — so they gain no forum key, and the export still reports no discussion for a problem that has some. That is the case issue data: problems.yaml carries no signal that a problem has a forum thread (894 of 1217 do) #370 was filed about. They are: 117, 119, 131, 319, 416, 424, 450, 538, 634, 653, 882, 959, and one more.
• 109 more have both, so their posts is understated. 245 proof-claim posts go unrecorded overall.

I don't think this is obviously a bug — there's a reasonable position that a proof claim is a distinct object and shouldn't be folded into a single posts count. But a reviewer reading the current description would believe coverage was complete, and it isn't, so it's worth settling explicitly.

Three options, roughly in increasing order of work:

1. Leave the behaviour as-is and correct the description and the parse_forum_index docstring to say that proof-claim threads are excluded too, and why.
2. Count proof-claim posts into posts, so the field means "posts on this problem across the forum".
3. Record them separately, e.g. forum: {posts: N, proof_claims: M}.

My inclination is (1) or (3) — folding them into one number loses a distinction that looks worth keeping. Your call, and Terry's.

One smaller note: unlike update_formalization_status.py, this script deletes forum for any problem absent from the listing. The zero-thread guard catches a total parse failure, but not a partial one — if the index ever paginates, problems below the fold would silently lose the field. Worth a comment noting the single-page assumption that guard depends on.

[Chessing234]

Thanks — you were right that the description undercounted the skips, and I went with option (3) in 8b6701f.

forum now carries up to three keys, each omitted when zero:

forum:
  posts: 13
  proof_claims: 2
  proof_claim_posts: 2

One thing I had to settle to make (3) work: the index badges a proof-claim item only once it has replies — 78 of the 235 carry a title="N posts" badge, and the badged wording ("(1 post)", "to a claim of …") makes clear it counts replies to the claim, not the claim itself. So proof_claims counts claims and proof_claim_posts counts replies, and an unanswered claim contributes 1 and 0. That is why the two numbers are separate rather than one figure.

Reproducing your measurements on today's index: 1153 items, 901 problem threads, 235 proof claims over 123 problems, 7 blog:, 10 named topic threads. The 13 problems whose only discussion is a proof claim come out as your list plus 1059, which I take to be your "one more". They now gain a forum key with proof_claims and no posts, so the schema no longer requires posts — instead each key is integer >= 1 with minProperties: 1, and additionalProperties: false is unchanged.

On the smaller note: the delete path now carries a comment saying it assumes the index is a single page, and that the zero-thread guard only catches a total parse failure, not a partial one.

The description and the parse_forum_index docstring are both rewritten to describe all three kinds of item and how each is counted, so option (1)'s correction is in there too. validate.py, derive_status.py --check and a second update_forum_status.py --check are all clean after the regeneration; the data diff is +202/-2, the two deletions being post counts that moved with the live index since the first commit.