Expose benchmark code, task IDs, and level-aggregation spec for the website data sync

[atharvas]

Expose benchmark code, task IDs, and level-aggregation spec for the website data sync

Context

The FormulaCode website (github.com/formula-code/formula-code.github.io, currently www/ in the monorepo) rebuilds remote/website_data/website_data.csv and seven derived data files via tasks/process_remote_data.py. The CSV today is a manually-curated input — 25,196 rows × 13 columns, last refreshed by hand.

We've wired up an npm run sync script that orchestrates a full refresh from api.formulacode.org. Step 2 of that script (tasks/fetch_remote_website_data.py) pulls the source rows that process_remote_data.py then fans out into the seven derived files. The fetcher is a skeleton today because three columns it needs aren't exposed by any Supabase table at api.formulacode.org/rest/v1/. All three are owned by datasmith.

This issue tracks closing those gaps so npm run sync becomes a true end-to-end refresh.

What's needed

1. benchmark_codes — a new public table or view

The CSV column carries the Python source of every benchmark function indexed by its fully-qualified name. Example value:

{
  "benchmarks.TimeArithmetic.time_abs": "class TimeArithmetic:\n    def time_abs(self, shape, axis):\n        execute(self.df.abs())\n\ndef setup(*args, **kwargs): ..."
}

This is the largest column in the CSV — ~1.3 MB of the 3.8 MB CSV — and powers the player page's "Benchmark code" tabs (src/routes/player/[...recordingPath]/+page.svelte).

Proposed shape: a benchmark_codes table keyed by (owner, repo, benchmark_without_params):

column	type	notes
owner	text	from the upstream repo
repo	text	from the upstream repo
benchmark_without_params	text	e.g. benchmarks.TimeArithmetic.time_abs
source	text	Python source of the benchmark function + its setup
last_scraped	timestamptz	when datasmith last refreshed it

If a benchmark function moves or is deleted upstream, keep the row but mark it stale (source may be from an older commit). The website can fall back gracefully.

Datasmith already scrapes ASV bench files during the synthesizer stage (src/datasmith/agents/synthesizer.py). The proposal is to persist the per-function source it parses, instead of throwing it away.

2. task_id mapping

The CSV column is the pipeline-internal task ID, e.g. modin_project-modin_1, pandas_dev-pandas_15, networkx_networkx_8. Construction looks like <owner_dashes_to_underscores>-<repo>_<seq>, where <seq> is a sequential index assigned by the pipeline.

This ID is the canonical join key between the website's rows and an agent recording path (see fc-eval issue) — and it's not currently in any API table. The closest thing is candidate_containers's (owner, repo, sha, issue_number) composite key.

Proposed shape — two options, pick whichever is least disruptive:

Option A (smaller change): add a task_id column to candidate_containers. Each container row already corresponds 1:1 with a website task; the column just exposes the sequence.

Option B (cleaner separation): new tasks table keyed by task_id, with (owner, repo, issue_number, container_sha, created_at) columns. candidate_containers references it via foreign key.

Either works. The website only needs to resolve (owner, repo, issue_number) → task_id.

3. Level-aggregation rule (documentation, not data)

The CSV has five level values: 0-None, 1-Params, 2-Func, 3-Class, 4-Module. Level-0 rows are per-workload measurements; levels 1–4 are aggregations grouped by progressively-coarser benchmark name keys:

• L1 (Params) — benchmark_name with the trailing (...) stripped
• L2 (Func) — drop the last dot-segment
• L3 (Class) — drop the next
• L4 (Module) — top-level module only

The website can compute these aggregations locally inside fetch_remote_website_data.py, but we need to know the aggregation statistic used today (geometric mean? arithmetic mean? median?) so the new rows match the existing CSV's numbers.

Ask: point us at the reference implementation in datasmith (file path + function name) or paste pseudocode here. One line of confirmation is enough.

4. benchmark_type — optional, nice-to-have

CSV column carrying time, mem, etc. We can derive this locally from the ASV class prefix (Time*, Mem*), but it's cleaner if datasmith emits it as part of benchmark_information (or wherever the per-workload metadata lands) so we don't maintain a parser.

Acceptance criteria

• api.formulacode.org/rest/v1/benchmark_codes returns rows with the schema above
• task_id is queryable from api.formulacode.org, either as a column on candidate_containers or via a new tasks table
• Datasmith repo (or this issue) documents the level-aggregation statistic so the website can replicate it locally
• (Optional) benchmark_type exposed alongside the per-workload data

Once these land, the website's tasks/fetch_remote_website_data.py becomes ~50 lines: fetch benchmark_information + benchmark_codes + tasks, join, compute levels locally, write remote/website_data/website_data.csv. The downstream process_remote_data.py already exists and works.

Related

• fc-eval issue: agent recording paths + oracle agent_id format (separate scope; both issues block the same npm run sync use case)
• Skeleton fetcher with the full per-column reconciliation: tasks/fetch_remote_website_data.py in the website repo