All issues

Search results

EvalPort adapter for ClawBench's run-meta.json / rescore-summary.json results

enhancementNew feature or requestNew feature or request

Status: Open.

#322 In TIGER-AI-Lab/ClawBench;

· adhabnr-ux opened on Aug 23, 2026

Proposal: isolated academia-registry-v0 suite for evidence-grounded registry research

questionFurther information is requestedFurther information is requested

Status: Open.

#317 In TIGER-AI-Lab/ClawBench;

· reacher-z opened on Aug 20, 2026

run_support/config.py resolves a container engine at import time, so any module importing it inherits a hard Docker dependency

questionFurther information is requestedFurther information is requested

Status: Open.

#315 In TIGER-AI-Lab/ClawBench;

· vaibhavdabas16 opened on Aug 20, 2026

feat(eval): support DeepSeek Harness and run a reproducible ClawBench benchmark

enhancementNew feature or requestNew feature or request

Status: Open.

Feature

#309 In TIGER-AI-Lab/ClawBench;

· reacher-z opened on Aug 19, 2026

One truncated run-meta.json aborts end-of-batch stats before batch-summary.json is written; metadata writes are not atomic

Status: Open.

#303 In TIGER-AI-Lab/ClawBench;

· reacher-z opened on Aug 18, 2026

A missing judge model kills the process after the agent run and loses run-meta.json entirely (SystemExit escapes both handlers)

Status: Open.

#302 In TIGER-AI-Lab/ClawBench;

· reacher-z opened on Aug 18, 2026

Stage-1 interceptor matching is duplicated between runtime-server and edgebench_judge, and the live copy has zero tests

Status: Open.

#301 In TIGER-AI-Lab/ClawBench;

· reacher-z opened on Aug 18, 2026

judge_llm.py has diverged from judge.py (no Gemini support, wrong responses endpoint, 800-token cap) — collapse the duplication

Status: Open.

#300 In TIGER-AI-Lab/ClawBench;

· reacher-z opened on Aug 18, 2026

Judge outages are scored as agent failures (match=None → exit 1 → 'failed')

Status: Open.

#299 In TIGER-AI-Lab/ClawBench;

· reacher-z opened on Aug 18, 2026

No host-side timeout on container wait — one wedged container stalls an entire batch indefinitely

Status: Open.

#298 In TIGER-AI-Lab/ClawBench;

· reacher-z opened on Aug 18, 2026

batch --resume skips failed jobs (log-file heuristic) and overwrites batch-summary.json with 'skipped' rows

Status: Open.

#297 In TIGER-AI-Lab/ClawBench;

· reacher-z opened on Aug 18, 2026

rescore: public reproducibility CLI defaults to a maintainer's home paths and silently no-ops elsewhere

Status: Open.

#296 In TIGER-AI-Lab/ClawBench;

· reacher-z opened on Aug 18, 2026