CodeAnchor

Public artifact for the ISSTA 2026 paper on CodeAnchor. CodeAnchor studies whether lightweight static structure, inserted as plain-text annotations, can make LLM code agents more stable and effective on SWE-bench style localization and repair tasks.

This release keeps the runnable experiment scripts, evaluation code, compact summaries, and paper-facing figures. Large generated traces, cloned benchmark repositories, raw repeated-run outputs, local indexes, and credentials are not committed.

What Is Included

.
├── assets/                 # Paper/artifact figures
├── docs/examples/          # Example prompts and usage notes
├── evaluation/             # Metrics, statistical tests, summaries, repair helpers
├── scripts/                # Runners for Codex localization and repair experiments
├── requirements.txt        # Python environment used by the artifact scripts
├── README.md
└── LICENSE

The main entry points are:

File	Purpose
scripts/swe_bench_codex.py	Run one localization task and write structured JSON plus prediction JSONL.
scripts/swe_bench_codex_batch.py	Dispatch many localization tasks in parallel.
scripts/run_codex.py	Older patch/localization runner used during the paper experiments.
evaluation/swe_bench_codex_eval.py	Evaluate localization predictions.
evaluation/analyze_rq4_statistics.py	Analyze repeated-run stability summaries.

Setup

git clone https://github.com/mathieu0905/Code-Anchor.git
cd Code-Anchor
python -m venv .venv
source .venv/bin/activate
pip install --upgrade pip
pip install -r requirements.txt

The runners call the Codex CLI and read the usual authentication/configuration from your local environment. At minimum, make sure your Codex/OpenAI credentials are configured before launching new agent runs.

Quick Start

Run one SWE-bench Lite localization instance:

python scripts/swe_bench_codex.py \
  --dataset swe-bench-lite \
  --split test \
  --index 0 \
  --prompt-mode swebenchlite \
  --no-annotation-mode \
  --save-trace

Run the annotated CodeAnchor condition:

python scripts/swe_bench_codex.py \
  --dataset swe-bench-lite \
  --split test \
  --index 0 \
  --prompt-mode swebenchlite \
  --annotation-mode \
  --prefer-local-annotation \
  --annotation-root auto \
  --save-trace

Outputs are written to ignored directories:

outputs*/<dataset>/structured/     # normalized localization JSON
outputs*/<dataset>/trace/          # Codex event traces
predictions*/<dataset>/            # JSONL records for evaluation

Batch Runs

Run a small batch with four workers:

python scripts/swe_bench_codex_batch.py \
  --dataset swe-bench-lite \
  --split test \
  --prompt-mode swebenchlite \
  --index-range 0:20 \
  --max-workers 4 \
  --annotation-root auto

By default, the batch runner skips instances whose prediction files already exist. Add --no-skip-existing to force a rerun.

CodeAnchor Inputs

The public repository does not ship benchmark worktrees or annotation indexes. For full reproduction, prepare one of the following:

• local annotated repositories under workspace/<dataset>/annotated_repos/;
• a manifest at dataset/annotated_repo_pairs.json;
• original SWE-bench checkouts generated by the runner when --checkout-if-missing is enabled.

The example prompt in docs/examples/callgraph_prompt_example.txt shows how the callgraph-MCP condition was expressed for one task.

Evaluation

Evaluate localization outputs:

python evaluation/swe_bench_codex_eval.py

Analyze repeated-run stability:

python evaluation/analyze_rq4_statistics.py \
  --summary-json evaluation/rq4_eval_summary.json \
  --out evaluation/rq4_summary.txt

Run lexical-similarity analysis:

bash evaluation/run_lexical_similarity_lite.sh

Repair submission helpers live in evaluation/repair/. They expect generated prediction files and local SWE-bench/SB-CLI tooling.

Reproducing Raw Outputs

To regenerate the raw artifacts omitted from git:

1. Install the Python dependencies.
2. Configure Codex/OpenAI credentials.
3. Prepare SWE-bench worktrees or annotated repositories.
4. Run scripts/swe_bench_codex.py or scripts/swe_bench_codex_batch.py.
5. Re-run the evaluation scripts over the generated outputs* and predictions* directories.

The .gitignore keeps regenerated traces, worktrees, predictions, indexes, and logs out of version control so the release stays small.

Citation

If you use this artifact, please cite the corresponding ISSTA 2026 paper. The final BibTeX entry will be added after proceedings metadata is available.