Create sandbox scoring pods on-demand - #55
bill-bateman wants to merge 2 commits into
Conversation
dadd10a to
183f72a
Compare
| "mem_limit": f"{AGENT_MEMORY_GIB}g", | ||
| "network_mode": "none", | ||
| }, | ||
| "comparator": { |
There was a problem hiding this comment.
leaving comparator sandbox environment alone for docker runs, since I didn't see any point bringing it up/down
183f72a to
65cdf25
Compare
|
Testing live, with the following yaml (based on YAML# Schema: https://github.com/METR/hawk/blob/main/hawk/hawk/api/EvalSetConfig.schema.json
#
# oeis-smoke-easy: end-to-end shakeout of the eval pipeline on the smoke_easy
# subset -- 10 OEIS conjectures with known outcomes (5 provable, 5 disprovable).
# Not a benchmark run; keep the caps small so a broken pipeline fails cheap.
name: oeis-smoke-easy-k8s-main
tasks:
- package: git+ssh://git@github.com/epoch-research/LeanOpenProblems.git
# @1122b2c563ca9506969cd719937fa6f819ff3f45
name: apn
items:
- name: apn_oeis
args:
subset: smoke_easy
agent_type: react
literature: false
sandbox_backend: k8s
models:
- package: git+https://github.com/epoch-research/benchmarks
name: epoch
items:
- name: claude-haiku-4-5-20251001
args:
config:
max_retries: 7
max_connections: 10
# 1 model x 10 samples x $2/sample cap = $20 max.
cost_limit: 2.0
working_limit: 600 # 10min * 60s
# $ per 1M tokens; keys must match Inspect's resolved <provider>/<model> names.
model_cost_config:
epoch/claude-haiku-4-5-20251001:
input: 1.00
output: 5.00
input_cache_read: 0.10
input_cache_write: 1.25
secrets:
- name: ANTHROPIC_API_KEY
- name: LEAN_OPEN_PROBLEMS_IMAGE_NAME
description: "The ECR repo containing the LeanOpenProblems sandbox images"
runner:
memory: 200Gi
environment:
# TODO: Remove once default in Hawk
INSPECT_LOG_CONDENSE: "true"
ANTHROPIC_BASE_URL: https://api.anthropic.com
HAWK_RUNNER_REFRESH_CLIENT_ID: ""
HAWK_RUNNER_REFRESH_TOKEN: ''
packages:
# Right now needs to be kept in sync with epoch-research/benchmarks@main.
# TODO Remove once this wart is no longer necessary.
- inspect-ai==0.3.259
retry_on_error: 2 # https://github.com/epoch-research/epoch-hawk/pull/13
Ideally I would at some point run it again and watch via edit: made a minor code change (making a param required when creating checker instead of giving a default), which is why the commit sha won't line up anymore |
c8f6a82 to
0e642cc
Compare
| from k8s_sandbox import K8sSandboxEnvironment, K8sSandboxEnvironmentConfig | ||
|
|
||
| task_name = _get_task_name() | ||
| config = K8sSandboxEnvironmentConfig(values=Path(values_path).resolve()) |
There was a problem hiding this comment.
Needs investigation: this release bypasses Hawk's sandbox patching. In production the runner (with HAWK_RUNNER_PATCH_SANDBOX, always set by the deployed job template) rewrites every sample's sandbox values in hawk/runner/run_eval_set.py::_patch_sample_sandbox. The comparator used to ride through that path; this release does not, so it gets none of:
- pod labels
inspect-ai.metr.org/job-id,sample-id,task-name,app.kubernetes.io/component: sandbox. Hawk monitoring and the viewer's pod list select sandbox pods by exactly these labels, so "the pods list only contains onedefaultper sample" is also what you'd see if the scoring pods exist but are unlabelled. The plannedkubectlwatch of the sandbox namespace should settle that. karpenter.sh/do-not-disrupt: "true". Without it Karpenter can consolidate the node under a running comparator and kill an up-to-1h check.- the
kubernetes.io/archnodeSelector and arm64/GPU tolerations. - every
isolationcontrol (runtime class, securityContext, egress). Our configs don't set an isolation level today, but if one ever does the scoring pod silently gets none of it. - the EKS-mirrored
corednsImage, andrestarted_container_behavior="raise"(Hawk sets that on agent sandboxes; here it defaults to"warn", so a comparator container restart mid-check warns and continues instead of failing the check).
One option that needs no Hawk change: at score time, read the patched agent values file from the active sample's sandbox config and copy its labels, annotations, nodeSelector and tolerations into the scoring values. At minimum, set restarted_container_behavior="raise" here.
There was a problem hiding this comment.
I'll look into this tmrw 👀
There was a problem hiding this comment.
Yeahhh this is an issue. I added a new commit to address this, since it took quite a bit of stuff. This got somewhat messy :/
tl;dr added a setup solver which reads the default sandbox's (patched) config, copies some info, and merges it with the scoring sandbox values, then stores that. the checker can then read that, put it in a tmp file, and call into sample_init.
(haven't yet tried running the new commit live on hawk fyi) - seems to work
| # but have a timeout so it doesn't hold forever | ||
| # and make the timeout longer than what the helm uninstall timeout itself is | ||
| with anyio.fail_after( | ||
| float(os.environ.get(_HELM_TIMEOUT_VAR, _HELM_TIMEOUT_DEFAULT_S)) * 2, |
There was a problem hiding this comment.
Hawk sets INSPECT_HELM_TIMEOUT to 24h in the runner env (hawk/api/run.py), and helm install runs with --wait. Two consequences:
- A scoring pod that can't schedule (no node with 32Gi free, autoscaler stuck) blocks the scorer for up to 24h per attempt, times 3 attempts. In gated mode the scorer runs inside the solver on every submit, so that wait counts against the 72h
working_limit. Before this PR the comparator was reserved at sample start, before any model spend; now a sample that has run for days can stall or fail at the very end depending on cluster capacity at that moment. There is no ResourceQuota on the sandbox namespace (checked the runner chart), so raw capacity is the constraint. Worth a scoring-specific bound around_install, and a decision about what a failed scoring install should do to the sample. - This shield timeout becomes 48h. And the METR fork of k8s_sandbox that Hawk actually runs (rev
ff1ef4a, 0.14.0) uses a separateINSPECT_HELM_UNINSTALL_TIMEOUT(default 600s) for uninstall, so deriving fromINSPECT_HELM_TIMEOUTisn't the relevant variable.
There was a problem hiding this comment.
so... just use INSPECT_HELM_UNINSTALL_TIMEOUT var instead?
There was a problem hiding this comment.
confirmed INSPECT_HELM_UNINSTALL_TIMEOUT is right https://github.com/METR/inspect_k8s_sandbox/blob/ff1ef4a6d845ed0abce87408e2ab2275d31c3687/src/k8s_sandbox/_helm.py#L632, I'll update that.
I don't think having a scoring-specific bound is useful?
And on failed scoring install, IIUC it would retry the sample from the top. Not sure how to get it to not do this, if we want it to not retry. 🤔
|
|
||
| On docker the comparator is a long-lived service, resolved by name. | ||
|
|
||
| On k8s the comparator is a Helm release of its own, installed as |
There was a problem hiding this comment.
Per-attempt pod churn in gated mode. Gated is the default and makes attempts effectively unlimited, so every submit now costs a helm install, scheduling, a pull of the Mathlib-sized comparator image if the node is cold, the check, and an uninstall. The 2.3s install in the test run was on a warm node. Worth measuring cold-start latency and checking how many submits per sample real runs make. If it's many, a per-sample release (installed on first check, torn down at sample end) is a middle ground, at the cost of some idle time again.
There was a problem hiding this comment.
Trueeee
Warm vs cold latency
Looking at all the scoring calls, can separate time into 2 groups, which I'm assuming is cold vs warm
Extra scoring call when correct?
Hey weird, looks like the one task that actually succeeded called scoring twice - once for the "intermediate score" which passed, and then it scored again (2 logs like "Installed scoring sandbox release" - link). I guess it's double scoring. I had assumed it didn't re-score when it already passed once.
How many scores real runs make?
Looking at this run of gpt-6-astra on oeis-lite (100 samples) - note 43 / 100 got cutoff due to some sort of limit
- oeis_365416_conjecture_0 does 2 scoring checks, oeis_306477_conjecture_1 does 3 (2 + 1 because correct), oeis_7918_conjecture_1 does 2, oeis_243512_conjecture_0 does 2, oeis_296075_conjecture_0 does 4 (3 + 1 because correct), oeis_340079_conjecture_0 does 3 (2 + 1 because correct)
- oeis_92243_conjecture does 2 but only because it was correct, same with oeis_a176477_conjecture, and 52 others
There was a problem hiding this comment.
Looked a lot more into this for some cost/savings estimation.
Real evals sometimes run very very very very very many submission attempts. Sometimes legitimately ~100, but sometimes it enters a doom loop and gets into the 1,000s. If it thinks the task is impossible, it will keep looping the same exact submission over and over until it runs out of budget. This is a big problem!
If we are in a doom loop, constantly bringing up/down sandbox pods could be worse than just leaving one up! (You may have to wait for a new node to start up, the node may hang around after you are done with it)
I put more info in this doc (and gsheet) but tl;dr if we don't do anything about the doom loops then we probably don't want to merge this in. One thing we could do about doom loops is cache the last submission attempt and the response - if it is the exact same, then we can return the previous response.
da693ec to
84bfac9
Compare
|
Tested again with the hawk config patcher commit + score cache commit ( 
It is the one that is being created. The name is pretty ambiguous right now though, it looks the same. 🤔 This doesn't really test the score cache tho, since there is no doom loop here. |
Previously we had 2 sandbox environments at all times. This meant the 32GiB scoring pod was sitting idle for most of the time, which costs money. Here, we spin up a scoring pod on-demand during the checker. Then we execute the same set of commands to run the scoring. We manage the scoring pods via K8sSandboxEnvironment calls directly - .sample_init and .sample_cleanup. Some notes: - docker remains unchanged - brings up 2 containers at the start - scoring pods do not count against max_sandboxes - scoring pods are brought up within the same sandbox namespace, so the janitor should clean them up if anything goes pear shaped - currently doesn't obey --no-sandbox-cleanup flag, i.e. all scoring sandbox pods get cleaned up
The on-demand scoring sandboxes bypass Hawk's sandbox patching. Basically it sets a bunch of values that are useful, like pod labels for tracking, karpenter do-not-disrupt, k8s nodeSelector, etc. Here, grab relevant values off the default sandbox (for agent tool use) and then merge them in with the scoring values.
84bfac9 to
5805924
Compare
Previously we had 2 sandbox environments at all times. This meant the 32GiB scoring pod was sitting idle for most of the time, which costs money.
Here, we spin up a scoring pod on-demand during the checker. Then we execute the same set of commands to run the scoring. We manage the scoring pods via K8sSandboxEnvironment calls directly - .sample_init and .sample_cleanup.
Some notes: