Skip to content

fix(pricing): claude-opus-4-7 is $5/$25, not $15/$75 (3x over) - #478

Open
roy-tong wants to merge 1 commit into
iris-eval:mainfrom
roy-tong:fix/opus-4-7-pricing
Open

fix(pricing): claude-opus-4-7 is $5/$25, not $15/$75 (3x over)#478
roy-tong wants to merge 1 commit into
iris-eval:mainfrom
roy-tong:fix/opus-4-7-pricing

Conversation

@roy-tong

@roy-tong roy-tong commented Sep 7, 2026

Copy link
Copy Markdown

Title: llm-judge pricing table lists claude-opus-4-7 at $15/$75 — actual rate is $5/$25 (3x overestimate, breaks judge cost cap)

Commit: 2f727e9a26d2a2f0edef013dc0bf09efafd82978

src/eval/llm-judge/pricing.ts:19 prices claude-opus-4-7 at $15 in / $75 out per 1M tokens, and docs/llm-as-judge.md repeats the same numbers. Anthropic has priced the Opus tier at $5/$25 since Opus 4.5; Opus 4.7 kept that rate (multiple public price trackers confirm, and Anthropic's model docs list $5/$25). $15/$75 is the pre-4.5 Opus rate.

Because this table backs the pessimistic pre-flight cap in evaluator.ts (estimateCostUsd first attempt + retry vs maxCostUsdPerEval, default $0.25), the 3x rate makes claude-opus-4-7 judge evals fail with CostCapError on inputs that would actually cost well under the cap:

  • 30k-token accuracy template, maxOutputTokens 512: current table → $0.4884 (rejected); at $5/$25 → $0.1628 (passes). In practice any opus-4-7 judge prompt over ~7k tokens is rejected at the default cap.
  • Stored/returned cost_usd is likewise 3x overstated for opus-4-7 evals (10k in + 1k out: $0.225 reported vs $0.075 actual).

Expected: inputUsdPer1M: 5, outputUsdPer1M: 25 for claude-opus-4-7, plus the same fix in the docs table. The file's own header notes these rates drive cost-cap decisions, so the correction is user-facing.

Minimal change: one line in src/eval/llm-judge/pricing.ts + the docs/llm-as-judge.md row (and any pinned test fixtures that assert 15/75). Happy to send a PR if that's easier to review.

元信息

  • 语言:issue 用英文(跟随仓库)。
  • 停止条件:维护者合并/拒绝即停;若指出 4.7 有特殊定价渠道(如 Bedrock 区域溢价)则核对后撤回。
  • 不确定性说明:opus-4.7 的 $5/$25 经三个独立价格源 + 两个独立代码库交叉确认;未直接打开 Anthropic 官方页(区域重定向),如需更强证据链可在主线程用可访问网络环境补一次官方页截图。

@roy-tong
roy-tong requested a review from irparent as a code owner September 7, 2026 00:56
@vercel

vercel Bot commented Sep 7, 2026

Copy link
Copy Markdown

@roy-tong is attempting to deploy a commit to the irparent's projects Team on Vercel.

A member of the Team first needs to authorize it.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant