Fix bcal CAPI evals: send M365 LLM API taxonomy headers; default to gpt-55-chat - #756
Merged
Merged
Conversation
…pt-55-chat The M365 LLM API's taxonomy/CoS rollout (aka.ms/llmapi/waves-timeline) began enforcing X-Taxonomy-Experience/Agent/InferenceStep/TrafficType headers between Jul 12 and Jul 19, breaking every bcal CAPI eval (InvalidInferenceInput "Required taxonomy data not provided"). Neither bc-eval's CapiModel nor the generated CAPI client sends them; BC production is unaffected because it calls Azure OpenAI directly, so only the CAPI eval path hits the gateway. Inject the four headers (env-overridable, defaults baked in) at the one shared point both code paths flow through -- CapiModel._get_common_capi_parameters, whose dict is splatted onto the generated operations that forward an unknown `headers` kwarg: - bcal eval path: bc_eval_capi_bridge.py patches it in the bridge subprocess. - nl2al lm_checklist judge path: evaluator/_capi_taxonomy.py, installed at import time from scores.py, which bceval exec_modules while resolving evaluators before running the judge. Also set gpt-55-chat-2026-04-29 as the default external-model (added to the choices) so the scheduled run and the new default both use it. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
The first test run confirmed the header injection works: the error advanced from "taxonomy not provided" to "Invalid taxonomy value: 'DynamicsBusinessCentral' is not allowed for X-Taxonomy-Experience. Allowed values: BizChat, WXPOAgents, AppCopilots, Cowork, Scout, WorkIQ." BC is an app copilot -> AppCopilots. Verified locally against CAPI for both the eval model (gpt-55-chat-2026-04-29) and the judge model (gpt-41); the other three headers (bcal / ChatCompletion / Test) were accepted as-is. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
Contributor
There was a problem hiding this comment.
Pull request overview
Adds required M365 LLM API taxonomy headers to bcal and judge CAPI requests, while updating the default external model.
Changes:
- Injects configurable taxonomy headers into both CAPI paths.
- Defaults evaluations to
gpt-55-chat-2026-04-29. - Adds taxonomy shim tests.
Reviewed changes
Copilot reviewed 6 out of 6 changed files in this pull request and generated 1 comment.
Show a summary per file
| File | Description |
|---|---|
.github/workflows/bcal-evaluation.yml |
Updates model defaults and taxonomy environment values. |
evaluator/_capi_taxonomy.py |
Adds the judge-side taxonomy shim. |
evaluator/scores.py |
Installs the judge-side shim. |
src/bcbench/agent/bcal/bc_eval_capi_bridge.py |
Injects headers into bridge requests. |
tests/test_bcal_capi_bridge.py |
Tests bridge taxonomy behavior. |
tests/test_capi_taxonomy_evaluator.py |
Tests evaluator shim installation. |
|
|
||
| @pytest.fixture | ||
| def stub_capi_model(monkeypatch): | ||
| """Stub bc_eval.capi.capi_model.CapiModel with the internal the taxonomy shim patches.""" |
Sun Haoran (haoranpb)
approved these changes
Jul 28, 2026
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Problem
All bcal CAPI evaluations have failed since ~Jul 19 (runs 29672964336, 30187493565, 30349142281).
Root cause
The M365 LLM API taxonomy/CoS rollout (aka.ms/llmapi/waves-timeline) began enforcing the
X-Taxonomy-Experience/Agent/InferenceStep/TrafficTypeheaders between the last green run (Jul 12)and Jul 19. Neither bc-eval's
CapiModelnor the generated CAPI client sends them, so CAPI rejectsevery call with
InvalidInferenceInput. BC production is unaffected because it calls Azure OpenAIdirectly (
BC-Platform .../CopilotApi/AL/AzureOpenAIClient.cs), bypassing the enforcing gateway —only BC-Bench's CAPI eval path hits it, so this was new and unfixed anywhere in BC.
Fix
Inject the four headers (env-overridable via
CAPI_TAXONOMY_*, defaults baked in) at the one pointboth code paths flow through —
CapiModel._get_common_capi_parameters, whose dict is splatted ontothe generated operations that forward an unknown
headerskwarg:bc_eval_capi_bridge.pypatches it in the bridge subprocess.evaluator/_capi_taxonomy.py, installed at import time fromscores.py, whichbcevalexec_modules while resolving evaluators before the judge runs(the judge runs in an isolated
uv toolenv that has bc-eval but not bcbench, hence a second,self-contained shim).
Values validated directly against CAPI (for both
gpt-55-chat-2026-04-29and thegpt-41judge):Experience=AppCopilots(BC is an app copilot; the LLM API's allowed set is BizChat, WXPOAgents,AppCopilots, Cowork, Scout, WorkIQ),
Agent=bcal,InferenceStep=ChatCompletion,TrafficType=Test.Also sets
gpt-55-chat-2026-04-29as the new default external-model (added to the choices and thescheduled-run fallback).
Validation
Co-authored-by: Copilot 223556219+Copilot@users.noreply.github.com