refactor(runtime): unify deferred loading + tool-source economy into ToolAvailabilityRuntime - #41
Conversation
ac60396 to
3951297
Compare
…o ToolAvailabilityRuntime Collapse the two parallel tool-visibility mechanisms — PR #30 deferred tool loading and PR #34 tool source economy — into one mechanism and one policy (issue #37): - One `economy` switch + configurable `coreToolNames`; grouping comes from `ToolAvailabilityConfig.groups`, not the per-tool `exposure` tag (removed from MakaTool). The now-dead `toolSource` tag is swept separately next. - One `load_tools` connector, built by the runtime. Same-turn activation honors only `load_tools` (and only its `group` arg). The historical `load_tool` (PR #30) and `connect_tool_source` (PR #34) names are accepted solely when re-seeding prior-turn activations from the durable ledger — never advertised, never live in the current turn. - One same-turn activation policy (prepareStep + execute-boundary gating); the next-request economy semantics are dropped. - Delete tool-source-economy.ts, deferred-activation.ts, load-tool.ts and their tests; add tool-availability.ts + tool-availability.test.ts. - Rename ToolSourceEconomyDiagnostic → ToolAvailabilityDiagnostic (record field toolSourceEconomy → toolAvailability) while keeping the historical *SourceIds shell vocabulary (a "source" id is a catalog group id). - Wire main.ts through a single ToolAvailabilityConfig (economy = !MAKA_DISABLE_DEFERRED_TOOLS; groups = rive/office/browser); the runtime now produces the connector, so builtinTools no longer include it.
With ToolSourceEconomyRuntime gone, nothing reads MakaTool.toolSource — the unified runtime groups via ToolAvailabilityConfig.groups, not per-tool tags. Remove the field (and its ToolSourceId import) plus the six builtin-tools tags that set it. Pure dead-code removal; no behaviour change. The ToolSourceId alias stays in @maka/core for the diagnostic's *SourceIds shell fields.
Recognize the unified `load_tools` connector (and the historical `load_tool`, PR #30 — it shipped and returns the same `{ loaded: [...] }` shape) for the localized "Load tools" card, reading the loaded group id from `group` with a `namespace` fallback so replayed pre-unification sessions still render. `connect_tool_source` (PR #34) is deliberately not presented: it never shipped, so no such result exists, and its `{ tools: [...] }` shape differs from this card's `{ loaded: [...] }`. The runtime still seeds from it for the durable ledger (a separate concern that reads call args, not results).
…o.logs() logs() rebuilt each row field-by-field but dropped toolSchemaChangeReason and toolAvailability, so the economy tool-surface diagnostic recorded on every LLM call was never queryable. Pass both through (insert + on-disk normalize already preserve them). Adds a precise telemetry-repo test round-tripping the diagnostic.
3951297 to
0df4a9d
Compare
Empirical validation: the economy
|
| provider | economy load-turn miss (×4 repeats) | cache-hit rate (all 3 modes) |
|---|---|---|
DeepSeek deepseek-chat |
81 / 75 / 79 / 206 tok | ~99.5–99.75% |
MiMo mimo-v2.5-pro |
118 / 126 / 104 / 93 tok | ~99.5–99.6% |
The ~33K-token system prefix is not invalidated; the miss is just the newly-added schemas, same order of magnitude as an ordinary turn. Quality: the load_tools dispatch was correct on every run (DeepSeek 4/4, MiMo 16/16 tool-call turns). The three modes' cache-hit rates are statistically indistinguishable — and full is actually the most expensive, since it carries every unused schema as cached input every turn.
(One early un-repeated MiMo run showed a 5377-token miss on the load turn — looked like a real break, did not reproduce across 4 seeded repeats. Server-side TTL/shard noise, not a deterministic consequence of the tools-array change. Independent seeds were essential.)
Mechanism (raw-HTTP probe — DeepSeek and MiMo gave identical signals)
These providers cache the tools field and the conversation as two independent prefixes rooted on the system prompt, not one linear stream:
- conversation cache = f(system, conversation) — excludes the
toolsfield, so aload_toolsthat grows the array leaves it intact. - tools cache = f(system, tools) — excludes the later conversation, so growing the conversation leaves it intact.
- both rooted on system — changing the system evicts both.
So a mid-conversation load_tools re-bills only the added schemas; system + history stay cached. This is the OpenAI-compatible-format-natural implementation: messages and tools are separate top-level JSON fields, hashed independently rather than inlined into one cascading stream.
Portability caveat — this is a DeepSeek/MiMo property, not universal
On OpenAI and Anthropic the opposite holds: tool definitions sit at the front of the cached prefix (OpenAI order: tools → system → few-shot → user message; Anthropic: tools → system → messages), so mutating the tools array mid-conversation cascades and evicts everything after it. That is exactly why both vendors ship native workarounds rather than letting you change the array:
- OpenAI
allowed_tools(tool_choice) — keep the full toolkit intoolsunchanged, restrict per-turn without busting the prefix. - OpenAI
defer_loading+ tool search (gpt-5.4+) — deferred tools are injected at the end of context when discovered, preserving the cached prefix (their benchmarks cite up to −47% prompt tokens). - Anthropic
tool-search-tool.
If Maka ever targets those providers, economy mode would need to be re-expressed via those native features instead of by mutating the tools array. (Verified against OpenAI docs — links below.)
Bottom line for this PR
The economy ToolAvailabilityRuntime delivers its schema-token savings with zero cache penalty on DeepSeek/MiMo — validated on the providers we actually ship to. A heavier message_defer indirection (full schemas in the load_tools result + a generic invoke_tool dispatcher) buys nothing here and was not pursued.
Sources: OpenAI prompt caching · Prompt Caching 201 · OpenAI tool search / deferred loading
How many schema tokens does economy mode actually save? (real production catalog)Companion to the cache comment above. That one showed the mid-conversation MethodReconstructed the exact Per-group deferred schema weight
Upfront tool-schema surface (same catalog,
|
| mode | DeepSeek tok | MiMo tok |
|---|---|---|
full — every tool, every turn |
4691 | 5817 |
economy — turn 1 (8 visible tools + connector) |
1705 | 2068 |
| kept off the wire every turn | 2986 | 3749 |
Takeaways
- Economy shrinks the upfront tool-schema surface by ~64% on both providers (4691→1705, 5817→2068), for a connector cost of only ~210–227 tokens.
- The full 12-tool, ~3.2K/4.0K-token deferred surface stays off the wire until a group is actually needed; loading one then costs only that group's 398–1252 tokens (and that group's schemas cache from then on).
- Because the mid-conversation load does not break the prefix cache on DeepSeek/MiMo (see the cache comment above), these are net savings — no cache penalty is paid back. On turn 1 the saved ~3K/3.7K tokens are avoided at full miss price; on every later turn
fullwould still carry them as cached input, so economy keeps paying off across the whole session. - This is also why
fullis the most expensive mode end-to-end: it pays the entire deferred surface as (cached) input on every single turn even when those capabilities are never used in the session.
(Numbers are deterministic — prompt_tokens is pure input counting, no sampling. Serialization uses zod's toJSONSchema, which is byte-close but not identical to the AI SDK's internal converter; expect single-digit-token drift vs. the exact wire bytes, immaterial at this scale.)
refactor(runtime): unify deferred loading + tool-source economy into ToolAvailabilityRuntime
refactor(runtime): unify deferred loading + tool-source economy into ToolAvailabilityRuntime
refactor(runtime): unify deferred loading + tool-source economy into ToolAvailabilityRuntime
refactor(runtime): unify deferred loading + tool-source economy into ToolAvailabilityRuntime
Closes #37.
Collapses the two parallel tool-visibility mechanisms — PR #30 (deferred tool loading) and PR #34 (tool source economy) — into a single
ToolAvailabilityRuntime: one mechanism, one policy.Commits
refactor(runtime): unify deferred loading and tool-source economy into ToolAvailabilityRuntime— the engine + wiring.refactor(runtime): drop the now-dead toolSource grouping tags— dead-code sweep the unification enables.refactor(ui): friendly presentation for the load_tools connector— renderer layer.fix(storage): carry tool-availability diagnostic through TelemetryRepo.logs()— closes the diagnostic loop so the economy tool-surface is queryable.What changed
economyboolean (default on); grouping comes fromToolAvailabilityConfig.groups, not per-tool tags. The model is exactly: a tool in a group is hidden behind the connector until loaded; every ungrouped tool is always visible. BothMakaTool.exposure(PR feat: Layer 1 deferred (on-demand) tool loading — defer heavy schemas via load_tool #30) andMakaTool.toolSource(PR Add Tool Source Economy v1 #34) are removed.load_tools(plural). Same-turn activation honors onlyload_tools(and only itsgrouparg). The historicalload_tool(PR feat: Layer 1 deferred (on-demand) tool loading — defer heavy schemas via load_tool #30) andconnect_tool_source(PR Add Tool Source Economy v1 #34) names are accepted solely when re-seeding prior-turn activations from the durable ledger — never advertised, never live in the current turn.prepareStep+ an execute-boundary gating guard. The next-request economy semantics are dropped.tool-source-economy.ts,deferred-activation.ts,load-tool.tsand their tests are gone;tool-availability.ts(+ test) replaces them. Net −347 lines.ToolSourceEconomyDiagnostic→ToolAvailabilityDiagnostic(record fieldtoolSourceEconomy→toolAvailability). The historical*SourceIdsshell vocabulary is kept — a "source" id is a catalog group id.main.tsuses oneToolAvailabilityConfig(economy = !MAKA_DISABLE_DEFERRED_TOOLS; groups = rive / office / browser); the runtime now produces the connector. The renderer presentsload_toolsand the shippedload_tool(same{ loaded: [...] }shape), reading the group id fromgroupwith anamespacefallback so replayed old sessions still render.Production behaviour is unchanged: prod only ever used the deferred mechanism, now expressed as
economy: true+ groups.Review (opencode GLM-5.2 + follow-ups)
Independently reviewed across several passes; no P0/P1. Addressed:
hiddenToolCountoff-by-one (telemetry regression): the connector was counted on thefullside but excluded on the visible side. Nowfull = visible + hidden; locked with a test.load_toolsconnector (and only itsgrouparg); historical connector names are ledger-seed-only. "One connector" is enforced, not just conventional. New test.load_tools+ the shippedload_tool(both{ loaded: [...] }).connect_tool_source(PR Add Tool Source Economy v1 #34) is deliberately not presented — it never shipped and its{ tools: [...] }shape differs; the runtime still seeds from it for the ledger (reads call args, not results).ToolAvailabilityDiagnostic.modeto'economy'(a diagnostic only exists in economy mode) and dropped the connector's dead empty-idsz.string()fallback (buildConnectoronly runs when at least one group is gated).coreToolNames: removed the config knob, the runtime branch, the diagnostic field and the test fixtures. It was a half-concept production never set — ungrouped tools are already always-visible, so "core" added nothing. The mechanism is now justeconomy + groups.ToolAvailabilityRuntime/LOAD_TOOLS_NAME/ config types now export directly fromtool-availability.tsthroughindex.ts, instead of bouncing throughai-sdk-backend.ts. External@maka/runtimeimport paths are unchanged.TelemetryRepo.logs()now passestoolSchemaChangeReason+toolAvailabilitythrough (insert and on-disk normalize already preserved them), so the recorded economy diagnostic is actually queryable. New storage test round-trips it.No open P3s.
Verification
npm run typecheck— clean across all workspaces.network/proxy-testtimeout assertion, which fails identically onmain(sandboxfetchreturnsfetch failedinstead of a timeout);network/is untouched by this PR.Base note
This branch is based on an ancestor of
main(it predates the credential-store refactor #32/#33). The three-dot PR diff shows only these commits, and amerge-treeprobe confirms a conflict-free merge — themain.tsedits here don't overlap the credential-store edits.