MCP-inside-rlm + return-shape cache (Blacksmith muscle memory, not V8) by justrach · Pull Request #623 · justrach/codegraff

justrach · 2026-08-25T10:58:05Z

Verdict

Muscle memory is a real token win vs --old. MCP-inside-rlm is real (each() ran). It is not a wall win this rep.

Warm D: 107k tok_in vs A 168k (−36%), same 10 API calls. Cold C lost wall (56.7s vs A 36.7s) — fat print() + import/for/len retries, same story as Blacksmith cold code mode. B (rlm, MCP structured-only) is the old gap: the model wrote each(...) and was refused. Do not hide bash/edit/read_file. Do not add V8/IPython/QuickJS. Do not make rlm the only catalog tool.

SuperGrok [usage] is $0.0000 flat-rate; the 60k input cut on D is the metered-key spend win.

Evidence: ADR 0029.

Live A/B (2026-08-25)

SuperGrok OAuth, grok-4.6, one rep, ReleaseSafe. Same prompt: 8 fat fixture issues + comments + counts + latest authors. No live Linear. tok_in is ordinary+cache_read+cache_write.

var	harness	pass	wall	RSS	in	cached	out	calls	notes
A `--old` structured MCP	graff-dev-old-nolean	✓	36.7s	10.6M	167692	128000	2033	10	classic 1+8 structured MCP
B rlm, MCP structured-only	graff-dev-rlm-struct	✓	43.3s	10.8M	179822	109312	2358	11	`each()` refused; then `import`
C rlm+MCP host, cold	graff-dev-nolean	✓	56.7s	10.6M	147948	130048	3313	13	`each()` worked; 3 rlm fails
D rlm+MCP host, warm	graff-dev-nolean + shapes	✓	41.1s	10.7M	107213	83840	2253	10	−36% in vs A; 1 `len` retry
E1 sidecar summarize	graff-dev-nolean / sidecar	✓	37.5s	10.7M	168164	120704	1998	11	MCP stayed structured on root
E2 two sibling children	graff-dev-nolean / split	✓	75.5s	12.3M	149156	77824	5490	18	children have no MCP; expensive

Retries were missing language (len/for/import), not wrong MCP field names. Shapes splice onto the load_tool_schemas result, never the always-on prefix (ADR 0011).

What this is

Blacksmith [code]smith mapped onto graff, not a JS/TS/V8 sandbox:

MCP-inside-rlm host calls. After load_tool_schemas unfolds a tool and consent already allows it, rlm scripts may call that MCP name (src/rlm_mcp.zig, exec_mod). Unloaded names refused. Deferral can only subtract. GRAFF_RLM_MCP=0 restores B.
Muscle memory. Infer keys + broad types (never values) after MCP results; persist .graff/mcp-shapes.json; splice on the next load/search result (src/mcp_shapes.zig).
each(arr, tool, field) maps a JSON array — smallest control flow for for issue in list_issues(): list_comments(id). Not a general language.

Fixture: scripts/linear_fixture_mcp.py (stdio MCP, 8 fat issues ISS-1..8, fat comments). Held-out check: graff-evals/hidden/check_linear_report.py.

How to rerun

zig build -Doptimize=ReleaseSafe
python3 scripts/eval-mcp-shapes.py
# or:
cd graff-evals && ./run.py --suite mcp --harness graff-dev-old-nolean,graff-dev-rlm-struct,graff-dev-nolean --model grok-4.6

Suite mcp is opt-in; default all stays core+rlm+swe.

Tests

Unit: rlm dispatches a loaded MCP name; unloaded refused; shape inference strips values; warm load_tool_schemas includes merged shapes; lean/consent unchanged. New modules imported from exec.zig test {} (main.zig is already at the 600 LOC ceiling). Suite 1646 pass / 1 skip / 1647 total (baseline 1600, slack 25).

After load_tool_schemas unfolds a tool, rlm scripts may call that MCP name via exec_mod (unloaded names refused; GRAFF_RLM_MCP=0 restores structured- only). Infer field names + broad types from results, persist under .graff/mcp-shapes.json, and splice them onto the next load_tool_schemas result — never the always-on prefix. each(arr, tool, field) maps a JSON array so the Linear-shaped bench can run as one script.

Live grok-4.6 one-rep: warm D is -36% tok_in vs --old; cold C loses wall; B (structured-only) is the old gap. How to rerun is in the record.

Second-round A/B for ADR 0029: F/G never print() fat MCP arrays; H/I drop the each() recipe so we can see whether shapes alone steer the script. --only filters the runner.

Second SuperGrok rep: A wall is stable (~36s). D token win did not hold (159k / 61s). Quiet-print (F/G) burned 21–29 calls on Python. No-hint H/I never used rlm and beat --old (28–31s, 7 calls).

justrach · 2026-08-25T11:21:19Z

Follow-up SuperGrok A/B (same day, ADR 0029):

var	pass	wall	in	calls	note
A-r2 `--old`	✓	36.4s	145k	9	wall stable
D-r2 warm + `each()` hint	✓	60.5s	159k	11	first-rep token win did not hold
F `each()` + never print() fat	✓	220s	462k	29	`def`/`for`/`len`; grepped graff src
G F + warm	✓	156s	280k	21	same dialect hole
H no `each()` recipe	✓	28.0s	112k	7	never used rlm; parallel structured MCP
I no recipe + warm	✓	31.1s	107k	7	same as H; shapes unused

The each() hint is a footgun on grok-4.6. Quiet-print made rlm worse. No-hint beat --old. Rerun: python3 scripts/eval-mcp-shapes.py --only A,D,F,G,H,I.

each() without a slim print made grok invent for/len (ADR 0029). len(x) and project(x, field) stay data helpers. eval-mcp-pareto.py keeps the undominated set on wall / tok_in / calls.

justrach · 2026-08-25T12:30:01Z

Pareto front on wall / tok_in / calls (minimize all three, pass-only). Added len(x) / project(x, field) and swept A,H,J,K live.

id	wall	in	calls	vertex
H	28.0s	112k	7	fastest (no-hint structured)
I	31.1s	107k	7	same path, fewer tokens
D-r1	41.1s	107k	10	lucky rlm+each rep
J-live	47.3s	84k	9	token vertex — `each` + `len`/`project`

This session: A --old 36s/133k/8 (dominated). H-live 70s (variance). K warm+reduce 75s (dominated by J).

No point wins wall and tokens. Rerun: python3 scripts/eval-mcp-pareto.py

len/project recipe (J) is the token vertex at 84k / 47s. H remains the wall vertex at 28s / 7 calls. Nothing wins both.

After remember() of the fat payload, takeSlim / print() keep identity keys and fold comments to n+latest_author. Two stored shapes add a # muscle: line on the load result (never the catalog prefix). L–Q sweep those paths against the existing Linear fixture.

The len/project advertise pushed the discovery note over the byte check (em-dash + signatures). Shorter wording; the catalog desc still has the full each/len/project signatures.

cursor · 2026-08-25T12:52:21Z

Learnt slim landed on this branch (ca9cd21 + 682675e). Shapes on the load result did not change grok-4.6's path, so the muscle is now applied in Zig:

After remember() of the fat payload, takeSlim / print() keep identity keys (id / identifier / title / name) and fold comment arrays to {n, latest_author}.
When two MCP shapes are stored, annotate adds a # muscle: playbook on the load result (still never the catalog prefix).
No new GRAFF_ knob.

Next: live SuperGrok L–Q sweep on the same Linear fixture (warm/cold × no-hint / each-hint / quiet / reduce) and a Pareto recompute. Will comment numbers here; not rewriting this PR body.

cursor · 2026-08-25T12:55:00Z

Follow-up in the agent queue, so the live L–Q SuperGrok sweep is deferred (ReleaseSafe binary is built). Unit suite is the gate for this revision. L–Q are wired in scripts/eval-mcp-shapes.py (linear-nohint-warm / linear-warm / linear-nohint / linear-quiet / linear-reduce / linear-reduce-warm against the new slim).

Lean skipped MCP connect and hid load_tool_schemas even when a workspace server was present, so every MCP one-shot needed --no-lean. Default -p (lean+yolo) now connects and folds: names on the meta tool, full schemas a load away. Empty -p still hides the meta tool.

cursor · 2026-08-25T13:06:27Z

Out-of-the-box gap was lean itself, not another hint:

Default -p is lean+yolo. Lean skipped MCP connect and hid load_tool_schemas even when .mcp.json existed — so every MCP one-shot needed --no-lean.
9ec5884: lean connects and folds. Empty -p still hides the meta tool. Consent unchanged.
Slim + muscle playbook stay on (previous commits). Host rlm MCP is on by default.

Next: live variant R = graff-dev (no --no-lean) + linear-nohint. That is the default binary on this task.

cursor · 2026-08-25T13:08:47Z

R live (SuperGrok grok-4.6, ReleaseSafe, one rep): 16.3s / 14,113 in / 4 calls / pass.

That is the default binary (graff-dev = -p --yolo, implied lean, no --no-lean, no each() hint). It dominates the previous wall vertex H (28s / 112k / 7 on --no-lean).

Out of the box was: connect + fold + slim on the default one-shot. Not another recipe.

Live SuperGrok grok-4.6: graff-dev + linear-nohint passed and dominates the prior --no-lean H/I/J front.

Lean shrinks the prefix (workflow/webfetch/full MCP listing off). R's 16s/14k/4 is not an A/B against H/J. Front stays graff-dev-nolean; next live points are N/L/P (slim on the full harness).

cursor · 2026-08-25T13:34:00Z

Full harness (graff-dev-nolean, --no-lean) with learnt slim, SuperGrok grok-4.6, one rep each:

var	vs	wall	in	calls
L warm + slim	I 31s / 107k / 7	14.8s	31,348	5
N cold + slim	H 28s / 112k / 7	20.8s	30,314	5
P reduce + slim	J 47s / 84k / 9	48.1s	66,260	9

L and N are the --no-lean front. R stays off it (lean catalog). P does not beat J on wall.

graff-dev-nolean SuperGrok: L 14.8s/31k/5 (warm), N 20.8s/30k/5 (cold). Both dominate H/I/J. P 48s/66k/9 does not. R stays off the front (lean catalog).

cursor · 2026-08-25T15:12:25Z

L/slim is on release/v0.0.277 (9e56aa6 merge + cfb1493 notes). No v0.0.277 tag. This PR (#623) stays the feature branch into main; the release branch is the cut.

cursoragent added 4 commits August 25, 2026 10:48

docs(adr): 0029 MCP-inside-rlm + return shapes A/B

a3acb0a

Live grok-4.6 one-rep: warm D is -36% tok_in vs --old; cold C loses wall; B (structured-only) is the old gap. How to rerun is in the record.

eval(mcp): quiet-print and no-hint Linear variants

edd0f4e

Second-round A/B for ADR 0029: F/G never print() fat MCP arrays; H/I drop the each() recipe so we can see whether shapes alone steer the script. --only filters the runner.

docs(adr): 0029 follow-up — each() hint is a footgun

51d5c58

Second SuperGrok rep: A wall is stable (~36s). D token win did not hold (159k / 61s). Quiet-print (F/G) burned 21–29 calls on Python. No-hint H/I never used rlm and beat --old (28–31s, 7 calls).

feat(rlm): len/project reducers + MCP Pareto sweep

9fc98e1

each() without a slim print made grok invent for/len (ADR 0029). len(x) and project(x, field) stay data helpers. eval-mcp-pareto.py keeps the undominated set on wall / tok_in / calls.

cursoragent added 3 commits August 25, 2026 12:30

docs(adr): 0029 Pareto front — H wall, J tokens

05a0d5d

len/project recipe (J) is the token vertex at 84k / 47s. H remains the wall vertex at 28s / 7 calls. Nothing wins both.

fix(rlm): keep system_note under the 360-byte prefix cap

682675e

The len/project advertise pushed the discovery note over the byte check (em-dash + signatures). Shorter wording; the catalog desc still has the full each/len/project signatures.

cursoragent added 2 commits August 25, 2026 13:09

docs(adr): 0029 R is the default -p vertex (16s / 14k / 4)

dd56121

Live SuperGrok grok-4.6: graff-dev + linear-nohint passed and dominates the prior --no-lean H/I/J front.

docs(eval): MCP bench is --no-lean; R is a different catalog

c4901bd

Lean shrinks the prefix (workflow/webfetch/full MCP listing off). R's 16s/14k/4 is not an A/B against H/J. Front stays graff-dev-nolean; next live points are N/L/P (slim on the full harness).

docs(adr): 0029 full-harness slim front is L/N

b47b5ae

graff-dev-nolean SuperGrok: L 14.8s/31k/5 (warm), N 20.8s/30k/5 (cold). Both dominate H/I/J. P 48s/66k/9 does not. R stays off the front (lean catalog).

This was referenced Aug 25, 2026

feat(xai): hosted x_search on grok-4.6 Responses #632

Merged

Showcase rlm when context crosses 50% of compactAt #633

Merged

justrach merged commit 9e56aa6 into main Aug 26, 2026
6 checks passed

justrach deleted the cursor/mcp-rlm-shapes-4ffc branch August 26, 2026 03:37

Navigation Menu

MCP-inside-rlm + return-shape cache (Blacksmith muscle memory, not V8) - #623

MCP-inside-rlm + return-shape cache (Blacksmith muscle memory, not V8)#623
justrach merged 12 commits into
mainfrom
cursor/mcp-rlm-shapes-4ffc

justrach commented Aug 25, 2026

Uh oh!

justrach commented Aug 25, 2026

Uh oh!

justrach commented Aug 25, 2026

Uh oh!

cursor Bot commented Aug 25, 2026

Uh oh!

cursor Bot commented Aug 25, 2026

Uh oh!

cursor Bot commented Aug 25, 2026

Uh oh!

cursor Bot commented Aug 25, 2026

Uh oh!

cursor Bot commented Aug 25, 2026

Uh oh!

cursor Bot commented Aug 25, 2026

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

2 participants

Conversation

justrach commented Aug 25, 2026

Verdict

Live A/B (2026-08-25)

What this is

How to rerun

Tests

Uh oh!

justrach commented Aug 25, 2026

Uh oh!

justrach commented Aug 25, 2026

Uh oh!

cursor Bot commented Aug 25, 2026

Uh oh!

cursor Bot commented Aug 25, 2026

Uh oh!

cursor Bot commented Aug 25, 2026

Uh oh!

cursor Bot commented Aug 25, 2026

Uh oh!

cursor Bot commented Aug 25, 2026

Uh oh!

cursor Bot commented Aug 25, 2026

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

2 participants