diff --git a/ARCHITECTURE.md b/ARCHITECTURE.md index 70bd8ed97..5582bda7c 100644 --- a/ARCHITECTURE.md +++ b/ARCHITECTURE.md @@ -7,7 +7,7 @@ text. It is the leverage the text-token floor denies the language track (see [IDEAL_AGENTIC_LANGUAGE.md](IDEAL_AGENTIC_LANGUAGE.md) for that analysis). > **Scope.** Everything below is implemented and test-covered in `prototype/` -> (**1,094 tests** green) and scored in the sibling `agentic-eval` crate (80 +> (**1,107 tests** green) and scored in the sibling `agentic-eval` crate (80 > tests, in the AetherShell repository and not verifiable from here). The one > deliberate non-feature is agent/swarm *execution* — see > [Honest boundaries](#honest-boundaries). @@ -175,7 +175,7 @@ are **five independent Cargo workspaces**: | Path | Crate | Tests | Notes | |---|---|--:|---| | `RecursiveMachineIntelligence/` | `rmi` | 1,380 | The low-level neurosymbolic framework. Feature-gated (`cpu` / `gpu` / `cuda`); build with `--no-default-features --features cpu` for the portable set | -| `prototype/` | `mage-prototype` | 1,094 | Compiler, evaluator, ABL, RAP server. Path-depends on `rmi` | +| `prototype/` | `mage-prototype` | 1,107 | Compiler, evaluator, ABL, RAP server. Path-depends on `rmi` | | `ribosome/` | `ribosome` | 164 | The distributed build engine. Depends on nothing in this repository — see below | | `germline/` | `germline` | 112 | Model succession, handoff, fallback — the RSI control plane. Path-depends on `ribosome` | | `forge/` | `forge` | 52 | The package registry, and only that | diff --git a/HANDOFF.md b/HANDOFF.md index 27e0f1752..54e61f8cb 100644 --- a/HANDOFF.md +++ b/HANDOFF.md @@ -15,7 +15,7 @@ run, and are pinned to their output. See "The example rewrite" below. | | | |---|---| -| Tests | **2,802** — rmi 1,380 · prototype 1,094 · ribosome 164 · germline 112 · forge 52 | +| Tests | **2,815** — rmi 1,380 · prototype 1,107 · ribosome 164 · germline 112 · forge 52 | | CUDA | **1,071 passing** on dual RTX 3090 Ti, driver 610.88 | | Warnings | 0 compiler, 0 clippy in the four owned crates (`rmi` keeps 2 — vendored) | | Vulnerabilities | 0 Rust across five lockfiles, 0 npm | @@ -109,7 +109,7 @@ Six of the eleven — #5, #6, #7, #8, #9, #11 — are the same class: a bug that **typechecks and then does not evaluate**. `--check` cannot find these. Only `--eval` can, which is why the pin now runs it. -Prototype tests **1,066 → 1,094**, all green. +Prototype tests **1,066 → 1,107**, all green. ### And a twelfth, from this list itself @@ -125,6 +125,40 @@ Surveyed first rather than assumed: every `guard` in the repository — two examples, four `prototype/examples/*.mg`, two parser tests — already returns explicitly, so nothing had come to depend on the fall-through. +### And the effect system got its elimination rule + +`handle { … } with E { … }`. The gap this closes was the one left in the +"found and left alone" list: effects could be declared, annotated, inferred, +and enforced, but never *discharged*, so `/ audit` propagated outward forever. + +Three things landed together, because none of them is useful alone: + +- **Introduction.** `Audit.record(x)` performs the operation and puts `audit` + in the calling function's effect set. Before this an `effect` block declared + operations that no analysis attributed to anyone and that the evaluator + rejected with `unknown function` — a thirteenth bug of the familiar + typechecks-then-does-not-evaluate kind. +- **Elimination.** `handle` removes the effect from the block it wraps, so a + function can be pure despite calling something effectful. The subtraction is + **per block, not per function**: an unhandled call sitting beside a handled + one still reports. Whatever the arm itself does is attributed honestly, so + handling `audit` by writing a file makes the handling function `/ fs`. +- **Declaration.** An operation the effect does not declare is an error, and + so is an effect annotation naming nothing. + `/ nte` used to be accepted as a *different effect* from `/ net`, enforced + consistently and matching nothing — a typo invented an effect instead of + failing. The operation check was added after the first version of this + feature shipped it broken: the analysis attributes the effect from the + *receiver* alone, so `Audit.recrod(x)` counted as performing `audit`, + checked clean, and died at run time. Building the fix for a bug class is no + protection against writing another instance of it one level down. + +Handlers do not resume. An operation call dispatches to its arm and returns +like an ordinary call, which is what a tree-walking evaluator can do without +capturing continuations. Handlers are found **dynamically** (innermost wins) +and evaluated **lexically** (the arm sees the scope the handler was written +in), and both are tested. + ### The examples are now pinned to their output, not to their exit status `check-examples.sh` used to record which examples typechecked. That bar was too @@ -142,9 +176,6 @@ the answer. `--print` regenerates the block after an intentional change. ### Found and left alone -- **There is no `handle` form.** `--check` reports `unresolved name: handle`. - Effects can be declared, annotated, inferred, and enforced, but never - *discharged* — the effect system has no elimination rule. - **`agent` and `unsafe` cannot be written as effect names**, because both lex as keywords: `/ agent` is a parse error. `rand` is not built in either — the built-in kind is `rng`, and anything else silently becomes `Effect::Custom`. diff --git a/MEASUREMENTS.md b/MEASUREMENTS.md index 877571853..90ccd9d38 100644 --- a/MEASUREMENTS.md +++ b/MEASUREMENTS.md @@ -6,8 +6,8 @@ numbers are machine-dependent; the shapes (throughput, scaling) are not. Date: 2026-06-10. Build: `release` for perf, `cargo test` for functionality. -> **Re-verified 2026-08-11** — all five crates tested: prototype **1,094**, rmi -> **1,380**, ribosome **164**, germline **112**, forge **52** = **2,802 passing, +> **Re-verified 2026-08-11** — all five crates tested: prototype **1,107**, rmi +> **1,380**, ribosome **164**, germline **112**, forge **52** = **2,815 passing, > 0 failing, 0 warnings**. No figure below has regressed. > > *A measurement that was wrong.* `BuildReport::cache_hit_ratio` was @@ -65,7 +65,7 @@ Date: 2026-06-10. Build: `release` for perf, `cargo test` for functionality. ### Test suites (all green) | Suite | Tests | Cmd | |---|---|---| -| MAGE prototype | **1094 pass** (+2 ignored perf harnesses) | `cargo test` | +| MAGE prototype | **1107 pass** (+2 ignored perf harnesses) | `cargo test` | | rmi (`cpu`) | **1380 pass** | `cargo test --no-default-features --features cpu` | | ribosome (build engine) | **164 pass** | `cargo test --manifest-path ribosome/Cargo.toml` | | germline (RSI control plane) | **112 pass** | `cargo test --manifest-path germline/Cargo.toml` | diff --git a/ROADMAP.md b/ROADMAP.md index 58b09c03f..ab5aec3ec 100644 --- a/ROADMAP.md +++ b/ROADMAP.md @@ -4,8 +4,8 @@ > Each step is a concrete, testable increment. > > **Last verified: 2026-08-11** — all five crates built and tested: prototype -> **1,094**, rmi **1,380**, ribosome **164**, germline **112**, forge **52** — -> **2,802 tests, 0 failures, 0 warnings**. The crate count went from three to +> **1,107**, rmi **1,380**, ribosome **164**, germline **112**, forge **52** — +> **2,815 tests, 0 failures, 0 warnings**. The crate count went from three to > five when the build engine (step 148) and the RSI control plane (step 149) > were extracted from `forge`; the total is unchanged by those moves, and > `forge`'s 52 is what the registry alone measured before they were parked in diff --git a/examples/effects-showcase/src/main.mg b/examples/effects-showcase/src/main.mg index 199c9a551..f6288cdb5 100644 --- a/examples/effects-showcase/src/main.mg +++ b/examples/effects-showcase/src/main.mg @@ -8,8 +8,8 @@ // - where that is *enforced*: private functions infer their effects, public // ones must declare them. The check is at the module boundary. // - composition, both `/ io, net` and `/ io + net` (the parser takes either) -// - a custom effect: any name that is not built in becomes one -// - `effect` declarations +// - a custom effect, declared by an `effect` block +// - performing an operation, and `handle … with` to *discharge* the effect // - `guard` for early exit, and `defer` // // The rule the checker enforces is *under*-declaration: a published function @@ -18,9 +18,10 @@ // description. Every claim in this file was checked by running // `mage-parse --check` on an edited copy, not by reading the compiler. // -// Not shown, because it does not exist yet: effect *handlers*. There is no -// `handle` form — `mage-parse --check` reports `unresolved name: handle`. An -// effect can be declared, annotated, inferred, and enforced, but not discharged. +// The elimination rule is `handle { … } with E { … }`: it removes an effect +// from the block it wraps, so a function can be *pure* despite calling +// something effectful. Handlers do not resume — an operation call dispatches to +// its arm and returns like an ordinary call. // // Run: forge run (or: mage-parse --eval src/main.mg main) @@ -28,10 +29,24 @@ // An `effect` block names an effect and the operations that belong to it. The // trailing semicolon on each signature is required. +// +// A declaration is not optional: an effect annotation naming nothing is an +// error. It used to be accepted, which meant `/ nte` was not a misspelling of +// `/ net` but a silently different effect that matched nothing. +// +// The declaration is `Audit`, the annotation is `/ audit`. The two spellings +// are matched case-insensitively. effect Audit { - fn record(entry: String); + fn record(entry: String) -> usize; } +// `db` is a custom effect too — it is not one of the built-in kinds (`io`, +// `net`, `fs`, `async`, `alloc`, `panic`, `ffi`, `env`, `time`, `gpu`, `npu`, +// `llm`, `evolve`, `learn`, `rng` — see `Effect::from_name` in `hir.rs`). It +// carries no operations, which is allowed: a bare `effect` block is how you +// name an effect you only want to track, not perform through. +effect Db {} + // ── Pure core ──────────────────────────────────────────────────────── // No annotation means no effects, and the checker holds this to it: calling @@ -77,10 +92,8 @@ fn jitter() -> i32 / rng { 17 } -// `db` is not one of the built-in effect kinds (`io`, `net`, `fs`, `async`, -// `alloc`, `panic`, `ffi`, `env`, `time`, `gpu`, `npu`, `llm`, `evolve`, -// `learn`, `rng` — see `Effect::from_name` in `hir.rs`). Any other name becomes -// a custom effect and is tracked the same way, so the system is open, not fixed. +// Custom effects propagate and are enforced exactly like built-in ones, so the +// system is open rather than fixed. fn persist(record: String) -> usize / db { len(chars(record)) } @@ -128,6 +141,33 @@ fn audit(entry: String) -> String / db { f"audited {len(chars(entry))} chars" } +// ── Performing an effect, and discharging it ───────────────────────── + +// `Audit.record(...)` *performs* the operation. That is what puts `audit` in +// this function's effect set — the annotation is checked against it, not the +// source of it. +fn transcribe(entry: String) -> usize / audit { + Audit.record(entry) +} + +// And here it is discharged. `handle { … } with Audit { … }` removes `audit` +// from the block, so this function is **pure** even though `transcribe` is not: +// `--check` reports `f summarize_audit: pure`. +// +// The subtraction is per-block, not per-function. A second, unhandled call to +// `transcribe` outside this `handle` would still be reported — handling one +// call does not launder the rest. +// +// Whatever the arm itself does is honestly attributed: make `record` call +// `persist` and this function becomes `/ db`, because that is what it now +// performs. A handler exchanges one effect for the effects of handling it. +fn summarize_audit(entry: String) -> String { + val n = handle { transcribe(entry) } with Audit { + record(e) => len(chars(e)) + } + f"recorded {n} chars" +} + // ── Entry point ────────────────────────────────────────────────────── // The union of everything reachable: `fs` and `net` from `check_host`, `time` @@ -140,6 +180,7 @@ pub fn main() -> String / fs, net, time, rng, db { val health = check_host("up.example", "app.toml") val stamped = stamp("up.example") val logged = audit(health) + val transcribed = summarize_audit(health) val codes = [200, 404, 503] val report = map(codes, fn(code) => severity(code)) @@ -150,6 +191,7 @@ pub fn main() -> String / fs, net, time, rng, db { health, stamped, logged, + transcribed, join(report, "/"), summarize(codes), ], diff --git a/prototype/src/ast.rs b/prototype/src/ast.rs index a216f9f9f..8c7420a09 100644 --- a/prototype/src/ast.rs +++ b/prototype/src/ast.rs @@ -284,6 +284,25 @@ pub enum Expr { scrutinee: Option>, arms: Vec, }, + /// `handle { body } with Audit { record(e) => … }` — the effect system's + /// elimination rule. + /// + /// Effects could be declared, annotated, inferred, and enforced, but never + /// *discharged*: there was no way to satisfy `/ audit` other than to keep + /// propagating it outward forever. This is what removes it. The handled + /// effect leaves the body's effect set; whatever the arms themselves do + /// takes its place, so handling `audit` by writing to a file is honestly + /// reported as `/ fs`. + /// + /// Handlers do not resume. An operation call dispatches to the matching arm + /// and returns its value like an ordinary call, which is what a + /// tree-walking evaluator can implement without capturing continuations. + Handle { + body: Block, + /// The declared `effect` block being discharged. + effect: String, + arms: Vec, + }, Loop { body: Block, }, @@ -366,6 +385,18 @@ pub struct MatchArm { pub body: Expr, } +/// One operation of a handled effect: `record(entry) => persist(entry)`. +/// +/// The parameters are bare names rather than patterns. Their types come from +/// the `effect` block's declaration of the operation, so writing them again +/// here would be a second place for them to disagree. +#[derive(Debug, Clone, Serialize, Deserialize)] +pub struct HandlerArm { + pub op: String, + pub params: Vec, + pub body: Expr, +} + #[derive(Debug, Clone, Serialize, Deserialize)] #[serde(tag = "type")] pub enum Pattern { diff --git a/prototype/src/effects.rs b/prototype/src/effects.rs index 359d1fbee..fabfe33e0 100644 --- a/prototype/src/effects.rs +++ b/prototype/src/effects.rs @@ -25,9 +25,39 @@ pub struct EffectInfer { call_graph: HashMap>, /// Currently being inferred (for cycle detection). in_progress: Vec, + /// Per function, the `handle` regions found in its body. + handled: HashMap>, + /// Regions collected while walking the function currently being collected. + /// Drained by `collect_function`; never meaningful between functions. + pending_regions: Vec, + /// Declared `effect` block names, lowercased. + /// + /// Lowercased because the two spellings of an effect are the declaration + /// (`effect Audit`) and the annotation (`/ audit`), and they have to name + /// the same thing. Keeping the fold here rather than at each use means the + /// convention is stated once. + effect_names: std::collections::HashSet, pub diagnostics: Vec, } +/// One `handle { … } with E { … }` occurrence. +/// +/// The calls inside the handled block are kept *out* of the function's ordinary +/// callee list and recorded here instead, because their effects have to be +/// resolved and then have `effect` removed before they join the enclosing +/// function's set. Discharging by simply deleting the effect from the whole +/// function would be unsound — an unhandled call to the same operation +/// elsewhere in the body would be silenced along with it. +#[derive(Debug, Clone)] +struct HandledRegion { + /// The effect this region discharges. + effect: Effect, + /// Functions called inside the handled block. + callees: Vec, + /// Effects performed directly inside the handled block. + local: EffectSet, +} + impl Default for EffectInfer { fn default() -> Self { Self::new() @@ -41,6 +71,9 @@ impl EffectInfer { inferred: HashMap::new(), call_graph: HashMap::new(), in_progress: Vec::new(), + handled: HashMap::new(), + pending_regions: Vec::new(), + effect_names: std::collections::HashSet::new(), diagnostics: Vec::new(), } } @@ -52,6 +85,16 @@ impl EffectInfer { // declarations are required here; private functions infer silently — // their effects still propagate to any public caller (see Pass 3). let mut boundary: std::collections::HashSet = std::collections::HashSet::new(); + + // Pass 0: the declared effects. Needed before anything else, because an + // operation call introduces its effect and an annotation naming no + // declared effect is an error — both need the full set up front. + for item in &module.items { + if let ast::ItemKind::Effect(ed) = &item.kind { + self.effect_names.insert(ed.name.to_lowercase()); + } + } + // Pass 1: collect function declarations and their call graphs. for item in &module.items { if let ast::ItemKind::Function(fd) = &item.kind { @@ -62,6 +105,34 @@ impl EffectInfer { } } + // Pass 1.5: every effect named in an annotation must exist — a built-in + // kind, or an `effect` block in this module. + // + // Anything else used to become `Effect::Custom` silently, so `/ nte` + // was not a misspelling of `/ net` but a *different effect*, enforced + // perfectly consistently and matching nothing. A typo invented an + // effect rather than failing, which is the most expensive way for a + // capability system to be wrong. + let mut named: Vec<(&String, &EffectSet)> = self.declared.iter().collect(); + named.sort_by_key(|(n, _)| n.as_str()); + for (func, effects) in named { + for effect in effects { + let Effect::Custom(name) = effect else { continue }; + if self.effect_names.contains(&name.to_lowercase()) { + continue; + } + self.diagnostics.push(Diagnostic::categorized( + crate::hir::Severity::Error, + format!( + "function `{func}` declares unknown effect `{name}` — it is not \ + a built-in kind, and no `effect {name} {{ … }}` declares it" + ), + DiagnosticCategory::UndeclaredEffect, + None, + )); + } + } + // Pass 2: infer effects bottom-up. let fn_names: Vec = self.call_graph.keys().cloned().collect(); for name in &fn_names { @@ -123,7 +194,12 @@ impl EffectInfer { // Build call graph for this function by walking its body. let mut callees = Vec::new(); let mut local_effects = EffectSet::new(); + self.pending_regions.clear(); self.collect_calls_in_block(&fd.body, &mut callees, &mut local_effects); + let regions = std::mem::take(&mut self.pending_regions); + if !regions.is_empty() { + self.handled.insert(fd.name.clone(), regions); + } self.call_graph.insert(fd.name.clone(), callees); @@ -134,7 +210,7 @@ impl EffectInfer { } fn collect_calls_in_block( - &self, + &mut self, block: &ast::Block, callees: &mut Vec, local_effects: &mut EffectSet, @@ -148,7 +224,7 @@ impl EffectInfer { } fn collect_calls_in_stmt( - &self, + &mut self, stmt: &ast::Stmt, callees: &mut Vec, local_effects: &mut EffectSet, @@ -178,7 +254,7 @@ impl EffectInfer { } fn collect_calls_in_expr( - &self, + &mut self, expr: &ast::Expr, callees: &mut Vec, local_effects: &mut EffectSet, @@ -205,7 +281,18 @@ impl EffectInfer { self.collect_calls_in_expr(arg, callees, local_effects); } } - ast::Expr::MethodCall { receiver, args, .. } => { + ast::Expr::MethodCall { receiver, method, args, .. } => { + // `Audit.record(x)` performs `audit`. This is the introduction + // rule: without it an `effect` block declared operations that + // no analysis attributed to anyone, so a function calling one + // inferred `pure` while claiming `/ audit`. + if let ast::Expr::Ident { name } = receiver.as_ref() { + let lowered = name.to_lowercase(); + if self.effect_names.contains(&lowered) { + let _ = method; + local_effects.insert(Effect::from_name(&lowered)); + } + } self.collect_calls_in_expr(receiver, callees, local_effects); for arg in args { self.collect_calls_in_expr(arg, callees, local_effects); @@ -230,6 +317,24 @@ impl EffectInfer { self.collect_calls_in_expr(&arm.body, callees, local_effects); } } + // The handled block's calls go into a *separate* bucket, so the + // effect can be removed from them alone. The arms are different: + // they really do run in the enclosing function, so whatever they + // perform belongs to it — handling `audit` by writing a file is + // honestly reported as `/ fs`. + ast::Expr::Handle { body, effect, arms } => { + let mut inner_callees = Vec::new(); + let mut inner_local = EffectSet::new(); + self.collect_calls_in_block(body, &mut inner_callees, &mut inner_local); + self.pending_regions.push(HandledRegion { + effect: Effect::from_name(&effect.to_lowercase()), + callees: inner_callees, + local: inner_local, + }); + for arm in arms { + self.collect_calls_in_expr(&arm.body, callees, local_effects); + } + } ast::Expr::Loop { body } => { self.collect_calls_in_block(body, callees, local_effects); } @@ -399,6 +504,23 @@ impl EffectInfer { effects.extend(callee_effects); } + // Handled regions, resolved the same way and then discharged. The + // subtraction happens per region rather than over the whole function, + // so a second, unhandled call to the same effect elsewhere in the body + // still surfaces. + for region in self.handled.get(name).cloned().unwrap_or_default() { + let mut region_effects = region.local.clone(); + for callee in ®ion.callees { + let mut callee_effects = self.infer_function(callee); + if let Some(declared) = self.declared.get(callee) { + callee_effects.extend(declared.iter().cloned()); + } + region_effects.extend(callee_effects); + } + region_effects.remove(®ion.effect); + effects.extend(region_effects); + } + self.in_progress.retain(|n| n != name); self.inferred.insert(name.to_string(), effects.clone()); effects @@ -674,3 +796,122 @@ mod tests { assert!(ei.effects_of("with_io").is_empty()); } } + +#[cfg(test)] +mod handler_tests { + use super::*; + use crate::lexer; + use crate::parser; + + fn infer_source(src: &str) -> EffectInfer { + let tokens = lexer::lex(src); + let module = parser::parse(&tokens).expect("parse failed"); + infer_effects(&module) + } + + fn errors(src: &str) -> Vec { + infer_source(src) + .diagnostics + .iter() + .map(|d| d.message.clone()) + .collect() + } + + const DECL: &str = "effect Audit { f record(e: str) -> usize; }\n"; + + /// The introduction rule. An `effect` block used to declare operations that + /// no analysis attributed to anyone, so a function calling one inferred + /// `pure` while claiming `/ audit`. + #[test] + fn performing_an_operation_introduces_its_effect() { + let src = format!("{DECL}+f w(e: str) -> usize {{ Audit.record(e) }}"); + let msgs = errors(&src); + assert!( + msgs.iter().any(|m| m.contains("undeclared effects") && m.contains("audit")), + "expected `audit` to be attributed to `w`, got {msgs:?}" + ); + } + + /// The elimination rule: the whole point. `main` is pure despite calling + /// something that performs `audit`. + #[test] + fn handle_discharges_the_effect_it_names() { + let src = format!( + "{DECL}f w(e: str) -> usize / audit {{ Audit.record(e) }}\n\ + +f main() -> usize {{ handle {{ w(\"x\") }} with Audit {{ record(e) => len(chars(e)) }} }}" + ); + assert!(errors(&src).is_empty(), "errors: {:?}", errors(&src)); + } + + /// The soundness property. Discharging by deleting the effect from the + /// whole function would silence this second, *unhandled* call — so the + /// subtraction is per handled block instead. + #[test] + fn an_unhandled_call_beside_a_handled_one_still_reports() { + let src = format!( + "{DECL}f w(e: str) -> usize / audit {{ Audit.record(e) }}\n\ + +f main() -> usize {{ v a = handle {{ w(\"x\") }} with Audit {{ record(e) => 1 }}\n\ + v b = w(\"y\")\n a + b }}" + ); + let msgs = errors(&src); + assert!( + msgs.iter().any(|m| m.contains("audit")), + "the unhandled call must still surface, got {msgs:?}" + ); + } + + /// A handler is not free: handling `audit` by touching the filesystem makes + /// the handling function perform `fs`. + #[test] + fn a_handlers_own_effects_are_attributed_to_the_handling_function() { + let src = format!( + "{DECL}f w(e: str) -> usize / audit {{ Audit.record(e) }}\n\ + f to_disk(e: str) -> usize / fs {{ 1 }}\n\ + +f main() -> usize {{ handle {{ w(\"x\") }} with Audit {{ record(e) => to_disk(e) }} }}" + ); + let msgs = errors(&src); + assert!( + msgs.iter().any(|m| m.contains("FS")), + "expected the handler's `fs` to surface, got {msgs:?}" + ); + } + + /// An effect annotation naming nothing used to be accepted, so `/ nte` was + /// a different effect rather than a misspelling of `/ net`. + #[test] + fn an_effect_annotation_naming_nothing_is_an_error() { + let msgs = errors("+f a() -> i32 / nte { 1 }"); + assert!( + msgs.iter().any(|m| m.contains("unknown effect")), + "expected an unknown-effect diagnostic, got {msgs:?}" + ); + } + + /// A misspelled operation. The effect analysis attributes the effect from + /// the *receiver* alone, so this was counted as performing `audit`, checked + /// clean, and then died at run time with `unknown function` — the same bug + /// one level down from the one this feature exists to fix. + #[test] + fn a_misspelled_operation_on_a_declared_effect_is_rejected() { + let tokens = crate::lexer::lex(&format!("{DECL}+f a() -> i32 / audit {{ Audit.recrod(1) }}")); + let module = crate::parser::parse(&tokens).expect("parse failed"); + let tc = crate::types::check(&module); + assert!( + tc.diagnostics.iter().any(|d| d.message.contains("declares no operation")), + "expected a misspelled-operation diagnostic, got {:?}", + tc.diagnostics + ); + } + + #[test] + fn a_declared_custom_effect_is_accepted() { + assert!(errors("effect Db {}\n+f a() -> i32 / db { 1 }").is_empty()); + } + + /// Built-in kinds need no declaration — the rule is about names that mean + /// nothing, not about forcing boilerplate for `fs`. + #[test] + fn builtin_effect_kinds_need_no_declaration() { + assert!(errors("+f a() -> i32 / fs, net, rng { 1 }").is_empty()); + } +} diff --git a/prototype/src/elision.rs b/prototype/src/elision.rs index b7a993f5e..a8d40b2f6 100644 --- a/prototype/src/elision.rs +++ b/prototype/src/elision.rs @@ -445,6 +445,18 @@ fn elide_expr(expr: &Expr) -> Expr { | Expr::Continue | Expr::Error { .. } => expr.clone(), + Expr::Handle { body, effect, arms } => Expr::Handle { + body: elide_block(body), + effect: effect.clone(), + arms: arms + .iter() + .map(|a| crate::ast::HandlerArm { + op: a.op.clone(), + params: a.params.clone(), + body: elide_expr(&a.body), + }) + .collect(), + }, Expr::Binary { op, left, right } => Expr::Binary { op: op.clone(), left: Box::new(elide_expr(left)), diff --git a/prototype/src/eval.rs b/prototype/src/eval.rs index 1f139f445..28e94883a 100644 --- a/prototype/src/eval.rs +++ b/prototype/src/eval.rs @@ -129,6 +129,7 @@ fn err(m: impl Into) -> Result { } /// Lexical environment: a stack of scopes. +#[derive(Clone)] struct Env { scopes: Vec>, } @@ -187,6 +188,26 @@ pub struct Interp { /// same (`Left { X }` and `Right { X }`), and keying by variant alone let the /// second registration evict the first, so one of the two stopped resolving. enum_variants: HashMap<(String, String), usize>, + /// Effect handlers currently installed, innermost last. + /// + /// A dynamic stack, because that is what an effect handler is: the call + /// that performs the operation may be many frames below the `handle` that + /// discharges it, and it does not know which handler it will reach. The + /// *arm bodies*, though, close over the environment where the handler was + /// written, so a handler is dynamically found and lexically evaluated. + /// + /// `RefCell` because `eval` takes `&self`; the alternative is threading a + /// handler stack through every expression form. + handlers: RefCell>, +} + +/// One installed `handle … with E { … }`. +struct HandlerFrame { + effect: String, + /// op name → (parameter names, body). + arms: HashMap, Expr)>, + /// The environment the handler was written in, captured at installation. + env: Env, } impl Interp { @@ -219,7 +240,24 @@ impl Interp { _ => {} } } - Interp { funcs, methods, enum_variants } + Interp { funcs, methods, enum_variants, handlers: RefCell::new(Vec::new()) } + } + + /// The innermost installed handler for `effect.op`, if any. + /// + /// Innermost-first, so a nested `handle` shadows an outer one for the same + /// effect — the ordinary scoping rule, and the reason the stack is searched + /// backwards rather than forwards. + fn handler_for(&self, effect: &str, op: &str) -> Option<(Vec, Expr, Env)> { + self.handlers + .borrow() + .iter() + .rev() + .find(|f| f.effect == effect && f.arms.contains_key(op)) + .map(|f| { + let (params, body) = &f.arms[op]; + (params.clone(), body.clone(), f.env.clone()) + }) } /// The type name a value dispatches on, for values that can carry methods. @@ -434,6 +472,26 @@ impl Interp { } } Expr::Block { block } => self.eval_block(block, env), + Expr::Handle { body, effect, arms } => { + self.handlers.borrow_mut().push(HandlerFrame { + effect: effect.clone(), + arms: arms + .iter() + .map(|a| (a.op.clone(), (a.params.clone(), a.body.clone()))) + .collect(), + // The environment the handler was *written* in. An arm + // must not see the locals of whatever frame happened to + // perform the operation. + env: env.clone(), + }); + // Bound rather than `?`-propagated: the frame has to come off + // the stack even when the body returns or errors, or a `return` + // out of a handled block would leave the handler installed for + // everything that ran afterwards. + let out = self.eval_block(body, env); + self.handlers.borrow_mut().pop(); + out + } Expr::Match { scrutinee, arms } => { let v = match scrutinee { Some(e) => self.eval(e, env)?, @@ -597,6 +655,33 @@ impl Interp { // shape, and is a constructor rather than a call: check it // first, since there is no receiver value to evaluate. if let Expr::Ident { name } = receiver.as_ref() { + // An effect operation: `Audit.record(x)`. Dispatched to the + // innermost installed handler. Before this, an operation + // declared in an `effect` block typechecked and then died + // with `unknown function`, because nothing ever read the + // block's operations. + if let Some((params, body, captured)) = self.handler_for(name, method) { + let mut av = Vec::with_capacity(args.len()); + for a in args { + av.push(self.eval(a, env)?); + } + if params.len() != av.len() { + return err(format!( + "effect operation `{name}.{method}` takes {} argument(s), \ + given {}", + params.len(), + av.len() + )); + } + // Evaluated in the handler's own environment, with the + // operation's arguments bound on top. + let mut henv = captured; + henv.push(); + for (p, v) in params.iter().zip(av) { + henv.define(p.clone(), v); + } + return self.eval(&body, &mut henv); + } if let Some((enum_name, arity)) = self.as_enum_variant(name, method) { if arity != args.len() { return err(format!( @@ -2023,3 +2108,69 @@ mod tests { assert_eq!(run(src, "s", &[]), Value::Int(25)); } } + +#[cfg(test)] +mod handler_eval_tests { + use super::*; + + fn run(src: &str, f: &str) -> Value { + run_source(src, f, &[]).expect("run failed") + } + + const DECL: &str = "effect Audit { f record(e: i32) -> i32; }\n"; + + /// An operation declared in an `effect` block used to typecheck and then + /// die with `unknown function \`record\``, because nothing read the block's + /// operations at run time. + #[test] + fn an_operation_dispatches_to_its_handler() { + let src = format!( + "{DECL}f w() {{ Audit.record(7) }}\n\ + f s() {{ handle {{ w() }} with Audit {{ record(e) => e * 3 }} }}" + ); + assert_eq!(run(&src, "s"), Value::Int(21)); + } + + /// A handler is found dynamically — `w` does not mention the handler and + /// sits a call frame below it. + #[test] + fn the_innermost_handler_wins() { + let src = format!( + "{DECL}f w() {{ Audit.record(1) }}\n\ + f s() {{ handle {{ handle {{ w() }} with Audit {{ record(e) => 20 }} }} \ + with Audit {{ record(e) => 10 }} }}" + ); + assert_eq!(run(&src, "s"), Value::Int(20)); + } + + /// …but evaluated lexically: the arm sees the environment the *handler* was + /// written in, not the frame that performed the operation. + #[test] + fn an_arm_body_sees_the_handlers_own_scope() { + let src = format!( + "{DECL}f w() {{ v hidden = 999\n Audit.record(1) }}\n\ + f s() {{ v scale = 5\n handle {{ w() }} with Audit {{ record(e) => e * scale }} }}" + ); + assert_eq!(run(&src, "s"), Value::Int(5)); + } + + /// The frame comes off the stack when the block ends, or everything after a + /// handled call would keep dispatching to a handler that is out of scope. + #[test] + fn a_handler_does_not_outlive_its_block() { + let src = format!( + "{DECL}f w() {{ Audit.record(1) }}\n\ + f s() {{ v a = handle {{ w() }} with Audit {{ record(e) => 4 }}\n a + w() }}" + ); + let out = run_source(&src, "s", &[]); + assert!(out.is_err(), "the second, unhandled call must fail: {out:?}"); + } + + #[test] + fn the_handled_block_still_produces_its_own_value() { + let src = format!( + "{DECL}f s() {{ handle {{ 6 * 7 }} with Audit {{ record(e) => 0 }} }}" + ); + assert_eq!(run(&src, "s"), Value::Int(42)); + } +} diff --git a/prototype/src/fmt.rs b/prototype/src/fmt.rs index b3a92ee2c..00d1ef055 100644 --- a/prototype/src/fmt.rs +++ b/prototype/src/fmt.rs @@ -1264,6 +1264,27 @@ fn emit_expr(buf: &mut String, expr: &Expr, mode: Mode) { buf.push('}'); } } + Expr::Handle { body, effect, arms } => { + match mode { + Mode::Agent => buf.push_str("hx "), + Mode::Human => buf.push_str("handle "), + } + buf.push_str("{\n"); + emit_block_body(buf, body, mode, 1); + buf.push_str("} with "); + buf.push_str(effect); + buf.push_str(" {\n"); + for arm in arms { + buf.push_str(" "); + buf.push_str(&arm.op); + buf.push('('); + buf.push_str(&arm.params.join(", ")); + buf.push_str(") => "); + emit_expr(buf, &arm.body, mode); + buf.push_str(",\n"); + } + buf.push('}'); + } Expr::Match { scrutinee, arms } => { match mode { Mode::Agent => buf.push_str("?= "), diff --git a/prototype/src/mlir.rs b/prototype/src/mlir.rs index bad680366..43b16f52f 100644 --- a/prototype/src/mlir.rs +++ b/prototype/src/mlir.rs @@ -630,6 +630,9 @@ impl<'a> EmitCtx<'a> { ast::Expr::Match { arms, .. } => { format!("MAGE.match({} arms)", arms.len()) } + ast::Expr::Handle { effect, arms, .. } => { + format!("MAGE.handle({effect}, {} ops)", arms.len()) + } ast::Expr::Loop { .. } => "MAGE.loop { ... }".to_string(), ast::Expr::While { .. } => "MAGE.while { ... }".to_string(), ast::Expr::For { .. } => "MAGE.for { ... }".to_string(), diff --git a/prototype/src/parser.rs b/prototype/src/parser.rs index 53cc34e86..3a8e9320b 100644 --- a/prototype/src/parser.rs +++ b/prototype/src/parser.rs @@ -1088,6 +1088,49 @@ impl<'a> Parser<'a> { // ── Effect ────────────────────────────────────────────── + /// `handle { body } with Audit { record(e) => …, flush() => … }` + /// + /// The effect name is required rather than inferred from the arms: an + /// operation name alone does not say which effect it belongs to, and two + /// effects are allowed to declare an operation with the same name. + fn parse_handle_expr(&mut self) -> Result { + self.advance(); // `handle` / `hx` + let body = self.parse_block()?; + + // `with` is not a keyword — the lexer leaves it a plain identifier — + // so it is matched on text, the same way `as` is for casts. + if self.peek() != TokenKind::Ident || self.current().text != "with" { + return Err(self.error("expected `with ` after `handle { … }`")); + } + self.advance(); + + let effect = self.expect_ident()?; + self.expect(TokenKind::LBrace)?; + + let mut arms = Vec::new(); + while self.peek() != TokenKind::RBrace && self.peek() != TokenKind::Eof { + let op = self.expect_ident()?; + self.expect(TokenKind::LParen)?; + let mut params = Vec::new(); + while self.peek() != TokenKind::RParen && self.peek() != TokenKind::Eof { + params.push(self.expect_ident()?); + if self.peek() == TokenKind::Comma { + self.advance(); + } + } + self.expect(TokenKind::RParen)?; + self.expect(TokenKind::FatArrow)?; + let arm_body = self.parse_expr()?; + arms.push(HandlerArm { op, params, body: arm_body }); + if self.peek() == TokenKind::Comma { + self.advance(); + } + } + self.expect(TokenKind::RBrace)?; + + Ok(Expr::Handle { body, effect, arms }) + } + fn parse_effect_def(&mut self) -> Result { self.expect(TokenKind::KwEffect)?; let name = self.expect_ident()?; @@ -3721,6 +3764,16 @@ impl<'a> Parser<'a> { // Same shape as the sum-type-constructor arm below: parse // as a plain Ident, with an optional call-args suffix so // `guard(x)` works too. + // `handle {` is the effect elimination form; anything else + // beginning with `handle` is the corpus using it as a plain name + // (`handle.spawn(...)`), which the arm below still allows. The + // brace is what separates them, and no identifier use can be + // followed by one — a `{` after a bare name would be a block, not + // a continuation of the expression. + TokenKind::KwHandle if self.peek_n(1) == TokenKind::LBrace => { + self.parse_handle_expr() + } + TokenKind::KwGuard | TokenKind::KwHandle | TokenKind::KwNet diff --git a/prototype/src/resolve.rs b/prototype/src/resolve.rs index 456aa6cbe..59f87449e 100644 --- a/prototype/src/resolve.rs +++ b/prototype/src/resolve.rs @@ -848,6 +848,21 @@ impl Resolver { self.pop_scope(); } } + // The handled body resolves in the enclosing scope; each arm gets + // its own, with the operation's parameters bound as plain names. + // Their types come from the `effect` declaration, so the resolver + // only has to make them visible. + ast::Expr::Handle { body, arms, .. } => { + self.resolve_block(body); + for arm in arms { + self.push_scope(); + for p in &arm.params { + self.define_value(p, SymbolKind::Variable { mutable: false }); + } + self.resolve_expr(&arm.body); + self.pop_scope(); + } + } ast::Expr::Loop { body } => { self.resolve_block(body); } diff --git a/prototype/src/token_budget.rs b/prototype/src/token_budget.rs index cbfe426fa..5504543bf 100644 --- a/prototype/src/token_budget.rs +++ b/prototype/src/token_budget.rs @@ -474,6 +474,14 @@ fn count_stmt_agent(stmt: &Stmt) -> u32 { fn count_expr_agent(expr: &Expr) -> u32 { match expr { + // `hx { … } with E { op(p) => … }` + Expr::Handle { body, arms, .. } => { + let mut n = 2 + count_block_agent(body); // `hx` + effect name + for arm in arms { + n += 3 + arm.params.len() as u32 + count_expr_agent(&arm.body); + } + n + } Expr::Literal { .. } => 1, Expr::Ident { .. } => 1, Expr::Binary { left, right, .. } => { @@ -920,6 +928,13 @@ fn count_stmt_human(stmt: &Stmt) -> u32 { fn count_expr_human(expr: &Expr) -> u32 { match expr { + Expr::Handle { body, arms, .. } => { + let mut n = 3 + count_block_human(body); // `handle` + `with` + name + for arm in arms { + n += 3 + arm.params.len() as u32 + count_expr_human(&arm.body); + } + n + } Expr::Literal { .. } => 1, Expr::Ident { .. } => 1, Expr::Binary { left, right, .. } => 1 + count_expr_human(left) + count_expr_human(right), diff --git a/prototype/src/types.rs b/prototype/src/types.rs index ba2e4f0e9..86d7982ce 100644 --- a/prototype/src/types.rs +++ b/prototype/src/types.rs @@ -307,6 +307,16 @@ pub struct TypeChecker { /// Enum definitions: enum name → its variant names. Used for match /// exhaustiveness checking. enum_defs: HashMap>, + /// Operations of each declared `effect` block: `(effect, op)` → parameter + /// types and return type. + /// + /// Before this the operations in an `effect` block were parsed, stored, and + /// read by nothing — an `effect` declaration was decoration. They are the + /// signature an operation call is checked against and the types a handler + /// arm's parameters are bound at, so they exist in exactly one place. + effect_ops: HashMap<(String, String), (Vec, Ty)>, + /// Names of declared `effect` blocks, in declaration order. + effect_defs: Vec, /// Interned names of user-defined types: name → id, and id → name. /// /// Gives `Ty::Named` real identity. Two vectors rather than one bimap @@ -344,6 +354,8 @@ impl TypeChecker { struct_defs: HashMap::new(), fn_sigs: HashMap::new(), enum_defs: HashMap::new(), + effect_ops: HashMap::new(), + effect_defs: Vec::new(), named_ids: HashMap::new(), named_names: Vec::new(), ret_stack: Vec::new(), @@ -520,6 +532,20 @@ impl TypeChecker { fn collect_item_sig(&mut self, item: &ast::Item) { match &item.kind { + ast::ItemKind::Effect(ed) => { + self.effect_defs.push(ed.name.clone()); + for op in &ed.operations { + let params: Vec = + op.params.iter().map(|p| self.lower_type(&p.ty)).collect(); + // An operation with no `->` returns unit, like a function. + let ret = match &op.return_type { + Some(t) => self.lower_type(t), + None => Ty::Unit, + }; + self.effect_ops + .insert((ed.name.clone(), op.name.clone()), (params, ret)); + } + } ast::ItemKind::Function(fd) => { let params: Vec = fd.params.iter().map(|p| self.lower_type(&p.ty)).collect(); // No return annotation → a fresh inference var, resolved from the @@ -1324,7 +1350,56 @@ impl TypeChecker { self.subst.apply(&ret) } - ast::Expr::MethodCall { receiver, args, .. } => { + ast::Expr::MethodCall { receiver, method, args, .. } => { + // `Audit.record(x)` — an effect operation, not a method. The + // receiver is the effect's name, so it is checked before the + // receiver is inferred as a value; there is no value there to + // infer, and treating it as one is what made an operation call + // return a fresh variable and accept any arguments at all. + if let ast::Expr::Ident { name } = receiver.as_ref() + && let Some((params, ret)) = + self.effect_ops.get(&(name.clone(), method.clone())).cloned() + { + if params.len() != args.len() { + self.emit_error(format!( + "effect operation `{name}.{method}` takes {} argument(s), \ + given {}", + params.len(), + args.len() + )); + } + for (arg, want) in args.iter().zip(params.iter()) { + let got = self.infer_expr(arg); + if let Err(e) = unify(&mut self.subst, &got, want) { + self.emit_error(format!( + "effect operation `{name}.{method}`: {e}" + )); + } + } + return ret; + } + // The receiver names a declared effect, but the operation is + // not one of its declarations. That is a misspelling, and it + // has to be an error here: the effect analysis attributes the + // effect on the *receiver* alone, so `Audit.recrod(x)` was + // accepted, counted as performing `audit`, and then died at run + // time with `unknown function`. Exactly the shape of bug this + // whole feature exists to stop being possible. + if let ast::Expr::Ident { name } = receiver.as_ref() + && self.effect_defs.contains(name) + { + let mut ops: Vec<&str> = self + .effect_ops + .keys() + .filter(|(e, _)| e == name) + .map(|(_, op)| op.as_str()) + .collect(); + ops.sort_unstable(); + self.emit_error(format!( + "effect `{name}` declares no operation `{method}` (it declares: {})", + if ops.is_empty() { "none".to_string() } else { ops.join(", ") } + )); + } self.infer_expr(receiver); for arg in args { self.infer_expr(arg); @@ -1560,6 +1635,54 @@ impl TypeChecker { result } + // `handle { body } with E { op(p) => … }`. The value is the body's, + // exactly as if the handler were not there — a handler discharges + // an effect, it does not change what the computation produces. + ast::Expr::Handle { body, effect, arms } => { + if !self.effect_defs.contains(effect) { + self.emit_error(format!( + "unknown effect `{effect}`: `handle … with` needs an \ + `effect {effect} {{ … }}` declaration" + )); + } + for arm in arms { + let Some((params, ret)) = + self.effect_ops.get(&(effect.clone(), arm.op.clone())).cloned() + else { + self.emit_error(format!( + "effect `{effect}` declares no operation `{}`", + arm.op + )); + continue; + }; + if params.len() != arm.params.len() { + self.emit_error(format!( + "handler for `{effect}.{}` binds {} parameter(s), the \ + operation declares {}", + arm.op, + arm.params.len(), + params.len() + )); + } + // The arm's parameters take their types from the effect + // declaration rather than from annotations on the arm, so + // there is only one place for them to be written. + self.env.push(); + for (name, ty) in arm.params.iter().zip(params.iter()) { + self.env.insert(name.clone(), ty.clone()); + } + let body_ty = self.infer_expr(&arm.body); + self.env.pop(); + if let Err(e) = unify(&mut self.subst, &body_ty, &ret) { + self.emit_error(format!( + "handler for `{effect}.{}` must produce what the \ + operation returns: {e}", + arm.op + )); + } + } + self.infer_block(body) + } ast::Expr::Loop { body } => { self.infer_block(body); // Loop type is determined by break expressions. diff --git a/scripts/check-examples.sh b/scripts/check-examples.sh index 7e4215a9a..d2b66ff5b 100755 --- a/scripts/check-examples.sh +++ b/scripts/check-examples.sh @@ -78,7 +78,7 @@ declare -A EXPECTED=( [cli-tool]='"-i alpha -> alpha beta | ALPHA delta; -c beta -> 2; no pattern -> usage error"' [cost-aware-optimizer]='"3 benchmark samples; x86-64: size | aarch64: balanced | riscv64: balanced | wasm32: balanced; x86-64: size | aarch64: size | riscv64: no candidate within budget | wasm32: balanced"' [data-structures]='"points=3, total_distance=15, closest=3"' - [effects-showcase]='"statuses=3 missing=0; 3 configured, live ok; up.example@1700000017; audited 21 chars; ok/warn/error; 3 samples, worst 503"' + [effects-showcase]='"statuses=3 missing=0; 3 configured, live ok; up.example@1700000017; audited 21 chars; recorded 21 chars; ok/warn/error; 3 samples, worst 503"' [hello-world]='"Hello, MAGE! (your name has 4 letters)"' [http-client]='"1: user 1 Ada (active=true); 2: rate limited, retry in 30s; 3: not found; 4: decode: expected 3 fields, got 1"' [live-compiler]='"live r2 passing 4; rollbacks 1; after explicit rollback r1 passing 3; type: handler:9 unresolved placeholder; revert placeholder@70; no repair proposed"' diff --git a/scripts/test-all.ps1 b/scripts/test-all.ps1 index 64813680d..70abcf692 100644 --- a/scripts/test-all.ps1 +++ b/scripts/test-all.ps1 @@ -8,12 +8,12 @@ This is the single entry point that covers everything CI covers: rmi (cpu) 1,380 tests - prototype 1,094 tests + prototype 1,107 tests ribosome 164 tests germline 112 tests forge 52 tests ------------------------- - total 2,802 tests, 0 warnings + total 2,815 tests, 0 warnings .PARAMETER Release Build and test in release mode (slower to build, much faster to run). diff --git a/scripts/test-all.sh b/scripts/test-all.sh index 1f1a58d2d..b409d876b 100755 --- a/scripts/test-all.sh +++ b/scripts/test-all.sh @@ -6,12 +6,12 @@ # This is the single entry point that covers everything CI covers: # # rmi (cpu) 1,380 tests -# prototype 1,094 tests +# prototype 1,107 tests # ribosome 164 tests # germline 112 tests # forge 52 tests # ------------------------- -# total 2,802 tests, 0 warnings +# total 2,815 tests, 0 warnings # # Usage: # scripts/test-all.sh # debug