Capstone: lead-finder as a pure declarative pipeline (#94) - #117
Merged
Conversation
…e proof (#94) Author the lead-finder as DATA, not code: lead-finder.json is a PipelineDef (source=geocode+http_request Places, map=reshape/derive, filter=no-website, dedupe_upsert by place_id, sink=`leads`), validated by validatePipeline and driven end-to-end through the real runner (executePipelineStep) + real map/filter/dedupe handlers in lead-finder.test.ts. Only the two I/O boundaries (outbound HTTP, the collection sink DO) are mocked; the composable spine + per-record audit trail (attachAudit) are proven with the right 2->1 no-website lead and place_id dedupe. Honest about the four expressibility gaps the composition hit (each asserted as a "GAP #N" test, documented in README.md, filed as #94 children): - #113 grid fan-out flatten (forEach -> array-of-arrays, no flatten primitive) - #114 dotted access into a forEach item ($param:"item.field") - #115 enrich-merge (join a forEach result back onto its records) - #116 map/responseMap type-select of Google addressComponents Reference-only: lives entirely in the platform repo; the running lead-finder agent + worker are untouched. api tsc clean; api suite 880 -> 889, 0 regressions. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Closed
5 tasks
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
2 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Part of #94 — the capstone proof. Reference definition + end-to-end composition test + gap map. The running lead-finder agent/worker is untouched; everything lives in
workers/api/src/lib/pipelines/.Proven composable spine (100% declarative, end-to-end through the real
executePipelineStep+ real map/filter/dedupe handlers; only outbound HTTP + sink DO mocked):geocode(city→centre) →http_requestplaces:searchNearby(vault key header, FieldMask, responseMap → typed rows) →map(derive category/status + geo) →filter(websiteUri missing) →dedupe_upsert(key place_id) → sinkleads, with a per-record{step,detail,at}audit trail. Test drives a realistic 3-business response (no-website / dead-site / live-site) → surviving lead is the no-website one, upserted once on re-run.Verdict: the spine yes, the full sweep not yet. Four runner/step primitives stand between the spine and the complete sweep — each asserted by a GAP test and filed as a #94 child:
arr[]$ref grammar for grid fan-out (child of #94) #113 flatten /arr[]$ref for grid fan-out (array-of-arrays from forEach)item($param:"item.field") (child of #94) #114 dotted access into a forEachitem($param:'item.field')map/responseMap type-predicate select for GoogleaddressComponentsNone require the running agent — all are platform primitives. A green test hiding any of these would have been the wrong outcome; each is proven present.
Verify:
tscclean · capstone 9/9 pass · full api suite 889 (880 baseline + 9, 0 regressions) · biome clean.🤖 Generated with Claude Code