Skip to content

Pipeline step library: enrich-merge (join a forEach result back onto its records) (child of #94) #115

Description

@serge-ivo

Gap found by the #94 capstone ("GAP #3" in lead-finder.test.ts).

Problem. The epic proof filter is "keep no-website OR unreachable." The "unreachable" half needs a reachability probe per place, then that {ok,code} merged back ONTO the place so filter can test it. Today http_reachable via forEach binds a parallel array [{ok,code},…] that sits beside the places with no way to correlate them: map/filter/dedupe_upsert each read a single items array, and there is no zip/join step. So only the "no-website" half of the filter is expressible; "unreachable" isn't.

Evidence. In the capstone the reachability array and the places array are the same length but nothing joins them by index; the shipped JSON filters on websiteUri missing only. Asserted as GAP #3.

Proposed fix (pick one):

  1. An enrich step: {items:$ref, tool:"http_reachable", input:{url:{$item:"websiteUri"}}, as:"reachable"} — runs the tool per item (concurrency-capped, like fan_out) and writes each result under as on the item. This is the general "call a connector per record and merge" primitive the epic's enrich stage wants (also covers [connectors] Web-search connector (Google Custom Search / SerpAPI) — enrichment #99 web_search→extract_contacts for socials/email).
  2. A zip/merge_by_index step joining two bound arrays.

Option 1 (a real enrich step) is the declarative primitive the epic's enrich bullet implies; it also depends on #94-child "dotted item access" for {$item:…} ergonomics. Part of #94.

Activity

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    No labels
    No labels

    Type

    No type

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions