Epic: declarative data pipelines — the "configure it, don't code it" half of #93
Evidence (lead-finder, 2026-08-02). The lead-finder is a deterministic data pipeline: geocode → grid → paginate Places → reachability-check → classify → dedupe → store. Today the ONLY ways to host that shape are a closed first-party Workflow or a standalone external Worker (#93) — i.e. code. That directly contradicts #58's operating principle: "build every agent as declarative data, never bespoke platform code."
#93 + #52 (Tier-2) host + parameterize creator CODE — a necessary escape hatch, but the logic stays code. This epic is the other half: a declarative pipeline where the logic itself is configuration/data, so the common scraper / ingest / enrich / sync class needs zero code.
Model
A pipeline = an ordered list of steps over connectors (#84) and collections (#91), run by a durable runner on a trigger (#92):
source (connector query) → paginate/fan-out → map/transform → filter → enrich (connector) → dedupe/upsert → sink (collection)
Each step is a registry tool (#85/#87). The pipeline is a JSON definition (data) — editable via UI/MCP/settings, no monorepo edit, no Worker.
The lead-finder, fully declarative (the proof)
- source: generic HTTP connector → Google Places nearby (city→geocode, type, radius = settings)
- paginate: grid cells around the geocoded centre
- map: addressComponents → {city, suburb, state, country}
- enrich: HTTP-reachability on websiteUri
- filter: keep no-website OR unreachable
- dedupe: upsert by place_id
- sink: instance collection
leads
- audit: per-record run log
Children
Relationship to #93 / #52
#93/#52 = host arbitrary creator code (escape hatch for logic that can't be declared). This epic = declarative pipelines for the common source→transform→sink class (the majority). Prefer declarative; fall back to Tier-2 only when a step genuinely can't be expressed. Both write to collections (#91) and fire from triggers (#92).
Part of #58. Builds on #84 (connectors), #91 (instance collections), #92 (triggers).
Epic: declarative data pipelines — the "configure it, don't code it" half of #93
Evidence (lead-finder, 2026-08-02). The lead-finder is a deterministic data pipeline: geocode → grid → paginate Places → reachability-check → classify → dedupe → store. Today the ONLY ways to host that shape are a closed first-party Workflow or a standalone external Worker (#93) — i.e. code. That directly contradicts #58's operating principle: "build every agent as declarative data, never bespoke platform code."
#93 + #52 (Tier-2) host + parameterize creator CODE — a necessary escape hatch, but the logic stays code. This epic is the other half: a declarative pipeline where the logic itself is configuration/data, so the common scraper / ingest / enrich / sync class needs zero code.
Model
A pipeline = an ordered list of steps over connectors (#84) and collections (#91), run by a durable runner on a trigger (#92):
source (connector query) → paginate/fan-out → map/transform → filter → enrich (connector) → dedupe/upsert → sink (collection)Each step is a registry tool (#85/#87). The pipeline is a JSON definition (data) — editable via UI/MCP/settings, no monorepo edit, no Worker.
The lead-finder, fully declarative (the proof)
leadsChildren
Relationship to #93 / #52
#93/#52 = host arbitrary creator code (escape hatch for logic that can't be declared). This epic = declarative pipelines for the common source→transform→sink class (the majority). Prefer declarative; fall back to Tier-2 only when a step genuinely can't be expressed. Both write to collections (#91) and fire from triggers (#92).
Part of #58. Builds on #84 (connectors), #91 (instance collections), #92 (triggers).