Core pipeline step library — map/filter/dedupe_upsert/fan_out/http_reachable/geocode (#96) - #107
Merged
Merged
Conversation
…/http_reachable/geocode (#96) First-party catalog of reusable pipeline steps, each a standard-tier ToolDef registered in FIRST_PARTY_TOOLS so the #97 runner composes them via runRegistryTool and they're callable at POST /v1/instances/:id/tools/:name. - map/filter/fan_out(grid) are pure; dedupe_upsert writes the instance's own collection via the AgentDO records path (respects the unique-key constraint by looking up before insert-or-update). - http_reachable + geocode reuse #95: reachable goes through safeFetch (SSRF guard), geocode delegates to the http_request tool so the vault api-key and fetch/SSRF handling are not re-implemented. - Proves the lead-finder's transform/filter/dedupe/reachability/geocode are all expressible from the catalog (composition test). Child of #94. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Closed
5 tasks
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
2 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Closes #96. Part of #94. Builds on #95 (http connector) + composes under #97 (pipeline runner).
Six reusable first-party steps (
tier:"standard", no connector), inworkers/api/src/lib/steps.ts, spread intoFIRST_PARTY_TOOLS→ dispatchable viarunRegistryTool(the #97 runner) and atPOST /v1/instances/:id/tools/:name:nextCursoruntil exhausted, cap ≤50).safeFetchprobe → {ok,code}; 4xx=live, 5xx/timeout/blocked=dead. Metadata IP rejected pre-network.http_requesttool (vault key + SSRF + responseMap all from [connectors] Generic HTTP/REST connector — call any API as configuration #95) → {lat,lng,country,state,locality}.geocode/http_reachable reuse #95 (no re-implemented fetch/SSRF);
getPathexported from http.ts for shared dotted-path grammar.Verify:
tscclean · 842 api tests pass (+24, 0 regressions) · lead-finder composition test drives map→filter→dedupe_upsert through the registry and asserts the right 2 leads survive+insert.Note: dedupe_upsert is scope
read(writes its own instance collection, no external connector → no write-consent gate, which keys offtool.connector).🤖 Generated with Claude Code