feat(reachr): iris reachr scrape — find people on public pages; --write adds them as leads - #80
Closed
mayoalexander wants to merge 7 commits into
Closed
mayoalexander wants to merge 7 commits into
mayoalexander wants to merge 7 commits into
Conversation
…rite adds them as leads
Reachr sequenced outreach to leads and had no way to get them. This is the acquisition end:
iris reachr scrape <bloq-id> <url...> [--follow …|--no-follow] [--max-pages N] [--next "Next"]
iris reachr scrape <bloq-id> <url...> --write # second, deliberate run
Extraction is the bridge's scrape-leads.sh (iris-daemon 9f525fc): throwaway Chrome, robots.txt
obeyed, polite delay, same-site links, deterministic, every field says how it was found. Public
pages only — LinkedIn/Instagram need a session and are separate lanes.
DRY RUN BY DEFAULT. --write creates each lead with a PUBLIC — not confirmed note (source URL,
evidence, how each field was found). Company contacts with no person are reported, never written
as people.
Verified by writing to a throwaway board and running it again — and the second run was wrong
three different ways before it was right:
- the email upsert (#137529) was only detected by NAME mismatch; a re-scrape returns the same
name, so the existing record counted as "created" and got a second note. Now also: a record
created before this run started is not one we created.
- duplicate check used `?search=` — an asynchronously updated index, so leads created a minute
earlier were invisible and a no-contact person was duplicated on every run. Now the board's
own list, loaded once, Prospected included (the default list omits it).
- that list is cached 30s in fl-api and never invalidated on write (#185994), and `_=` is
stripped from the key; two runs 20s apart still duplicated. A differently-named param reads
fresh. Back-to-back runs 8s apart: 3 created, then 3 recognised, 0 duplicates.
Also: report lines all on stdout — UI.println (stderr) mixed with printKV (stdout) printed a
person's details before their name whenever output was piped. Same fix in `iris browser`.
Test data (3 leads, board anomalyco#692) deleted and verified absent.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_018EozxzAioSnD4afkYcw8Zm
…s, profiles, or your DM inbox
iris reachr scrape <board> --instagram https://www.instagram.com/p/ABC/ # a post's commenters
iris reachr scrape <board> --instagram @account # its followers
iris reachr scrape <board> --instagram "@A,@b,@c" # those profiles
iris reachr scrape <board> --instagram inbox # people who DM you
… --ig-account <saved session> --max-profiles N --write
A front door onto the Instagram scraper that already exists (tests/e2e/leadgen-scraper.spec.ts
in the Freelabel checkout), not a second one. It is always run DRY and asked for a result file;
writing goes through reachr scrape's own path, so Instagram leads get the same board dedupe,
upsert detection and PUBLIC-not-confirmed note as web leads — stored in the shape the SOM
leadgen runner has always used (nickname "@handle", contact_info.instagram/instagram_url,
source leadgen:instagram:<mode>), so the two recognise each other. Dedupe matches the handle first.
Refuses, saying what to do, when there is no checkout, no saved session, an ambiguous or unknown
--ig-account, or no sign-in. A scrape that could not look (session expired, login wall) is exit 2,
never "no leads".
Verified live as @heyiris.io on a throwaway board: 2 profiles created with real display names,
then a second run immediately after created nothing. That run found a bug in the scraper — the
last handle of a profiles list came back as "handle/" (fixed at the source in the monorepo, and
stripped here). The provenance note no longer claims "followers: 0" when stats were not fetched,
nor calls a brand account a person. Test leads and board deleted, verified absent.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_018EozxzAioSnD4afkYcw8Zm
… to your board's leads Reads the saved Instagram session's DM inbox, matches each conversation to a lead on the board, and prints who replied, how long ago, opt-outs, and the lead's next step. Read-only by default; --write-back adds notes and replied/no-response tags. Never sends a message and never posts to Discord (discord_posted is read from the scan's own log). A browser that closes mid-scan exits 2 — could not measure. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_018EozxzAioSnD4afkYcw8Zm
… LinkedIn inbox
A front door onto tests/e2e/linkedin-scraper.spec.ts, like the Instagram lane:
always dry, and --write goes through reachr's own path (board dedupe by the /in/
slug, upsert detection, PUBLIC-not-confirmed provenance note). Stores leads in the
runner's shape (contact_info.linkedin = profile URL) so the two recognise each other.
Title and company come from the headline ("CEO at Blu Creative Agency"), falling
back to LinkedIn's own summary line ("… at X in Dallas since 2018"). --max-profiles
is capped at 100: it browses as you, and LinkedIn restricts accounts that pull more.
Measured on a scratch board: 3 created with notes, rerun → 3 "already on board".
Also: reachr-playwright.ts runs every reachr spec (IG, LinkedIn, inbox) and retries
ONCE for one failure — no result file + "browser has been closed" + died fast.
3 of 9 CLI runs failed that way on the first navigation; each rerun passed.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_018EozxzAioSnD4afkYcw8Zm
…ead's comms thread A reply the scan saw existed only in its log: the board Communications inbox, the reply-suggestion pane and the agents' thread tools all read lead_comms, and all showed the lead as silent. --log-comms (implied by --write-back) writes each reply there as an inbound Instagram comm from @handle. Idempotent: text already on the lead's Instagram thread is skipped — including rows the older inbox-check path wrote as `[inbox reply] IG reply from @x: "…"`. The server's own dedupe keys on the timestamp, which Instagram often hides. Measured on board 38: 28 logged + 3 recognised as already there; two reruns logged 0. (A first run logged our own DMs as their replies — a scanner bug fixed in the monorepo and iris-daemon 7418b2b; those 54 rows were deleted.) Errors now say how many attempts the spec runner made. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_018EozxzAioSnD4afkYcw8Zm
…ound Every judgement reachr acts on moves to reachr-core.ts (no I/O): scrape result → leads (IG, LinkedIn), already-on-the-board, the written payload, upsert detection, the provenance note, which replies to log, when to retry a run. The commands keep the browsing and HTTP and call these. Mutation-checked: 7 of 8 mutants fail a test (the survivor is noted in the test). Found by writing the scenarios first: - A second person with a known name was reported "already on the board" and never written. A name match is now refused when both records carry a different profile on the same platform (a nickname counts only if "@handle"). - Three of the four legacy reply shapes in lead_comms were not recognised (combined [DM Reply] notes, previews cut at 100/120 chars, LinkedIn's "reply detected from Name"), so a lead scanned again would get its replies twice. knownReplies() parses each; long previews match as prefixes, short ones never. Audit of the 28 rows logged on board 38: 0 duplicates of legacy rows. Live after the refactor: LinkedIn dry run 3/3 with companies; inbox rerun logged 0, 31 already there. bun test src/cli/cmd: 1515 pass. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_018EozxzAioSnD4afkYcw8Zm
…d a real reply? Before #186186 was fixed the inbox scan tagged leads "DM Replied" on our own DMs, wrote the pitch into a [DM Reply] note under sender "me", and a backfill copied it into lead_comms as an inbound message. auditRepliedLead() reads what a tagged lead carries and returns replied / false_reply / unknown — never "false" without having read something. Read-only; --fix (false leads only) removes the tag, deletes the inbound comms rows that are only our words, and notes why. Measured, read-only: board 38 — 21 tagged, 11 false; board 80 — 17 tagged, 1 false (12 of 38 overall). Also found: boardLeads stopped after 20 pages without a word, so on a 6,005-lead board the last 2,005 were invisible — scrape --write could duplicate them and the audit saw 3 of board 80's 17 tagged leads. collectPages reads to the end or throws at a cap; tested. reachr-core.test.ts: 57 scenarios. bun test src/cli/cmd: all pass. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_018EozxzAioSnD4afkYcw8Zm
mayoalexander
added a commit
that referenced
this pull request
Sep 19, 2026
…+ iris browser check (PRs #80, #72) Brings in `iris reachr scrape` (public pages, Instagram, LinkedIn; --write adds leads), `iris reachr inbox` (+ --log-comms), `iris reachr audit-replies`, reachr-core with 57 tests, and its base branch feat/iris-browser-command (`iris browser check <url>`). Verified on the merged tree before pushing: - typecheck (tsgo --noEmit): 0 errors - reachr-core.test.ts: 57/57 - full suite: 3268 pass / 231 fail — the same 231 as the PR's CI run. Grouped by file, every failing file is outside this branch (provider, snapshot, config, agent, …). The three that still fail in isolation (platform-commands 13, bug-regressions 7, provider 9) fail IDENTICALLY on main 51c3cb9 — pre-existing, whole-suite leak tracked in #185847. - only conflict: capabilities.json (generated) — regenerated from the merged source. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_0121oCnNCeZBewSyiQSQ8bin
Author
Author
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Stacked on #72 (
iris browser) — it reuses that PR's script lookup. Merge #72 first; this PR then retargets tomain.Reachr sequenced outreach to leads and had no way to get them. This is the acquisition end:
Extraction is the bridge's
scrape-leads.sh(iris-daemon9f525fc): throwaway headless Chrome, robots.txt obeyed, polite delay, same-site links only, deterministic — every field says how it was found, and every person says what evidence (photo / role title / contact) made it a person. Public pages only; LinkedIn/Instagram need a logged-in session and are separate lanes.Measured on real pages: ycombinator.com/people — v1 returned 3 department headings as people and missed ~100; now 103 people, 101 with photo + role title, 0 junk in a sample, 0 duplicates.
--write, verified against a throwaway board, run twice back to back:It took three fixes to get there, each found by the second run:
?search=is an async index — freshly created leads are invisible, so no-contact people were duplicated. Now the board's own list, Prospected included._=stripped from the key. A differently-named param reads fresh.Each created lead carries a PUBLIC — not confirmed note: source URL, evidence, and how each field was found.
Also fixes report ordering in both
iris browserand this command (stderr/stdout interleaving scrambled output when piped). Test data deleted and verified absent.🤖 Generated with Claude Code
https://claude.ai/code/session_018EozxzAioSnD4afkYcw8Zm