Repository navigation
vapi: pin search backend to origin IP (bypass broken CF colo path) - #926
Conversation
vapi reaches the search backend via api.hivesearcher.com (Cloudflare-proxied). A bad CF colo<->origin leg can make that hop hang for minutes while the origin is healthy — on 2026-06-08 the rebuilt US origin's vapi hung 12-25s via the CF PDX colo, breaking all US-routed search and read-next (SSR /similar prefetch aborted at the 2s cap, client fetch at 4s). Origin-direct was 1.2s. Pin api.hivesearcher.com -> origin IP in the vapi service so it connects directly (keeps the hostname for SNI/cert, uses the whitelisted IPv4). The IP comes from the HIVESEARCHER_ORIGIN_IP secret (wired through the deploy envs in master.yml/staging.yml) so it never lands in this public repo; the compose fails the deploy loudly if the secret is missing. vision-api's deploy updates only the image (docker service update --image), so it preserves this.
![chatgpt-codex-connector[bot]](https://avatars.githubusercontent.com/in/1144995?s=60&v=4){kind=small}
There was a problem hiding this comment.
💡 Codex Review
Here are some automated review suggestions for this pull request.
Reviewed commit: 7c94781152
ℹ️ About Codex in GitHub
Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you
- Open a pull request for review
- Mark a draft as ready
- Comment "@codex review".
If Codex has suggestions, it will comment; otherwise it will react with 👍.
Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".
| HIVESIGNER_SECRET: ${{secrets.HIVESIGNER_SECRET}} | ||
| SEARCH_API_ADDR: ${{secrets.SEARCH_API_ADDR}} | ||
| SEARCH_API_SECRET: ${{secrets.SEARCH_API_SECRET}} | ||
| HIVESEARCHER_ORIGIN_IP: ${{secrets.HIVESEARCHER_ORIGIN_IP}} |
There was a problem hiding this comment.
In the inspected deploy jobs, adding HIVESEARCHER_ORIGIN_IP only under env: does not make it available inside the remote script; appleboy/ssh-action forwards only variables named in with.envs, and the envs: lists below still omit this new variable. Because the compose files now require ${HIVESEARCHER_ORIGIN_IP:?…}, docker-compose config will fail with the variable unset on the EU/US/SG production deploys (and staging has the same omission) even when the GitHub repo secret exists.
Useful? React with 👍 / 👎.
Greptile SummaryThis PR pins
Confidence Score: 5/5Safe to merge — the fix is complete, well-scoped, and the previous review's forwarding gap has been resolved in all three deploy jobs. All three propagation layers (env block, SSH envs list, remote export) are correctly added for every origin job in master.yml and for staging.yml. The Docker Compose extra_hosts entry uses :? fail-loud semantics so a missing secret surfaces immediately at deploy time rather than silently falling back to a broken CF path. No IP literal is committed to the repo. The change is narrowly targeted with no application code touched. No files require special attention. Important Files Changed
Sequence DiagramsequenceDiagram
participant GHA as GitHub Actions
participant SSH as appleboy/ssh-action
participant Remote as Remote Host
participant DC as docker compose
participant vapi as vapi container
participant Origin as api.hivesearcher.com (origin IP)
participant CF as Cloudflare CDN
Note over GHA,Remote: Before this PR (broken path)
GHA->>SSH: deploy (HIVESEARCHER_ORIGIN_IP missing from envs)
SSH->>Remote: SSH script (var not forwarded)
Remote->>DC: docker stack deploy (no extra_hosts)
vapi->>CF: DNS → api.hivesearcher.com
CF-->>vapi: route via degraded colo (12–25s hang)
Note over GHA,Origin: After this PR (fixed path)
GHA->>SSH: deploy (HIVESEARCHER_ORIGIN_IP in env+envs+export)
SSH->>Remote: SSH script (var forwarded correctly)
Remote->>DC: docker stack deploy (extra_hosts set)
DC->>vapi: /etc/hosts: api.hivesearcher.com → origin IP
vapi->>Origin: "direct TLS (SNI=api.hivesearcher.com, ~1.2s)"
Note over vapi,Origin: Cloudflare colo bypassed entirely
Reviews (2): Last reviewed commit: "Forward HIVESEARCHER_ORIGIN_IP through s..." | Re-trigger Greptile |
…deploy) Setting it only under the step `env:` makes it available on the GitHub runner, not in the remote deploy shell — so `docker-compose config` would see it empty and the extra_hosts `:?` guard would fail every deploy. Add it to each job's ssh-action `envs:` forwarding list and `export` it in the script, matching the SEARCH_API_* pattern. (Addresses the PR review P1.)
|
Caution Review failedPull request was closed or merged during review No actionable comments were generated in the recent review. 🎉 ℹ️ Recent review info⚙️ Run configurationConfiguration used: Organization UI Review profile: CHILL Plan: Pro Run ID: 📒 Files selected for processing (4)
📝 WalkthroughWalkthroughThis PR introduces support for pinning the VAPI service directly to an origin IP address to mitigate Cloudflare connectivity hangs. The ChangesVAPI Origin IP Pinning
Estimated code review effort🎯 2 (Simple) | ⏱️ ~12 minutes Poem
🚥 Pre-merge checks | ✅ 5✅ Passed checks (5 passed)
✏️ Tip: You can configure your own custom pre-merge checks in the settings. ✨ Finishing Touches🧪 Generate unit tests (beta)
Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out. Comment |
1 participant
Problem
vapi proxies search to
api.hivesearcher.com(Cloudflare-proxied → the search origin). A degraded CF colo↔origin leg can make that server-to-server hop hang for minutes while the origin itself is healthy.On 2026-06-08 the rebuilt US origin's vapi →
api.hivesearcher.comhung 12–25s via the CF PDX colo (origin-direct was 1.2s). Since the CF worker routes US traffic to the US origin, all US-routed search + read-next broke: the SSR/similarprefetch aborted at the 2s SSR cap (no strip) and the client fetch aborted at the 4s cap (the4000ms status-0request in GTmetrix HARs). sin2/eu were fine (healthy colos).Fix
Pin
api.hivesearcher.com→ the origin IP in thevapiservice viaextra_hosts, so vapi connects directly (bypassing CF), keeping the hostname for SNI/LE-cert and using the whitelisted IPv4.HIVESEARCHER_ORIGIN_IPsecret, wired through the deployenvsinmaster.yml(EU/US/SG) andstaging.yml— so it never appears in this public repo.:?) if the secret is missing, rather than silently degrading.docker service update --image(image-only), so it preserves thisextra_hosts— no change needed there.Applied as a runtime
--host-addon all 3 origins already (immediate fix: US 12s→1.2s); this makes it survivedocker stack deploy.Before merge
The
HIVESEARCHER_ORIGIN_IPrepo secret is set. Update it if the search box IP ever changes.Test plan
docker compose configresolvesextra_hoststo the IP with the var set, and fails loudly without it.Summary by CodeRabbit