ddgs-api

Local web search and page fetch for Pi. Runs in Docker, no API keys, no rate limits.

The fetch path has three layers: a fast static HTTP client (curl_cffi), real headless Chrome, and Camoufox (a Firefox fork that's painful for anti-bot stacks to detect). The extension layers them automatically so the agent gets useful text out of most pages without thinking about it.

Setup

Clone the repo and start the backend:

docker compose up -d --build

First build takes 5–10 minutes and the image is around 3GB. It downloads Chrome, Camoufox, and a GeoIP database, so be patient. Once it's up:

curl http://localhost:8091/health
# {"status":"ok"}

Then install the extension into Pi:

pi install npm:@arcadia64/pi-ddgs

Reload Pi. You'll have four tools available: web_search, web_search_news, get_search_results, and fetch_url.

Config

Drop a JSON file at ~/.pi/ddgs.json to override defaults. Every field is optional.

{
  "ddgsApiUrl": "http://localhost:8091",
  "timeoutMs": 30000,
  "fetchTimeoutMs": 20000,
  "maxResults": 10,
  "safesearch": "off",
  "region": "us-en",
  "defaultEngine": "chrome",
  "searchCacheSize": 50
}

Field	Default	Notes
ddgsApiUrl	http://localhost:8091	Backend URL. Point this at a remote box if you split machines.
timeoutMs	30000	Search request timeout.
fetchTimeoutMs	20000	Static fetch timeout. Render paths multiply this by 3 (Chrome) or 5 (Camoufox).
maxResults	10	Default result count.
safesearch	"off"	"off", "moderate", "on".
region	"us-en"	DDG region code.
defaultEngine	"chrome"	"chrome" is fast, "camoufox" is slower but beats more anti-bot.
searchCacheSize	50	LRU bound on cached search results.

What it can and can't fetch

Static HTML, server-rendered SPAs, and most blogs/docs work fine through the fast static path. Pure client-rendered SPAs fall back to Chrome. Cloudflare's standard managed challenge falls through to Camoufox and usually gets the page. Reddit fetches route through Redlib (a mobile-OAuth-spoofing frontend) instead of reddit.com, which bypasses Reddit's VPN-IP blocklist — see Host rewrites to change or disable this.

Things it doesn't do: Cloudflare Turnstile, interactive challenges, and sites that block on IP reputation or behavioral signals. Those need residential proxies or real auth, which is out of scope.

---

Beyond the basics

Container hygiene

Each fetch creates a fresh browser context and destroys it before returning, so no cookies, history, or session state persists between requests.

Memory and CPU are bounded in layers, because headless browsers under burst load will otherwise eat everything:

• Concurrency caps. Rendered fetches queue behind per-engine semaphores (Chrome 2, Camoufox 1). A burst of render requests waits as cheap coroutines instead of stacking browser processes, so peak memory is a constant rather than a function of load.
• Resource blocking. Renders abort image, media, and font requests. Text extraction never needs them, and they are most of the memory and CPU on news/media sites.
• Load shedding. A render is refused with 503 when the container is above 85% of its cgroup memory limit (MEMORY_PRESSURE_RATIO to tune). Failing one request fast beats wedging the box.
• Deadline + client-disconnect cancellation. Every render has a hard wall-clock ceiling (50s) and is torn down as soon as the caller disconnects. Previously a caller that timed out left the server rendering an abandoned page.
• Browser rotation. Long-lived browsers leak slowly even when idle, so each engine is relaunched periodically (Chrome every 500 requests / 2h, Camoufox every 50 / 1h) without interrupting in-flight requests.
• cpus: 4, pids_limit: 1024, mem_limit: 3g, no swap. The CPU cap bounds the blast radius no matter what goes wrong inside. Swap headroom was tried and made things worse: at the memory cap the kernel spent ~96% of the container's CPU time swapping browser pages for minutes across every host core while /health still passed. Without swap the OOM killer drops one renderer instantly instead.
• init: true reaps orphaned browser child processes so zombies don't accumulate.
• Healthcheck + autoheal. A /health probe marks the container unhealthy if it stops responding, and the autoheal sidecar restarts it (plain Docker only labels unhealthy containers; restart-on-unhealthy is Swarm-only). Worst case a wedged container is back in ~2.5 minutes without intervention. restart: unless-stopped still covers plain crashes. /health also reports memory_ratio for quick inspection.

Backend endpoints

Path	Purpose
/search?q=&max_results=&safesearch=&region=	DDG web search
/search/news?q=&max_results=&safesearch=&region=	DDG news search
/fetch?url=&render=&engine=	Fetch + clean text. render=true uses a headless browser. engine is chrome (default) or camoufox.
/health	Liveness

Host rewrites

app/main.py has a small map that rewrites hosts before fetching:

HOST_REWRITES = {
    "reddit.com":      "redlib.privacyredirect.com",  # Redlib proxy (spoofs mobile OAuth)
    "www.reddit.com":  "redlib.privacyredirect.com",
}

The default Redlib instance is set via REDDIT_PROXY_HOST env var (redlib.privacyredirect.com). Set it to a different Redlib instance URL (see github.com/redlib-org/redlib-instances), or to reddit.com to disable the rewrite entirely. Add others if useful — medium.com → scribe.rip, twitter.com → nitter.net, etc.

Installing the extension from a local path

If you're hacking on the extension and don't want to bump npm versions every time, point Pi at the cloned directory instead:

{
  "packages": ["C:/path/to/ddgs-api/pi-ddgs"]
}

Reload Pi. Edit pi-ddgs/index.ts freely.

Split setup (extension on machine A, backend on machine B)

Run docker compose up -d --build on machine B and make sure port 8091 is reachable from machine A. On machine A, set "ddgsApiUrl": "http://<machine-B-ip>:8091" in ~/.pi/ddgs.json, install the extension, reload Pi.

Layout

ddgs-api/
├── docker-compose.yml
├── Dockerfile
├── app/
│   └── main.py             # FastAPI backend
└── pi-ddgs/                # Pi extension
    ├── package.json
    ├── index.ts
    └── src/
        ├── ddgs.ts
        └── config.ts