Grabit Engine

A simple, plugin-based engine for scraping media streams and subtitles. Load provider plugins from GitHub, local files, or directly in code — with health tracking, auto-updates, caching, and more built right in. Works in Node.js, browsers, React and React Native.

---

📚 Documentation

Guides live in docs/ — the README keeps just the overview and a quick start.

Guide	What it covers
Installation	install + React Native / Expo setup
Provider Sources	GitHub, local, and registry sources
Creating a Provider Plugin	config.ts / stream.ts / subtitle.ts / index.ts
Bundling Providers	bundling plugins into standalone modules
Testing Providers	the test-provider CLI
Configuration	manager + scrape config, metrics & health
Examples	end-to-end usage examples
React Hook (useSources)	the React/React Native hook
API Reference	full API surface

New capabilities (see What's new): HTTP hardening · Challenge solver · Source flags · Lazy sources · full changelog

---

✨ Features

Core

• 🔌 Plugin system — add or remove providers anytime
• 🌍 Runs anywhere — Node.js, browsers, React Native
• 🎯 Pick a provider — scrape from one specific provider by its scheme
• ⚡ Run in parallel — scrape from multiple providers at the same time
• 🏁 Stop early — quit as soon as enough providers have responded
• ⏱️ Timeouts — never wait forever for a slow provider

Reliability

• 📊 Health tracking — see how each provider is doing (errors, successes)
• 🔴 Auto-disable — bad providers get turned off on their own
• 🔄 Auto-update — remote providers refresh themselves on a timer
• ♻️ Warm Puppeteer pool — reuse browser processes as tabs instead of spawning a browser for every request
• 💾 Built-in cache — save results in memory so you don't repeat work
• 🔁 Retries — automatically retry failed providers
• ✅ Validation — checks that plugins are set up correctly before loading

---

🚀 Quick Start

import { GrabitManager } from "grabit-engine";

// Create the manager with a registry source (simplest approach)
const manager = await GrabitManager.create({
	source: {
		type: "registry",
		name: "my-providers",
		providers: {
			"my-provider": myProviderModule
		}
	},
	tmdbApiKeys: ["your-tmdb-api-key"]
});

// Scrape streams for a movie — minimal: only tmdbId is required!
// TMDB service auto-fills title, year, duration, imdbId, etc.
const streams = await manager.getStreams({
	media: {
		type: "movie",
		tmdbId: "27205"
	},
	targetLanguageISO: "en"
});

// Scrape from a specific provider by scheme
const targeted = await manager.getStreamsByScheme("my-provider", request);

See Provider Sources for GitHub/local sources and Creating a Provider Plugin to build your own.

---

🆕 What's new

Recent additions — each is opt-in and documented in its own guide:

• HTTP hardening — ctx.xhr gains a cookie jar, plus per-host concurrency, 429 rate-limit handling, and request coalescing (on by default from config.xhr). Proxy is host config on the manager — a proxy agent or a URL resolver — see Configuration → Proxy.
• Challenge solver — ctx.solveChallenge(url, …) (puppeteer on Node; inject a hidden RN WebView or FlareSolverr via setChallengeSolver).
• Source flags — xhr.flags: SourceFlag[] (replaces haveCorsPolicy) so a source states exactly how the host must play it.
• Lazy sources — return { lazy: { id } } and a resolveLazy worker to defer final-URL resolution to play time.

Full details in IMPROVEMENTS.md.

---

🧭 Browser access: ctx.puppeteer vs ctx.solveChallenge

Two ways to drive a real browser from a provider, with different reach:

• ctx.puppeteer hands you the live Puppeteer page (and browser) leased from the shared pool. Use it only when you truly need the page object: listening to network requests to capture the media URL, or injecting / interacting directly in the browser. It runs on Node only, so a provider that uses it must set env: "node" in its manifest.json entry. Off Node (browser / React Native) the engine only runs env: "universal" providers, so a node-only provider is correctly skipped there instead of failing at runtime.
• ctx.solveChallenge(url, requester, opts) returns just { html, cookies, cookieMap, userAgent }. Use it when you only need the rendered HTML (for example to pass a Cloudflare interstitial and read the DOM). It works everywhere: on Node it drives Puppeteer, and a host can inject an RN hidden WebView or FlareSolverr solver via setChallengeSolver, so these providers stay env: "universal".

Rule of thumb: need the page or a network listener, use ctx.puppeteer with env: "node". Only need the solved HTML, use ctx.solveChallenge with env: "universal".

When reusing a solved result, forward the returned userAgent (and cookies) on later requests: Cloudflare binds cf_clearance to the exact User-Agent and IP that earned it.

---

📜 License

ISC — see LICENSE. For educational / personal use.