A toolkit for CDX indices such as Common Crawl and the Internet Archive's Wayback Machine
-
Updated
Sep 7, 2026 - Python
#
cdx
Here are 10 public repositories matching this topic...
Summarize web archive capture index (CDX) files.
-
Updated
Mar 28, 2026 - Python
Python tools to retrieve text from CommonCrawl WARC files based on cdx index.
-
Updated
Feb 18, 2022 - Python
Fetch and normalize parameterized URLs from the Wayback CDX API (OSINT, inspired by ParamSpider).
python url cli security osint pypi pip recon bugbounty wayback-machine kali-linux wayback parrot cdx pypi-package
-
Updated
Sep 3, 2026 - Python
Enables Mac roundtrip editing for ChemDraw scheme-contaning PowerPoints made in Windows
-
Updated
Jun 10, 2021 - Python
Zero-install, server-preserving web archiver and ISO 28500 forensic engine for offline replication and digital preservation.
python offline-first warc web-archiving digital-preservation cdx anti-ddos mcp-server solo-maintainer server-preservation iso-28500
-
Updated
Sep 11, 2026 - Python
View cdx and warc files, caching them locally as needed
-
Updated
May 30, 2025 - Python
A web crawler built from scratch in Python, one concept per stage: frontier, robots.txt, politeness, async workers, dedup, rendering, freshness, a distributed frontier, WARC archives, a CDX index and replay. Thirteen stages, 222 tests, verified against a corpus that declares its own ground truth.
python crawler web-crawler simhash teaching web-scraping robots-txt asyncio learning-by-doing warc web-archiving cdx
-
Updated
Sep 8, 2026 - Python
The solution to extend the deadline for the virtual machines on CDX.
-
Updated
Nov 16, 2020 - Python
Add this topic to your repo
To associate your repository with the cdx topic, visit your repo's landing page and select "manage topics."