DataSnap

A high-performance, local-first Chrome Extension (Manifest V3) for visual web scraping, structured catalog extraction, and in-browser spreadsheet analysis.

Overview
- The Problem DataSnap Solves
- Who It Is For
Features
- Implemented & Verified Features
- Planned & Experimental Features
Technology Stack
Installation
- Prerequisites
- Step-by-Step Installation
Development Setup
- Verified Commands
- Inspecting Extension Surfaces
How to Use
Architecture & Internal Design
Project Structure
Permissions and Privacy
- Manifest Permissions Justification
- Privacy & Data Security Model
Troubleshooting
Roadmap
Contributing
License

Overview

DataSnap is an open-architecture, client-side web data extraction platform built directly into the Google Chrome Side Panel. It bridges the gap between crude, error-prone manual copy-pasting and complex, server-heavy scraping frameworks like Python/Playwright/Selenium.

Operating entirely within the context of an authenticated browser session, DataSnap enables users to visually select web elements, automatically identify repeating structured items (such as product search results, directory feeds, and article listings), deep-scrape individual detail pages across hundreds of URLs, clean data inline within an Excel-inspired spreadsheet, and export production-ready datasets.

The Problem DataSnap Solves

Manual Copy-Paste Overhead: Aggregating hundreds of product prices, specifications, or contact records manually takes hours and leads to transcription errors.
Brittle Selectors & Markup Inconsistency: Dynamic e-commerce platforms frequently rotate class names, inject sponsored advertisements, and shuffle container tags. DataSnap employs structural scoring and semantic normalization to identify genuine data cards and strip extraneous DOM noise.
Multi-Page Detail Extraction Friction: Scrapers often collect top-level URLs but require external scripts to crawl individual product pages. DataSnap provides a native 3-step deep-scraping pipeline with background queue execution.
Manifest V3 Service Worker Freezes: In Chrome Manifest V3, background service workers terminate after ~30 seconds of inactivity or 5 minutes of continuous run time, freezing multi-page scrapes at approximately 73 URLs. DataSnap implements an alarm-driven keepalive heartbeat and persistent queue state to reliably process extensive URL batches.
Cross-Domain Workspace Contamination: Switching tabs during extraction frequently mixes data across websites. DataSnap partitions datasets and configurations by domain, ensuring strict session isolation.
External Tool Dependency for Data Verification: Most scraping extensions dump raw unformatted CSVs that require external spreadsheet programs to inspect. DataSnap includes a virtualized, in-panel spreadsheet with in-place cell editing, column management, and sorting.

Who It Is For

Market Researchers & E-Commerce Analysts: Extract competitor catalog pricing, stock status, ratings, and technical specifications directly from Amazon, Flipkart, Meesho, and custom web storefronts.
Data Analysts & Business Intelligence Teams: Build structured datasets with normalized types, clean international currency symbols, and export directly to native .xlsx workbooks or UTF-8 CSVs.
QA Engineers & Frontend Developers: Inspect DOM structures, generate relative CSS selectors, and validate dynamic web layouts using Raw Extraction mode.
Recruiters & Sales Operations: Compile contact lists, public rosters, and event directories, annotating records with manual custom columns before exporting.

Features

Implemented & Verified Features

1. Dual Extraction Modes

Product List Mode (Default):
- Powered by semantic normalization engines (product-normalizer.ts) with specialized adapters for major platforms (Amazon, Flipkart, Meesho).
- Automatically identifies, extracts, and normalizes attributes: Product Name, Brand, Price, Original Price, Discount, Rating, Reviews, Availability, Product URL, and Image URL.
- Intelligently filters out navigation headers, promotional banners, carousels, and cart widgets.
Raw Extraction Mode:
- Technical DOM inspection engine (scraper-engine.ts) extracting every node within the targeted container.
- Generates exact CSS selectors and preserves raw DOM attributes (innerText, href, src, alt, custom data attributes) while filtering out noisy SVG path elements and icon wrappers.
- Switching modes preserves both datasets in memory without data loss.

2. Visual Element Picking & Automatic Detection

Semantic Container Scoring: Automatically evaluates repeating DOM patterns, ranking candidate containers by child element density, text distributions, and link patterns (list-detector.ts).
Interactive Visual Element Picker: Full-page isolated SVG overlay (highlight-overlay.ts) and floating control banners (element-picker.ts) that highlight containers on hover with instant match counters (e.g., Items: 24).
Relative CSS Selector Generation: Automatically derives minimal, robust relative selectors (.title a, span.price, img.product-image) avoiding fragile absolute DOM paths (selector-generator.ts).

3. Deep Extraction: Scrape Page Details

3-Step Guided Workflow:
- Step 1 (Select URLs): Choose URLs directly from the active Scrape List session (with automatic URL column detection) or upload an external CSV file (Step1SelectUrls.tsx).
- Step 2 (Select Data): Load an active test URL, interactively pick target fields using on-page overlays, or define custom CSS selectors with designated extraction types (text, link, image, or attribute) (Step2SelectData.tsx).
- Step 3 (Extract Data): Configure request delays, page readiness timeouts, error recovery policies, and execute the run (Step3ExtractData.tsx).
Resilient Execution Engine:
- Orchestrated by DetailJobManager and DetailScraperRunner.
- Background tab processing maintains the user's focus on the active workspace.
- Automatic error isolation logs failed URLs without crashing the broader queue.

4. Pagination & Repeated Record Extraction

Pagination Strategies:
- Next Page Button: Automatically clicks pagination buttons (rel="next", "Next >", page numbers) and awaits DOM re-rendering.
- Load More Button: Identifies and triggers in-page expansion buttons, accumulating newly rendered cards.
- Infinite Scroll: Progressively scrolls the viewport to trigger dynamic lazy-loading.
Throttling & Limits: Configurable delay intervals (800ms–3000ms), max page limits, and max record thresholds prevent rate limiting.

5. Virtualized Spreadsheet Workspace

High-Density Data Grid: Virtualized rendering (SpreadsheetWorkspace.tsx) handling thousands of rows smoothly at 60 FPS.
In-Place Cell Editing: Double-click or press Enter to modify cell values inline, with full keyboard navigation (Arrow keys, Tab, Shift+Tab, Escape).
Column Operations: Drag header boundaries to resize, double-click headers to rename, toggle column visibility, and sort ascending/descending.
Custom Columns: Add manual annotation columns (e.g., Lead Status, Notes) or on-the-fly web extraction columns with custom CSS selectors (AddFieldModal.tsx).

6. Exporting & Deduplication

Export Formats (export-utils.ts):
- Microsoft Excel (.xlsx): Native multi-column workbooks generated via SheetJS.
- CSV: Standard comma-separated values with UTF-8 BOM encoding to preserve special characters and currency symbols across Microsoft Excel and Google Sheets.
- JSON: Formatted structured arrays preserving primitive data types.
- Clipboard (TSV): One-click tab-separated copying for direct pasting into active spreadsheets.
Intelligent URL Deduplication: Normalizes URLs by stripping noisy tracking query parameters (e.g., iid, ppt, marketplace) while strictly preserving distinct product variants (record-identity.ts).

7. Local Persistence & Session Management

Domain-Partitioned Sessions: State is isolated per domain (useScraperStore.ts). Navigating between sites snapshots previous datasets and restores site-specific sessions automatically.
Saved Scrapes Library: Save, load, and manage named dataset snapshots and configuration templates directly from Chrome local storage (SavedScrapesModal.tsx).

Planned & Experimental Features

The following capabilities are architected in the project roadmap (docs/implementation_checklist.md) but are not yet implemented in the active codebase:

AI-Assisted Schema Extraction: Integration with LLM providers (AIExtractor interface) for zero-selector automated schema extraction.
Scheduled Background Scraping: Automatic cron-style recurring scrape jobs triggered independently of browser focus.
Direct Cloud Sync: Automated export streaming to Google Drive, OneDrive, or remote webhook endpoints.

Technology Stack

Layer	Technology	Version	Purpose in DataSnap
Extension Platform	Google Chrome Manifest V3	v3	Uses `sidePanel`, `storage`, `unlimitedStorage`, `alarms`, `scripting`, `tabs`, and `downloads` APIs.
UI Framework	React	19.2.8	Declarative component architecture for the Side Panel workspace, modals, and detail stepper.
Language	TypeScript	~6.0.2	Strict static typing across DOM analyzers, messaging protocols, and storage models.
Build Tool	Vite	^8.3.0	Multi-target compilation for the React workspace, background service worker, and isolated content scripts.
Styling	Tailwind CSS	^3.4.19	Utility-first styling with dark/light theme support and high-density spreadsheet gridlines.
State Management	Zustand	^5.0.15	Centralized reactive store coordinating active datasets, column layouts, and domain sessions.
Data Export	SheetJS (`xlsx`)	^0.18.5	Client-side generation of native Microsoft Excel `.xlsx` spreadsheets.
Schema Validation	Zod	^4.6.5	Runtime validation for configuration contracts, datasets, and cross-context messages.
Iconography	Lucide React	^1.52.0	Consistent vector iconography across toolbars and dialog interfaces.
Testing Framework	Vitest	^5.0.3	Automated testing suite with JSDOM browser simulation and Chrome extension API mocking.
Linter	Oxlint	^1.81.0	Fast static analysis and linting for code quality assurance.

Installation

Prerequisites

Google Chrome v116 or higher (or Chromium-based browsers supporting the Manifest V3 Side Panel API, such as Brave, Microsoft Edge, or Opera).
Node.js v18.0.0 or higher.
npm (bundled with Node.js).

Step-by-Step Installation

Clone the Repository:

git clone https://github.com/your-username/datasnap.git
cd datasnap

Install Dependencies:
```
npm install
```
Build the Extension:
```
npm run build
```
This compiles the TypeScript code, bundles the React workspace, and outputs the production extension into the dist/ directory.
Load into Google Chrome:
1. Open Google Chrome and navigate to chrome://extensions/.
2. Enable Developer mode using the toggle in the upper-right corner.
3. Click the Load unpacked button in the top toolbar.
4. Select the dist directory inside the DataSnap project folder (e.g., path/to/DataSnap/dist).
  
  Important: Select the dist folder, not the repository root folder.
Verify the Installation:
- Confirm DataSnap appears in your extensions list.
- Click the puzzle icon in Chrome's toolbar and pin DataSnap for quick access.
- Click the DataSnap icon to open the Side Panel workspace.

Development Setup

Verified Commands

All development commands are defined in package.json and have been verified against the codebase:

# Install all production and development dependencies
npm install

# Compile TypeScript and build the production extension into dist/
npm run build

# Start the Vite local development server (for UI and component preview)
npm run dev

# Run Oxlint across the codebase for static code analysis
npm run lint

# Run Vite preview server for compiled assets
npm run preview

# Execute the complete automated Vitest test suite (28 test files, 373 tests)
npx vitest run

Inspecting Extension Surfaces

Chrome extensions execute across distinct isolated environments. Use the following techniques to inspect and debug each surface:

Extension Surface	Source Location	Inspection Method
Side Panel Workspace	`src/workspace/`	Right-click anywhere inside the DataSnap Side Panel and select Inspect. DevTools opens directly for the React workspace.
Background Service Worker	`src/background/`	Navigate to `chrome://extensions/`, find DataSnap, and click the service worker link under "Inspect views" to view background logs, message routing, and alarms.
Content Script	`src/content/`	On the target webpage, press `F12`, switch to the Console tab, and select `DataSnap` from the execution context dropdown.
Keepalive Alarms	`datasnap_job_keepalive`	Open `chrome://serviceworker-internals/` to inspect active service worker registration and background alarm executions.

How to Use

Workflow 1: Scrape List (Repeating Elements)

Use this workflow to extract multi-item feeds, search results, or catalogs from the active webpage.

Navigate to the Target Page: Open any website containing repeating items (e.g., an e-commerce search page, news feed, or business directory).
Open DataSnap: Click the DataSnap icon in the Chrome toolbar to open the Side Panel.
Select or Detect Containers:
- Auto Detect: Click Auto Detect in the toolbar. DataSnap analyzes repeating DOM structures and automatically highlights the primary listing grid.
- Visual Pick: Alternatively, click Select, hover over any repeating card on the page, and click to designate it as the template container.
Verify Match Count: The toolbar displays the number of matching items found on the page (e.g., Items: 24).
Configure Pagination (Optional):
- Click Action to load more items to expand pagination settings.
- Select your strategy: Click button to load more items on same page, Click Next Page button to navigate, or Infinite Scroll.
- Click Verify to interactively pick the pagination button on the page.
- Adjust the Page delay (ms) (recommended: 800ms–2000ms for dynamic pages).
Execute Scraping: Click Start Scraping. DataSnap extracts records across pages, displaying them live in the spreadsheet.

Workflow 2: Scrape Page Details (Multi-URL Deep Extraction)

Use this workflow to navigate to individual pages and extract detailed specifications, descriptions, or attributes.

Open Scrape Page Details: Click the Scrape Page Details tab in the TopBar.
Step 1: Select URLs:
- From Current List: Click Use Current List to import URLs directly from your active Scrape List session. Select the target column (e.g., Product URL).
- From External CSV: Click Upload CSV, choose a file from your computer, and designate the column containing webpage links.
- Click Continue.
Step 2: Select Data to Extract:
- DataSnap loads the first URL as a reference test page.
- Click Pick Field to launch the visual picker on the page, hover over elements (e.g., technical specifications, description, SKU), click, and assign field names.
- Alternatively, click Add Custom Field to manually specify CSS selectors and extraction targets (text, link, image, or attribute).
- Click Continue.
Step 3: Extract Data:
- Set execution options: Delay Between URLs (ms), Page Timeout (ms), and Stop on Error.
- Click Start Scraping.
- Monitor the live progress bar, processed counters, and error logs. You may pause or stop at any time; completed records are retained.

Workflow 3: In-Place Spreadsheet Editing & Field Management

Inline Editing: Double-click any cell in the spreadsheet grid or press Enter to edit values directly. Press Enter to commit or Escape to discard.
Column Resizing: Hover over column boundary borders in the header and drag horizontally.
Renaming Headers: Double-click any column header to rename it.
Adding Custom Columns: Click Add Field to append:
- A Web Extraction Field: Evaluates a CSS selector across all rows.
- A Manual Column: Creates an empty column for manual notes, categorization, or lead scoring.

Workflow 4: Multi-Format Export

Click Download in the workspace toolbar.
Select your desired format:
- Excel (.xlsx): Generates a multi-column workbook via SheetJS.
- CSV: Generates a UTF-8 BOM-encoded .csv file.
- JSON: Generates a structured JSON array.
Alternatively, click Copy to copy the current table to your clipboard as Tab-Separated Values (TSV) for direct pasting (Ctrl+V / Cmd+V) into Google Sheets or Excel.

Architecture & Internal Design

flowchart TD
    subgraph Browser ["Target Webpage"]
        DOM["Live DOM Tree"]
        CS["Content Script (content.ts)"]
        PICKER["Interactive Visual Picker (highlight-overlay.ts)"]
        OVERLAY["Floating Live Overlay (live-extraction-overlay.ts)"]
    end

    subgraph ServiceWorker ["Background Service Worker"]
        ROUTER["Message Router (message-router.ts)"]
        JOB_MGR["Detail Job Manager (detail-job-manager.ts)"]
        ALARM["Chrome Alarms (datasnap_job_keepalive)"]
        TAB_MGR["Tab Lifecycle Manager (tab-lifecycle-manager.ts)"]
    end

    subgraph Workspace ["Side Panel Workspace"]
        STORE["Zustand Store (useScraperStore.ts)"]
        GRID["Spreadsheet Workspace (SpreadsheetWorkspace.tsx)"]
        STEPPER["Page Details Stepper (Step1 -> Step2 -> Step3)"]
        EXPORT["Export Engine (export-utils.ts)"]
    end

    subgraph Storage ["Chrome Local Storage"]
        SESSIONS["Domain Site Sessions"]
        ACTIVE["Active Workspace Session"]
        DETAILS["Page Details Job State"]
    end

    DOM <--> |DOM Inspection & Highlighting| CS
    CS <--> |Chrome Runtime Messaging| ROUTER
    ROUTER <--> |Job Dispatch & Progress Stream| STORE
    STORE <--> |State Rendering & In-Place Edits| GRID
    STORE <--> |Auto-Save Persistence| ACTIVE
    JOB_MGR <--> |Tab Creation & Reuse| TAB_MGR
    ALARM --> |Wakeup Heartbeat & Resume| JOB_MGR
    JOB_MGR <--> |Queue Index Sync| DETAILS
    GRID --> |Generate XLSX / CSV / TSV| EXPORT

Component Architecture

DataSnap is organized into four decoupled subsystems:

Side Panel Workspace (src/workspace/):
- A React 19 single-page application hosted in src/workspace/index.html rendered through Chrome's sidePanel API.
- Manages interactive spreadsheet grids, step-by-step detail wizards, modal dialogs, and user preferences.
- State is coordinated via Zustand (useScraperStore.ts), utilizing debounced synchronization with chrome.storage.local.
Background Service Worker (src/background/):
- Central message router (message-router.ts) mediating communication between content scripts and the UI.
- Coordinates multi-URL queue execution (detail-job-manager.ts).
- Manages tab creation and isolation (tab-lifecycle-manager.ts).
Content Script (src/content/):
- Compiled as a self-contained IIFE script injected into webpages.
- Houses the listing detector (list-detector.ts), relative selector generator (selector-generator.ts), and visual hover overlay (highlight-overlay.ts).
Core Extraction Engine (src/core/):
- Platform normalizers (product-normalizer.ts), URL deduplication algorithms (record-identity.ts), and export formatters (export-utils.ts).

Overcoming Manifest V3 Service Worker Limits

In Chrome Manifest V3, background service workers are terminated automatically after 30 seconds of event inactivity or 5 minutes of total execution. Multi-URL scrapes requiring 3–5 seconds per page (to load dynamic scripts and await DOM stability) traditionally terminate prematurely around URL 73 ($73 \times 4.1\text{s} \approx 300\text{s}$).

DataSnap solves this through a multi-tier resilience architecture:

Recurring Alarms: Upon starting a scraping job, DetailJobManager schedules a datasnap_job_keepalive alarm every 0.4 minutes using chrome.alarms.
Automatic Wakeup & Resume: If Chrome suspends the service worker, the alarm fires, waking the worker and invoking detailJobManager.checkAndResumeJob().
Persistent Queue Indexing: The current queue index (currentIndex) is continuously written to chrome.storage.local. Resumed jobs pick up from the exact index without restarting or dropping records.
Idle Timer Resets: Each URL iteration executes lightweight Chrome API calls (chrome.runtime.getPlatformInfo()), resetting Chrome's internal idle countdown.
Teardown: Alarms are cleared cleanly upon job completion, error termination, or user cancellation.

Tab Lifecycle & Workspace Window Isolation

To ensure background scraping does not interfere with user browsing or accidentally close the Side Panel:

Workspace Window Registration: On startup, DataSnap stores its own workspace window ID in storage (datasnap_workspace_window_id).
Normal Browser Windows: Scraping tabs are explicitly created in standard browser windows with active: false.
Protected Tab Registry: Created tab IDs are tracked in an internal Set. DataSnap will never close an unmanaged user tab or the extension workspace window.
Idempotent Tab Cleanup: Tab closures are wrapped in safety handlers to absorb cases where the user manually closed a scraping tab.

Domain-Partitioned Session State

To prevent data contamination when navigating between different websites:

When the active tab changes, DataSnap extracts the current hostname.
If data exists for the previous domain, a snapshot is saved under siteSessions[previousDomain].
If a session exists for the new domain, it is restored into the active workspace; otherwise, a clean workspace is initialized.

Project Structure

DataSnap/
├── dist/                           # Compiled production-ready Chrome Extension (built output)
├── docs/                           # Architecture specifications and implementation checklists
│   ├── architecture.md
│   └── implementation_checklist.md
├── public/                         # Static extension assets and Manifest
│   ├── extension_icon.png          # Master high-resolution brand asset
│   ├── favicon.ico                 # Multi-resolution ICO (16x16 to 256x256)
│   ├── favicon.svg                 # Scalable vector icon
│   ├── icons/                      # Resampled raster icons (icon16, icon32, icon48, icon128, icon512)
│   └── manifest.json               # Chrome Extension Manifest V3 configuration
├── src/
│   ├── background/                 # Background Service Worker subsystem
│   │   ├── detail-job-manager.ts   # Multi-URL job runner with keepalive alarms
│   │   ├── message-router.ts       # Central runtime message dispatcher
│   │   ├── service-worker.ts       # Service worker entry point
│   │   └── window-manager.ts       # Workspace window registration & isolation
│   ├── content/                    # Webpage Content Script subsystem
│   │   ├── content.ts              # Content script bootstrap and message listener
│   │   ├── dom/                    # DOM analysis and selector generation
│   │   │   ├── field-detector.ts   # Field and attribute extraction
│   │   │   ├── list-detector.ts    # Repeating container detection & scoring
│   │   │   └── selector-generator.ts # Relative CSS selector generator
│   │   ├── overlay/                # On-page status overlays
│   │   │   ├── live-extraction-overlay.ts # Floating progress indicator
│   │   │   └── record-highlighter.ts      # Active record bounding box
│   │   └── picker/                 # Visual element picker interfaces
│   │       ├── detail-element-picker.ts # Visual field picker for detail pages
│   │       ├── element-picker.ts        # Container card selector banner
│   │       ├── field-element-picker.ts  # In-page field picker
│   │       └── highlight-overlay.ts     # SVG hover highlight overlay
│   ├── core/                       # Core extraction, normalization, and export engines
│   │   ├── detection/
│   │   │   └── page-type-detector.ts    # Page classification heuristics
│   │   ├── export/
│   │   │   └── export-utils.ts          # SheetJS Excel, UTF-8 CSV, and TSV copy
│   │   ├── scraper/                     # Scraping algorithms and platform adapters
│   │   │   ├── amazon-adapter.ts        # Amazon-specific extraction rules
│   │   │   ├── column-filter.ts         # Noise filter for raw columns
│   │   │   ├── csv-url-parser.ts        # CSV parsing and URL identification
│   │   │   ├── dataset-model.ts         # Dataset typing and canonical models
│   │   │   ├── detail-scraper-runner.ts # Sequential URL runner
│   │   │   ├── detail-scraper-types.ts  # Types for detail scrape state
│   │   │   ├── field-namer.ts           # Automatic column labeling
│   │   │   ├── flipkart-adapter.ts      # Flipkart-specific extraction rules
│   │   │   ├── meesho-adapter.ts        # Meesho-specific extraction rules
│   │   │   ├── pagination-detector.ts   # Next button and load-more heuristics
│   │   │   ├── product-normalizer.ts    # E-commerce field normalizer
│   │   │   ├── record-identity.ts       # URL deduplication and variant tracking
│   │   │   ├── record-normalizer.ts     # Tabular record cleaning
│   │   │   ├── schema-builder.ts        # Dynamic column schema builder
│   │   │   ├── scraper-config.ts        # Default settings and thresholds
│   │   │   ├── scraper-engine.ts        # Primary Scrape List execution engine
│   │   │   └── tab-lifecycle-manager.ts # Background tab creation & teardown
│   │   └── storage/
│   │       └── persistence-manager.ts   # Chrome storage persistence layer
│   ├── shared/                     # Shared types and message passing contracts
│   │   ├── types/message.ts        # Discriminated union message interfaces
│   │   └── utils/message-utils.ts  # Safe message transmission helpers
│   ├── tests/                      # Automated Vitest test suites (28 files, 373 tests)
│   └── workspace/                  # Side Panel React 19 application
│       ├── App.tsx                 # Workspace root component
│       ├── index.html              # Side panel HTML entry point
│       ├── main.tsx                # React DOM render bootstrap
│       ├── theme-init.ts           # Theme synchronization & favicon handler
│       ├── components/             # UI components and dialog modals
│       │   ├── ActionFooter.tsx    # Pagination & scrape action controls
│       │   ├── AddFieldModal.tsx   # Custom field & manual column dialog
│       │   ├── ChangeListModal.tsx # Container switcher dialog
│       │   ├── ConfigArea.tsx      # Scraping configuration controls
│       │   ├── ExportPreviewModal.tsx # Export sample preview dialog
│       │   ├── FieldManagerModal.tsx  # Column manager & renamer dialog
│       │   ├── Header.tsx          # Workspace header and controls
│       │   ├── HelpModal.tsx       # Built-in documentation modal
│       │   ├── PaginationControls.tsx # Pagination configuration panel
│       │   ├── PreviewTable.tsx    # Live extraction preview table
│       │   ├── SavedScrapesModal.tsx  # Saved templates & history dialog
│       │   ├── ScrapingToolbar.tsx # Primary toolbar (Auto Detect, Select, Scrape)
│       │   ├── SettingsModal.tsx   # Preferences dialog (theme, density)
│       │   ├── Sidebar.tsx         # Quick action navigation sidebar
│       │   ├── SpreadsheetWorkspace.tsx # Virtualized spreadsheet grid
│       │   ├── TopBar.tsx          # Top bar with mode pills & domain status
│       │   └── details/            # Scrape Page Details 3-step components
│       │       ├── DetailStepper.tsx
│       │       ├── ScrapePageDetailsWorkflow.tsx
│       │       ├── Step1SelectUrls.tsx
│       │       ├── Step2SelectData.tsx
│       │       └── Step3ExtractData.tsx
│       └── store/
│           └── useScraperStore.ts  # Central Zustand workspace store
├── package.json                    # Project configuration, scripts, and dependencies
├── tsconfig.json                   # TypeScript compiler configuration
├── tsconfig.node.json              # TypeScript Vite configuration
└── vite.config.ts                  # Vite build setup with multi-target output

Permissions and Privacy

Manifest Permissions Justification

DataSnap requests only the permissions strictly required to execute local web scraping and data export:

Permission	API	Purpose in DataSnap
`sidePanel`	`chrome.sidePanel`	Renders the primary spreadsheet workspace alongside active web pages.
`activeTab`	`chrome.tabs`	Provides secure temporary access to the active tab's DOM upon user interaction.
`tabs`	`chrome.tabs`	Reads the active tab's URL/title for domain session isolation and manages background tabs during detail scraping.
`scripting`	`chrome.scripting`	Programmatically injects the element picker, highlight overlays, and content scripts into target pages.
`storage`	`chrome.storage.local`	Persists workspace sessions, column configurations, saved scrape templates, and history.
`unlimitedStorage`	`chrome.storage`	Removes the default 5MB storage limit to support multi-thousand-row datasets without quota errors.
`alarms`	`chrome.alarms`	Schedules background keepalive heartbeats (`datasnap_job_keepalive`) preventing service worker termination during large jobs.
`downloads`	`chrome.downloads`	Initiates client-side file downloads for generated `.xlsx`, `.csv`, and `.json` files.
`<all_urls>`	Host Permission	Enables the scraper to operate across any public web domain requested by the user.

Privacy & Data Security Model

100% Client-Side Processing: All web scraping, DOM parsing, data normalization, and export generation happen entirely inside your local browser.
Zero Remote Telemetry: DataSnap does not include external analytics, tracking pixels, or remote logging services.
No External Servers: Extracted data, imported CSVs, and URLs are never transmitted to external cloud servers or third parties.
Sandboxed Local Storage: Datasets are stored strictly within Chrome's sandboxed extension storage on your local disk.

Troubleshooting

"Manifest file is missing or unreadable"

Cause: Selecting the root project folder instead of dist/ when loading the unpacked extension in Chrome.
Solution: Run npm run build to compile the project. In chrome://extensions/, click Load unpacked and select the dist directory (path/to/DataSnap/dist).

Scrape Page Details Stops at Approximately 73 URLs

Cause: Chrome Manifest V3 service worker was put to sleep by the browser's idle timeout.
Solution: Ensure you are running the latest build containing the chrome.alarms keepalive engine. Open chrome://extensions/, click the service worker link under DataSnap, and verify that datasnap_job_keepalive is firing. If interrupted, click Check & Resume in the details workflow.

Previous Website's Data Appears on a New Domain

Cause: Active tab changed without triggering the domain sync listener.
Solution: DataSnap automatically partitions sessions by domain. Check the domain badge in the TopBar (e.g., amazon.com). If the domain did not update, click the reload icon in the TopBar to refresh tab detection.

"Could not establish connection. Receiving end does not exist"

Cause: The content script is not injected on the active page (e.g., restricted browser pages like chrome://extensions/, Chrome Web Store, or pages loaded prior to extension installation).
Solution: Refresh the target webpage (F5 or Ctrl+R) so Chrome can inject assets/content.js. Note that Chrome security policies prohibit content script injection into browser internal pages.

Blank Values Extracted from Dynamic Webpages

Cause: Single-page applications (SPAs) or lazy-loaded elements require additional rendering time before appearing in the DOM.
Solution: In pagination or detail scrape settings, increase the Page delay (ms) from the default 800ms to 2000ms–3000ms to allow dynamic JavaScript scripts to render before extraction.

Clipboard Copy Fails

Cause: Side panel window lost focus or browser clipboard permission was restricted.
Solution: Click inside the DataSnap Side Panel before clicking Copy. DataSnap includes an automatic fallback to an offscreen textarea copy routine if the Clipboard API is unavailable.

Updating After Making Code Changes

Cause: Chrome runs the cached build from dist/.
Solution: Run npm run build in your terminal. Then go to chrome://extensions/ and click the circular Reload icon on the DataSnap card.

Roadmap

The following enhancements represent planned directions derived from the project architecture:

AI-Assisted Field Extraction: Integration of an AIExtractor interface for heuristic, zero-selector schema extraction on arbitrary pages.
Advanced Pagination Recovery: Enhanced DOM mutation observers to detect asynchronous page replacements on heavily obfuscated single-page applications.
Direct Cloud Storage Sync: Optional client-side OAuth export integration for Google Drive and Microsoft OneDrive.
Webhook Integration: Native POST dispatch of extracted records to custom REST endpoints.
Headless Scheduled Execution: Background cron-based scraping triggers for automated catalog price monitoring.

Contributing

Contributions to DataSnap are welcome! Please adhere to the following workflow:

Fork the Repository: Create your personal fork on GitHub.
Create a Feature Branch:
```
git checkout -b feature/my-feature-name
```
Install Dependencies:
```
npm install
```
Implement Code: Write clean, modular TypeScript adhering to the existing project architecture.

Verify Quality:

# Run the full test suite
npx vitest run

# Run the linter
npm run lint

# Verify production build compilation
npm run build

Submit a Pull Request: Push your branch to GitHub and open a Pull Request with clear reproduction steps and test verification notes.

License

No license has been specified for this project.

All rights are reserved by the original author. You may not distribute, modify, sublicense, or publish this software without explicit written permission from the copyright holder.

Name		Name	Last commit message	Last commit date
public		public
src		src
.gitignore		.gitignore
.oxlintrc.json		.oxlintrc.json
README.md		README.md
package-lock.json		package-lock.json
package.json		package.json
postcss.config.js		postcss.config.js
tailwind.config.js		tailwind.config.js
tsconfig.app.json		tsconfig.app.json
tsconfig.json		tsconfig.json
tsconfig.node.json		tsconfig.node.json
vite.config.ts		vite.config.ts

Latest commit

History

Folders and files

Repository files navigation