document-intelligence

Production-ready semantic search and Q&A over documents. Upload text, PDF, or DOCX files, ask questions in natural language, get grounded answers with source citations. Built on Gemini AI + Chroma vector database. Free to run — $0.00.

What It Does

Splits documents into overlapping chunks, embeds each chunk with Gemini's embedding model, and stores them in a local Chroma vector database. When you ask a question, it finds the most semantically relevant chunks and returns them ranked by similarity. Week 3 adds Gemini generation on top — grounded answers with source citations. Week 4 adds a Streamlit web interface — a chat UI on top of the same ingest/retrieve/generate pipeline.

Scripts

Script	What it does	When to run it
semantic_search.py	Embeds 8 sentences, ranks by cosine similarity	Learning demo — proves embeddings work
chroma_store.py	Stores corpus in Chroma, queries by meaning	Test Chroma in isolation
chunker.py	Splits any text file into overlapping chunks	Inspect chunks before ingesting
ingest_text.py	Chunks a file and stores all chunks in Chroma	Main ingestion script
document_loader.py	Extracts text from PDF and DOCX files	Used by ingest_text.py --pdf/--docx
generator.py	Generates grounded answers with citations from retrieved chunks via Gemini	Core generation step
qa.py	Single-command CLI — ingest a PDF (if needed) and answer a question with citations	Main entry point (PDF only)
app.py	Streamlit web UI — upload a PDF or DOCX, chat, view relevance-ranked sources, dark/light theme	streamlit run app.py

Setup

git clone https://github.com/AHTISHAM327/document-intelligence.git
cd document-intelligence
python3 -m pip install -r requirements.txt
cp .env.example .env

# Add your Gemini API key from aistudio.google.com to .env

Quick Start

python3 qa.py --pdf your_document.pdf --question "your question here"

That's it — one command. It ingests the PDF on first run and reuses the index on later questions. Add --reingest to force a fresh ingest, or --top-k 5 to retrieve more context chunks.

Example:

python3 qa.py --pdf sample.pdf --question "what is the refund policy?"

Refund requests must be submitted within 30 days of purchase.
Contact billing@nexusanalytics.com with your invoice number. (Source: sample.pdf)

Web interface

streamlit run app.py

A chat interface over the same ingest → retrieve → generate pipeline:

• Upload a PDF or DOCX from the sidebar — it's chunked, embedded, and indexed automatically (replacing any previously loaded document).
• Ask questions in the chat box and get answers grounded only in the document, with invented facts and inline (Source: …) citations stripped out.
• Expand "View N sources" under any answer to see the exact passages the answer drew from, each tagged High / Medium / Low relevance.
• Copy any message with the per-message copy button.
• Toggle Dark / Light theme and Clear conversation from the sidebar; the session panel tracks the loaded document and question count.

The Gemini API key is read from .env locally, or from Streamlit secrets (.streamlit/secrets.toml / Streamlit Cloud) when deployed.

Live demo: [add your Streamlit Cloud URL here after deploying]

Usage

Ingest a document:

python3 ingest_text.py --file your_document.txt

Ingest a PDF document:

python3 ingest_text.py --pdf your_document.pdf

Ingest a DOCX document:

python3 ingest_text.py --docx your_document.docx

Then query it:

python3 chroma_store.py --mode query --question "your question about the document"

Re-ingest a different document:

rm -rf chroma_db/
python3 ingest_text.py --pdf new_document.pdf

Query it:

python3 chroma_store.py --mode query --question "what is the refund policy?"

Inspect chunks before ingesting:

python3 chunker.py --file your_document.txt --verbose

To re-ingest (start fresh):

rm -rf chroma_db/
python3 ingest_text.py --file your_document.txt

Example

$ python3 ingest_text.py --file sample_long.txt
📄 Loaded 'sample_long.txt' → 7 chunks (size=500, overlap=100)
✅ Ingested 7/7 chunks into 'support-docs' collection.

$ python3 chroma_store.py --mode query --question "what is the refund policy?"
🔍 Query: what is the refund policy?

Rank | Similarity | Result
------------------------------------------------------------
   1 | 0.7421     | ...Refund requests must be submitted within 30 days...
   2 | 0.6834     | ...Nexus Analytics offers monthly and annual billing plans...

Test the generator standalone (uses mock chunks):

python3 generator.py --question "what is the refund policy?"

Full pipeline — PDF to grounded answer:

# After ingesting a PDF with ingest_text.py --pdf sample.pdf:
from chroma_store import query_collection
from generator import generate_answer

chunks = query_collection("your question here", n_results=3)
answer = generate_answer("your question here", chunks)
print(answer)
# Output: grounded answer with (Source: filename.pdf) citations

Testing

.venv/bin/pytest test_document_intelligence.py -v

12 regression tests covering the documented contracts of the core pipeline functions — no mocking of network calls beyond a MagicMock() stand-in for genai_client where a function accepts one but the path under test never calls it:

• Chunking — correct chunk count/boundaries, invalid overlap raises ValueError, and consecutive chunks genuinely share overlapping characters (not just the right count)
• Retrieval — query_collection()'s return schema, empty-collection returns [] with no embedding call, and the collection is actually configured for cosine distance (structural check + a numeric cosine-vs-L2 regression test)
• DOCX — load_docx() extracts real paragraph text from a generated .docx file; ingest_file(is_docx=True) extracts, chunks, and embeds a real .docx end to end
• Ingestion — ingest_file() skips re-ingesting an already-populated collection
• End-to-end — run_qa() returns a clear error string (not an exception) for a missing PDF; generate_answer() returns its documented message for empty chunks and refuses out-of-scope questions

60-page retrieval stress test: Standard/Premium gig tiers promise support for documents up to 60 pages. test_assets/handbook_60pages.pdf is a synthetic company handbook (generated by test_assets/generate_test_pdf.py, via reportlab) with 3 planted, exactly-checkable facts spread near the start, middle, and end — the middle placement matters most, since that's typically where naive chunking degrades first. Run the check yourself:

test_assets/check_planted_facts.sh

It asks a natural-language question for each planted fact through the real qa.py pipeline and asserts the answer contains the expected value. Last run: 3/3 passed, including the middle-page fact ranked as the #1 retrieved chunk.

How It Works

Text is split into overlapping chunks (default: 500 chars, 100-char overlap). Each chunk is embedded into a 768-dimensional vector by Gemini. Vectors are stored in a local Chroma database. When you query, your question is embedded the same way and Chroma returns the chunks whose vectors are closest — matching by meaning, not keywords.

Project Structure

document-intelligence/
├── semantic_search.py    # Day 6 — pure-Python embedding + cosine similarity demo
├── chroma_store.py       # Day 7 — Chroma ingest + query
├── chunker.py            # Day 7 — splits text into overlapping chunks with metadata
├── ingest_text.py        # Day 7 — orchestrates chunker → Chroma (use this for documents)
├── document_loader.py    # Day 8 — PDF (PyPDF2) + DOCX (python-docx) text extraction
├── generator.py          # Day 9 — Gemini generation with source citations
├── qa.py                 # Day 10 — single-command CLI (ingest + retrieve + generate; PDF only)
├── app.py                # Day 11 — Streamlit web UI (chat interface, dark/light theme, PDF/DOCX upload)
├── test_document_intelligence.py  # 12 regression tests — pytest test_document_intelligence.py -v
├── sample.pdf            # Test document — PDF version
├── sample_long.txt       # Test document — Nexus Analytics support manual
├── test_assets/          # 60-page retrieval stress test
│   ├── generate_test_pdf.py       # Builds handbook_60pages.pdf with 3 planted facts (reportlab)
│   ├── handbook_60pages.pdf       # Synthetic 60-page handbook, generated (not hand-edited)
│   └── check_planted_facts.sh     # Queries qa.py for each planted fact, asserts correct answers
├── chroma_db/            # Local vector database (gitignored — generated by ingest)
├── .streamlit/           # Streamlit config + secrets (secrets.toml is gitignored)
├── requirements.txt      # google-genai, python-dotenv, httpx, chromadb, streamlit, PyPDF2, python-docx, reportlab (test-only)
├── .env.example          # Copy to .env, add GEMINI_API_KEY
└── README.md

Roadmap

• Semantic search — pure Python cosine similarity (Day 6)
• Chroma vector storage and retrieval (Day 7)
• Text chunking with configurable overlap (Day 7)
• PDF document loading — PyPDF2 text extraction (Day 8)
• PDF ingestion pipeline end-to-end (Day 8)
• Gemini generation with retrieved context (Day 9)
• Source citations in answers (Day 9)
• Single-command Q&A CLI (Day 10)
• Streamlit web interface (Day 11)
• DOCX document loading and ingestion — python-docx text extraction, --docx CLI flag, Streamlit upload support
• Multi-document ingestion support (future)

Tech Stack

• Embeddings: Google Gemini (text-embedding-004),Google Gemini (gemini-embedding-001)
• Vector DB: Chroma (local, open-source, no account required)
• Generation (Week 3): Google Gemini (gemini-flash-latest)
• Web UI (Week 4): Streamlit
• Cost: $0.00

License

MIT