Restructure docs and simplify main README

README.md

@@ -2,82 +2,23 @@
 
 ![Python°](app/static/python.png)
 
-> Production ready, open-source FastAPI application with PostgreSQL and blazing-fast full-text search
+This project is a FastAPI-based backend for collecting, organizing, and serving business data. It brings together PostgreSQL storage, API endpoints, and a few practical automation features such as AI prompt handling, email sending, and data integration with services like GitHub, Flickr, and YouTube.
 
-#### Overview
+The app is designed to be a reliable backend layer for internal tools, admin workflows, or front-end applications that need structured data and simple API access.
 
-This project provides a scalable API backend using FastAPI and PostgreSQL, featuring:
+## Table of contents
 
-- Automatic full-text search on all text fields (via tsvector)
-- Endpoints for health checks, product management, prompt handling (via `/prompt`), notify email, and prospect management
-- Efficient ingestion and processing of large CSV files
+- [Project overview](docs/overview.md)
+- [Architecture](docs/architecture.md)
+- [Setup and development](docs/setup.md)
+- [API reference](docs/api.md)
+- [Integrations](docs/integrations.md)
+- [Database](docs/database.md)
+- [Testing](docs/testing.md)
+- [Deployment](docs/deployment.md)
 
-#### Features
+## Quick note
 
-- **Python 3.11+**
-- **FastAPI** — Modern, high-performance REST API
-- **PostgreSQL** — Robust relational database
-- **tsvector + GIN** — Superfast full-text search
-- **Uvicorn** — Lightning-fast ASGI server
-- **Pytest** — Comprehensive testing
+If you want to get started, the best place to begin is the [setup guide](docs/setup.md). If you want to understand the system as a whole, start with the [overview](docs/overview.md).
 
-#### Install & Use
-
-#### 1. Clone & Setup Environment
-
-```bash
-git clone https://github.com/goldlabelapps/python.git
-cd python
-cp .env.sample .env  # Add your Postgres credentials and settings
-python -m venv venv
-source venv/bin/activate
-pip install -r requirements.txt
-```
-
-#### 2. Run the App
-
-```bash
-uvicorn app.main:app --reload
-```
-
-Visit [localhost:8000](http://localhost:8000) or [onrender](https://nx-ai.onrender.com)
-
-#### API Documentation
-
-FastAPI auto-generates interactive docs:
-
-- [Swagger UI](https://nx-ai.onrender.com/docs)
-- [ReDoc](https://nx-ai.onrender.com/redoc)
-
-#### Notable Endpoints
-
-- `GET /health` — Health check
-- `GET /prompt` or `GET /prompts` — Prompt table metadata (`record_count`, `columns`)
-- `POST /prompt` — LLM prompt completion (formerly `/llm`)
-- `GET/POST /notify/email` — Send email via Resend API (see implementation in `app/api/notify/email.py`)
-- `GET /prospects` — Paginated prospects
-- `POST /prospects/process` — Bulk CSV ingestion
-
-#### Full-Text Search (tsvector)
-
-The `prospects` table includes a `search_vector` column (type: tsvector) computed from all text fields on insert/update. A GIN index enables fast, scalable full-text search:
-
-```sql
-SELECT * FROM prospects WHERE search_vector @@ plainto_tsquery('english', 'search terms');
-```
-
-**How it works:**
-- On every insert/update, `search_vector` is computed using PostgreSQL's `to_tsvector('english', ...)`.
-- The GIN index (`idx_prospects_search_vector`) enables efficient search across large datasets.
-
-#### Processing Large CSV Files
-
-The `/prospects/process` endpoint supports robust ingestion of large CSVs (e.g., 1300+ rows, 300KB+), following the same normalization and insertion pattern as `/prospects/seed` but optimized for scale.
-
-#### Contributing
-
-Contributions welcome. Please open issues or submit pull requests.
-
-#### License
-
-This project is licensed under the MIT License. See [LICENSE](LICENSE) for details.
+Before deployment, make sure the frontend origin is included in `ALLOWED_ORIGINS`; otherwise browser requests from that domain will be rejected by CORS.

app/__init__.py

@@ -1,5 +1,5 @@
 """Python° - FastAPI, Postgres, tsvector"""
 
 # Current Version
-__version__ = "3.1.3"
+__version__ = "3.1.4"
 

app/main.py

@@ -15,27 +15,28 @@
     version=__version__,
 )
 
-# CORS middleware for development
+def get_allowed_origins() -> list[str]:
+    configured_origins = os.getenv("ALLOWED_ORIGINS", "")
+    if configured_origins:
+        return [origin.strip() for origin in configured_origins.split(",") if origin.strip()]
+
+    return [
+        "http://localhost:3000",
+        "http://localhost:8000",
+        "http://127.0.0.1:3000",
+        "http://127.0.0.1:8000",
+        "https://goldlabel.pro"
+    ]
+
+
+# CORS middleware with an explicit, environment-driven allow-list.
 app.add_middleware(
     CORSMiddleware,
-    allow_origins=[
-        "http://localhost:1999", 
-        "http://localhost:1998", 
-        "http://localhost:1975", 
-        "http://localhost:1980", 
-        "http://localhost:2027",
-        "http://localhost:2020",
-        "http://localhost:2000",
-        "https://goldlabel.pro",
-        "https://nx-admin.goldlabel.pro",
-        "https://free.goldlabel.pro",
-        "https://listingslab.com",
-        "https://ed-tech.co",
-        "https://notheretofuckspiders.art",
-    ],
-    allow_credentials=True,
-    allow_methods=["*"],
-    allow_headers=["*"]
+    allow_origins=get_allowed_origins(),
+    allow_origin_regex=os.getenv("CORS_ALLOW_ORIGIN_REGEX"),
+    allow_credentials=False,
+    allow_methods=["GET", "POST", "PUT", "PATCH", "DELETE", "OPTIONS"],
+    allow_headers=["Accept", "Accept-Language", "Content-Language", "Content-Type", "Authorization", "X-API-Key"],
 )
 
 

docs/README.md

@@ -0,0 +1,25 @@
+# Project Documentation
+
+This directory contains the main documentation for the Python backend service in this repository.
+
+## Documentation map
+
+- [Overview](overview.md) — What the application does and why it exists
+- [Architecture](architecture.md) — Application structure, runtime flow, and major components
+- [Setup](setup.md) — Installation, environment variables, and local development
+- [API Reference](api.md) — Routes, request patterns, and response shape
+- [Integrations](integrations.md) — Gemini, email, and third-party data connectors
+- [Database](database.md) — PostgreSQL usage, schemas, and search capabilities
+- [Testing](testing.md) — How the project is tested and how to run tests
+- [Deployment](deployment.md) — Render-style deployment considerations and runtime configuration
+
+## Quick start
+
+1. Install dependencies with `pip install -r requirements.txt`
+2. Create a local environment file with the required variables
+3. Start the app with `uvicorn app.main:app --reload`
+4. Open the interactive documentation at `/docs`
+
+## Project summary
+
+This repository is a FastAPI-based backend that exposes APIs for data storage, retrieval, and automation. It is designed to support business workflows involving prospects, prompts, orders, queue operations, and integrations with external services.

docs/api.md

@@ -0,0 +1,66 @@
+# API Reference
+
+## Core endpoints
+
+### Root
+
+- `GET /` — returns basic service metadata such as title, version, and base URL
+
+### Health
+
+- `GET /health` — health check endpoint used to confirm the service is available
+
+### Prompt endpoints
+
+- `GET /prompt` or `GET /prompts` — returns metadata for the prompt table, including row count and columns
+- `POST /prompt` — accepts a prompt payload and returns either cached output or a generated response from Gemini
+
+### Prospects
+
+- `GET /prospects` — returns paginated prospects, with optional filtering and search
+- `GET /prospects/{id}` — returns one prospect and any related prompt records
+- `PATCH /prospects/{id}` — updates flag and hide state
+- `PATCH /prospects/factoryreset` — resets prospect flags and hidden state
+
+### Orders
+
+- `GET /orders` — returns paginated and filterable order data
+
+### Queue routes
+
+The queue module exposes routes for creating, reading, deleting, emptying, and altering queue-related data.
+
+### Notifications
+
+- `GET /notify/email` — returns usage information for the email endpoint
+- `POST /notify/email` — sends an email through Resend
+
+### External data endpoints
+
+- `GET /github` — returns GitHub-related table data
+- `GET /flickr` — returns Flickr-related table data
+- `GET /youtube` — returns YouTube-related table data
+
+## Response style
+
+Most endpoints return a response object shaped like:
+
+```json
+{
+  "meta": {
+    "status": "success",
+    "message": "..."
+  },
+  "data": {}
+}
+```
+
+## Authentication
+
+Some routes depend on an API key header:
+
+```http
+X-API-Key: your_key
+```
+
+The key is validated through the shared authentication utility.

docs/architecture.md

@@ -0,0 +1,66 @@
+# Architecture
+
+## Runtime stack
+
+The application is built around the following core components:
+
+- FastAPI for HTTP routing and request handling
+- PostgreSQL for persistent storage
+- Pydantic for request/response validation
+- Uvicorn as the ASGI server
+- Python dotenv for environment configuration
+
+## Application entry point
+
+The main application is initialized in [app/main.py](../app/main.py). It creates the FastAPI app, configures CORS, mounts static files, and includes the API router.
+
+## Router structure
+
+The main router is assembled in [app/api/routes.py](../app/api/routes.py). It includes multiple route modules for:
+
+- root metadata
+- health checks
+- prompt endpoints
+- prospects
+- orders
+- queue routes
+- notifications
+- GitHub, Flickr, and YouTube integrations
+
+## Request flow
+
+A typical request follows this pattern:
+
+1. The FastAPI app receives an HTTP request
+2. A route handler validates or parses input
+3. The handler connects to PostgreSQL through the database utilities
+4. Queries or updates are executed
+5. A standardized response payload is returned using the shared metadata helper
+
+## Core modules
+
+### app/main.py
+
+Defines the application object and global middleware.
+
+### app/api
+
+Contains the route modules and feature-specific endpoints.
+
+### app/utils
+
+Contains shared support code for:
+
+- database connections
+- API-key authentication
+- response metadata
+- health checks
+
+## Design characteristics
+
+The architecture favors a simple, service-oriented approach:
+
+- route modules are feature focused
+- database access is centralized
+- shared metadata responses keep output consistent
+- integrations are isolated into dedicated modules

docs/database.md

@@ -0,0 +1,32 @@
+# Database
+
+## Storage approach
+
+The application relies on PostgreSQL for persistent storage. Database connection helpers are defined in [app/utils/db.py](../app/utils/db.py).
+
+## Main data areas
+
+The app uses several logical data areas:
+
+- prospects
+- prompt history
+- orders
+- queue-related records
+- platform-specific tables for GitHub, Flickr, and YouTube
+
+## Search capabilities
+
+The README describes PostgreSQL full-text search support for prospects using `tsvector` and a GIN index. This allows efficient search across text fields.
+
+## Why the database is central
+
+The database is the system of record for most application features. It provides:
+
+- reliable persistence
+- filtering and pagination support
+- search ability
+- historical storage for AI prompt outputs and business records
+
+## Operational note
+
+The app expects database connection settings to be present in the environment. If the database is unavailable, many endpoints will not function properly.

docs/deployment.md

@@ -0,0 +1,30 @@
+# Deployment
+
+## Deployment target
+
+The project is compatible with deployment platforms such as Render. The repository includes a [render.yaml](../render.yaml) configuration file.
+
+## Runtime considerations
+
+For deployment, ensure the following are configured:
+
+- database environment variables
+- `PYTHON_KEY` if protected routes are used
+- `GEMINI_API_KEY` for prompt generation
+- `RESEND_API_KEY` for email sending
+- `BASE_URL` for environment-aware metadata
+- `ALLOWED_ORIGINS` with the exact frontend origin(s) that will call the API from the browser
+
+> Deployment gotcha: CORS will block browser requests unless the frontend URL is explicitly allowed. Before deploying, add the production frontend URL to `ALLOWED_ORIGINS` (for example, `https://your-app.example.com`). If you use a different subdomain or preview URL, include that exact origin as well.
+
+## Recommended deployment checklist
+
+1. Set all required environment variables
+2. Ensure PostgreSQL is available and reachable
+3. Install Python dependencies
+4. Run the application with Uvicorn or the deployment platform's startup command
+5. Verify core endpoints such as `/health` and `/docs`
+
+## Notes
+
+Because the app depends on external services and a database, deployment should be treated as a full-stack environment rather than a simple static app.