diff --git a/README.md b/README.md index e2786e2..1d68921 100644 --- a/README.md +++ b/README.md @@ -2,82 +2,23 @@ ![Python°](app/static/python.png) -> Production ready, open-source FastAPI application with PostgreSQL and blazing-fast full-text search +This project is a FastAPI-based backend for collecting, organizing, and serving business data. It brings together PostgreSQL storage, API endpoints, and a few practical automation features such as AI prompt handling, email sending, and data integration with services like GitHub, Flickr, and YouTube. -#### Overview +The app is designed to be a reliable backend layer for internal tools, admin workflows, or front-end applications that need structured data and simple API access. -This project provides a scalable API backend using FastAPI and PostgreSQL, featuring: +## Table of contents -- Automatic full-text search on all text fields (via tsvector) -- Endpoints for health checks, product management, prompt handling (via `/prompt`), notify email, and prospect management -- Efficient ingestion and processing of large CSV files +- [Project overview](docs/overview.md) +- [Architecture](docs/architecture.md) +- [Setup and development](docs/setup.md) +- [API reference](docs/api.md) +- [Integrations](docs/integrations.md) +- [Database](docs/database.md) +- [Testing](docs/testing.md) +- [Deployment](docs/deployment.md) -#### Features +## Quick note -- **Python 3.11+** -- **FastAPI** — Modern, high-performance REST API -- **PostgreSQL** — Robust relational database -- **tsvector + GIN** — Superfast full-text search -- **Uvicorn** — Lightning-fast ASGI server -- **Pytest** — Comprehensive testing +If you want to get started, the best place to begin is the [setup guide](docs/setup.md). If you want to understand the system as a whole, start with the [overview](docs/overview.md). -#### Install & Use - -#### 1. Clone & Setup Environment - -```bash -git clone https://github.com/goldlabelapps/python.git -cd python -cp .env.sample .env # Add your Postgres credentials and settings -python -m venv venv -source venv/bin/activate -pip install -r requirements.txt -``` - -#### 2. Run the App - -```bash -uvicorn app.main:app --reload -``` - -Visit [localhost:8000](http://localhost:8000) or [onrender](https://nx-ai.onrender.com) - -#### API Documentation - -FastAPI auto-generates interactive docs: - -- [Swagger UI](https://nx-ai.onrender.com/docs) -- [ReDoc](https://nx-ai.onrender.com/redoc) - -#### Notable Endpoints - -- `GET /health` — Health check -- `GET /prompt` or `GET /prompts` — Prompt table metadata (`record_count`, `columns`) -- `POST /prompt` — LLM prompt completion (formerly `/llm`) -- `GET/POST /notify/email` — Send email via Resend API (see implementation in `app/api/notify/email.py`) -- `GET /prospects` — Paginated prospects -- `POST /prospects/process` — Bulk CSV ingestion - -#### Full-Text Search (tsvector) - -The `prospects` table includes a `search_vector` column (type: tsvector) computed from all text fields on insert/update. A GIN index enables fast, scalable full-text search: - -```sql -SELECT * FROM prospects WHERE search_vector @@ plainto_tsquery('english', 'search terms'); -``` - -**How it works:** -- On every insert/update, `search_vector` is computed using PostgreSQL's `to_tsvector('english', ...)`. -- The GIN index (`idx_prospects_search_vector`) enables efficient search across large datasets. - -#### Processing Large CSV Files - -The `/prospects/process` endpoint supports robust ingestion of large CSVs (e.g., 1300+ rows, 300KB+), following the same normalization and insertion pattern as `/prospects/seed` but optimized for scale. - -#### Contributing - -Contributions welcome. Please open issues or submit pull requests. - -#### License - -This project is licensed under the MIT License. See [LICENSE](LICENSE) for details. +Before deployment, make sure the frontend origin is included in `ALLOWED_ORIGINS`; otherwise browser requests from that domain will be rejected by CORS. diff --git a/app/__init__.py b/app/__init__.py index 1a00f3c..805b7c3 100644 --- a/app/__init__.py +++ b/app/__init__.py @@ -1,5 +1,5 @@ """Python° - FastAPI, Postgres, tsvector""" # Current Version -__version__ = "3.1.3" +__version__ = "3.1.4" diff --git a/app/main.py b/app/main.py index 5dcad57..f0c013f 100644 --- a/app/main.py +++ b/app/main.py @@ -15,27 +15,28 @@ version=__version__, ) -# CORS middleware for development +def get_allowed_origins() -> list[str]: + configured_origins = os.getenv("ALLOWED_ORIGINS", "") + if configured_origins: + return [origin.strip() for origin in configured_origins.split(",") if origin.strip()] + + return [ + "http://localhost:3000", + "http://localhost:8000", + "http://127.0.0.1:3000", + "http://127.0.0.1:8000", + "https://goldlabel.pro" + ] + + +# CORS middleware with an explicit, environment-driven allow-list. app.add_middleware( CORSMiddleware, - allow_origins=[ - "http://localhost:1999", - "http://localhost:1998", - "http://localhost:1975", - "http://localhost:1980", - "http://localhost:2027", - "http://localhost:2020", - "http://localhost:2000", - "https://goldlabel.pro", - "https://nx-admin.goldlabel.pro", - "https://free.goldlabel.pro", - "https://listingslab.com", - "https://ed-tech.co", - "https://notheretofuckspiders.art", - ], - allow_credentials=True, - allow_methods=["*"], - allow_headers=["*"] + allow_origins=get_allowed_origins(), + allow_origin_regex=os.getenv("CORS_ALLOW_ORIGIN_REGEX"), + allow_credentials=False, + allow_methods=["GET", "POST", "PUT", "PATCH", "DELETE", "OPTIONS"], + allow_headers=["Accept", "Accept-Language", "Content-Language", "Content-Type", "Authorization", "X-API-Key"], ) diff --git a/app/static/SVGIcon.sketch b/app/static/SVGIcon.sketch deleted file mode 100644 index 9a3e3a1..0000000 Binary files a/app/static/SVGIcon.sketch and /dev/null differ diff --git a/docs/README.md b/docs/README.md new file mode 100644 index 0000000..5ba1eeb --- /dev/null +++ b/docs/README.md @@ -0,0 +1,25 @@ +# Project Documentation + +This directory contains the main documentation for the Python backend service in this repository. + +## Documentation map + +- [Overview](overview.md) — What the application does and why it exists +- [Architecture](architecture.md) — Application structure, runtime flow, and major components +- [Setup](setup.md) — Installation, environment variables, and local development +- [API Reference](api.md) — Routes, request patterns, and response shape +- [Integrations](integrations.md) — Gemini, email, and third-party data connectors +- [Database](database.md) — PostgreSQL usage, schemas, and search capabilities +- [Testing](testing.md) — How the project is tested and how to run tests +- [Deployment](deployment.md) — Render-style deployment considerations and runtime configuration + +## Quick start + +1. Install dependencies with `pip install -r requirements.txt` +2. Create a local environment file with the required variables +3. Start the app with `uvicorn app.main:app --reload` +4. Open the interactive documentation at `/docs` + +## Project summary + +This repository is a FastAPI-based backend that exposes APIs for data storage, retrieval, and automation. It is designed to support business workflows involving prospects, prompts, orders, queue operations, and integrations with external services. diff --git a/docs/api.md b/docs/api.md new file mode 100644 index 0000000..2532f9a --- /dev/null +++ b/docs/api.md @@ -0,0 +1,66 @@ +# API Reference + +## Core endpoints + +### Root + +- `GET /` — returns basic service metadata such as title, version, and base URL + +### Health + +- `GET /health` — health check endpoint used to confirm the service is available + +### Prompt endpoints + +- `GET /prompt` or `GET /prompts` — returns metadata for the prompt table, including row count and columns +- `POST /prompt` — accepts a prompt payload and returns either cached output or a generated response from Gemini + +### Prospects + +- `GET /prospects` — returns paginated prospects, with optional filtering and search +- `GET /prospects/{id}` — returns one prospect and any related prompt records +- `PATCH /prospects/{id}` — updates flag and hide state +- `PATCH /prospects/factoryreset` — resets prospect flags and hidden state + +### Orders + +- `GET /orders` — returns paginated and filterable order data + +### Queue routes + +The queue module exposes routes for creating, reading, deleting, emptying, and altering queue-related data. + +### Notifications + +- `GET /notify/email` — returns usage information for the email endpoint +- `POST /notify/email` — sends an email through Resend + +### External data endpoints + +- `GET /github` — returns GitHub-related table data +- `GET /flickr` — returns Flickr-related table data +- `GET /youtube` — returns YouTube-related table data + +## Response style + +Most endpoints return a response object shaped like: + +```json +{ + "meta": { + "status": "success", + "message": "..." + }, + "data": {} +} +``` + +## Authentication + +Some routes depend on an API key header: + +```http +X-API-Key: your_key +``` + +The key is validated through the shared authentication utility. diff --git a/docs/architecture.md b/docs/architecture.md new file mode 100644 index 0000000..f8c209c --- /dev/null +++ b/docs/architecture.md @@ -0,0 +1,66 @@ +# Architecture + +## Runtime stack + +The application is built around the following core components: + +- FastAPI for HTTP routing and request handling +- PostgreSQL for persistent storage +- Pydantic for request/response validation +- Uvicorn as the ASGI server +- Python dotenv for environment configuration + +## Application entry point + +The main application is initialized in [app/main.py](../app/main.py). It creates the FastAPI app, configures CORS, mounts static files, and includes the API router. + +## Router structure + +The main router is assembled in [app/api/routes.py](../app/api/routes.py). It includes multiple route modules for: + +- root metadata +- health checks +- prompt endpoints +- prospects +- orders +- queue routes +- notifications +- GitHub, Flickr, and YouTube integrations + +## Request flow + +A typical request follows this pattern: + +1. The FastAPI app receives an HTTP request +2. A route handler validates or parses input +3. The handler connects to PostgreSQL through the database utilities +4. Queries or updates are executed +5. A standardized response payload is returned using the shared metadata helper + +## Core modules + +### app/main.py + +Defines the application object and global middleware. + +### app/api + +Contains the route modules and feature-specific endpoints. + +### app/utils + +Contains shared support code for: + +- database connections +- API-key authentication +- response metadata +- health checks + +## Design characteristics + +The architecture favors a simple, service-oriented approach: + +- route modules are feature focused +- database access is centralized +- shared metadata responses keep output consistent +- integrations are isolated into dedicated modules diff --git a/docs/database.md b/docs/database.md new file mode 100644 index 0000000..9d207b4 --- /dev/null +++ b/docs/database.md @@ -0,0 +1,32 @@ +# Database + +## Storage approach + +The application relies on PostgreSQL for persistent storage. Database connection helpers are defined in [app/utils/db.py](../app/utils/db.py). + +## Main data areas + +The app uses several logical data areas: + +- prospects +- prompt history +- orders +- queue-related records +- platform-specific tables for GitHub, Flickr, and YouTube + +## Search capabilities + +The README describes PostgreSQL full-text search support for prospects using `tsvector` and a GIN index. This allows efficient search across text fields. + +## Why the database is central + +The database is the system of record for most application features. It provides: + +- reliable persistence +- filtering and pagination support +- search ability +- historical storage for AI prompt outputs and business records + +## Operational note + +The app expects database connection settings to be present in the environment. If the database is unavailable, many endpoints will not function properly. diff --git a/docs/deployment.md b/docs/deployment.md new file mode 100644 index 0000000..bf22bbf --- /dev/null +++ b/docs/deployment.md @@ -0,0 +1,30 @@ +# Deployment + +## Deployment target + +The project is compatible with deployment platforms such as Render. The repository includes a [render.yaml](../render.yaml) configuration file. + +## Runtime considerations + +For deployment, ensure the following are configured: + +- database environment variables +- `PYTHON_KEY` if protected routes are used +- `GEMINI_API_KEY` for prompt generation +- `RESEND_API_KEY` for email sending +- `BASE_URL` for environment-aware metadata +- `ALLOWED_ORIGINS` with the exact frontend origin(s) that will call the API from the browser + +> Deployment gotcha: CORS will block browser requests unless the frontend URL is explicitly allowed. Before deploying, add the production frontend URL to `ALLOWED_ORIGINS` (for example, `https://your-app.example.com`). If you use a different subdomain or preview URL, include that exact origin as well. + +## Recommended deployment checklist + +1. Set all required environment variables +2. Ensure PostgreSQL is available and reachable +3. Install Python dependencies +4. Run the application with Uvicorn or the deployment platform's startup command +5. Verify core endpoints such as `/health` and `/docs` + +## Notes + +Because the app depends on external services and a database, deployment should be treated as a full-stack environment rather than a simple static app. diff --git a/docs/integrations.md b/docs/integrations.md new file mode 100644 index 0000000..d03dfcf --- /dev/null +++ b/docs/integrations.md @@ -0,0 +1,35 @@ +# Integrations + +## Gemini / Google AI + +The prompt endpoint uses the Google GenAI client to generate completions when no suitable cached response is found. + +Key points: + +- the application reads `GEMINI_API_KEY` from the environment +- prompt requests can be cached in the database +- generated responses are stored with metadata such as model and prompt hash + +## Resend email + +The notify module sends email messages through Resend. + +Key points: + +- the application reads `RESEND_API_KEY` from the environment +- the endpoint accepts recipient, subject, and HTML content +- a template wrapper is used for consistent outbound email formatting + +## GitHub, Flickr, and YouTube + +Separate route modules expose endpoints that read from database tables associated with those platforms. + +These integrations are designed to provide a simple API layer over data that has already been imported or synced into the system. + +## General design + +The integration modules are intentionally isolated so that: + +- external services can be replaced or extended easily +- database access remains centralized +- route code stays simple and focused on HTTP behavior diff --git a/docs/overview.md b/docs/overview.md new file mode 100644 index 0000000..6db307b --- /dev/null +++ b/docs/overview.md @@ -0,0 +1,44 @@ +# Overview + +## Purpose + +This project is a backend application built with FastAPI. Its main purpose is to serve as a data and automation layer for a broader product or business workflow. + +Rather than acting as a standalone website, it focuses on exposing reliable API endpoints that can: + +- store and retrieve business data +- support search and filtering +- connect to external services +- power internal tools or front-end applications + +## What the app does + +The application currently supports several functional areas: + +- health checks and basic service metadata +- prompt handling with optional AI completion +- prospect management and search +- order retrieval and filtering +- queue-related operations for CSV and data processing +- email sending +- integrations with GitHub, Flickr, and YouTube data endpoints + +## Why it exists + +The codebase suggests a goal of combining several operational needs into one backend service: + +1. Centralize data access for multiple sources +2. Provide a consistent API for front-end or admin tools +3. Add automation features such as AI-generated content and notifications +4. Use PostgreSQL for structured storage and search + +## High-level concept + +Think of this app as a service-oriented backend that acts like a hub between: + +- a database +- external APIs +- automation tasks +- business data workflows + +It is especially useful when data needs to be collected, normalized, searched, and surfaced through a simple API interface. diff --git a/docs/setup.md b/docs/setup.md new file mode 100644 index 0000000..200e862 --- /dev/null +++ b/docs/setup.md @@ -0,0 +1,59 @@ +# Setup and Local Development + +## Requirements + +- Python 3.11 or newer +- PostgreSQL access +- Optional: environment variables for AI and email services + +## Installation + +From the repository root: + +```bash +python -m venv venv +source venv/bin/activate +pip install -r requirements.txt +``` + +## Environment variables + +Create a local environment file and configure the required values: + +```bash +DB_HOST=localhost +DB_PORT=5432 +DB_NAME=your_database +DB_USER=your_user +DB_PASSWORD=your_password +BASE_URL=http://localhost:8000 +PYTHON_KEY=your_api_key +GEMINI_API_KEY=your_gemini_key +RESEND_API_KEY=your_resend_key +ALLOWED_ORIGINS=http://localhost:3000,http://localhost:8000 +``` + +> Important: if the API is called from a browser frontend, the frontend origin must be explicitly listed in `ALLOWED_ORIGINS`. This is a common deployment gotcha. If your app is hosted at a production URL such as `https://your-app.example.com`, add that exact origin to the environment variable before deploying. + +## Running the app + +Start the development server: + +```bash +uvicorn app.main:app --reload +``` + +The service will then be available at: + +- http://localhost:8000 +- http://localhost:8000/docs for Swagger UI + +## Static assets + +Static files are mounted under `/static` from the application’s static folder. + +## Notes + +- Some endpoints require the API key header `X-API-Key` +- If the database is not configured correctly, many endpoints will fail at runtime +- The app expects the database schema to exist before it can serve data reliably diff --git a/docs/testing.md b/docs/testing.md new file mode 100644 index 0000000..5f10882 --- /dev/null +++ b/docs/testing.md @@ -0,0 +1,31 @@ +# Testing + +## Test framework + +The project uses `pytest` for automated tests. + +## Running tests + +From the repository root: + +```bash +pytest +``` + +## Existing test areas + +The repository includes tests for: + +- GitHub integration behavior +- health endpoints +- metadata helpers +- orders +- prompts +- prospects +- queue routes +- resend email behavior +- route registration + +## Testing approach + +The tests appear to validate behavior at the route and utility level, focusing on expected API responses and core functionality rather than UI interaction.