diff --git a/.gitignore b/.gitignore index 6168fba..2af18ef 100644 --- a/.gitignore +++ b/.gitignore @@ -62,3 +62,4 @@ terraform-provider-*.log .coverage .coverage.* htmlcov/ +dist/ diff --git a/README.md b/README.md index 5af1606..bbd9ff7 100644 --- a/README.md +++ b/README.md @@ -39,7 +39,7 @@ - **Rate limiting:** in-process only, so it suits a single API instance. - **Frontend audit page:** shows this browser's local import history, not the server audit log (see the naming note under Frontend Features). -- **Offline install:** no documented air-gapped install procedure yet. +- **Offline install:** procedure in `docs/offline-install.md`, rehearsed with networking off. The clean-VM dry run (#60) is still to do. ## For Reviewers diff --git a/docs/offline-install.md b/docs/offline-install.md new file mode 100644 index 0000000..8d75988 --- /dev/null +++ b/docs/offline-install.md @@ -0,0 +1,116 @@ +# Offline install (Ubuntu 24.04, no internet) + +This is the procedure for installing the platform on a machine with no network access (#60). It covers the API, the frontend, PostgreSQL 16 and the T&E schema. + +It uses two scripts: + +| Script | Runs on | Does | +|---|---|---| +| `scripts/offline/build_bundle.sh` | A **connected** Ubuntu 24.04 x86_64 machine | Builds one `.tar.gz` holding everything the offline machine needs | +| `scripts/offline/install.sh` | The **offline** Ubuntu 24.04 x86_64 VM | Installs from that file alone, then runs `smoke_test.sh` | + +**Status:** rehearsed here with networking switched off, but not yet run on a clean VM. That VM run is the evidence #60 needs (see [Evidence for #60](#evidence-for-60)). + +## What the bundle contains + +| Part | Contents | +|---|---| +| `app/` | The repository at the bundle's commit, without the frontend source | +| `wheels/` | Python wheels for CPython 3.12 on x86_64 Linux: everything in `api/requirements.txt`, plus pip | +| `frontend/` | The frontend, prebuilt to run under Node.js | +| `debs/` | Ubuntu 24.04 packages for PostgreSQL 16, `python3.12-venv` and Node.js 18, with dependencies, plus an index so apt can use the folder as a repository | +| `SHA256SUMS` | A checksum for every file in the bundle | + +The bundle is about 140 MB. + +## 1. Build the bundle (connected machine) + +You need Ubuntu 24.04 x86_64 with `git`, `python3-pip`, Node.js 20 or later and `apt-utils`. WSL Ubuntu 24.04 on Windows works. + +```bash +cd ~/PostgreDataMigrationAppWithCSVLoader +git checkout main && git pull +bash scripts/offline/build_bundle.sh +``` + +This produces two files, `dist/offline-bundle-.tar.gz` and `dist/offline-bundle-.tar.gz.sha256`. + +## 2. Move it to the offline VM + +Copy both files by your approved transfer method. Then, on the VM, check that the bundle arrived intact and unpack it: + +```bash +cd ~ +sha256sum -c offline-bundle-.tar.gz.sha256 +tar -xzf offline-bundle-.tar.gz +``` + +## 3. Install (offline VM) + +```bash +cd ~/offline-bundle- +sudo bash app/scripts/offline/install.sh localhost 2>&1 | tee ~/install.log +``` + +The argument is the address browsers use to reach the API. Use `localhost` when the browser runs on the VM itself; otherwise use the VM's hostname or IP address. + +The script runs six steps: +1. Checks every file against `SHA256SUMS`. +2. Installs the Ubuntu packages, using the bundle as the only apt source. The VM's own apt sources are ignored. +3. Copies the app to `/opt/te-migration`, creates the `te-app` system user, and builds a Python virtual environment from `wheels/` without network access. +4. Deploys database `te_mgmt_prod` with schema `te_prod` and its login role `te_prod_user`, using a generated password. +5. Writes `/opt/te-migration/env` (mode 600) with that password and a generated API key, and starts the `te-api` (port 8000) and `te-frontend` (port 3000) services under systemd. +6. Runs the smoke test. + +During step 2, apt may print `W: Download is performed unsandboxed as root`. This only means apt's sandbox user can't read the bundle folder, and it's harmless. + +## 4. Verify + +The install finishes by printing these five lines. Every one must read PASS: + +``` +PASS API health +PASS API rejects a missing key +PASS upload a CSV (2 rows inserted) +PASS upload listed with persisted counts +PASS frontend serves the app +``` + +You can re-run the check at any time: + +```bash +cd /opt/te-migration +sudo bash app/scripts/offline/smoke_test.sh "$(sudo sed -n 's/^API_KEY=//p' env)" +``` + +## Evidence for #60 + +Attach the following to #60: + +1. `~/install.log` from step 3, showing all five PASS lines. +2. Proof that the VM has no outside network. This command should print `000` or a connection error: + ```bash + cd ~ + ip route; curl -s -m 5 -o /dev/null -w '%{http_code}\n' https://pypi.org + ``` +3. The commit in the bundle name, the VM's `lsb_release -d`, and the date. + +Once that evidence is on #60, the README can move to the "fully offline" wording in `DECISIONS/2026-10-04-sovereignty-wording.md`. + +## Known limits + +- **Fresh VM only.** Re-running `install.sh` replaces `/opt/te-migration`. The database script has only been run against an empty database. +- **The API key is shared and visible to browsers.** The frontend still sends one API key from the browser, so anyone who can load the page can read it. Per-user sign-in replaces it in ART Milestone 2 (#92, #96). Until then, run this only on a trusted network. +- **The API address is fixed at install.** It's written into the frontend files. To change it, re-run the install. +- **Ubuntu 24.04 x86_64 only.** Other operating systems need different packages and wheels. + +## How it was rehearsed (26 Oct 2026) + +The rehearsal ran in Claude's Ubuntu 24.04 workspace, inside a Linux network namespace that had only a loopback interface. `curl https://pypi.org` returned `000` from inside it. + +- **What ran:** `install.sh` end to end. Checksums verified (468 files), packages installed from the bundle's local repository, the venv built from wheels, `te_mgmt_prod` deployed with 12 T&E tables, both services started, and the smoke test passed 5 of 5. The smoke test then passed twice more against the running install. +- **What differed from a real VM:** + - systemd was replaced by a stand-in that starts each unit's `ExecStart` as its `User`. + - PostgreSQL 16 was already installed in the workspace, so only Node.js was newly installed from the bundle. + +These differences are why the clean-VM run is still needed. diff --git a/scripts/offline/build_bundle.sh b/scripts/offline/build_bundle.sh new file mode 100755 index 0000000..5b3a333 --- /dev/null +++ b/scripts/offline/build_bundle.sh @@ -0,0 +1,49 @@ +#!/usr/bin/env bash +# build_bundle.sh: build the offline install bundle (#60). +# +# Run on an INTERNET-CONNECTED Ubuntu 24.04 x86_64 machine with git, python3-pip, +# Node.js 20+ and apt-utils installed, from the repo root: +# bash scripts/offline/build_bundle.sh +# Output: dist/offline-bundle-.tar.gz plus its .sha256 file. +# Copy both to the offline VM and follow docs/offline-install.md. +set -euo pipefail + +ROOT="$(cd "$(dirname "${BASH_SOURCE[0]}")/../.." && pwd)" +cd "$ROOT" +SHA="$(git rev-parse --short HEAD)" +OUT="$ROOT/dist/offline-bundle-$SHA" +rm -rf "$OUT"; mkdir -p "$OUT"/{wheels,debs} + +echo "== 1/4 application source (git HEAD $SHA)" +git archive --format=tar --prefix=app/ HEAD | tar -x -C "$OUT" +rm -rf "$OUT/app/csv-table-hub-main" # shipped prebuilt below + +echo "== 2/4 Python wheels for CPython 3.12, manylinux x86_64" +python3 -m pip download -q -r api/requirements.txt -d "$OUT/wheels" \ + --only-binary=:all: --python-version 3.12 --implementation cp \ + --platform manylinux2014_x86_64 --platform manylinux_2_17_x86_64 \ + --platform manylinux_2_28_x86_64 --platform any +python3 -m pip download -q pip -d "$OUT/wheels" --only-binary=:all: --python-version 3.12 + +echo "== 3/4 frontend, built for a plain Node server" +# Placeholders are replaced by install.sh with the real API address and key. +( cd csv-table-hub-main + [ -d node_modules ] || npm ci + rm -rf .output + NITRO_PRESET=node-server VITE_API_BASE=__OFFLINE_API_BASE__ VITE_API_KEY=__OFFLINE_API_KEY__ npm run build >/dev/null + cp -r .output "$OUT/frontend" ) + +echo "== 4/4 Ubuntu packages: PostgreSQL 16, Python venv, Node.js, with dependencies" +PKGS="postgresql-16 postgresql-client-16 python3.12-venv nodejs" +# ponytail: downloads the whole dependency closure, including packages a stock +# Ubuntu 24.04 already has; apt skips those on install. Prune if size matters. +DEPS=$(apt-cache depends --recurse --no-recommends --no-suggests --no-conflicts \ + --no-breaks --no-replaces --no-enhances $PKGS | grep '^[a-z0-9]' | sort -u) +( cd "$OUT/debs" && apt-get download $DEPS >/dev/null 2>&1 || true ) +for p in $PKGS; do ls "$OUT/debs/${p}_"*.deb >/dev/null || { echo "missing deb: $p" >&2; exit 1; }; done +( cd "$OUT/debs" && apt-ftparchive packages . > Packages ) # local apt repository index + +( cd "$OUT" && find . -type f ! -name SHA256SUMS -print0 | sort -z | xargs -0 sha256sum > SHA256SUMS ) +tar -C "$ROOT/dist" -czf "$OUT.tar.gz" "offline-bundle-$SHA" +( cd "$ROOT/dist" && sha256sum "offline-bundle-$SHA.tar.gz" > "offline-bundle-$SHA.tar.gz.sha256" ) +echo "Bundle: $OUT.tar.gz ($(du -h "$OUT.tar.gz" | cut -f1))" diff --git a/scripts/offline/install.sh b/scripts/offline/install.sh new file mode 100755 index 0000000..d841711 --- /dev/null +++ b/scripts/offline/install.sh @@ -0,0 +1,91 @@ +#!/usr/bin/env bash +# install.sh: install from the offline bundle on an Ubuntu 24.04 VM with no internet (#60). +# +# From inside the extracted bundle directory: +# sudo bash app/scripts/offline/install.sh [API_HOST] +# API_HOST is the address browsers use to reach the API (default: localhost). +# Installs to /opt/te-migration, deploys database te_mgmt_prod (schema te_prod), +# writes /opt/te-migration/env, and installs systemd units te-api and te-frontend. +set -euo pipefail + +BUNDLE="$(cd "$(dirname "${BASH_SOURCE[0]}")/../../.." && pwd)" +API_HOST="${1:-localhost}" +DEST=/opt/te-migration + +echo "== 1/6 verify bundle checksums" +( cd "$BUNDLE" && sha256sum --quiet -c SHA256SUMS ) + +echo "== 2/6 install Ubuntu packages from the bundle only" +# Use the bundle as the only apt source; the VM's own sources are ignored. +APTDIR="$(mktemp -d)"; mkdir -p "$APTDIR/lists/partial" +echo "deb [trusted=yes] file:$BUNDLE/debs ./" > "$APTDIR/sources.list" +APT_OPTS=(-o Dir::Etc::SourceList="$APTDIR/sources.list" -o Dir::Etc::SourceParts=/dev/null + -o Dir::State::Lists="$APTDIR/lists") +apt-get "${APT_OPTS[@]}" update -qq +apt-get "${APT_OPTS[@]}" install -y -qq --no-install-recommends \ + postgresql-16 postgresql-client-16 python3.12-venv nodejs >/dev/null +systemctl enable --now postgresql >/dev/null + +echo "== 3/6 application files and Python environment" +id te-app >/dev/null 2>&1 || useradd --system --home "$DEST" --shell /usr/sbin/nologin te-app +rm -rf "$DEST"; mkdir -p "$DEST" +cp -r "$BUNDLE/app" "$DEST/app"; cp -r "$BUNDLE/frontend" "$DEST/frontend" +python3.12 -m venv "$DEST/venv" +"$DEST/venv/bin/pip" install -q --no-index --find-links "$BUNDLE/wheels" -r "$DEST/app/api/requirements.txt" + +echo "== 4/6 database, T&E schema and application role" +DB_PASS="$(openssl rand -hex 24)"; API_KEY="$(openssl rand -hex 32)" +# env_prod.sql creates te_mgmt_prod, schema te_prod and login role te_prod_user. +sed "s/__INJECT_AT_DEPLOY__/$DB_PASS/" "$DEST/app/build/environments/env_prod.example.sql" \ + > "$DEST/app/build/environments/env_prod.sql" +( cd "$DEST/app" && sudo -u postgres env PGHOST=/var/run/postgresql bash build/deploy_all.sh prod >/dev/null ) +rm "$DEST/app/build/environments/env_prod.sql" # it holds the password + +echo "== 5/6 configuration and services" +cat > "$DEST/env" < /etc/systemd/system/te-api.service < /etc/systemd/system/te-frontend.service </dev/null + +echo "== 6/6 smoke test" +bash "$DEST/app/scripts/offline/smoke_test.sh" "$API_KEY" diff --git a/scripts/offline/smoke_test.sh b/scripts/offline/smoke_test.sh new file mode 100755 index 0000000..3de05a8 --- /dev/null +++ b/scripts/offline/smoke_test.sh @@ -0,0 +1,25 @@ +#!/usr/bin/env bash +# smoke_test.sh [API_URL] [FRONTEND_URL]: prove the installed stack works. +# Prints PASS/FAIL per check and exits non-zero on any failure. +set -uo pipefail +KEY="$1"; API="${2:-http://localhost:8000}"; FE="${3:-http://localhost:3000}" +fail=0 +pass() { echo "PASS $1"; } +nope() { echo "FAIL $1"; fail=1; } +for _ in $(seq 1 30); do curl -sf "$API/api/health" >/dev/null && break; sleep 1; done + +curl -sf "$API/api/health" >/dev/null && pass "API health" || nope "API health" + +code=$(curl -s -o /dev/null -w '%{http_code}' "$API/api/csv/files") +[ "$code" = 401 ] && pass "API rejects a missing key" || nope "API rejects a missing key (got $code)" + +TAG="$(date +%s%N)" # unique name and content, so the test can be re-run +body=$(printf '{"fileName":"smoke_%s.csv","content":"a,b\\n1,%s\\n2,y\\n"}' "$TAG" "$TAG") +res=$(curl -s -H "X-API-Key: $KEY" -H 'content-type: application/json' -d "$body" "$API/api/csv/upload") +echo "$res" | grep -q '"insertedRows":2' && pass "upload a CSV (2 rows inserted)" || nope "upload a CSV: $res" + +curl -sf -H "X-API-Key: $KEY" "$API/api/csv/files" | grep -q '"total_rows":2' \ + && pass "upload listed with persisted counts" || nope "upload listed with persisted counts" + +curl -sf "$FE/" | grep -qi '