stacs

Documentation: https://www.newgraphenvironment.com/stacs/

Register a STAC catalogue into pgstac and prove it arrived.

stacs works on a catalogue after its items exist. It does not read source data or build items: that belongs to the repository that owns the source. Anything source-agnostic that is done to a published catalogue belongs here.

Status: pre-release. Extracted from stac_dem_bc, where the registration and verification code has run against a 102,460-item collection.

What it does

• Verify what a STAC API serves against what is published: item id sets compared in both directions, and every item body compared by digest. Never by count: a STAC API omits ids it does not have without reporting an error, so equal counts can hide unequal sets. A body that could not be read is an error, never "unchanged".
• Register with upserts only, collection before items. Nothing deletes. drift registers only what the API is missing or serves differently, which keeps it stateless: a skipped run is picked up by the next one. Every write is followed by a check that the API now serves the body that was sent.
• Audit items before they are registered: each names the collection it is going into, and carries the asset keys the catalogue declares (require) and none it has retired (forbid).
• Validate items with pystac.

Body digests are SHA-256 over RFC 8785 (JCS) canonical JSON, with links removed (the API rewrites them) and null members dropped (pgstac strips them). research/pgstac_round_trip.md records what pgstac changes between load and serve, and why each rule exists.

Install

Managed with uv. Install from git rather than PyPI: the stacs name on PyPI belongs to an unrelated, archived project.

[tool.uv.sources]
stacs = { git = "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/NewGraphEnvironment/stacs", tag = "v0.2.0" }

Without uv: pip install "stacs @ git+https://github.com/NewGraphEnvironment/stacs@v0.2.0".

Configure

The host, database, API and bucket are always the caller's. stacs has no defaults that point at any deployment, and reads a config only when one is named with --config.

# stacs.toml -- committed in the catalogue's own repository. Values are examples.
[catalogue]
api = "https://stac.example.org"            # STAC API base URL
collection_id = "my-collection"
bucket_url = "https://my-bucket.example.org" # serves collection.json and the items

[assets]
require = "data"        # every item must carry this asset key
forbid = ["legacy"]     # no item may carry any of these

[transport]             # only for commands that write
host = "user@stac-host.example.org"
db = "pgstac"
env_file = "/srv/stac/.env"           # sourced on the host before loading
workdir = "/srv/stac"                 # cd here on the host
path_prepend = "/usr/local/bin"
pg_host = "localhost"
pg_port = 5432
pg_user = "pgstac"
password_env = "POSTGRES_PASSWORD"    # the NAME of a variable the env_file defines
pypgstac = ["uv", "run", "pypgstac"]  # the launcher on the host

[tuning]                # optional
page_size = 10000       # keyset page size for full enumerations
chunk = 500             # ids per lookup when checking what was just written
fetch_workers = 20

[queryables."my:property"]  # optional: item properties clients can filter on
type = "string"             # a JSON Schema fragment, one table per property
title = "My property"

Flags override the catalogue and transport settings. The asset rules can be added to by a flag but never loosened: a declared require cannot be replaced, and --forbid-asset adds keys. Pointing at another collection id does not drop them.

Queryables are loaded for the catalogue's own collection_id and no other. pgstac serves /collections/<id>/queryables from its own table, never from the collection body, and its loader replaces every row of a name that the collection is served, including a shared or global one. So each catalogue declares its own, and rows for different collections coexist. register loads a queryable only when it is missing or served with a different definition, after the items, and never with an index (each load replaces the row, and with it any index set on that name by hand). Removing a declaration leaves its row in place: nothing is deleted. Queryables are loaded last, once the collection and items are verified and the collection is registered. A name the collection is then served with a different definition may come from a shared or global row, so register reads every other collection's definition of it just before the load and again after it, whatever happened in between, and fails if one was lost or changed. The host needs pypgstac 0.9.6 or later.

No setting is a secret. Writes go over ssh to the STAC host, which runs pypgstac load --method upsert there; the database password stays on the host and is named (password_env), never given. Each load first asks that connection which database it reached and refuses anything but db. The config refuses unknown keys, so a password line is an error rather than a silently ignored secret.

Use

stacs verify   --config stacs.toml --out-dir verify_report/   # changes nothing
stacs register --config stacs.toml --mode drift               # what the API lacks or serves stale
stacs register --config stacs.toml --mode all --dryrun
stacs register --config stacs.toml --mode ids --ids-file ids.txt
stacs audit    --config stacs.toml --dir build/items/ --expect 120
stacs validate --dir build/items/

# Repositories that register their own build output directly -- collection first:
stacs load collection --config stacs.toml build/items/collection.json
find build/items -name '*.json' ! -name collection.json \
  | stacs load items --config stacs.toml

load checks every item against the declared collection and asset rules before sending, and refuses an id that appears twice; it exits 1 if there was nothing to load.

--dryrun writes nothing, and it still reaches the host, and for register the API too. register asks the API for one item of the collection, and both commands run ssh <host> true, so a dryrun that passes means they answered, and one that cannot reach them exits 1. It then stops: all and ids before fetching any item body, drift after comparing every body with the API but before the asset audit, load after its checks. None of them runs the script on the host, so the env file, pypgstac and the database check are first exercised by a real load. When verify reports the collection same, stacs load collection with a downloaded copy of the published collection.json is the smallest real load: it upserts the body the database already holds.

verify exits 1 on any drift and writes missing.txt, changed.txt, orphaned.txt and collection_state.txt to --out-dir, and queryables_missing.txt and queryables_changed.txt when queryables are declared. register refuses before writing wherever it can: a collection id that does not match the published collection.json, an item link duplicated or pointing at a body that names another id, a body that could not be fetched, an item that fails the audit, an unreachable host.

ids mode checks only the ids it registered. If collection.json changed, run verify after it: pgstac serves items hydrated against their collection, so a collection change can change how untouched items read back. all, and a drift whose collection changed, re-compare the whole catalogue after writing for that reason.

What it assumes

• The STAC API is stac-fastapi on pgstac, or speaks its paging dialect: POST /search with collections, ids, limit and fields, the continuation token in the next link's body.
• Items are linked from collection.json with rel: item, and an item's id is its file name without .json, with %20 read as a space. Child catalogues are refused, not skipped.
• The STAC host has bash and a pypgstac that can reach the database.
• Where queryables are declared, the API has the filter extension enabled. A 404 for an existing collection's queryables is reported as the extension missing, never as "none served".

Development

uv sync
uv run pytest

The tests run offline: a stub STAC API on loopback, a stub ssh that runs the remote script locally against a fake pypgstac, and a guard that refuses any DNS lookup outside loopback.

License

MIT