Skip to content

Bring geoprocessor source and tests into cumulus-api - #798

Open
oskarhurst wants to merge 2 commits into
cwbi-devfrom
feature/integrate-geoprocessors
Open

oskarhurst wants to merge 2 commits into
cwbi-devfrom
feature/integrate-geoprocessors

Conversation

@oskarhurst

Copy link
Copy Markdown
Contributor

The geoprocess worker currently downloads the processor package from USACE/cumulus-geoproc@main during its Docker build. Processor changes therefore require a separate repository push and can change an image without a corresponding Cumulus API commit.

This PR brings processor development and builds into this repository. Editing async_geoprocess/geoproc/src/cumulus_geoproc/processors/ and rebuilding geoprocess now uses that exact local source.

Changes

  • Import the package, license, SNODAS resource, unit/integration tests, development runner, and documentation configuration from USACE/cumulus-geoproc at 0a25999. The first commit contains the unchanged upstream files; the second wires them into this repository. Historical commits remain in the source repository.
  • Replace the remote Git ADD with a local package COPY. Remove the obsolete processor Git-ref build argument from Compose and image workflows.
  • Let local Compose use the Dockerfile's GDAL 3.13.1 default, matching production and processor tests, instead of its old 3.9.1 override.
  • Add a Docker test target, standalone test Compose service, VS Code devcontainer, and a PR workflow running the imported processor tests.
  • Update local-development instructions. Keep the existing worker entrypoint, Python import name, and SQS message contract.

Reference grids remain in USACE/cumulus-geoproc-test-data, using the same 2025-11-04 release as the upstream test suite. Only the test image downloads that archive and installs development dependencies. Existing generated documentation stays in docs/async_geoprocess; the upstream generated test report is not imported.

Validation

  • Production image: docker build -t cumulus-geoproc-migration-review async_geoprocess — passed.
  • Test image: docker compose -f async_geoprocess/docker-compose.tests.yml build geoproc — passed.
  • Full imported suite: docker compose -f async_geoprocess/docker-compose.tests.yml run --rm geoproc — 16 passed in 66 seconds, including reference-grid conversions and COG validation. Three warnings were emitted: one NumPy binary-size warning and two Python warnings about continue inside finally in imported processors.
  • Production image smoke check: worker imports, all 66 processor plugins load, and the packaged SNODAS TIFF is present; default entrypoint and non-root user are retained.
  • Main Compose, test Compose, and devcontainer Compose configuration validation — passed. VS Code itself was not launched.
  • All 94 imported package/development-tool files match upstream byte for byte; 89 Python files parse successfully. Whitespace checks pass for the integration commit; existing whitespace in the unchanged import is preserved.

This validates processor execution with reference data, not a live SQS-to-API end-to-end run.

The original cumulus-geoproc repository is not archived or deleted by this PR. After merge, processor changes should be submitted here.

This branch has not been deployed

No deployments
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant