Add new jwt_secret for ooniprobe service - #160
Closed
LDiazN wants to merge 1 commit into
Closed
Conversation
Contributor
Author
|
After talking with @hellais we realized that we still need to use the same token for all services. So we have to take a different approach, for now we will rely on the |
aagbsn
added a commit
that referenced
this pull request
Aug 18, 2026
Extends the harness with a second, additional CI job that verifies the actual OONI data pipeline -- not just synthetic seed data -- survives the production upgrade path. Design per direct instruction: download real data exactly once per run, re-verify integrity + query correctness at each hop, and walk PRODUCTION_HOPS (the real 4-hop runbook) rather than every diagnostic bisection waypoint. - harness/real_data.py: new module. Snapshots real-data tables (row count + order-independent cityHash64 checksum) across all 3 nodes, loads real OONI measurements once via ooni/data#160's downloader/fastpath containers, takes a golden snapshot, then per hop: upgrades all 3 nodes (reusing scenarios.upgrade_node_step unmodified), diffs against the golden snapshot, and re-runs ooni/data's own pytest suite against the real api-oonimeasurements service. - docker-compose.real-data.yml: new overlay adapting ooni/data#160's tests/integration stack onto this project's existing 3-node replicated cluster instead of a single throwaway node. Pinned to the add_end_to_end_tests branch since #160 is still unmerged -- update the ref once it merges. - sql/001_schema.sql: adds fingerprints_dns, fingerprints_http, obs_web_ctrl, obs_http_middlebox, obs_openvpn -- the tables oonipipeline's observations workflow and fastpath actually write, with per-table provenance notes and manual signedness fixes where oonipipeline's own DDL generator is known-wrong (Int32/Int8 vs the already-verified obs_web table's UInt32/UInt8 convention). - harness/compose.py: multi-compose-file support (files= param throughout) plus inspect_exit_code()/run_oneoff() for polling one-shot containers and running a fresh `verify` pytest pass per hop. - harness/scenarios.py: extracted apply_schema() from load_schema_and_seed() so the real-data scenario can reuse the schema load without the synthetic seed rows; generalized step_ok() to cover every non-upgrade-node step shape via a single top-level "ok" fallback. - harness/report.py: render_ci_step() now handles the real-data scenario's step shapes (setup-real-data, load-real-data, golden-snapshot, verify-e2e, real-data-hop). - ci_step.py: 6 new subcommands (setup-real-data, load-real-data, golden-snapshot, verify-e2e, real-data-hop, teardown-real-data) wiring harness/real_data.py into discrete, individually pass/fail-able CI steps, matching the existing setup/upgrade-node/verify-ddl pattern. - .github/workflows/clickhouse_upgrade_test.yml: new real-data-upgrade job (checkout ooni/data at add_end_to_end_tests, sanity-check PRODUCTION_HOPS alignment, setup/load/snapshot/4-hop-loop/report/ teardown). Deliberately not run on pull_request -- depends on an unmerged external branch and real network downloads -- only on push to main and workflow_dispatch (scenario: real-data or all). - README.md: documents the new job's design, the load-once/verify-per-hop rationale, the external-branch caveat, and marks PR #477 review point 4 ("run the target version against the real API + data pipeline") as addressed. Verified offline (no Docker daemon available in this sandbox, consistent with every prior patch in this project): all Python modules compile, all YAML/compose files parse and interpolate, SQL is structurally sound (balanced parens/backticks, unique table names), and every new step shape round-trips correctly through step_ok()/render_ci_step(). Real validation deferred to an actual GitHub Actions run.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
This is related to this PR in backend: ooni/backend#923
We have to add a new
jwt_secretencryption key for ooniprobe, since the current ooniprobe service provides toy login tokens without authentication. If we don't use a different encryption key, you can get a session token from ooniprobe that can be used in ooniauth to login without authentication.Note: For this PR to work we have to create the key in Parameter Storage in AWS