feat(gates): gate 99 — demo data for every schema, and it must be valid - #590
Merged
Conversation
added 3 commits
August 27, 2026 14:20
ADR-111 rule 1. An app installed from the App Store opens on an empty list, and
the person evaluating it must author objects by hand against a schema they do
not know yet. Measured 2026-08-27: the fleet declares 598 schemas and 16 have
any demo data.
Three objects per schema, not one: one cannot show a list as a list, cannot
distinguish an empty state from a populated one, and leaves a detail page with
no sibling to page to.
🔴 COUNTING IS NOT CHECKING. Demo data that fails its own schema fails at
import, in front of whoever asked for the demo, so every object is validated
with jsonschema against the schema that declares it. ARM 4 of the acceptance
suite pins exactly that: three objects present, one `status` outside its own
enum, still a failure.
DIFF-SCOPED, keyed on HAVE_DELTA_BASE — not a judgement call. Twenty of
twenty-one apps have no demo data at all; a full-tree version would redden every
branch in the fleet on the day it shipped. gate-98 made that exact mistake a day
earlier by keying on `SCOPE_TO_DIFF` (which defaults to 0), so the note is
written into this gate's header rather than left to be rediscovered.
THE CHECKER IS THE GENERATOR. `generate_mock_register.py` both writes demo data
and validates it, in one file, because a validator and a producer that each own
a copy of the rules drift apart silently — and this package has a gate about
that. Writing it moves to `--check` reading the same schemas.
TWO BUGS THE ACCEPTANCE SUITE FOUND, both of which made every app look empty:
1. `basename(app_dir)` with the gate's `.` produced app id `"."`, so the
lookup target became `./lib/Settings/._mock_register.json` — every app
reported zero demo objects, including apps that have them.
2. It keyed on a FILENAME CONVENTION at all. OpenRegister's five mocks are
`bag_register.json`, `brp_register.json`, `dso_register.json`… — a checker
looking for `{app}_mock_register.json` calls all five missing. It now keys
on `x-openregister.type: mock`, the same marker the register-descriptor
inventory uses.
Covered by `test_gate99_demo_data_scope.sh` over a real two-commit history and
registered in COVERED-ELSEWHERE.md — a delta gate cannot be covered by a
gate-acceptance bundle, which has no git and would only ever report NOT
APPLICABLE.
Verified: suite ALL PASS (4 arms); acceptance matrix passed 190 / failed 0 with
gate-99 in the covered set; empty-scope suite ALL PASS.
ADR-111 rule 4. An app installed from the App Store opens on an empty list, and
the only question its first reader has is "can I see this working?" — which they
cannot answer, because authoring objects requires a schema they do not know yet.
ADR-042 already built every piece: `CnSetupWizard` renders
`manifest.setup.steps[]`, the manifest schema types the six step kinds, and
`POST /api/setup/action/{action}` runs a step's privileged server-side work.
Measured 2026-08-27: ten fleet apps declare setup steps, and every one opens
with `welcome`.
A welcome screen tells you what an app is. The demo-data offer lets you SEE it.
There is no reason the first step cannot also say hello.
🔴 IT ASSERTS ON THE DECLARATION, NOT ON VUE SOURCE — and that is the whole
reason this gate is strong. The first draft of ADR-111 measured adoption by
scanning `src/**/*Wizard*.vue`, found one app, and was wrong by 10x: the
walkthrough is not a file an app writes, it is a declaration an app makes, and a
shared component renders it. Counting the artefact instead of the declaration
would also have justified rebuilding a component that already exists.
DOES NOT ENFORCE PRESENCE, deliberately. Twenty of thirty manifests declare no
`setup` at all; failing them would block every unrelated manifest edit in the
fleet on the day this ships. That is how a gate gets switched off, and gate-98
did exactly it a day earlier. Adoption is a rollout, tracked as work — this gate
stops the ten that exist from drifting and stops new ones landing wrong.
Presence becomes enforceable in a follow-up, and that ordering is deliberate.
An unreadable or non-object manifest IS a finding rather than a silent skip: a
gate that shrugs at a broken input reports a clean bill of health over a file
nothing could parse.
Verified against three real fleet manifests before wiring: opencatalogi (opens
with `welcome`) FAILs, openregister (no setup block) passes, and a diff touching
no manifest exits 4.
Covered by `test_gate100_setup_demo_first_scope.sh` over a real two-commit
history, registered in COVERED-ELSEWHERE.md. ARM 3 is the load-bearing one — the
baseline deliberately ships a manifest with no setup block, which is what most
of the fleet looks like.
Verified: suite ALL PASS (4 arms); acceptance matrix passed 190 / failed 0 with
gate-100 covered; empty-scope suite ALL PASS.
…rries Rolling ADR-111 out to five real apps found five defects, all in the generator, all of which its own `--check` caught. 242 schemas across those apps now generate and validate clean; before these fixes, 10 did not. 1. 🔴 IT PAIRED EVERY REGISTER WITH EVERY SCHEMA IN THE SAME FILE. Fleet descriptors are MODULAR — pipelinq declares one register across twenty files, each contributing a few schemas, and one file declares three registers at once. The cross-product produced SIX objects for `client` instead of three, attributed schemas to registers that do not carry them, and validated objects against a same-named schema defined in a different file. A register's own `schemas` list is the authority and is now read. 2. THE FIRST FIX FOR (1) WAS ALSO WRONG, and worse: "declared list ELSE fallback" dropped scholiq from 118 schemas to 1 and launchpad to 0, because a modular register names its list in SOME files and omits it in others. It is a union, not a choice. Caught by re-measuring rather than by assuming the fix worked. 3. A GROUPED ALTERNATION defeated the pattern walk. `^([01]\d|2[0-3]):[0-5]\d$` — a clock time — could not be stepped over, so two schemas got values failing their own pattern. Resolved to the first branch, then verified against the FULL pattern so a wrong branch is caught rather than shipped. 4. AN OPEN QUANTIFIER IGNORED minLength. `+` expanded to three characters, so `minLength: 20` with `^[A-Za-z0-9+/=]+$` produced "ABC" — pattern-valid, length-invalid. The early return was the real culprit: it accepted any value the pattern matched without consulting length at all. One constraint checked, the other not. 5. 🔴 `nullable: true` IS OPENAPI, NOT JSON SCHEMA — and this one had the validator wrong rather than the data. Every fleet descriptor declares `"openapi": "3.0.0"` and uses the keyword; jsonschema does not know it, so a nullable property given `null` by the schema's OWN declared `default` was reported as `None is not of type 'string'`. That is the more dangerous direction of error: it would have had an author "fix" data that was already correct. Translated to `type: [T, "null"]`, which is what it means. Also removed an `if True:` scaffold left from an indentation-preserving edit. Verified: gate-99's four-arm suite still ALL PASS; larpingapp 10, portaliq 13, scholiq 118, pipelinq 100 and launchpad 1 schema all generate and validate with zero findings.
This was referenced Aug 27, 2026
added 3 commits
August 27, 2026 15:19
…led its own rule
Rolling ADR-111 to three more apps found two more gaps, both in the pattern
synthesiser, both caught by `--check` rather than by review.
1. 🔴 AN ESCAPED LITERAL MATCHED NEITHER BRANCH. The tokeniser accepted `\d`,
`\w` and `\s` — and nothing else escaped. `\.` therefore matched neither the
class branch nor the escape branch, the walk stopped, and the whole pattern
was abandoned: `^[0-9]+\.[0-9]+\.[0-9]+$` produced "Voorbeeld Version 1".
Every semver-shaped property in the fleet was affected. An escaped
anything-else is a literal of itself.
2. AN OPTIONAL GROUP could not be stepped over at all.
`^\d+\.\d+\.\d+(?:[-+][\w.-]+)?$` — semver with an optional pre-release —
left the walk stuck the same way. An optional group contributes nothing to
the shortest matching string, so it is dropped; a bare non-capturing group
becomes its contents.
Both were invisible until an app that uses them was generated, which is the
argument for rolling out breadth-first rather than perfecting one app: five
apps found five defects, three more found two.
Verified: all five pattern families now synthesise and self-verify —
`^[0-9]+\.[0-9]+\.[0-9]+$`, `^\d+\.\d+\.\d+(?:[-+][\w.-]+)?$`,
`^([01]\d|2[0-3]):[0-5]\d$`, `^CVE-\d{4}-\d{4,}$`, `^#[0-9A-Fa-f]{6}$`.
Seven apps regenerate with ZERO findings across 287 schemas: larpingapp 10,
portaliq 13, scholiq 118, pipelinq 100, launchpad 1, decidesk 39, openbuild 6.
Both gate suites still ALL PASS.
gate-60 (icon-vocabulary) failed the very PRs that added demo data: the
generator wrote `Voorbeeld Icon 1` into any property named `icon`, and an icon
outside the canonical vocabulary "renders blank wherever it is not aliased
locally" (ADR-077 rule 1).
🔴 THE FIX READS gate-60's OWN VOCABULARY FILE. `schemas/semantic-icons.json` is
the file the gate validates against, so the generator loads it rather than
carrying a second list. A producer and its judge holding separate copies of the
same rule is the defect this package corrected once already today; repeating it
one commit later would have been remarkable.
🔴 AND IT READS THE VALUES, NOT THE KEYS — which took a second attempt. The map
is keyed by SEMANTIC name and valued by MDI component name:
"documentation": "BookOpenVariantOutline"
The first fix emitted `documentation`, and gate-60 rejected it in the same words
as the invented name it replaced: "a kebab-case or lowercase spelling of an MDI
name resolves to nothing". Half-reading a vocabulary is not reading it, and the
failure looked identical from the outside.
An UNREADABLE vocabulary yields an empty list and the value is left alone, so
gate-60 reports it. A generator that invents an icon when it cannot read the
rules is worse than one that admits it.
Verified: portaliq gate-60 PASS with zero gate failures; seven apps regenerate
with zero validation findings; both gate-99 and gate-100 suites ALL PASS.
…eir schemas I reported 55 schemas across softwarecatalog and docudesk as "not valid JSON Schema" and wrote it into ADR-111 as a Consequence those apps must fix before they can comply. Cloning both and looking properly: 106 findings, and almost all of them were this checker being wrong about OpenRegister's dialect. FOUR DIALECT DIFFERENCES, each measured before being accepted: 1. `required: true` ON A PROPERTY is OpenAPI 2.0; JSON Schema puts `required` as an ARRAY on the parent. 62 findings. 🔑 I checked whether it was decorative or load-bearing before deciding: 38 of the 42 properties carrying the flag are ALSO named in their parent's array, so it is redundant and consistent. Dropped. (The four that are NOT mirrored are a genuine finding — a field its author marked required that nothing enforces — reported, not silently promoted, because promoting them changes what saves.) 2. BUILDER PLACEHOLDERS. These schemas come from a form builder that writes EVERY keyword, using `null` or `[]` for the ones that do not apply: `"minLength": null, "maximum": null, "oneOf": []`. Read as written, `oneOf: []` means "validate against exactly one of NOTHING" and no value can satisfy it — 144 properties in one app. Read as intended it is absent, which is what the author meant and what OpenRegister does. 3. OpenRegister's TYPE VOCABULARY IS WIDER THAN JSON SCHEMA'S. `PropertyValidatorHandler::$validTypes` accepts `file`, `geo`, `color`, `recurrence`, `NcFile`, `NcMail`, `NcContact`, `NcNote`, `NcTodo`, and handles `type: 'file'` at line 244. Twelve legitimate declarations were called invalid by a validator that only knows the seven JSON Schema types. 4. `nullable: true` — already handled, same family. 🔴 AND THE FIRST FIX FOR (2) WAS AN ENUMERATION, WHICH MISSED. I listed minLength/maxLength/minimum/maximum/minItems/maxItems/pattern/format and still had twenty failures, because the builder also writes `multipleOf: null` and `exclusiveMinimum: null`. A list of the keywords I happened to think of is the same mistake as the icon lookup table two commits ago. It is now the general rule — any keyword holding null — with `default` and `const` exempt, since null is a legitimate value for both. RESULT, measured: softwarecatalog 36 -> 0 across 38 schemas; docudesk 70 -> 0 across 75. The seven already-shipped apps stay at zero. Both gate suites ALL PASS. The direction of this error is the dangerous one: a validator that calls conformant data invalid sends an author to "fix" schemas that were right, and I had already written that conclusion into an ADR.
This was referenced Aug 27, 2026
added 4 commits
August 27, 2026 18:20
Each produced a plausible, valid-looking result and exited 0. 1. DISCOVERY WAS NARROWER THAN COLLECTION. Schema definitions were swept only from files that also DECLARE a register. humaniq declares its register with a list of 54 schema names in one file and defines all 54 across thirty others that declare none, so every name resolved to nothing and the tool said "declares no schemas — nothing to generate". Three apps were written off on the strength of that sentence; humaniq alone has 54 schemas and now ships 162 demo objects. 2. A SCHEMA SPLIT ACROSS FILES WAS TRUNCATED to whichever file sorted first. pipelinq ships a base `ticket` plus a CTI overlay whose own comment says the overlay must land there "or OpenRegister's magic-table columns for these fields never exist". Definitions are now merged, properties and `required` unioned — the "an extended schema extends its base" semantics the descriptors already assumed. Four apps had been generating against half a schema. 3. THE APP ID CAME FROM THE CHECKOUT DIRECTORY. `x-openregister.app` is what the descriptor inventory resolves a register to an app by. Five of eight fleet apps already ship an `<id>` that differs from their directory name — larpinq, learniq, humaniq, decidiq, buildiq — so those descriptors named an app that does not exist, and the `--app=` hint they print resolved nothing. A cross-app id is a runtime lookup: it finds nobody rather than erroring. The delta scope for --check moves with it: it keyed on register-declaring files, so in a modular app a PR adding a schema put that schema OUT of scope and the gate passed a PR shipping a schema with no demo data. test_generate_mock_register.py: 12 tests. Verified as controls by reverting each defect in turn — 5, 2 and 1 failures respectively. The app-id arms initially stayed GREEN under revert because they called the helper directly and never ran main(); that gap is closed by an end-to-end arm asserting the filename and attribution that land on disk.
`components.registers` is not always a slug-keyed map — openregister ships
`avg-bundle.json` and `report-bundle.json` with a LIST of register objects.
The old discovery required a dict, so those files never reached the code that
reads `.keys()` off them. Widening discovery to schema-defining files let them
through and `--check` died on the app that owns the gate:
AttributeError: 'list' object has no attribute 'keys'
A crashed checker is not a finding. Guarded via `_as_dict`; the bundles still
contribute their schema definitions and contribute no register slug. This does
not teach the generator the list dialect — it stops it dying on one.
openregister now checks 15 schemas instead of crashing, and reports the two it
is actually missing. Two test arms, verified as a control: without the guard
the suite errors with that same AttributeError.
openregister ships `n8n_workflows.openregister.json`, which declares
`x-openregister.app: n8n` — the n8n app's register, carried in openregister's
tree so it installs alongside. The generator seeded it anyway, putting five
foreign schemas into openregister's own mock descriptor under a register
openregister does not own. openregister's `validate-register` then rejected
them for missing `slug`, a field n8n's schemas do not declare:
[validate-register] ERROR openregister_mock_register.json › workflow:
missing string "slug"
Two CI checks failed on one root cause.
The descriptor INVENTORY already attributes a register by its DECLARING app —
that was the fix for `n8n-workflows` being reported as openregister's missing
register. The generator did not apply the same rule, so the two tools
described different fleets. It does now.
Blast radius measured across all fifteen apps carrying demo data: openregister
is the only one affected. It drops from 15 schemas to 10 — the 5 removed are
n8n's — and its data now validates.
Three test arms, verified as a control: without the filter, two fail.
planninq's Hydra Gates went red on the demo data:
[gate-60] icon-vocabulary: FAIL — 1 icon(s) outside the canonical
vocabulary (ADR-077)
FAIL src/icons.js: 3 icon name(s) used by the manifests are NOT
registered — they render with no icon at all, not a fallback:
BookOpenVariantOutline, MapMarkerPath, ViewDashboardOutline
All three are canonical vocabulary names. None is in planninq's src/icons.js.
`CnAppNav` resolves an MDI name only through the registry `registerIcons()`
populates, with no fallback — the very defect ADR-077 exists for — so being IN
the vocabulary was never sufficient. The app has to have imported it.
Measured as a control: with the generated file removed the same checker
reported 2 manifests and 0 failures. The demo data was the whole finding, and
this is mine.
The generator now intersects the vocabulary with what the app registers, using
gate-60's OWN `_registered_icon_names()` — imported, not reimplemented, so the
producer and the judge cannot drift into disagreeing about what "registered"
means. That is the same rule that stopped `--check` and `build()` disagreeing
about which (register, schema) pairs exist.
Falls back to the full vocabulary when an app has no bootstrap to inspect:
emitting nothing would trade a gate failure for a blank icon, which is worse —
the gate goes quiet and the demo shows a gap.
Blast radius measured across all sixteen apps carrying demo data: only pipelinq
and planninq change, and every app now reports 0 gate-60 failures.
Two test arms, verified by control: without the scope, one fails.
rubenvdlinde
added a commit
that referenced
this pull request
Aug 28, 2026
…615) Two different gates both declared number 99: 10818: # GATE 99 — manifest-l10n-coverage (#604, 2026-08-27) 10997: # GATE 99 — demo-data-coverage (#590, earlier) demo-data-coverage held 99 first and COVERED-ELSEWHERE.md records it there, so manifest-l10n-coverage is the one that moves. WHAT IT BROKE. The declared inventory is built with ... | sort -n -k1,1 -s | awk '!seen[$1]++' first declaration of each NUMBER wins — so one of the two gates was absent from the inventory entirely, and `_declared_n` counted 80 where 81 gates exist. The coverage line then printed arithmetic that cannot be true. From a real pipelinq run: COVERAGE: 72 of 80 declared gates reported a result (9 not applicable; 72 of 71 applicable gates ran) 72 of 71. That is the duplicate, surfacing as a number nobody can act on. It also made `test_gate99_demo_data_scope.sh` unpassable: it greps the runner's gate-99 output and received manifest-l10n-coverage's NOT APPLICABLE verdict on all three arms, so `Package invariants` has been red on main — masking any real regression in this repository behind a failure nobody could fix. Verified: - `bash -n` on the runner - test_gate99_demo_data_scope.sh: ALL PASS (was 3 FAILURES) — all four arms - test_gate_acceptance_matrix.sh: 193 passed, 0 failed - the fixtured-gate count moves 80 -> 81, which is the shadowed gate reappearing in the inventory - no duplicate gate number remains - demo-data-coverage's five call sites are untouched Co-authored-by: Ruben van der Linde <juan.claude@conduction.nl>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Implements ADR-111 rule 1 (ConductionNL/hydra
feat/mock-data-generator).Why
An app installed from the App Store opens on an empty list. The person evaluating it — a municipal CISO, a VNG architect, a developer sizing an integration — has to author objects by hand before they can see whether it does anything, and the schema they must satisfy is exactly the thing they don't know yet.
Measured 2026-08-27 across the fourteen fleet apps checked out locally:
Three objects per schema, not one: one object cannot show a list as a list, cannot distinguish an empty state from a populated one, and leaves a detail page with no sibling to page to.
🔴 Counting is not checking
Demo data that fails its own schema fails at import, in front of whoever asked for the demo. So every object is validated with
jsonschemaagainst the schema that declares it.ARM 4 of the acceptance suite pins exactly this: three objects present, one
statusoutside its own enum — still a failure. A gate that only counted would have passed it.Diff-scoped, and not as a judgement call
Twenty of twenty-one apps have no demo data at all. A full-tree version would redden every branch in the fleet the day it shipped.
gate-98 made that exact mistake a day earlier by keying on
SCOPE_TO_DIFF(which defaults to0), so this gate keys onHAVE_DELTA_BASEand the reasoning is written into its header rather than left to be rediscovered.The checker is the generator
generate_mock_register.pyboth writes demo data and validates it, in one file — because a validator and a producer that each own a copy of the rules drift apart silently, and this package ships a gate about precisely that.Two bugs the acceptance suite found
Both made every app look empty:
basename(app_dir)with the gate's.produced app id".", so the lookup target became./lib/Settings/._mock_register.json. Every app reported zero demo objects — including apps that have them.bag_register.json,brp_register.json,dso_register.json… — a checker looking for{app}_mock_register.jsoncalls all five missing. It now keys onx-openregister.type: mock, the same marker the register-descriptor inventory uses.Coverage
test_gate99_demo_data_scope.shover a real two-commit history, registered inCOVERED-ELSEWHERE.md. A delta gate cannot be covered by agate-acceptance/bundle — that format has no git, so the gate could only ever report NOT APPLICABLE there.🤖 Generated with Claude Code