fix(spp_studio): remove or rewrite pack items with non-evaluable CEL expressions

[gonzalesedwin1123]

Fixes #431.

Problem

24 of the 106 spp.studio.pack.item records shipped in spp_studio/data/packs/ carry filter
expressions that cannot work on any install: every load-bearing identifier ultimately points at a
registrant field that exists in no module — in some cases the identifier has no spp.cel.variable
at all, and in others the variable IS catalogued but its source_field is dangling (the broader
variable-catalogue defect is #446; this PR fixes the shipped pack items that consume it). They also fail
spp_cel_load_testing's strict studio-validation test
(test_all_pack_cel_expressions_translate), which gates the merge of #432 and would fail the
weekly ci-full run once that migration lands.

Approach

Per-item rewrite where a real near-equivalent survives, remove the rest — no placeholder
variables were invented, and no test was weakened. Every candidate rewrite was validated
empirically through the exact resolve+translate path the test uses (odoo shell against an
installed DB), because translation stops at the first bad identifier — terms after it in the
original expressions had never actually been validated.

Fixed properly (1)

Exclusion Criteria · Institutional Residence Exclusion keeps its shipped expression
(!in_institutional_care): the item only lacked a variable definition — core spp_registry
already defines the in_institutional_care field on individuals. This PR adds the missing
in_institutional_care standard variable over that field. standard_variables.xml is not
noupdate, so existing databases get the variable through the regular upgrade data load.

Rewritten (3) — every rewrite is strictly narrowing

Item	Old expression	New expression
OVC · Child Age Eligibility	age < child_age_limit || (age < 25 && is_in_education)	age < child_age_limit
OVC · Vulnerable Household Check	is_elderly_headed || is_child_headed || is_skipped_generation || has_chronically_ill_head	is_elderly_headed || is_child_headed
Public Works · Poverty Status Check	per_capita_income <= poverty_threshold || is_food_insecure	per_capita_income <= poverty_threshold

All three drop OR-branches, so the predicate only gets stricter — a template can lose reach, never
silently grant eligibility. Descriptions updated where the semantics narrowed. (GMI Residency
Requirement was initially rewritten here too, but that rewrite dropped an AND-conjunct —
fail-open — so per review it is removed instead.)

Removed (20)

No meaningful sub-expression survives — every load-bearing identifier is backed by nothing:

• Disability Assistance · Disability Status Verification
• OVC Support · Orphan Status Check (the !has_mother && !has_father fallback expands to
  members.*, which doesn't exist in individual context — no rescue)
• OVC Support · School Enrollment Requirement
• Social Pension · No Formal Pension Check
• Guaranteed Minimum Income · Employment Barriers Check (3 of its 4 criteria are broken —
  is_single_parent also fails validation; the single surviving term isn't a near-equivalent)
• Guaranteed Minimum Income · Residency Requirement (is_legal_resident exists nowhere; dropping
  it would leave residency_months >= 12 alone, which is MORE permissive than the shipped rule —
  the one fail-open case in the set, so per review it is removed rather than rewritten)
• Public Works · Physical Work Capability
• Public Works · Seasonal Availability
• Conditional Cash Transfer · Health Conditionality Compliance
• Geographic Targeting · Limited Service Access
• Exclusion Criteria · Government Employee, Formal Sector, Vehicle Ownership, Business Ownership,
  Housing Quality, Pension Receipt, Duplicate Program, Income Tax Payer, Bank Balance,
  Livestock Ownership (10 items; the pack keeps Land Ownership and Institutional Residence)

Migration

Pack data is noupdate="1", so existing databases keep the broken records after the XML change.
migrations/19.0.2.0.2/post-migration.py applies the same 20 removals and 3 rewrites directly.
Per review, every migration write is guarded on the item still carrying the known-broken shipped
expression (verified byte-for-byte against origin/19.0): noupdate exists so local changes
survive upgrades, so a deployment that already repaired an item (e.g. by defining the missing
variables itself) keeps its version, with a log line. The restored Institutional Residence item
needs no data fix — its new variable arrives via the regular (non-noupdate) data load — but the
migration narrows its context_type from both to individual to match the variable's scope
(also guarded). The new in_institutional_care variable uses applies_to="individual" per review.

Installed artifacts are deliberately left alone: installed_logic_id points at
spp.cel.expression records copied at pack-install time; once installed they are the deployment's
own data and may be referenced by programs, so the migration does not follow and delete them. They
were never evaluable, and cleaning them up is the deployment's call. The decision is documented in
the migration docstring.
Version 19.0.2.0.1 → 19.0.2.0.2 + HISTORY fragment.

Verification

All local (Docker), with spp_cel_load_testing taken from PR #432 @ 419752d6 since the strict test lives there:

1. Baseline (red) — fresh install with original data: test_all_pack_cel_expressions_translate fails with exactly the 24 items from spp_studio: shipped logic packs contain filter expressions referencing nonexistent fields/variables (24 items) #431 (14 translation errors + 10 unresolved-variable items).
2. Fresh install with this fix (green) — 0 failed, 0 error(s) of 6 tests; Translated 30/25 CEL expressions (0 errors, 0 with unresolved variables).
3. Upgrade path (green, including the guard) — DB installed with the ORIGINAL pack data at 19.0.2.0.1 (pristine origin/19.0), with ONE item locally modified pre-upgrade to simulate an admin repair (Child Age Eligibility set to age < 18 via SQL). The migration skips it with a warning and processes the rest: removed 20 and rewrote 2; item count 106 → 86; the in_institutional_care variable is created by the data load with applies_to=individual; the restored item lands as individual | !in_institutional_care; the locally-repaired item keeps age < 18 and still translates; strict suite 0 failed of 6 on the upgraded DB.
4. spp_studio own suite — Passed: 225 | Failed: 0 | Errors: 0.
5. Lint — ruff / ruff-format / prettier all pass on the changed files. README.rst deliberately not regenerated locally (CI's pinned generator is the authority; its diff will be applied verbatim).

Out of scope / follow-ups

• Several standard variables remain defined against nonexistent fields (is_orphan,
  owns_car, land_hectares, residency_years, is_single_parent, …). They are the same
  defect one level down but are not exercised by the strict test unless a pack item references
  them; needs its own issue.
• Re-home candidates: two removed concepts have real data in optional modules —
  Disability Status Verification maps cleanly onto spp_disability_registry
  (has_disability, disability_severity_id, assessments), and livestock/land exclusions onto
  spp_farmer_registry/spp_land_record. Since pack items are plain data records, those modules
  can ship their own items into the shipped packs, defined against their own fields — the right
  long-term home for domain templates. Needs its own issue.
• The CEL translator validates boolean field references against the model but lets numeric
  comparisons through unvalidated (r.land_hectares <= 2 translates even though the field exists
  nowhere) — latent quirk worth a look.
• required_modules support for "template" packs (issue spp_studio: shipped logic packs contain filter expressions referencing nonexistent fields/variables (24 items) #431 Option 3) if aspirational packs are
  wanted as a product feature.

Reviewer caveat

Neither this PR's CI nor #432's exercises the strict test (per-module jobs don't co-install
spp_studio and spp_cel_load_testing). Only a manual ci-full run after both merge does.
The local runs documented above are the substitute evidence.

[codecov]

Codecov Report

✅ All modified and coverable lines are covered by tests.
✅ Project coverage is 74.81%. Comparing base (3939520) to head (ff3e1df).
⚠️ Report is 4 commits behind head on 19.0.

Additional details and impacted files

@@            Coverage Diff             @@
##             19.0     #444      +/-   ##
==========================================
+ Coverage   74.76%   74.81%   +0.05%     
==========================================
  Files         509      565      +56     
  Lines       35008    39497    +4489     
==========================================
+ Hits        26172    29551    +3379     
- Misses       8836     9946    +1110     

Flag	Coverage Δ
spp_base_common	91.07% <ø> (ø)
spp_cel_event	85.34% <ø> (?)
spp_dci_demo	94.28% <ø> (ø)
spp_dci_indicators	96.23% <ø> (ø)
spp_farmer_registry	88.58% <ø> (?)
spp_farmer_registry_cr	61.24% <ø> (ø)
spp_farmer_registry_demo	63.43% <ø> (ø)
spp_mis_demo_v2	70.38% <ø> (ø)
spp_programs	65.53% <ø> (+0.25%)	⬆️
spp_registry	87.22% <ø> (+0.07%)	⬆️
spp_security	69.56% <ø> (ø)
spp_starter_sp_mis	88.88% <ø> (ø)
spp_studio	62.92% <ø> (?)
spp_studio_api_v2	95.35% <ø> (?)
spp_studio_change_requests	84.85% <ø> (ø)

Flags with carried forward coverage won't be shown. Click here to find out more.
see 61 files with indirect coverage changes

🚀 New features to boost your workflow:

• ❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.
• 📦 JS Bundle Analysis: Save yourself from yourself by tracking and limiting bundle sizes in JS merges.

[emjay0921]

One change requested. The mechanics of this PR are clean and I verified them item by item — notes in a separate comment, along with a root-cause finding that is bigger than this PR.

The GMI rewrite is the only fail-open change in the set, and it reads like the others

Three of the four rewrites drop OR-branches, which makes the predicate stricter — fewer registrants qualify. That is the safe direction for a shipped eligibility template. The GMI one drops an AND-conjunct:

is_legal_resident && residency_months >= 12   →   residency_months >= 12

That makes the rule more permissive: it now passes registrants the policy author wrote it to exclude. The new description — "Minimum residency duration requirement (12 months)." — drops any mention of legal residency, so nothing downstream records that the requirement was there and was removed. The PR body lists it in the same table as the three narrowing rewrites, and the note that "descriptions updated where the semantics narrowed" does not cover this case.

I checked whether the conjunct could be rescued the way Institutional Residence was: it cannot. There is no is_legal_resident field on any model (ir_model_fields on a dev DB, and a repo-wide field index), so the two consistent options are the same ones applied to the other 19 items:

1. Remove the item, like the 19 whose load-bearing terms have nothing behind them. This is the option I would take — a template that silently grants eligibility is worse than no template.
2. Keep item and expression as shipped, accepting that it does not evaluate until the field exists, and track it under the variable-catalogue issue (#446).

If there is a reason to prefer the narrowed-but-permissive form, that is a policy call rather than a data-hygiene one, and it should be stated explicitly in the PR body and the HISTORY entry so it is not read as an equivalent rewrite.

Everything else I found is non-blocking.

[emjay0921]

Non-blocking findings, plus the verification I did on the mechanics.

The root cause survives this fix — filed as #446

The 19 removed pack items were consumers of the defect, not the defect. spp.cel.variable records for is_legal_resident, is_food_insecure, is_in_education, is_skipped_generation, has_chronically_ill_head and is_single_parent all exist in standard_variables.xml; what is missing is the fields they point at. Across the catalogue, 25 of the 27 source_type="field" variables have a source_field that exists on no model — verified against a repo-wide index of 2,837 field definitions and then against ir_model_fields on a dev database. Details and the full list are in #446.

The practical consequence is that Studio's variable picker still offers these to no-code users, so the first filter anyone builds on one reproduces #431 from the UI. Worth keeping in mind when #431 is closed: the pack items are fixed, the class is not.

It also means the framing in the PR body is inverted for these six. It says the items reference "studio variables that exist in no module" and that "every load-bearing identifier is backed by nothing" — the variables exist, their fields do not. That is worth being precise about because it is the exact mirror of the in_institutional_care case this PR handles correctly (field existed, variable was missing, so adding the variable fixed it). The inverse case has a different set of fixes, which is what #446 is for.

Removed items leave their installed artifacts behind

spp.studio.pack.item.installed_logic_id is a Many2one to spp.cel.expression, so unlink() on a pack item does not touch logic already installed from it. On a database where one of the 19 packs was installed, the broken spp.cel.expression record survives this migration — and #432's TestStudioLogicValidation validates exactly those records, not just pack items. Worth deciding whether the migration should follow installed_logic_id for the removed set, or whether installed artifacts are considered the deployment's own data at that point.

The migration overwrites local edits without checking

REWRITTEN_ITEMS is applied with json.dumps({"cel_expression": expression}), which replaces the whole logic_data blob. Pack data is noupdate="1" precisely so local changes survive upgrades, so an admin who already repaired one of these four locally — plausible, since a deployment could have defined the missing variables itself — loses that silently, and any other keys in logic_data go with it. Guarding the write on "current expression still matches the known-broken shipped value" would make it safe to run anywhere.

Minor

The new var_in_institutional_care uses applies_to="both". Among the demographic-category variables, 17 use individual and 1 uses both, and the field is defined on individuals in spp_registry — so individual looks like the right value, and both offers it in group context where it cannot mean anything.

Verified

• The XML removals and the migration's REMOVED_ITEMS match exactly — 19 and 19, no drift in either direction. (My first extraction counted 18 and I nearly reported a mismatch; that was my regex missing prettier's multi-line <record> formatting, not a defect here.)
• All four rewritten expressions are identical between the pack XML and the migration, so fresh installs and upgraded databases converge on the same state.
• The migration guards if not version: return so it is skipped on fresh installs, uses env.ref(..., raise_if_not_found=False) throughout, and is idempotent on re-run.
• in_institutional_care genuinely exists at spp_registry/models/individual.py:40 and resolves on res.partner in an installed schema, and standard_variables.xml is not noupdate, so the claim that existing databases pick up the new variable through the regular data load holds.
• Version 19.0.2.0.1 → 19.0.2.0.2 matches the migration directory; the HISTORY entry names every removed item rather than summarising, which is the right call for a data-removal change.
• 1 + 4 + 19 = 24, reconciling with spp_studio: shipped logic packs contain filter expressions referencing nonexistent fields/variables (24 items) #431's inventory.

One carry-over

Neither this PR's CI nor #432's runs the assert that would actually validate this fix — test_all_pack_cel_expressions_translate only executes when spp_studio and spp_cel_load_testing are installed together, which no per-module job does. A manual ci-full run once both land is still the verification step.

[gonzalesedwin1123]

All points addressed — thank you for the review; the fail-open catch and the guard suggestion both made this materially safer.

Blocker — GMI Residency Requirement: removed (your option 1). You're right that it was the one fail-open change dressed up as a narrowing rewrite: the other three drop OR-branches, this one dropped an AND-conjunct. Rewrites are now 3 (all strictly narrowing), removals 20. The PR body and HISTORY now state the fail-open rationale explicitly instead of listing it alongside the narrowing rewrites.

Migration now guards every write — removals included. Each of the 20 removals and 3 rewrites only proceeds if the item still carries the known-broken shipped expression (verified byte-for-byte against origin/19.0 pack data by script); locally repaired items are left in place with a log line. I extended your suggestion to the removals too, since a deployment that defined the missing variables itself would have working items we must not delete. Rewrites also now update the cel_expression key inside logic_data rather than replacing the whole blob. Verified on a pristine 19.0.2.0.1 database with one simulated local repair: the migration skips it with a warning, processes the rest, and the strict suite stays green.

applies_to → individual, and to keep the restored item consistent with its variable's scope, its context_type is narrowed both → individual as well (in the pack XML for fresh installs, and in the migration — guarded — for existing databases). Without that, the strict test would probe the item in group context and fail on the individual-scoped variable.

Installed artifacts: deliberately left alone, now documented. installed_logic_id targets spp.cel.expression records copied at install time — after that point they're the deployment's own data and may be referenced by programs, so the migration doesn't follow and delete them. They were never evaluable, so they can't have working consumers; cleanup is the deployment's call. Decision recorded in the migration docstring and PR body. If you think the strict-test exposure on long-lived databases warrants an opt-in cleanup instead, happy to discuss — but I'd rather not delete user-owned records in a mandatory migration.

Framing fixed. The PR body now distinguishes the two defect shapes: identifiers with no spp.cel.variable at all vs. catalogued variables with dangling source_field — and points at #446 for the class. Thanks for filing that; the 25/27 figure is sobering, and you're right that the variable picker keeps offering these to no-code users until #446 is handled.

[emjay0921]

Approved. The blocker is resolved the way I'd hoped — removal rather than a narrowed-but-permissive rewrite — and the rationale is now recorded in both the PR body and the HISTORY entry, so the next reader sees why that item is in the removed set and not the rewritten one.

I re-derived the claims rather than reading them:

• Every guard matches shipped data. I parsed the 106 pack items out of origin/19.0's XML and compared each guard's expected expression against what actually shipped: 23/23 exact, zero mismatches. That was the failure mode worth checking — a guard whose expected string differed by a space would silently skip a broken item and leave it in place on upgrade, which is exactly the kind of bug that only shows up in production databases.
• XML and migration agree. 20 items gone from the pack XML, 20 in REMOVED_ITEMS, no drift in either direction; all 3 rewritten expressions identical between XML and migration. Fresh installs and upgraded databases converge on the same state.
• pack_gmi_item_residency is gone from the pack XML and now carries its shipped expression as a removal guard. 1 fixed + 3 rewritten + 20 removed = 24.
• applies_to="individual" on the new variable, and context_type="individual" on the restored item in both the XML and the guarded migration write. Narrowing the item's context alongside the variable's scope is the right call and the reason is sound — the strict test picks its translation profile from context_type, so leaving it both would probe an individual-scoped variable in group context.

Extending the guard to the removals was the better call, and it's a genuine improvement on what I suggested. I'd only argued for guarding the rewrites; you're right that a deployment which defined the missing variables itself would have working items, and deleting those would be worse than leaving broken ones. Mutating only the cel_expression key instead of replacing the blob closes the other half of it.

On installed artifacts — agreed, and no need to revisit. Not following installed_logic_id is the right default: those records were never evaluable so they can't have working consumers, and a mandatory migration shouldn't delete data the deployment owns. The residual exposure is bounded, too — ci-full builds fresh databases, so the strict suite never meets them there; only a long-lived deployment that installed one of the removed packs would, and as documented that's its own cleanup. Recording the decision in the docstring is what makes it reviewable later, which is the part that matters.

CI is green (34 checks; Trivy skipped as usual).

Merge order from here: this lands and closes #431 → #432 unblocks → a manual ci-full run is what actually proves test_all_pack_cel_expressions_translate passes on corrected data, since no per-module job installs spp_studio and spp_cel_load_testing together. #446 stays open for the variable catalogue behind all of this.

[gonzalesedwin1123]

Post-merge verification verdict for the strict suite, with one correction to the merge-order note above.

test_all_pack_cel_expressions_translate passes on the corrected data — but ci-full did not and cannot prove it. Run 32689039289 (19.0 @ 614cad7, after both this PR and #432 landed) never executed the test: the demo jobs install only each demo module's dependency closure (-i spp_mis_demo_v2 etc.), and no module in the repo depends on spp_cel_load_testing, so its test tag selects nothing. The assumption that a manual ci-full run would exercise spp_studio + spp_cel_load_testing together doesn't hold — no CI job anywhere installs that pair.

So the verdict was produced directly: a local run on 19.0 @ 614cad7 with -i spp_cel_load_testing,spp_studio --test-tags /spp_cel_load_testing — 0 failed, 0 errors of 101 tests, with TestStudioLogicPackValidation.test_all_pack_cel_expressions_translate confirmed in the executed set.

For the record, the ci-full failure itself is not related to this PR: diffing the three demo jobs' failure sets against the 2026-08-20 baseline run 32350162352, GRM and SP-MIS are identical to the known-red baseline, and DRIMS adds exactly the one known #438 test bug (test_sla_hours_are_keyed_by_real_priority_codes, _codes_in vs _canonical_codes_in). Zero regressions from this PR or #432.

Follow-up worth considering: give ci-full (or a per-module job) a leg that installs spp_studio + spp_cel_load_testing together, so the strict pack gate actually runs in CI instead of only on demand.