fix(sql): TRY_CAST in a computed column, and the type rules the assembly exposed (#382)

[fupelaqu]

Closes #382

What was broken

TRY_CAST / SAFE_CAST in a computed column had never worked, in any shape. The generated ingest script put the assignment inside a catch, with no right-hand side:

CREATE TABLE t (name KEYWORD, c BIGINT SCRIPT AS (TRY_CAST(name AS BIGINT)))
  def param1 = ctx.name; try { (param1 != null ? (def)(Long.parseLong(param1)…) : null) }
  catch (Exception e) { return null; ctx.c = }        -- Elasticsearch: compile error

fromScript re-derived the prologue/expression boundary by splitting the rendered Painless on ;. A ; inside a literal was #373/#375; a ; inside a block is this one. The cure is not a smarter scanner — it is to stop throwing the boundary away and then hunting for it.

Measuring it found the family is wider, and one member was silent: TRY_CAST(name AS DOUBLE) + 1 dropped the conversion and computed on the raw string, HTTP 200.

What changed

#	Change	Example
1	The assembly is handed the prologue and the expression (IngestScript.assemble)	c BIGINT SCRIPT AS (TRY_CAST(name AS BIGINT)) → '125'→125, 'abc'→null
2	A safe cast hoists its try/catch into the prologue in every context with one	CONCAT/COALESCE/CASE/UPPER over a TRY_CAST were def param2 = try { … }; compile errors
3	Arithmetic over a cast computes instead of concatenating	CAST(raw AS BIGINT) + m, '125'+7 stored 1257, now 132
4	A narrowing cast still narrows the OPERAND — but / itself always yields DOUBLE	CAST(x AS INTEGER) / 2 over 7.5 → 3.5 (the cast narrows 7.5 to 7), against 3.75 uncast
5	A failed safe cast is a NULL operand, not the text "null"	CONCAT(TRY_CAST(raw AS BIGINT),'x') over 'abc' stored "nullx", now NULL
6	NULLIF over a cast or a nullable function works	NULLIF(TRIM(description), '') — an example our own docs publish — now gives ' hello '→'hello', ' '→NULL
7	🔴 NULLIF over a compound argument was a silent wrong answer	NULLIF(CASE WHEN n>1 THEN 1 ELSE 2 END, 1) over n=5 stored 1 where SQL says NULL
8	A NULLIF argument is evaluated once, not three times	NULLIF(UPPER(s),'X') rendered UPPER(s) 3×
9	🔴 NULLIF refuses every pair it cannot compare, derived from the arms	a date against a number, a boolean against anything, a date against a string — all accepted and silently always-false before
10	🔴 / always yields DOUBLE, and a zero divisor is NULL, on every path	SELECT n / m → 3.5; n / 0 → NULL instead of a rejected document
11	🔴 A pipeline diff no longer drops, nor re-keys, a processor	see below
12	ScriptTarget.of no longer reads a Painless comment — or a regex literal — as code	ctx.c = 1 /* } ctx.zzz = 9 */ reported column zzz; RLIKE 'a/*b' blanked the script

#11, in full — an ALTER that silently did nothing

A script processor has no field, so Elasticsearch hands a hand-authored one back anonymous and its identity is recovered from the last ctx.<name> = in the source. Two processors claiming the same column made the diff indexed by Seq.toMap drop the loser — the ALTER did nothing and the stored script kept computing the old expression. Content-based disambiguation was tried and is wrong (only one side collides, so a change becomes an add plus two removals). A colliding member falls back to its pre-recovery identity, and — found by review — that fallback now reserves the keys nobody had to derive, so it cannot displace a processor that did not collide.

The engine decision behind #3 and #4

They are one method, argTypeOf: an operand's type is what it renders, not what its column is declared as. Reverting one reverts the other (measured: the 1257 concatenation returns, 6 tests red).

The / ruling (lead, 2026-09-23) is separate and stronger: the result type of a division is a property of the operator, not of its arguments. CAST(5.0 AS INTEGER) / 2 — measured, not quoted: PostgreSQL 16 → 2, MySQL 8 → 2.5, DuckDB 1.5.5 → 2.5. We follow MySQL and DuckDB. PostgreSQL is the outlier that truncates, and it spells truncation separately (DIV, //) — we have no such operator, so the documented way to truncate is to compute the quotient into a column and cast THAT column.

⚠️ This branch CLOSES a pre-existing engine-vs-DuckDB divergence. A JOIN query evaluates its SELECT list in DuckDB (JoinPlanner hands it field.identifier.sql), and DuckDB's / is always decimal — so on main SELECT n / m answered 3 outside a JOIN and 3.5 inside one, for the same two INTEGER columns. Aligning / removes the class, not one instance of it.

Release notes (0.24.0)

1. TRY_CAST in a computed column works. An unconvertible row is indexed with the column NULL — _source carries "c": null, no doc value — so a plain SELECT returns the row with an empty value while IS NOT NULL and aggregations skip it.
2. 🔴 A cast decides the arithmetic around it, in a computed column, a WHERE and a projection alike. A WHERE over such an expression matches a different row set. Arithmetic with no cast and a cast to the column's own type are unchanged (0 of 232 corpus rows move for the cast rule; the / rule below moves more).
3. 🔴 / always yields DOUBLE. SELECT n / m over two INTEGER columns answers 3.5, not 3. This changes projections, computed columns and WHERE row sets for every statement that divides. +, -, * and % are unchanged. There is no truncating-division operator: compute the quotient into a column and cast that column.
4. 🔴 A zero divisor yields NULL instead of failing. Integer division threw (HTTP 400 in a search, a silently absent column on ingest) and floating division produced Infinity, which Elasticsearch refuses to index — rejecting the whole document. That was data loss on any double column. SELECT 1/0 now returns NULL. % is NOT covered and still throws (integer) or yields NaN and loses the document (floating) — see known limitations.
5. 🔴 NULLIF refuses every pair it cannot compare. Comparable means both text, both temporal, both numeric, both boolean, or one side NULL / an unresolved column. NULLIF(<date>, <number>), NULLIF(<boolean>, <anything else>) and NULLIF(<date>, '<string literal>') were ACCEPTED before and silently returned the first argument for every row — Painless == answers false for those pairs without failing. A well-formed date literal is refused too, saying so explicitly; cast it (NULLIF(created_at, CAST('2025-01-01' AS DATE))). A follow-up story is scoped for the coercion. ⚠️ The refusal applies to schema-resolved SELECTs; a wildcard or multi-index FROM, and DML, do not reach it.
6. 🔴 A cast to a TEXT type over a numeric column is now a rejection where it used to work — NULLIF(CAST(qty AS KEYWORD), 0) stored a value before. Remedy: cast the other side, or drop the cast.
7. ⚠️ ALTER churn, two sources. ScriptProcessor.source is persisted in _meta and IngestPipeline.diff compares it. (a) A stored computed column of a moved shape (cast-arithmetic, NULLIF(<nullable function>, …)) is reported ProcessorChanged. (b) Every stored column that divides moved too: n / 2 → (param1 / ((double) 2)), and n / m gained a (double) cast and a zero guard. Re-run CREATE TABLE / ALTER … SET SCRIPT AS and reindex for affected tables; downstream repositories pinning generated JSON need updating.
8. IngestPipeline.diff no longer drops a processor on a recovered-identity collision, and no longer re-keys one that did not collide. On ES 6.8 (no processor description) such a collision churns rather than reporting a change.
9. New, all additive and source-compatible: app.softnetwork.elastic.sql.schema.IngestScript.assemble, Function.rendersNullOnFailure, and the type-family predicates on SQLType (isText, isNumber, isTemporal, isBoolean, isUnknown). New ResolveCostProbe in core/src/test.

Two decisions the spec required be stated here

• PainlessOperandForm.splitStatements has no production caller since the assembly is handed its two halves. It is kept for two differential specs and its scaladoc says so. Lead decision requested: keep or remove.
• The case Array(single) if single.trim.startsWith("return ") arm was REMOVED, proved unreachable as a property rather than argued (\breturn\b absent across the 13-shape sweep). The softclient4es-extensions copy has the same arm and is deleted with it on that branch.

Evidence

• sql/test 1567/1567 · core/test 1141/1141
• Executed on real Elasticsearch: es8java 94/94 (8.18.3), es6rest 92/92 and es6jest 92/92 (6.8.23, 2 assume-skipped each)
• ++ 2.12.20 compiling sql/main, sql/test and core/test · the exact CI lint line (headerCheck scalafmtSbtCheck scalafmtCheck test:scalafmtCheck), files staged first
• Not run locally: es7 and es9 — left to CI, which runs sbt compile test at the root
• Every new assertion was seen RED by the mutation it guards; reverting the NULLIF comparability rule to its hand-written predecessor reddens 4 assertions

🔴 What the integration legs found that the unit suites could not

Both are recorded because they are the argument for running them at all.

• ES 6.8 refuses the zero guard when the division is the whole script. ((((double) 0) == 0) ? null : …) → illegal_argument_exception: Extraneous conditional statement. Well-formed on 8.18, and the byte pin said so. Fixed by folding a literal-zero divisor to null — same answer everywhere, one less runtime guard. Divisions over columns were always fine, because the parameter declarations precede the conditional.
• A contradiction between the unit spec and its executed twin. NULLIF(created, '2024-01-15') was asserted to succeed in GatewayApiIntegrationSpec while the unit spec had already been amended to expect a refusal — and sql/test plus core/test were both green with that disagreement in the tree.

Parse, render and resolve cost

• Parse / render: differential probe over 169 statements, branch vs a clone at origin/main. Parsing: exactly one change (NULLIF(CAST(n AS KEYWORD), 0), the shape release note 6 names). Rendering: zero changes. Cost: no signal — interleaved medians 523 ms vs 522 ms, ranges 493–654 vs 493–682, overlapping.
• Resolution, measured for the first time (new ResolveCostProbe, asked for by the lead): resolution is 0.4–5 % of the parse it follows — 0.6–8.4 µs against a 69–356 µs parse. validateResolved is 0.1–0.9 µs, so the second AST walk this branch adds is not a cost concern; the whole update(Some(schema)) rebuild is 0.5–1.5 µs. The only stage that registers is TemporalLiterals at ~2.7 µs. 🔴 An un-LIMITed row query pays the chain twice (search → scrollRows → scroll). ⚠️ Absolute measurement; excludes loadSchema and the round-trip, and is not an attribution against a 0.23.x control.

Recorded, not fixed

• HAVING <function>(<aggregate>) silently drops the entire HAVING clause — every group is returned #389 — HAVING <function>(<aggregate>) silently drops the whole clause. Filed, documented, identical before and after this branch.
• The arithmetic-operand gap is UNFILED — CAST(a / b AS INTEGER), FLOOR(a / b), COALESCE(a / b, 0) and CASE WHEN … THEN a / b END are all parse errors, directionally (arithmetic over a function is fine). It is not CAST/TRY_CAST/CONVERT reject bare literal operands — CAST('125' AS BIGINT) does not parse #267, which is a different defect (a bare literal operand), closed 2026-09-04. Documented in known_limitations.md. Needs a lead decision: file it, or accept an explicitly-unfiled dependency, since a shipped design argument leans on it.
• % by zero is not guarded — integer operands throw; floating operands yield NaN and Elasticsearch rejects the whole document, the same data loss the / guard fixes. Documented.
• softclient4es-web publishes no operators, precedence or conditional-functions page, so the / rule reaches no published page. Belongs to that repo.
• NULLIF(<numeric literal>, NULL) emits 0 == null — byte-identical on main, degenerate SQL, pinned as a characterisation.
• copyBridge (build.sbt:340) races sbt's own source glob: when it loses, a bridge compiles from zero sources, publishLocal reports success, and a 318-byte empty jar lands in ~/.ivy2/local. Reproduced three times.

Sequencing

softclient4es-extensions has a companion branch that deletes its private copy of this assembly and calls IngestScript.assemble. Its PR is deliberately not open: it cannot compile until this merges and 0.24.0-SNAPSHOT is published (the snapshot on JFrog carries no IngestScript, and release.yml fires only on workflow_dispatch or a v* tag — merging alone publishes nothing). Verify jar contents, not the exit code, when publishing.

🤖 Generated with Claude Code