[tkaymak]

The Spark 4 Structured Streaming stateful ParDo grouped by the Beam key object, so a null key (VoidCoder, WithKeys.of((Void) null), Combine.globally) became a null Spark grouping key and transformWithState failed with INTERNAL_ERROR_TWS "Implicit key not found in state store" on the first state access.

The translator now groups by CoderHelpers.toByteArray(key, keyCoder) with Encoders.BINARY(), and BeamStatefulProcessor decodes the key once per handleInputRows and handleExpiredTimer call. Byte equality is Beam key equality, ParDo already rejects non deterministic key coders for stateful DoFns, and keys with custom coders were already grouped as bytes through EncoderHelpers.binaryEncoder.

The grouping key state schema changes from the key type to BinaryType, so a query restarted from a checkpoint written by #40281 or #40447 fails with STATE_STORE_KEY_SCHEMA_NOT_COMPATIBLE. That code is unreleased.

Per review, the structured streaming CoderHelpers is merged into the common org.apache.beam.runners.spark.coders.CoderHelpers, so all structured streaming callers now encode in nested context.

testNullKey fails on master with INTERNAL_ERROR_TWS and passes with this change.

Part of #36841.

[tkaymak]

R: @Abacn

[github-actions]

Stopping reviewer notifications for this pull request: review requested by someone other than the bot, ceding control. If you'd like to restart, comment assign set of reviewers

[codecov]

Codecov Report

✅ All modified and coverable lines are covered by tests.
✅ Project coverage is 65.07%. Comparing base (76f6e6d) to head (5980f56).
⚠️ Report is 221 commits behind head on master.

Additional details and impacted files

@@             Coverage Diff              @@
##             master   #40516      +/-   ##
============================================
+ Coverage     56.16%   65.07%   +8.90%     
- Complexity     2288    13991   +11703     
============================================
  Files          1124     1728     +604     
  Lines        178336   108509   -69827     
  Branches       1489    11313    +9824     
============================================
- Hits         100170    70610   -29560     
+ Misses        75645    33690   -41955     
- Partials       2521     4209    +1688     

Flag	Coverage Δ
go	?
java	65.07% <ø> (-10.72%)	⬇️
python	?

Flags with carried forward coverage won't be shown. Click here to find out more.

☔ View full report in Codecov by Harness.
📢 Have feedback on the report? Share it here.

🚀 New features to boost your workflow:

• ❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.
• 📦 JS Bundle Analysis: Save yourself from yourself by tracking and limiting bundle sizes in JS merges.

[Abacn]

Thanks, just one minor comment

[Abacn]

Thanks!