Skip to content

branch-4.1:[feature](iceberg) Enforce NULL defaults for VARIANT - #67323

Open
hubgeter wants to merge 1 commit into
apache:branch-4.1from
hubgeter:feature/iceberg-variant-null-defaults-branch-4.1
Open

branch-4.1:[feature](iceberg) Enforce NULL defaults for VARIANT#67323
hubgeter wants to merge 1 commit into
apache:branch-4.1from
hubgeter:feature/iceberg-variant-null-defaults-branch-4.1

Conversation

@hubgeter

Copy link
Copy Markdown
Contributor

What problem does this PR solve?

Related PR: #66538
Problem Summary:
Iceberg V3 requires both initial-default and write-default of a VARIANT field to be NULL. After adding general Iceberg V3 default-value support, Doris still needs explicit VARIANT handling to avoid accepting unsupported non-NULL defaults or reconstructing a compute-V2 VARIANT as the legacy type.
This PR:

  • Rejects non-NULL VARIANT initial-default and write-default values in FE.
  • Adds the same validation to the BE read path for malformed or externally forged Iceberg metadata.
  • Preserves the compute-V2 VARIANT type when reconstructing the current schema from Thrift metadata.
  • Supports typed SQL NULL defaults for optional VARIANT fields, including VARIANT fields nested inside structs.
  • Adds FE and BE unit coverage.
  • Adds an external Iceberg regression covering historical Parquet/ORC rows, INSERT, UPDATE, MERGE, explicit DEFAULT, SQL NULL versus JSON null, invalid DDL, and Spark interoperability.

Release note

None

Check List (For Author)

  • Test

    • Regression test
    • Unit Test
    • Manual test (add detailed scripts or steps below)
    • No need to test or manual test. Explain why:
      • This is a refactor/code format and no logic has been changed.
      • Previous test can cover this change.
      • No code files have been changed.
      • Other reason
  • Behavior changed:

    • No.
    • Yes.
  • Does this need documentation?

    • No.
    • Yes.

Check List (For Reviewer who merge this PR)

  • Confirm the release note
  • Confirm test cases
  • Confirm document
  • Add branch pick label

@hubgeter
hubgeter requested a review from yiguolei as a code owner August 31, 2026 02:49
@hello-stephen

Copy link
Copy Markdown
Contributor

Thank you for your contribution to Apache Doris.
Don't know what should be done next? See How to process your PR.

Please clearly describe your PR:

  1. What problem was fixed (it's best to include specific error reporting information). How it was fixed.
  2. Which behaviors were modified. What was the previous behavior, what is it now, why was it modified, and what possible impacts might there be.
  3. What features were added. Why was this function added?
  4. Which code was refactored and why was this part of the code refactored?
  5. Which functions were optimized and what is the difference before and after the optimization?

@hubgeter

hubgeter commented Sep 1, 2026

Copy link
Copy Markdown
Contributor Author

run buildall

@hubgeter

hubgeter commented Sep 1, 2026

Copy link
Copy Markdown
Contributor Author

/review

@github-actions github-actions Bot left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Review status: complete. Approval opinion: looks good from a code-review perspective; no blocking issue was found on head d2b6763.

Critical checkpoints:

  • Iceberg defaults: optional VARIANT initial and write defaults remain typed SQL NULL, while non-NULL defaults are rejected in FE and defensively in BE. Explicit Variant values, including a Variant payload representing JSON null, remain distinct from SQL NULL.
  • Schema and type identity: the FE to TField carrier remains recursive, and the BE correctly preserves the compute-V2 VARIANT representation across the legacy TColumnType boundary for root and nested fields.
  • Reader materialization: missing historical fields, required-field failures, nested struct defaults, schema-history lookups, and equality-delete consumers follow the same table-layer default contract. Historical ORC rows remain supported without enabling physical ORC Variant.
  • Architecture and lifecycle: the change stays within schema/default and TableReader responsibilities, adds no reverse format-layer dependency, and does not alter split, EOF, cleanup, caching, or profile behavior.
  • Performance and observability: the added checks run during schema/default setup, not in per-row or batch hot paths; no counter or attribution path is removed.
  • Coverage: focused FE and BE tests cover top-level and nested null/default behavior and malformed metadata. The external regression covers historical Parquet and ORC rows, INSERT, UPDATE, MERGE, DEFAULT, SQL NULL versus Variant JSON null, invalid DDL atomicity, and Spark interoperability. No local builds or tests were run, per the review-runner instruction.

User focus: no additional focus points were provided.

Convergence: the normal FE/integration review, normal BE/materialization review, and separate risk-focused review all returned NO_NEW_VALUABLE_FINDINGS in Round 1. The final changed-file, unresolved-candidate, live-head, and live-inline-thread sweep was clean. No inline comments are needed.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants