Skip to content

fix(avro): read TIME, BLOB, MULTISET and non-string-key map columns - #724

Merged
JingsongLi merged 1 commit into
apache:mainfrom
jackylee-ch:fix/avro-reader-missing-types
Aug 20, 2026
Merged

fix(avro): read TIME, BLOB, MULTISET and non-string-key map columns#724
JingsongLi merged 1 commit into
apache:mainfrom
jackylee-ch:fix/avro-reader-missing-types

Conversation

@jackylee-ch

Copy link
Copy Markdown
Contributor

Reading a table whose data files are Avro fails or loses data for three column
types that Java Paimon happily writes.

TIME, BLOB and MULTISET have no arm in the Avro reader's build_column,
so a query over such a column aborts with Avro reader does not support data type: ... even though the same column reads fine from Parquet, .row and
Mosaic. Java writes all three: AvroSchemaConverter maps TIME to an int with
the time-millis logical type (rejecting precision above 3), BLOB to bytes
alongside BINARY/VARBINARY, and MULTISET<T> to a map from the element to
an int count.

A map whose key is not a string is worse than a hard failure. Avro supports
string keys natively only, so Java encodes those as an array of {key, value}
records (AvroSchemaConverter#isArrayMap). build_map_column only matched
Value::Map and the fallback arm pushed an unchanged offset, so every such row
decoded to a non-null map with zero entries, silently dropping the data.

Fix: add the three missing type arms, and decode the array-map shape into
the same Arrow Map array. Entries missing either field are skipped rather than
shifting the offsets of later rows. MAP and MULTISET declare key nullability
differently in paimon_type_to_arrow, so the shared helper takes both flags
explicitly and the new tests go through build_target_arrow_schema to pin it.

VECTOR is still unsupported here: Java maps it to an Avro array, but its
Arrow type is FixedSizeList rather than List, so it needs its own builder.
Happy to follow up separately.

@JingsongLi JingsongLi left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

+!

@JingsongLi
JingsongLi merged commit dbb44c4 into apache:main Aug 20, 2026
13 checks passed
jerry-024 added a commit to jerry-024/paimon-rust that referenced this pull request Aug 21, 2026
* main:
  perf: vectorize raw vector search (apache#734)
  feat(file_index): add predicate evaluation foundation (apache#721)
  feat(go): add postpone fixed-bucket write bindings (apache#722)
  perf(vindex): split build timing logs by phase (apache#723)
  fix(avro): read TIME, BLOB, MULTISET and non-string-key map columns (apache#724)
  fix(datafusion): surface tag create-time and retention in $tags (apache#728)
  [core] Support multivalue global index (apache#731)
  feat: add Java-compatible array predicate pushdown (apache#732)
  fix: serialize unbounded varchar as string (apache#730)
  perf(vindex): decouple vector read threads and remove chunk barrier (apache#720)
  feat(vindex): add DiskANN and IVF-SQ/RQ support (apache#726)

# Conflicts:
#	crates/paimon/src/table/data_file_reader.rs
#	crates/paimon/src/table/vindex_index_build_builder.rs
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants