Skip to content

fix(datafusion): keep metadata queries available without table engines - #747

Open
shyjsarah wants to merge 2 commits into
apache:mainfrom
shyjsarah:fix/information-schema-unregistered-engine
Open

fix(datafusion): keep metadata queries available without table engines#747
shyjsarah wants to merge 2 commits into
apache:mainfrom
shyjsarah:fix/information-schema-unregistered-engine

Conversation

@shyjsarah

@shyjsarah shyjsarah commented Aug 26, 2026

Copy link
Copy Markdown
Contributor

Purpose

After external table routing was made fail-closed, DataFusion metadata queries such as information_schema.columns could fail if a catalog contained any external table whose engine was not registered. This also surfaced through statements that inspect catalog metadata.

Changes

  • preserve catalog field metadata when classifying external tables
  • return a metadata-only provider when the declared table engine is unavailable
  • expose the real table schema to DataFusion metadata queries
  • make table_exist() agree with the metadata-only provider for catalog tables without a registered engine
  • bypass read-specific session option validation only for metadata-only loading; registered-engine reads still reject unsupported time travel, incremental scan, and query-auth options
  • keep scans and writes fail-closed with the existing no table engine is registered error
  • cover SHOW TABLES, unrelated and external information_schema.columns queries, direct reads, destructive statements, and table_exist

Behavior

Operation Unregistered external table
SHOW TABLES succeeds and lists the table
information_schema.columns succeeds using catalog metadata
SessionContext::table_exist returns true for the catalog table
direct SELECT fails when the scan is planned
writes fail closed

Tests

  • cargo fmt --all -- --check
  • cargo +1.98.0 clippy --locked -p paimon -p paimon-datafusion --all-targets --features fulltext,vortex -- -D warnings
  • cargo test -p paimon-datafusion --test table_type_routing

@shyjsarah
shyjsarah force-pushed the fix/information-schema-unregistered-engine branch from 8075d43 to ce43b27 Compare August 27, 2026 02:51
@JingsongLi

Copy link
Copy Markdown
Contributor
  • When no engine is present, table() returns a metadata-only provider, yet table_exist() returns false, violating the existence semantics of the DataFusion SchemaProvider.
  • Consequently, the same external table can be queried via information_schema but appears non-existent when checked via SessionContext::table_exist.

Comment thread crates/integrations/datafusion/src/catalog.rs
@shyjsarah

Copy link
Copy Markdown
Contributor Author

Addressed in d3200f4: when no engine is registered, table_exist() now returns true to match the metadata-only provider returned by table(). The regression also verifies metadata loading under a session time-travel option.

@jerry-024 jerry-024 left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

+1

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants