[spark] Cover the partition operations left over on format tables - #9119
Open
sundapeng wants to merge 1 commit into
Open
[spark] Cover the partition operations left over on format tables#9119sundapeng wants to merge 1 commit into
sundapeng wants to merge 1 commit into
Conversation
TRUNCATE TABLE, TRUNCATE TABLE PARTITION and ALTER TABLE ... RENAME TO PARTITION all refuse a format table with catalog-managed partitions, and these tests pin that they refuse without moving anything: a registration pointing at a directory that is gone, or data under a spec nobody registered, would both be worse than the refusal. The refusal TRUNCATE PARTITION produced described the table wrongly. "Only FileStoreTable supports partitions" is reached by a table that has partitions and lists them through this very trait, which sends the reader looking for the wrong problem; it now says which operations do manage those partitions. A null partition value registers under the default partition name, keeps a directory of that name, reads back as null, is discovered by MSCK REPAIR and is dropped by that name. SHOW PARTITIONS prints it as dt=null rather than the default name, which is what Spark's own ShowPartitionsExec does with a null value; a native Paimon table prints the same thing, so the test records where that behaviour comes from instead of working around it here. The remaining probes cover ADD/DROP PARTITION idempotence, prefix DROP, partial specs in SHOW PARTITIONS, case-insensitive partition column names and empty string partition values, all of which already behaved this way.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Purpose
TRUNCATE TABLE,TRUNCATE TABLE PARTITIONandALTER TABLE ... RENAME TO PARTITIONall refuse aformat table with catalog-managed partitions. These tests pin that they refuse without moving
anything: a registration pointing at a directory that is gone, or data under a spec nobody
registered, would both be worse than the refusal.
The one behaviour change is an error message.
Only FileStoreTable supports partitionsis reached bya table that has partitions and lists them through this very trait, which sends the reader looking
for the wrong problem. It now names the operations that do manage those partitions.
The remaining probes cover
ADD/DROP PARTITIONidempotence, prefixDROP, partial specs inSHOW PARTITIONS, case-insensitive partition column names, null partition values and empty stringpartition values — all of which already behaved this way and are now held in place.
Tests
New:
CatalogManagedPartitionEdgeParityTest,CatalogManagedPartitionDdlParityTest.The probes are mapped from Spark's own
TruncateTableSuiteBase,AlterTableRenamePartitionSuiteBase,AlterTableAddPartitionSuiteBase,AlterTableDropPartitionSuiteBaseandShowPartitionsSuiteBase, so the reference for "what shouldhappen" is Spark's rather than ours.
One finding recorded rather than worked around:
SHOW PARTITIONSprints a null partition value asdt=nullrather than the default partition name. That is Spark's ownShowPartitionsExecwith a v2table, and a native Paimon partitioned table prints the same thing — so a test pins where the
behaviour comes from instead of special-casing it here.
API and Format
No public API change. No format change.
Documentation
None needed; the only user-visible change is an error message.