feat(go): add postpone fixed-bucket write bindings - #722
Merged
JingsongLi merged 4 commits intoAug 20, 2026
Merged
Conversation
Expose the dedicated fixed-bucket write path for bucket = -2 tables through typed builder, writer, commit-message, and committer handles so they cannot be mixed with the standard write path. The caller supplies a resolved partition-to-bucket-count plan; Go performs no planning.
XiaoHongbo-Hope
marked this pull request as draft
August 17, 2026 05:49
Collapse the bucket-plan helper into an inline example and merge the constraint prose. Same coverage, fewer words. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Extend the example through WriteArrowBatch, PrepareCommit, NewCommit, and Commit so it is runnable as-is, and drop the now-redundant lifecycle reference. Per review feedback on apache#722. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
XiaoHongbo-Hope
marked this pull request as ready for review
August 17, 2026 07:22
XiaoHongbo-Hope
requested review from
JingsongLi
and
a balanced review from Copilot
August 18, 2026 11:40
There was a problem hiding this comment.
Pull request overview
Warning
Copilot couldn't run its full agentic review because it didn't start before the timeout. Make sure your repository has a runner available, or add a copilot-code-review.yml file specifying one with the runs-on attribute. See the docs for more details.
Adds Go bindings for the “postpone fixed-bucket” (bucket = -2) write path, including APIs, docs, and integration testing to write real buckets directly using a resolved partition→bucket-count plan.
Changes:
- Add new Go FFI bindings and typed Go wrappers for postpone fixed-bucket builder/write/commit/messages.
- Document how to provide and use a resolved bucket plan for bucket = -2 tables.
- Add Spark provisioning and Go integration tests for same-process multi-writer commits.
Reviewed changes
Copilot reviewed 6 out of 6 changed files in this pull request and generated 5 comments.
Show a summary per file
| File | Description |
|---|---|
| docs/src/go-binding.md | Adds usage docs and a Go example for postpone fixed-bucket writes with a bucket plan. |
| dev/spark/provision.py | Provisions a new bucket = -2 PK table for Go fixed-bucket write tests. |
| bindings/go/types.go | Introduces new FFI handle/result types for the postpone fixed-bucket path. |
| bindings/go/tests/paimon_test.go | Adds compile-time type isolation test and a same-process multi-writer integration test. |
| bindings/go/postpone_fixed_bucket_write_ffi.go | Adds the libffi symbol bindings for postpone fixed-bucket builder/write/commit/messages. |
| bindings/go/postpone_fixed_bucket_write.go | Adds the public Go API wrappers for postpone fixed-bucket writes and commits. |
💡 Add a code-review agent skill or configure MCP servers for context-aware, tailored reviews. Learn more in the docs.
Comment on lines
+29
to
+45
| var ffiTableNewPostponeFixedBucketWriteBuilder = newFFI(ffiOpts{ | ||
| sym: "paimon_table_new_postpone_fixed_bucket_write_builder", | ||
| rType: &typeResultWriteBuilder, | ||
| aTypes: []*ffi.Type{&ffi.TypePointer}, | ||
| }, func( | ||
| ctx context.Context, | ||
| ffiCall ffiCall, | ||
| ) func(*paimonTable) (*paimonPostponeFixedBucketWriteBuilder, error) { | ||
| return func(table *paimonTable) (*paimonPostponeFixedBucketWriteBuilder, error) { | ||
| var result resultPostponeFixedBucketWriteBuilder | ||
| ffiCall(unsafe.Pointer(&result), unsafe.Pointer(&table)) | ||
| if result.error != nil { | ||
| return nil, parseError(ctx, result.error) | ||
| } | ||
| return result.writeBuilder, nil | ||
| } | ||
| }) |
Comment on lines
+213
to
+231
| var ffiPostponeFixedBucketTableWritePrepareCommit = newFFI(ffiOpts{ | ||
| sym: "paimon_postpone_fixed_bucket_table_write_prepare_commit", | ||
| rType: &typeResultPrepareCommit, | ||
| aTypes: []*ffi.Type{&ffi.TypePointer}, | ||
| }, func( | ||
| ctx context.Context, | ||
| ffiCall ffiCall, | ||
| ) func(*paimonPostponeFixedBucketTableWrite) (*paimonPostponeFixedBucketCommitMessages, error) { | ||
| return func( | ||
| write *paimonPostponeFixedBucketTableWrite, | ||
| ) (*paimonPostponeFixedBucketCommitMessages, error) { | ||
| var result resultPostponeFixedBucketPrepareCommit | ||
| ffiCall(unsafe.Pointer(&result), unsafe.Pointer(&write)) | ||
| if result.error != nil { | ||
| return nil, parseError(ctx, result.error) | ||
| } | ||
| return result.messages, nil | ||
| } | ||
| }) |
| if err != nil { | ||
| return nil, err | ||
| } | ||
| tw.lib.acquire() |
Comment on lines
+204
to
+217
| // Merge appends a copy of source's messages. Both handles must belong to the | ||
| // same process, and both builders must use the same table, commit user, and | ||
| // overwrite mode. | ||
| func (m *PostponeFixedBucketCommitMessages) Merge( | ||
| source *PostponeFixedBucketCommitMessages, | ||
| ) error { | ||
| if m.inner == nil { | ||
| return ErrClosed | ||
| } | ||
| if source == nil || source.inner == nil { | ||
| return ErrClosed | ||
| } | ||
| return ffiPostponeFixedBucketCommitMessagesMerge.symbol(m.ctx)(m.inner, source.inner) | ||
| } |
| if err := builder.WithBucketPlan(plan); err != nil { | ||
| log.Fatal(err) | ||
| } | ||
| writer, err := builder.NewWrite() // fails without a plan |
jerry-024
added a commit
to jerry-024/paimon-rust
that referenced
this pull request
Aug 21, 2026
* main: perf: vectorize raw vector search (apache#734) feat(file_index): add predicate evaluation foundation (apache#721) feat(go): add postpone fixed-bucket write bindings (apache#722) perf(vindex): split build timing logs by phase (apache#723) fix(avro): read TIME, BLOB, MULTISET and non-string-key map columns (apache#724) fix(datafusion): surface tag create-time and retention in $tags (apache#728) [core] Support multivalue global index (apache#731) feat: add Java-compatible array predicate pushdown (apache#732) fix: serialize unbounded varchar as string (apache#730) perf(vindex): decouple vector read threads and remove chunk barrier (apache#720) feat(vindex): add DiskANN and IVF-SQ/RQ support (apache#726) # Conflicts: # crates/paimon/src/table/data_file_reader.rs # crates/paimon/src/table/vindex_index_build_builder.rs
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Purpose
Expose the dedicated postpone fixed-bucket write path to Go.
Changes
Scope
This API writes real buckets for
bucket = -2tables. Planning, shuffling,preclustering, and cross-process commit-message serialization remain the
integration's responsibility. Writers sharing one commit must use the same plan
and commit user, with one owner per
(partition, bucket).This PR adds Go bindings for the Rust/C primitives introduced by #659 and does
not change the storage format.