Repository navigation
Conversation
2 of 3 tasks
1 of 17 tasks
Contributor
Author
The reflection based row coder used the varint encoding for int16 and uint16 fields. Java and Python encode the INT16 schema type as 2 big endian bytes, so rows with such fields could not be exchanged with the other SDKs. Fixes apache#40151. Part of apache#39684.
Eliaaazzz
force-pushed
the
go-logical-types-instants-decimal
branch
from
October 7, 2026 22:53
5672f19 to
31d5ef9
Compare
Contributor
|
Assigning reviewers: R: @shunping for label go. This pull request likely touches a core component ("core" label). Please review with scrutiny. Note: If you would like to opt out of this review, comment Available commands:
The PR bot will only process comments in the main thread (not review comments). |
Logical types registered with an argument now emit argument_type and argument in their FieldType, and schemas from other SDKs are matched on the URN and the argument value. A logical type registered without an argument still matches any argument, so pass through types keep working. Top level logical types carry the argument in a schema option. Part of apache#39684.
… micros_instant types schema.RegisterLogicalTypeConversion registers a logical type identified by a URN together with the conversions between its Go type and its storage type, and derives the row coder from the storage type. schema.RegisterLogicalTypeCoder does the same for logical types whose wire encoding differs from the row encoding of the storage type. The standard beam:logical_type:date:v1 and beam:logical_type:micros_instant:v1 logical types are registered by default as schema.Date and schema.MicrosInstant. The standard coder regression test now builds the expected row from the schema in the coder payload, decodes payload bytes above 0x7f, counts encoding mismatches as failures, and runs the micros_instant case. Part of apache#39684.
schema.MillisInstant encodes as the Java InstantCoder does: 8 big endian bytes of the milliseconds shifted by the minimum int64. schema.Decimal encodes as the Java BigDecimalCoder does: the scale as a varint followed by the length prefixed two's complement bytes of the unscaled value. Both cases of standard_coders.yaml now run in the Go regression test. Part of apache#39684.
Eliaaazzz
force-pushed
the
go-logical-types-instants-decimal
branch
from
October 8, 2026 00:13
31d5ef9 to
8fdbfd5
Compare
shunping
requested changes
Oct 8, 2026
shunping
left a comment
Collaborator
There was a problem hiding this comment.
Thanks for the contribution, @Eliaaazzz!
Since the prior PRs in this series are still being reviewed, I'd prefer to wait until those go through the pipeline first to avoid any cascading changes you might have to make here.
Eliaaazzz
marked this pull request as draft
October 8, 2026 01:18
Contributor
Author
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Part of #39684, and the Go side of #19817. Depends on #40154 (the first three commits); review the last commit.
Both types have a wire encoding that differs from the row encoding of their representation type, so they use
schema.RegisterLogicalTypeCoder:schema.MillisInstant(beam:logical_type:millis_instant:v1), atime.Timewith millisecond precision. The representation is INT64, but Java (InstantCoder) and Python (TimestampCoder, special cased inrow_coder.py) encode it as 8 big endian bytes shifted by the minimum int64, so the byte order matches the chronological order. The Go encoding uses the same shift ascoder.EncodeEventTime. Encoding a value with sub millisecond precision returns an error.schema.Decimal(beam:logical_type:decimal:v1), an arbitrary precision decimal withUnscaled *big.IntandScale int32. The representation is BYTES, encoded as Java'sBigDecimalCoderand Python'sDecimalCoderdo: the scale as an unsigned 32 bit varint (the INT32 varint form, so negative scales such as 1E+3 round trip with Java) followed by the length prefixed two's complement bytes of the unscaled value.ParseDecimalandStringconvert to and from plain notation.The
millis_instantanddecimalcases of standard_coders.yaml now run in the Go regression test.MillisInstantandDecimalare registered withruntime.RegisterTypelike the other standard types.Tests: byte level row encoding against the yaml examples for both types (including the negative decimal and the shifted millis value), two's complement conversion at the byte boundaries, decimal parsing and formatting, and the sub millisecond error.
Thank you for your contribution! Follow this checklist to help us incorporate your contribution quickly and easily:
addresses #123), if applicable. This will automatically add a link to the pull request in the issue. If you would like the issue to automatically close on merging the pull request, commentfixes #<ISSUE NUMBER>instead.CHANGES.mdwith noteworthy changes.See the Contributor Guide for more tips on how to make review process smoother.
To check the build health, please visit https://github.com/apache/beam/blob/master/.test-infra/BUILD_STATUS.md
GitHub Actions Tests Status (on master branch)
See CI.md for more information about GitHub Actions CI or the workflows README to see a list of phrases to trigger workflows.