Repository navigation
Conversation
2 of 3 tasks
1 of 17 tasks
Contributor
Author
The reflection based row coder used the varint encoding for int16 and uint16 fields. Java and Python encode the INT16 schema type as 2 big endian bytes, so rows with such fields could not be exchanged with the other SDKs. Fixes apache#40151. Part of apache#39684.
Eliaaazzz
force-pushed
the
go-logical-types-passthrough
branch
from
October 7, 2026 22:53
1d937c9 to
311a8fe
Compare
Contributor
|
Assigning reviewers: R: @lostluck for label go. This pull request likely touches a core component ("core" label). Please review with scrutiny. Note: If you would like to opt out of this review, comment Available commands:
The PR bot will only process comments in the main thread (not review comments). |
Logical types registered with an argument now emit argument_type and argument in their FieldType, and schemas from other SDKs are matched on the URN and the argument value. A logical type registered without an argument still matches any argument, so pass through types keep working. Top level logical types carry the argument in a schema option. Part of apache#39684.
… micros_instant types schema.RegisterLogicalTypeConversion registers a logical type identified by a URN together with the conversions between its Go type and its storage type, and derives the row coder from the storage type. schema.RegisterLogicalTypeCoder does the same for logical types whose wire encoding differs from the row encoding of the storage type. The standard beam:logical_type:date:v1 and beam:logical_type:micros_instant:v1 logical types are registered by default as schema.Date and schema.MicrosInstant. The standard coder regression test now builds the expected row from the schema in the coder payload, decodes payload bytes above 0x7f, counts encoding mismatches as failures, and runs the micros_instant case. Part of apache#39684.
schema.MillisInstant encodes as the Java InstantCoder does: 8 big endian bytes of the milliseconds shifted by the minimum int64. schema.Decimal encodes as the Java BigDecimalCoder does: the scale as a varint followed by the length prefixed two's complement bytes of the unscaled value. Both cases of standard_coders.yaml now run in the Go regression test. Part of apache#39684.
…gth string and bytes logical types schema.RegisterPassThroughLogicalType maps a logical type URN to the Go type of its representation without mapping that Go type back, so a schema field with the URN decodes with the plain row encoding and the length argument is ignored. The standard fixed_char, var_char, fixed_bytes and var_bytes logical types are registered this way, and their standard_coders.yaml case now runs in the Go regression test. Part of apache#39684.
Eliaaazzz
force-pushed
the
go-logical-types-passthrough
branch
from
October 8, 2026 00:13
311a8fe to
6020d83
Compare
Eliaaazzz
marked this pull request as draft
October 8, 2026 01:18
2 of 3 tasks
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Part of #39684, the Go side of #23526. Depends on #40155 (the first four commits); review the last commit.
beam:logical_type:fixed_char:v1,var_char:v1,fixed_bytes:v1andvar_bytes:v1are STRING and BYTES on the wire; their INT32 argument is the fixed or maximum length. The Go row coder is built from the Go type alone, so a plainstringor[]bytefield is the natural Go representation, and it must keep producing an atomic STRING or BYTES schema field.schema.RegisterPassThroughLogicalType(urn, goType)maps a URN to the Go type of its representation without mapping that Go type back. A schema field with the URN decodes with the plain row encoding of the Go type, and the argument is ignored (the length is not enforced). Any argument matches because the arguments PR falls back to the registration without an argument.stringand[]byte, so a JDBC or SQL row from Java with CHAR, VARCHAR, BINARY or VARBINARY columns maps tostringand[]bytefields (nullable columns map to pointers).Tests:
ToTypeof the yaml schema gives*stringand*[]bytefields, decoding the yaml example with nulls, and plainstringand[]bytefields still produce atomic schema fields.Thank you for your contribution! Follow this checklist to help us incorporate your contribution quickly and easily:
addresses #123), if applicable. This will automatically add a link to the pull request in the issue. If you would like the issue to automatically close on merging the pull request, commentfixes #<ISSUE NUMBER>instead.CHANGES.mdwith noteworthy changes.See the Contributor Guide for more tips on how to make review process smoother.
To check the build health, please visit https://github.com/apache/beam/blob/master/.test-infra/BUILD_STATUS.md
GitHub Actions Tests Status (on master branch)
See CI.md for more information about GitHub Actions CI or the workflows README to see a list of phrases to trigger workflows.