[Task]: Go SDK portable logical type support

What needs to happen?

Logical types defined in https://github.com/apache/beam/blob/master/model/pipeline/src/main/proto/org/apache/beam/model/pipeline/v1/schema.proto are not supported in Go SDK. This prevents Go SDK to use schema transforms in a meaningful way.

Currently, only PYTHON_CALLABLE appear to be supported,

beam/sdks/go/pkg/beam/transforms/xlang/python/external.go

Line 42 in 1318bfe

    
           beam.RegisterSchemaProviderWithURN(pcsType, &callableSourceProvider{}, pythonCallableUrn)

however, through a different mechanism. Likely we need to first implement a framework for portable logical types in Golang, then implement each of them

Issue Priority

Priority: 2 (default / most normal work should be filed as P2)

Issue Components

Some AI insights. Likely we need a series of changes to support it

Architectural Comparison: Go SDK vs. Python SDK

In Python (schemas.py), a LogicalType is a unified class containing:

The URN (urn())
The Language Type (language_type())
The Representation Wire Type (representation_type())
Bidirectional Converters: to_representation_type(val) and to_language_type(val)
Argument Specs: argument_type() and argument() for parameterized types

When RowCoder encodes/decodes rows in Python, it automatically invokes to_representation_type() / to_language_type().

In Go (logicaltypes.go), LogicalType is only a static struct containing goT, storageT, argT, argV—with no conversion methods attached. Custom encoding/decoding is offloaded to RegisterSchemaProviders or SchemaProvider instances (like callableSourceProvider in external.go:L79).

Key Gaps in Go SDK for Portable Logical Type Support

Gap 1: Lack of Bidirectional Conversion Methods in `schema.LogicalType`

Problem: schema.LogicalType in Go (logicaltypes.go) defines types (goT and storageT), but does not provide ToStorageValue(goValue) or ToGoValue(storageValue) functions.
Impact: Automatic conversion cannot occur during RowEncoder / RowDecoder execution. Every new logical type requires writing manual BuildEncoder and BuildDecoder closures using coder.EncodeStringUTF8 or reflection, rather than specifying how to map a Go value to its underlying storage representation type.

Gap 2: Fragmented Registries across Packages

Problem: Logical type metadata is split across three ununified registries:
1. schema.RegisterLogicalType: Registers goT <-> storageT in schema.Registry (logicaltypes.go:L36).
2. coder.RegisterSchemaProviders: Registers encoder/decoder functions in RowEncoderBuilder / RowDecoderBuilder (row.go:L43).
3. beam.RegisterSchemaProviderWithURN: Ties a URN to a SchemaProvider interface (schema.go:L87).
Impact: Adding a new logical type requires multiple calls across schema, coder, and beam packages rather than registering a single logical type object.

Gap 3: Missing Argument & Payload Support for Parameterized Types

Proto Specification (schema.proto:L152-L165):

message LogicalType {
  string urn = 1;
  bytes payload = 2;
  FieldType representation = 3;
  FieldType argument_type = 4;
  FieldValue argument = 5;
}

Problem: In schema.go:L580, argument handling is explicitly marked as unhandled:
```
// TODO(BEAM-9615): Handle type Arguments.
```
Neither reflectTypeToFieldType (encoding) nor fieldTypeToReflectType (decoding) sets or parses argument_type or argument.
Impact: Parameterized logical types defined in Beam standard schemas—such as TIMESTAMP (precision arg), DECIMAL (precision/scale arg), FIXED_BYTES (length arg), and VAR_CHAR (max_length arg)—cannot be serialized or deserialized with their parameters.

Gap 4: Local Identifier Keying vs. Standard URN Lookup

Problem: Go SDK uses identifier as the map key in registry.logicalTypes[identifier].
- For Go's primitive type overrides (int, uint16), identifier is "int", "uint16" (logicaltypes.go:L166-L171).
- For time.Time, identifier is "time.Time" (encoding.go:L52).
Impact: When decoding a row schema sent from a cross-language runner or Python/Java SDK containing standard URNs (e.g. beam:logical_type:timestamp:v1 or beam:logical_type:date:v1), fieldTypeToReflectType in schema.go:L796 attempts r.logicalTypes["beam:logical_type:timestamp:v1"] and fails with:
```
unknown logical type: beam:logical_type:timestamp:v1
```

Gap 5: Non-Standard Wire Representations for Standard Types (e.g., `time.Time`)

Problem: In encoding.go:L52-L53, L274, Go's time.Time is registered using a custom Go-specific logical type ("time.Time") that encodes time using t.MarshalText() (string format) inside a wrapper struct struct{ EncodedBeamData string }.
Impact: Standard Beam portable timestamps expect either beam:logical_type:micros_instant:v1 (ROW<seconds: INT64, micros: INT64>), beam:logical_type:millis_instant:v1 (INT64), or beam:logical_type:timestamp:v1. Go's time.Time cannot interoperate out of the box with cross-language rows containing standard Beam timestamps.

Gap 6: `FieldValue` <-> `reflect.Value` Conversion Framework Missing

Problem: FieldValue protos in schema.proto are used to encode default values, option values, and logical type arguments. Go SDK only supports basic toggle options (optGoNillableUrn, optGoEmbeddedUrn) and string options (optGoLogicalUrn) (schema.go:L408-L500).
Impact: Arguments or payloads stored inside pipepb.FieldValue cannot be automatically converted to Go reflect.Value instances.

Summary of Requirements for a Full Portable Logical Type Framework in Go

To bring Go SDK to parity with Python/Java for portable logical types, the following framework enhancements are needed:

Extend LogicalType interface/struct in logicaltypes.go to include ToStorageValue(goVal) and ToGoValue(storageVal) conversion hooks, allowing RowEncoder / RowDecoder to automatically convert values without requiring custom SchemaProvider boilerplate.
Support argument_type and argument (pipepb.FieldValue) in schema.go when translating LogicalType to/from pipepb.FieldType.
Register standard Beam URNs (beam:logical_type:timestamp:v1, beam:logical_type:date:v1, beam:logical_type:decimal:v1, etc.) by default in schema.defaultRegistry.
Map time.Time to beam:logical_type:micros_instant:v1 or timestamp:v1 (or provide standard aliases) for cross-language compatibility.

.take-issue

Update Beam website roadmap pages (#39899)

.take-issue

Hi Yi, I went through the Go schema code against the six gaps above and they all check out on current master. Two extra findings on the way: the Go row coder encodes INT16 as a varint while Java and Python use 2 byte big endian, and the Go standard coder regression test skipped every logical type row case with one "logical" filter, so the 7 yaml cases can serve as acceptance tests.

I split the work into a series of PRs, each one commit stacked on the previous one:

[Go SDK] Encode INT16 row fields as big endian to match Java and Python #40152 fixes the INT16 row encoding. This is the separate bug [Bug]: Go SDK row coder encodes INT16 schema fields as varint instead of big endian #40151, and timestamp:v1 below precision 5 needs it.
[Go SDK] Convert logical type arguments to and from schema protos #40153 plumbs argument_type and argument through the schema proto conversion and keys the registry by URN and argument.
[Go SDK] Add portable logical type registration and standard date and micros_instant types #40154 adds a single registration call for portable logical types, schema.RegisterLogicalTypeConversion with its parameterized and coder variants, and the standard date and micros_instant types.
[Go SDK] Add the standard millis_instant and decimal logical types #40155 adds millis_instant and decimal, whose wire encodings differ from their representation types.
[Go SDK] Add pass through registration for the fixed and variable length string and bytes logical types #40156 maps fixed_char, var_char, fixed_bytes and var_bytes to string and []byte as pass through types.
[Go SDK] Add the standard timestamp logical type with millisecond, microsecond and nanosecond precision #40157 adds timestamp:v1 as TimestampMillis, TimestampMicros and TimestampNanos.
[Go SDK] Document the standard logical types and the registration API #40158 updates the programming guide.

With the last one, every logical type row case in standard_coders.yaml runs in the Go regression test.

Two decisions I made that are easy to change if you prefer otherwise. First, time.Time keeps its current Go specific encoding and the standard types are separate Go types, so nothing existing breaks. Second, because the Go row coder is built from the Go type alone, each URN and argument value maps to one named Go type, so timestamp:v1 covers precisions 3, 6 and 9, and the string and bytes types do not enforce their length argument.

If the other assignees have work in progress on any part of this, I am happy to rebase or drop the overlapping PR.

Update: #40155 and #40156 are open again. #40157 and #40158 stay closed for now to keep the review load small, and their branches stay in my fork.

[Go SDK] Document the standard logical types and the registration API

[Go SDK] Encode INT16 row fields as big endian to match Java and Python

[Go SDK] Convert logical type arguments to and from schema protos

[Go SDK] Add portable logical type registration and standard date and…

[Go SDK] Add the standard millis_instant and decimal logical types

[Go SDK] Add pass through registration for the fixed and variable len…

[Go SDK] Document the standard logical types and the registration API

[Go SDK] Add portable logical type registration and standard date and…

[Go SDK] Encode INT16 row fields as big endian to match Java and Pyth…

[Go SDK] Convert logical type arguments to and from schema protos (#4…

[Go SDK] Add portable logical type registration and standard date and…

Navigation Menu

Repository navigation

[Task]: Go SDK portable logical type support #39684

What needs to happen?

Issue Priority

Issue Components

Architectural Comparison: Go SDK vs. Python SDK

Key Gaps in Go SDK for Portable Logical Type Support

Gap 1: Lack of Bidirectional Conversion Methods in `schema.LogicalType`

Gap 2: Fragmented Registries across Packages

Gap 3: Missing Argument & Payload Support for Parameterized Types

Gap 4: Local Identifier Keying vs. Standard URN Lookup

Gap 5: Non-Standard Wire Representations for Standard Types (e.g., `time.Time`)

Gap 6: `FieldValue` <-> `reflect.Value` Conversion Framework Missing

Summary of Requirements for a Full Portable Logical Type Framework in Go

Metadata

Assignees

Labels

Type

Projects

Milestone

Relationships

Development

[Task]: Go SDK portable logical type support #39684

Description

What needs to happen?

Issue Priority

Issue Components

Activity

Abacn commented on Aug 7, 2026

Architectural Comparison: Go SDK vs. Python SDK

Key Gaps in Go SDK for Portable Logical Type Support

Gap 1: Lack of Bidirectional Conversion Methods in schema.LogicalType

Gap 2: Fragmented Registries across Packages

Gap 3: Missing Argument & Payload Support for Parameterized Types

Gap 4: Local Identifier Keying vs. Standard URN Lookup

Gap 5: Non-Standard Wire Representations for Standard Types (e.g., time.Time)

Gap 6: FieldValue <-> reflect.Value Conversion Framework Missing

Summary of Requirements for a Full Portable Logical Type Framework in Go

lalitium commented on Aug 7, 2026

dhruv-dagar commented on Sep 11, 2026

Eliaaazzz commented on Sep 17, 2026

Eliaaazzz commented on Sep 17, 2026

Metadata

Metadata

Assignees

Labels

Type

Projects

Milestone

Relationships

Development

Issue actions

Gap 1: Lack of Bidirectional Conversion Methods in `schema.LogicalType`

Gap 5: Non-Standard Wire Representations for Standard Types (e.g., `time.Time`)

Gap 6: `FieldValue` <-> `reflect.Value` Conversion Framework Missing