Repository navigation
sync with open source how - #118
Draft
lesterhaynes wants to merge 10000 commits into
Draft
lesterhaynes wants to merge 10000 commits into
lesterhaynes wants to merge 10000 commits into
Conversation
njeevan233-spec
approved these changes
Feb 1, 2026
Signed-off-by: Ilyas Ahsan <meilyasahsan@gmail.com>
…ounded pipelines (#40293) * [Python] Fix ParquetIO silently corrupting data across windows _RowDictionariesToArrowTable buffered rows from every window into a single flat buffer and overwrote self._window on each process() call. At finish_bundle() the whole mixed-window table was emitted in one WindowedValue carrying only the last window seen. In unbounded pipelines a bundle routinely spans windows (the convert_fn ParDo runs before the window-grouping GroupByKey in WriteImpl._expand_unbounded), so rows from earlier windows were silently attributed to the latest window: data loss for the earlier windows and corrupted downstream aggregations. Buffer record batches per window and emit one WindowedValue per window at finish_bundle() (and on mid-bundle row-group flushes), mirroring how _WriteWindowedBundleDoFn keys its writers by window. Pane info is intentionally not propagated: rows from many panes may land in one table, and the downstream file sink keys writers by window only. Adds regression tests driving the DoFn directly with interleaved windows. Fixes #40284 * style: apply Python formatter * Fix _ParquetSink file-handle leak across windows + add streaming e2e test The _RowDictionariesToArrowTable fix exposed a second bug further down the write path: _ParquetSink stored the open file handle on the sink (self._file_handle), but several writers may be open on one sink at once (one per window in streaming writes). The wrong handle got closed, producing Parquet files with missing footers ("Parquet magic bytes not found"). Make the file handle travel with its writer: open() now returns a (writer, file_handle) tuple, and write_record()/close() unpack it. Also adds the end-to-end regression test from review: one bundle spanning 3 windows must land each row in its own window's file. Co-authored-by: claudevdm (review feedback) * Fix formatting: remove extra blank line before new streaming test
Fix Playground Frontend cloud build
* [Python] Reset BigQuery retries for each batch * [Python] Link BigQuery retry fix in changelog * [Python] Move BigQuery retry fix to unreleased changelog
Bumps [golang.org/x/sys](https://github.com/golang/sys) from 0.48.0 to 0.49.0. - [Commits](golang/sys@v0.48.0...v0.49.0) --- updated-dependencies: - dependency-name: golang.org/x/sys dependency-version: 0.49.0 dependency-type: direct:production update-type: version-update:semver-minor ... Signed-off-by: dependabot[bot] <support@github.com> Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
Bumps [github/codeql-action](https://github.com/github/codeql-action) from 4.38.2 to 4.38.3. - [Release notes](https://github.com/github/codeql-action/releases) - [Changelog](https://github.com/github/codeql-action/blob/main/CHANGELOG.md) - [Commits](github/codeql-action@v4.38.2...v4.38.3) --- updated-dependencies: - dependency-name: github/codeql-action dependency-version: 4.38.3 dependency-type: direct:production update-type: version-update:semver-patch ... Signed-off-by: dependabot[bot] <support@github.com> Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
Bumps [google.golang.org/api](https://github.com/googleapis/google-api-go-client) from 0.300.0 to 0.301.0. - [Release notes](https://github.com/googleapis/google-api-go-client/releases) - [Changelog](https://github.com/googleapis/google-api-go-client/blob/main/CHANGES.md) - [Commits](googleapis/google-api-go-client@v0.300.0...v0.301.0) --- updated-dependencies: - dependency-name: google.golang.org/api dependency-version: 0.301.0 dependency-type: direct:production update-type: version-update:semver-minor ... Signed-off-by: dependabot[bot] <support@github.com> Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
Bumps [cloud.google.com/go/spanner](https://github.com/googleapis/google-cloud-go) from 1.95.1 to 1.96.1. - [Release notes](https://github.com/googleapis/google-cloud-go/releases) - [Changelog](https://github.com/googleapis/google-cloud-go/blob/main/CHANGES.md) - [Commits](googleapis/google-cloud-go@spanner/v1.95.1...spanner/v1.96.1) --- updated-dependencies: - dependency-name: cloud.google.com/go/spanner dependency-version: 1.96.1 dependency-type: direct:production update-type: version-update:semver-minor ... Signed-off-by: dependabot[bot] <support@github.com> Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
Bumps [golang.org/x/sync](https://github.com/golang/sync) from 0.23.0 to 0.24.0. - [Commits](golang/sync@v0.23.0...v0.24.0) --- updated-dependencies: - dependency-name: golang.org/x/sync dependency-version: 0.24.0 dependency-type: direct:production update-type: version-update:semver-minor ... Signed-off-by: dependabot[bot] <support@github.com> Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
…ks (#40522) Bumps [github.com/aws/aws-sdk-go-v2/config](https://github.com/aws/aws-sdk-go-v2) from 1.33.6 to 1.33.8. - [Release notes](https://github.com/aws/aws-sdk-go-v2/releases) - [Commits](aws/aws-sdk-go-v2@config/v1.33.6...config/v1.33.8) --- updated-dependencies: - dependency-name: github.com/aws/aws-sdk-go-v2/config dependency-version: 1.33.8 dependency-type: direct:production update-type: version-update:semver-patch ... Signed-off-by: dependabot[bot] <support@github.com> Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
…0153) * [Go SDK] Encode INT16 row fields as big endian to match Java and Python The reflection based row coder used the varint encoding for int16 and uint16 fields. Java and Python encode the INT16 schema type as 2 big endian bytes, so rows with such fields could not be exchanged with the other SDKs. Fixes #40151. Part of #39684. * [Go SDK] Convert logical type arguments to and from schema protos Logical types registered with an argument now emit argument_type and argument in their FieldType, and schemas from other SDKs are matched on the URN and the argument value. A logical type registered without an argument still matches any argument, so pass through types keep working. Top level logical types carry the argument in a schema option. Part of #39684. * Update CHANGES.md restore whitespace --------- Co-authored-by: Jack McCluskey <34928439+jrmccluskey@users.noreply.github.com>
* Fix BufferingDoFnRunner.checkpointCompleted() leveraging polymorphism in BufferedElement * Add a test for the fix * refine the logic introducing minTimestampPerBuffer and improve the test
* [Python] Return every group from Regex.find_all with Regex.ALL * Update sdks/python/apache_beam/transforms/util.py Co-authored-by: Danny McCormick <dannymccormick@google.com> * Test outputEmpty=False with two groups and with empty matches * Gate Regex.ALL outputEmpty on the full match and note the change in CHANGES.md * Update sdks/python/apache_beam/transforms/util.py Co-authored-by: Danny McCormick <dannymccormick@google.com> * Apply suggestion from @damccorm Co-authored-by: Danny McCormick <dannymccormick@google.com> * Drop the leftover outputEmpty line and fix the docstring indent * Update CHANGES.md Co-authored-by: Danny McCormick <dannymccormick@google.com> --------- Co-authored-by: Danny McCormick <dannymccormick@google.com>
* [Go SDK] Attribute a failed last data/timer send like Flush * Fix timer writer error handling in tests
* Adds an end-to-end example for migrating change data (CDC) from Delta Lake to GCP Lakehouse * updates * updates * Address reviewer comments
…rides (#40532) * Make --aws_* options defaults and --s3_* options S3 overrides * Reformat
* Add bigquery_to_bigtable yaml blueprint example and e2e test * update example * address reviewer's comment
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
20 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Please add a meaningful description for your change here
Thank you for your contribution! Follow this checklist to help us incorporate your contribution quickly and easily:
addresses #123), if applicable. This will automatically add a link to the pull request in the issue. If you would like the issue to automatically close on merging the pull request, commentfixes #<ISSUE NUMBER>instead.CHANGES.mdwith noteworthy changes.See the Contributor Guide for more tips on how to make review process smoother.
To check the build health, please visit https://github.com/apache/beam/blob/master/.test-infra/BUILD_STATUS.md
GitHub Actions Tests Status (on master branch)
See CI.md for more information about GitHub Actions CI.