fix(storage): do not retry permanent errors in async writer resume - #16340
fix(storage): do not retry permanent errors in async writer resume#16340v-pratap wants to merge 1 commit into
Conversation
There was a problem hiding this comment.
Code Review
This pull request introduces OpenTelemetry tracing and metrics support to the asynchronous OpenObject class, adding a new OpenObjectTelemetry helper class and corresponding unit tests. It also updates the buffered and resumed writer connections to prevent resume attempts on permanent failures. The review feedback correctly identifies a build issue where GOOGLE_CLOUD_CPP_HAVE_OPENTELEMETRY is unconditionally defined as PUBLIC in CMake, which would break builds when OpenTelemetry is disabled. Additionally, the reviewer suggests correcting the header inclusion in open_object.cc to include the trace provider instead of the metrics provider, as the file only retrieves the current span and does not record metrics directly.
ecf19dc to
34f75fb
Compare
AsyncWriterConnectionResumed and AsyncWriterConnectionBuffered previously unconditionally invoked Resume() when stream operations returned an error, even if the error was a permanent failure such as FAILED_PRECONDITION. For appendable uploads (BidiWriteObject), issuing a new BidiWriteObject RPC on Resume() caused the client to request writer exclusivity again, resulting in concurrent writers endlessly stealing exclusivity back and forth from each other on FAILED_PRECONDITION errors. This change checks if the error status is a permanent failure (via AsyncRetryPolicy / AsyncStatusTraits) before attempting Resume(). If permanent, the upload immediately terminates with the error status.
34f75fb to
4af6792
Compare
Codecov Report❌ Patch coverage is
Additional details and impacted files@@ Coverage Diff @@
## main #16340 +/- ##
=======================================
Coverage 92.23% 92.23%
=======================================
Files 2227 2227
Lines 209283 209378 +95
=======================================
+ Hits 193026 193125 +99
+ Misses 16257 16253 -4 ☔ View full report in Codecov by Harness. 🚀 New features to boost your workflow:
|
Summary
This PR fixes a bug in Storage async writer connections (
AsyncWriterConnectionResumedandAsyncWriterConnectionBuffered) where permanent errors (such asFAILED_PRECONDITION) were being retried during streaming write operations.Cause
Previously, when a stream error occurred during
Write,Flush,Close,Finalize, orQuery,Resume(Status const& s)was called unconditionally without checking ifswas a permanent failure. For appendable uploads (BidiWriteObject), issuing a newBidiWriteObjectRPC onResume()requested writer exclusivity from the server, causing concurrent writers to continuously steal exclusivity back and forth from each other in an infinite loop upon receivingFAILED_PRECONDITION.Fix
Resume(Status const& s)inwriter_connection_resumed.ccandwriter_connection_buffered.ccto check ifsis a permanent failure (viaAsyncRetryPolicy/AsyncStatusTraits) before attemptingResume().sis a permanent failure (such asFAILED_PRECONDITION), the connection immediately terminates with statussviaSetError(...)without opening a new stream.WriteConnectionResumed.PermanentErrorNoResumeandWriteConnectionBuffered.PermanentErrorNoResumeto verify that permanent errors on active streams do not trigger resume attempts.