fix(secret-scan): exponential backoff and 2-minute retry cap for TruffleHog download - #214
Merged
Merged
Conversation
Fixed 5s retry delay x3 gave ~15s of retry coverage against GitHub's transient release-asset 500s. Drop --retry-delay so curl uses its default exponential backoff (1s, 2s, 4s, ...) and cap total retry time at 120s with --retry-max-time, applied to the release probe and all four TruffleHog asset downloads. Co-Authored-By: Ming Chen <ming.chen@sentry.io>
- Replace sigstore/cosign-installer with a pinned, checksummed curl using the same backoff, since the action's retries are hard-coded and one of this week's failures was its download 504ing. - Drop --retry-all-errors from the release probe so a not-yet-published release (404) falls through to the next tag immediately instead of retrying for two minutes. - Print effective URL and status on each download so the log shows which hop failed.
mchen-sentry
changed the title
Out of scope for the retry bump; noted the gap in the PR body instead.
mchen-sentry
changed the title
mchen-sentry
enabled auto-merge (squash)
September 10, 2026 22:43
kenzoengineer
approved these changes
Sep 10, 2026
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
2 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Examples
Secret Scan failing on the TruffleHog download, all
curl: (22) The requested URL returned error: 500four times in a row, then exit 22:Same class of blip on the cosign download (504, not covered by this PR):
None of these had a GitHub status-page incident at the time.
Context
The org-wide secret-scan check has failed multiple times with GitHub returning
500/504while downloading TruffleHog release assets — not caused by the PR under scan. Current curl calls use--retry 3 --retry-delay 5(or--retry 2 --retry-delay 3for the release probe), which is a fixed 5s delay between attempts and only ~15s of total retry coverage.Change
--retry-delayso curl falls back to its default exponential backoff (1s, 2s, 4s, ...).--retry-max-time 120to cap the total retry window at 2 minutes.--retrycount to 8 so the exponential schedule can reach the 2-minute cap..pem,.sig,.tar.gz).--retry-all-errorsoff the release probe so a not-yet-published release (404) falls through to the next tag immediately instead of retrying for two minutes.Why
GitHub's transient 500s on release asset downloads are usually short blips; a longer exponential-backoff window gives them time to clear without needing a manual job rerun, while still failing after two minutes of retrying.
Worst case during a real outage the job now spends up to ~12 minutes retrying (two probe tags plus four downloads, each capped at 2 minutes) before failing, versus ~1 minute today.
Not covered:
sigstore/cosign-installerdownloads cosign from GitHub Releases with its own hard-coded ~7s retry window, which is what failed on Sep 7. Left as-is; changing it is a separate call for the workflow owners.via Ming Chen.
--
View Junior Session [Sentry]