Stop writing fileblob's .attrs sidecar by montehurd · Pull Request #328 · git-pkgs/proxy

montehurd · 2026-09-08T21:28:04Z

fileblob keeps per-object metadata in an .attrs sidecar and rewrites it with os.Create, truncating in place outside the atomic rename that protects the blob. A read overlapping a write to one key decodes a partial file and fails with opening reader: EOF, served as a 502. One writer against four readers on a single key failed 408 of 2000 reads.

Nothing in the proxy reads what the sidecar holds (gocloud.dev/blob is imported only by internal/storage, Store sets no ContentType, and the only Attributes field read is Size), and getAttrs already defaults a missing one, so metadata=skip removes the hazard instead of locking around it. That is the first commit, one line.

The second is separable. metadata=skip stops rewriting sidecars without deleting them, so one already left partial by an interrupted write would be stuck for good, where a later store used to repair it. Store therefore clears the sidecar for the key it writes. Deriving that path uses filepath.Localize, which also keeps the removal inside the cache directory. Drop this commit in favour of documenting a find <cache> -name '*.attrs' -delete sweep if that is preferred.

Each commit carries the tests that prove it, and each fails without its own change. Green on ubuntu, macOS and Windows.

fileblob stores blob metadata in an ".attrs" file per object and rewrites it with os.Create, truncating in place outside the atomic rename that protects the blob. A read overlapping a write decodes a partial file and fails with "opening reader: EOF", served as a 502. One writer against four readers on a single key failed 408 of 2000 reads. cacheMetadataBlob is most exposed to it, rewriting a key on every refresh while readers are served from it. Nothing in the proxy reads what the sidecar holds. gocloud.dev/blob is imported only by internal/storage, Store sets no ContentType, and Attributes is used only for Size, which comes from os.Stat. A missing sidecar already defaults cleanly, so "metadata=skip" removes the hazard rather than locking around it, and saves a write per store.

metadata=skip stops fileblob rewriting sidecars but does not delete ones already on disk, so a sidecar left partial by an interrupted write now fails every read of its key for good. Before, a later store repaired it by rewriting. Store therefore removes the sidecar for the key it writes. Removal is atomic where the rewrite was not, so a concurrent reader gets the whole old file or nothing. Delete already removes sidecars, so the two paths drain a cache between them. Deriving that path is necessary because fileblob's key escaping is unexported. It is the identity for a plain key and parts from one only for keys that are not valid local paths, which is what filepath.Localize rejects. That also keeps the removal inside the cache directory: without it a key holding ".." resolves outside. The clearing test runs one key per storage path the proxy builds, seeded through a bucket that still writes sidecars so the path under test is fileblob's own.

Copilot

🟡 Changes recommended

Address the Windows cleanup gap and propagate writer errors in the concurrency test.

Get a fresh assessment by requesting another Copilot review.

Pull request overview

This PR prevents .attrs sidecar corruption during concurrent fileblob access and cleans up legacy sidecars.

Changes:

Enables metadata=skip for local storage.
Removes legacy sidecars during writes.
Adds cleanup and concurrency regression tests.

File summaries

File	Summary
`internal/storage/blob.go`	Configures metadata skipping and sidecar cleanup. Critical (2 votes): Windows key localization can leave corrupted legacy sidecars in place.
`internal/storage/blob_test.go`	Tests sidecar behavior and concurrency. Moderate (2 votes): writer errors are ignored, allowing the test to pass without successful writes.

Review details

Suppressed comments (2)

internal/storage/blob.go:146

This unconditional removal can delete a real cached object: for a valid key x, the legacy sidecar path is the same filesystem path used by the object key x.attrs. Storing x after x.attrs has been cached removes x.attrs before writing the new blob, so subsequent reads of that unrelated object return not found. The cleanup needs a migration strategy that can distinguish a legacy sidecar from a legitimate object at the same path.

	if sidecar := b.legacySidecarPath(path); sidecar != "" {
		_ = os.Remove(sidecar)

internal/storage/blob.go:147

Ignoring os.Remove errors lets Store return success even when a stale sidecar could not be removed (for example, due to permissions or platform file-sharing rules). With metadata=skip, that sidecar remains the only metadata source and future reads can still fail with the same decode error. Please ignore only not-exist and propagate other removal errors, or otherwise make failed cleanup observable.

	if sidecar := b.legacySidecarPath(path); sidecar != "" {
		_ = os.Remove(sidecar)
	}

Files reviewed: 2/2 changed files
Comments generated: 2
Review effort level: Lite

💡 Add a code-review agent skill or configure MCP servers for context-aware, tailored reviews. Learn more in the docs.

montehurd · 2026-09-15T06:18:39Z

+	rel, err := filepath.Localize(key)
+	if err != nil {
+		return ""


Fixed in 623ff3e: the sidecar path is derived with fileblob's own escaping, so keys with a colon are cleared on Windows too. The table test now proves that for OCI and Debian keys on the Windows job.

montehurd · 2026-09-15T06:18:41Z

+			if _, _, err := b.Store(ctx, key, strings.NewReader(payload)); err != nil {
+				return
+			}


Fixed in a4be9e0. Reporting the error showed the writer had been dying at its first collision on Windows, where Go opens files without FILE_SHARE_DELETE, so the test now skips there with that reason.

Move the removal into clearLegacySidecar and say why its error is dropped rather than returned. A failed removal leaves exactly the state this change inherited, while failing the write would turn a cleanup miss into a failed request. Windows makes that concrete: Go opens files with FILE_SHARE_READ and FILE_SHARE_WRITE but not FILE_SHARE_DELETE, so a reader holding the sidecar open blocks deletion, and that reader is the workload this change exists to protect. Propagating would fail stores during exactly the overlap being fixed. The next store of the key retries. A test pins it, using a non-empty directory at the sidecar path to make os.Remove fail with something other than not-exist on any platform.

The writer returned silently when Store failed, so the test could pass with no concurrent writes at all. Its error is now reported, and the test also checks that at least one write completed. Reporting it showed the writer had been dying on Windows at its first collision: Go opens files without FILE_SHARE_DELETE, so a reader holding the file open makes the writer's rename fail with access denied. The test now skips there, since it cannot contend a writer with readers on that platform.

legacySidecarPath declined any key filepath.Localize rejects, which on Windows is every key with a colon: OCI digests and Debian epochs. Their sidecars were never cleared there, and a truncated one kept failing reads, since fileblob still reads a sidecar it finds under metadata=skip. fileblob hex-escapes such characters on the way to disk. The path is now derived the same way, so the sidecar is looked for where fileblob wrote it. Localize still validates the escaped form, which keeps the removal inside the cache directory.

montehurd · 2026-09-15T06:18:45Z

On the suppressed comments:

os.Remove errors: kept non-fatal, with the reasoning in the code and a test that Store succeeds when the sidecar cannot be removed (f4f74ce). Propagating anything but not-exist would fail the store whenever a Windows reader holds the sidecar open, since Go opens files without FILE_SHARE_DELETE, and that reader is the workload this change protects. The next store of the key retries.

"x.attrs" collision: fileblob rejects any key ending in ".attrs" on every operation, read or write (errAttrsExt: file extension ".attrs" is reserved), so no such object can exist in a fileblob bucket. The path the cleanup removes is one fileblob could never have written as an object.

623ff3e made legacySidecarPath escape keys the way fileblob does, so colon-bearing keys are now cleared on Windows and the OCI and Debian rows in TestStoreClearsLegacyAttrsSidecar prove it. The comment described the behaviour before that commit.

ghost

Applied 1f75f1b to drop the stale colon-key comment left over from before 623ff3e. Full suite passes locally. The earlier Windows failure was TestStartUsesConfiguredLoopbackUpstreams in internal/server timing out at 250ms — unrelated to this change; the push retriggers CI.

montehurd added 2 commits September 7, 2026 14:12

This was referenced Sep 8, 2026

Fix duplicate fetches and 502s on concurrent cache misses #329

Merged

Stop writing fileblob's .attrs sidecar montehurd/proxy#3

Closed

andrew requested review from andrew and a lite review from Copilot and removed request for andrew September 13, 2026 16:01

Copilot started reviewing on behalf of andrew September 13, 2026 16:02

View session

Copilot AI reviewed Sep 13, 2026

View reviewed changes

montehurd added 3 commits September 14, 2026 16:19

andrew approved these changes Sep 15, 2026

View reviewed changes

andrew merged commit 0089917 into git-pkgs:main Sep 15, 2026

andrew mentioned this pull request Sep 15, 2026

Separate OCI request timeout from readiness probes in loopback test #347

Merged

Navigation Menu

Repository navigation

Stop writing fileblob's .attrs sidecar - #328

Stop writing fileblob's .attrs sidecar#328
andrew merged 6 commits into
git-pkgs:mainfrom
montehurd:skip-fileblob-metadata-sidecar

montehurd commented Sep 8, 2026

Uh oh!

Copilot AI left a comment

Uh oh!

montehurd Sep 15, 2026

Uh oh!

montehurd Sep 15, 2026

Uh oh!

montehurd commented Sep 15, 2026

Uh oh!

ghost left a comment

Uh oh!

Reviewers

Uh oh!

Assignees

Labels

Projects

Milestone

Development

Uh oh!

3 participants

Conversation

montehurd commented Sep 8, 2026

Uh oh!

Copilot AI left a comment

Choose a reason for hiding this comment

🟡 Changes recommended

Suppressed comments (2)

Uh oh!

montehurd Sep 15, 2026

Choose a reason for hiding this comment

Uh oh!

montehurd Sep 15, 2026

Choose a reason for hiding this comment

Uh oh!

montehurd commented Sep 15, 2026

Uh oh!

ghost left a comment

Choose a reason for hiding this comment

Uh oh!

Reviewers

Uh oh!

Assignees

Labels

Projects

Milestone

Development

Uh oh!

3 participants