Skip to content
This repository was archived by the owner on Jul 4, 2026. It is now read-only.

fix(snapshot): bump FetchSnapshotConfig read cap from 1 MiB to 64 MiB - #3

Merged
tonicmuroq merged 1 commit into
mainfrom
fix/snapshot-config-limit
May 6, 2026
Merged

tonicmuroq merged 1 commit into
mainfrom
fix/snapshot-config-limit

Conversation

@tonicmuroq

Copy link
Copy Markdown
Contributor

Fixes #2.

Summary

pull.go:159 reads the snapshot config blob via io.ReadAll(io.LimitReader(body, 1<<20)) with the comment "config blob is tiny", but the SnapshotConfig embeds per-file SparseMap fields. A 4 GiB Windows guest running an Electron app with a live Firebase WebSocket signed in produces sparse maps in the high-hundred-KB range per fragmented file, and the config blob easily exceeds 1 MiB once memory-ranges and overlay.qcow2 are both accounted for.

Observed live:

field bytes
files["memory-ranges"].sparseMap 580,107
files["overlay.qcow2"].sparseMap 611,926
base config fields ~few KB
total config blob 1,367,436

Symptom on the wake side:

ERR wake default/<vm> error="pull hibernation snapshot <name>:
    stream snapshot:
    fetch snapshot config:
    parse snapshot config: unexpected end of JSON input"

io.LimitReader returns EOF after 1,048,576 bytes; io.ReadAll swallows the EOF; the caller hands the truncated buffer to json.Unmarshal, which fails on the unterminated structure.

Fix

  • Raise the cap to 64 MiB (still bounded — defends against a malicious manifest pointing at an arbitrarily large config blob).
  • Pre-check desc.Size so we reject oversize blobs before any byte is read.
  • Add a streaming overflow guard for the case where the descriptor lies about Size.

The reader path is the only thing that changes; pushers and on-disk format are untouched.

Relationship to cocoonstack/cocoon#23

Independent of #23. #23 addressed the writer-side tar PAX cap (a single file's sparse map > 1 MiB couldn't be encoded). This is the reader-side config cap (aggregate sparse maps + config metadata > 1 MiB couldn't be parsed). Both bugs share the same triggering condition (live Firebase agent → fragmented memory) but live in different layers.

Test plan

  • TestFetchSnapshotConfigOverOneMiB — builds a real ~1.4 MB SnapshotConfig with two fragmented sparse maps and asserts parse succeeds. Would have failed under the old 1 MiB cap with "unexpected end of JSON input".
  • TestFetchSnapshotConfigRejectsOversizeDescriptor — descriptor advertising > 64 MiB is rejected before any read.
  • go test ./snapshot/ — all existing tests still pass alongside the new ones.

🤖 Generated with Claude Code

The previous cap was set with the comment "config blob is tiny", but
the SnapshotConfig embeds a per-file SparseMap for every file in the
snapshot. A 4 GiB Windows VM whose guest has many small live
allocations (e.g. an Electron app with a Firebase WebSocket signed in)
produces sparse maps in the high-hundred-KB range per fragmented file,
and the config blob easily exceeds 1 MiB once memory-ranges and
overlay.qcow2 are both accounted for. Observed live: 580 KB +
612 KB + base fields = 1.37 MB total config blob.

Wake-side symptom was

    pull hibernation snapshot ...:
        stream snapshot:
        fetch snapshot config:
        parse snapshot config: unexpected end of JSON input

— `io.ReadAll(io.LimitReader(body, 1<<20))` truncated the JSON exactly
at 1 MiB, leaving an unterminated structure.

Bump the cap to 64 MiB (still defends against pathological / malicious
manifests) and add a descriptor.Size pre-check so we reject oversize
blobs before reading. Add a streaming overflow guard for the case
where the descriptor lies about Size.

Tests:
  - TestFetchSnapshotConfigOverOneMiB: builds a real ~1.4 MB
    SnapshotConfig with two fragmented sparse maps and asserts parse
    succeeds (would have failed under the old 1 MiB cap).
  - TestFetchSnapshotConfigRejectsOversizeDescriptor: rejects a
    descriptor advertising > 64 MiB up front.

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
@tonicmuroq
tonicmuroq force-pushed the fix/snapshot-config-limit branch from dc0745d to 0315e8a Compare May 6, 2026 02:17
@tonicmuroq
tonicmuroq merged commit d94f215 into main May 6, 2026
2 checks passed
@tonicmuroq
tonicmuroq deleted the fix/snapshot-config-limit branch May 6, 2026 02:19
tonicmuroq added a commit to cocoonstack/vk-cocoon that referenced this pull request May 6, 2026
Picks up tombstack/epoch#3, which raises the snapshot-config blob
read cap from 1 MiB to 64 MiB. Wake-side parse failed with
"unexpected end of JSON input" when the SnapshotConfig — including
per-file SparseMap fields for memory-ranges and overlay.qcow2 — grew
past 1 MiB, which a 4 GiB Windows guest with a Firebase-active agent
hits routinely.

No vk-cocoon source changes; just the go.mod / go.sum bump and a
rebuild. Existing tests pass.

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
tonicmuroq added a commit to cocoonstack/vk-cocoon that referenced this pull request May 6, 2026
Picks up tombstack/epoch#3, which raises the snapshot-config blob
read cap from 1 MiB to 64 MiB. Wake-side parse failed with
"unexpected end of JSON input" when the SnapshotConfig — including
per-file SparseMap fields for memory-ranges and overlay.qcow2 — grew
past 1 MiB, which a 4 GiB Windows guest with a Firebase-active agent
hits routinely.

No vk-cocoon source changes; just the go.mod / go.sum bump and a
rebuild. Existing tests pass.
tonicmuroq added a commit to cocoonstack/vk-cocoon that referenced this pull request May 6, 2026
Picks up tombstack/epoch#3, which raises the snapshot-config blob
read cap from 1 MiB to 64 MiB. Wake-side parse failed with
"unexpected end of JSON input" when the SnapshotConfig — including
per-file SparseMap fields for memory-ranges and overlay.qcow2 — grew
past 1 MiB, which a 4 GiB Windows guest with a Firebase-active agent
hits routinely.

No vk-cocoon source changes; just the go.mod / go.sum bump and a
rebuild. Existing tests pass.
CMGS pushed a commit to cocoonstack/vk-cocoon that referenced this pull request May 6, 2026
Picks up tombstack/epoch#3, which raises the snapshot-config blob
read cap from 1 MiB to 64 MiB. Wake-side parse failed with
"unexpected end of JSON input" when the SnapshotConfig — including
per-file SparseMap fields for memory-ranges and overlay.qcow2 — grew
past 1 MiB, which a 4 GiB Windows guest with a Firebase-active agent
hits routinely.

No vk-cocoon source changes; just the go.mod / go.sum bump and a
rebuild. Existing tests pass.
CMGS pushed a commit to cocoonstack/vk-cocoon that referenced this pull request May 6, 2026
Picks up tombstack/epoch#3, which raises the snapshot-config blob
read cap from 1 MiB to 64 MiB. Wake-side parse failed with
"unexpected end of JSON input" when the SnapshotConfig — including
per-file SparseMap fields for memory-ranges and overlay.qcow2 — grew
past 1 MiB, which a 4 GiB Windows guest with a Firebase-active agent
hits routinely.

No vk-cocoon source changes; just the go.mod / go.sum bump and a
rebuild. Existing tests pass.
CMGS added a commit that referenced this pull request Jun 28, 2026
Pin the merged cocoon-common main (cross-node migration API surface,
PR #3) via pseudo-version until v0.2.2 is tagged.
CMGS added a commit that referenced this pull request Jun 28, 2026
Pin the merged cocoon-common main (cross-node migration API surface,
PR #3) via pseudo-version until v0.2.2 is tagged.
CMGS added a commit that referenced this pull request Jun 28, 2026
* feat(server): redirect blob GETs to presigned object-store URLs

v2GetBlob proxies every blob byte through the epoch process: it reads
from the object store and io.Copy's the body back to the client, paying
TLS on both sides for multi-GiB VM disk/memory blobs. A single stream
tops out at ~20 MB/s even with 3 CPUs, so a 10.5 GB snapshot pull takes
~9 min, and concurrent transfers saturate the pod.

When EPOCH_BLOB_REDIRECT is set, v2GetBlob now responds with a 307 to a
presigned object-store URL (minio-go PresignedGetObject), so blob bytes
flow client<->storage directly and epoch leaves the data path entirely.
GCS honors the SigV4-signed URL and serves Range requests natively.

Measured on a same-region GCE VM pulling the 5.7 GB memory-ranges blob:

  path                         single-stream    10.5 GB pull
  proxy (1 CPU, before)        3.4 MB/s         ~51 min
  proxy (3 CPU)                20 MB/s          ~9 min
  presigned direct-to-GCS      275 MB/s         ~38 s

~13x over the proxy with zero client changes — any redirect-following
OCI client (registryclient/vk-cocoon, oras, crane, docker) benefits
transparently, and CopyBlobExact still verifies the digest end to end.
Intra-blob Range parallelism (measured 1.28 GB/s at 8 connections) would
need a client-side downloader and is out of scope here.

The flag defaults off. On any server-side failure (existence check or
presign) the handler falls back to streaming. Note there is no fallback
once the 307 is sent: every client hitting /v2/blobs must have egress to
the object-store host when the flag is enabled.

* chore(deploy): enable blob redirect and right-size epoch-server

Turn on EPOCH_BLOB_REDIRECT so pulls bypass the proxy. Bump resources
from 1 CPU / 512Mi to 3 CPU / 4Gi: pushes still stream through epoch
(this PR only redirects GETs) and the redirect fallback path also
proxies, so the pod still needs headroom for multi-GiB transfers.

* chore(deps): bump cocoon-common to v0.2.2 (merged #3)

Pin the merged cocoon-common main (cross-node migration API surface,
PR #3) via pseudo-version until v0.2.2 is tagged.

* fix(server): /code review hardening of blob redirect

- clamp EPOCH_BLOB_REDIRECT_TTL to [1s, 7d]: an out-of-range value fails
  every presign and silently degrades to proxying, so fall back to the
  default below 1s (the minio presign floor) and clamp above 7d
- read the redirect env via commonk8s.EnvBool / EnvDuration, dropping the
  hand-rolled strconv/ParseDuration to match the package convention
- mark the blobRedirectTTL config field; note why redirectBlob HEADs
  before presigning; document the EPOCH_BLOB_REDIRECT_TTL knob

---------

Co-authored-by: CMGS <ilskdw@gmail.com>
Sign up for free to subscribe to this conversation on GitHub. Already have an account? Sign in.

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

FetchSnapshotConfig truncates config blob > 1 MiB → wake fails on highly fragmented snapshots

1 participant