Skip to content

registrypush: push core for exporting cached images to remote registries - #348

Open
chruffins wants to merge 8 commits into
mainfrom
hypeship/registry-push-lib
Open

registrypush: push core for exporting cached images to remote registries#348
chruffins wants to merge 8 commits into
mainfrom
hypeship/registry-push-lib

Conversation

@chruffins

@chruffins chruffins commented Aug 5, 2026

Copy link
Copy Markdown

Layer 1 of remote registry push support (first layer of a stack).

What

Hypeman can pull from remote registries and accept pushes into its embedded registry, but has no outbound push path. This adds the registry-agnostic push core that later layers (manager integration, API endpoint) build on.

lib/ocicache (new)

  • ImageFromCache reconstructs a v1.Image from the OCI blob cache, extracted from lib/registry (previously the private imageFromBlobStore). Preserves the transparent Docker v2 → OCI conversion and adds a typed ErrNotFound for missing digests.
  • lib/registry now uses this package; the inbound push path is otherwise unchanged.

lib/registrypush (new)

  • Push / PushFromCache: write a cached image to any registry via remote.Write. Blobs are pushed as stored, so manifest digests are preserved end to end.
  • Provider interface for credential resolution — the modularity point for registry-specific auth (e.g. ECR token exchange in a later phase). Ships with:
    • KeychainProvider — docker config.json incl. credential helpers
    • StaticProvider — fixed credentials
    • Multi — per-host routing with default fallback
  • Errors classified via images.ClassifyRegistryError (exported from lib/images, renamed from wrapRegistryError) so rate limits and missing repos map to the existing typed errors.

Tests

  • Hermetic: in-process go-containerregistry registry as push destination, no external deps.
  • Covers: round-trip through the cache, digest preservation on push, Docker v2 → OCI conversion, bearer auth, rate-limit and not-found classification, keychain config resolution, per-host provider routing.

Verification

  • go build -tags containers_image_openpgp ./... — full tree builds
  • go test -tags containers_image_openpgp ./... — all packages pass, including the full lib/images suite (with mkfs.erofs installed) and lib/builds, except 42 VM-instance/integration tests that require Docker Hub pulls (anonymous rate limit) and instance networking. Verified these 42 fail identically on main (same test names, side-by-side run), so they are environmental, not caused by this change.
  • Wrote a test program utilizing the public functions to push an image to an AWS ECR repository
  • Pulled the pushed image from AWS ECR using hypeman

Note

Medium Risk
Touches registry error mapping, embedded-registry GC roots, and a large new cache read path; behavior changes are mostly additive but misclassification or GC regressions could break pulls/pushes.

Overview
Adds the outbound push foundation: read images from hypeman's OCI blob cache and write them to any remote registry, with pluggable credentials and shared error typing for API layers.

lib/ocicache extracts cache-backed v1.Image construction (formerly private registry code). It validates digest paths, converts Docker v2 manifests to OCI on read, and serves layers with proper gzip handling. lib/registry delegates layout append to this package and broadens in-memory tag tracking from BuildKit-only cacheTags to pushedTags for every accepted manifest so OCI GC keeps blobs still served by tag/digest pulls.

lib/registrypush implements Push / PushFromCache via remote.Write, plus Provider implementations (Docker keychain, static creds, per-host Multi). Push auth failures map to ErrUnauthorized; other registry errors use exported images.ClassifyRegistryError, which now prefers typed transport.Error status/codes so URLs or tag names (e.g. v404) cannot be misclassified as not-found.

ClassifyRegistryError replaces the private wrapRegistryError at pull/mirror/oci call sites, with expanded tests for transport-shaped errors.

Reviewed by Cursor Bugbot for commit 7471a0c. Bugbot is set up for automated code reviews on this repo. Configure here.

@chruffins chruffins changed the title hypeship/registry push lib registrypush: push core for exporting cached images to remote registries Aug 5, 2026
Comment thread lib/registrypush/push.go Outdated
Comment thread lib/images/errors.go
@chruffins

chruffins commented Aug 6, 2026

Copy link
Copy Markdown
Author

reviewed — clean extraction of the OCI cache reader into lib/ocicache with a tidy provider/push seam and solid tests. all bugs, nits, and structural items on this layer are fixed across 05de62e3 and a3b3d9e8 (typed push error classification, digest path validation, real Uncompressed/DiffID, parse-once v1.Manifest with field preservation, and GC-rooting of pushed manifests); one structural note remains open by choice.

Structural / Maintainability (open)

  • lib/registrypush/provider.goMulti has no consumers outside tests and its host-dispatch semantics are speculative; kept for now by decision, revisit if a real per-host routing requirement appears.

status: all review findings addressed except Multi, kept intentionally

@chruffins
chruffins marked this pull request as ready for review August 6, 2026 17:09
@chruffins
chruffins force-pushed the hypeship/registry-push-lib branch from 101c783 to 56e884f Compare August 6, 2026 17:09

@cursor cursor Bot left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Cursor Bugbot has reviewed your changes using high effort and found 1 potential issue.

Fix All in Cursor

❌ Bugbot Autofix is OFF. To automatically fix reported issues with cloud agents, enable autofix in the Cursor dashboard.

Reviewed by Cursor Bugbot for commit 56e884f. Configure here.

Comment thread lib/ocicache/image.go
@chruffins
chruffins requested a review from sjmiller609 August 6, 2026 20:09

@sjmiller609 sjmiller609 left a comment

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

lib/ocicache/image.go:216-230 — actual DockerForeignLayer media types are left unchanged inside the converted OCI manifest. map them to OCIRestrictedLayer and update the test to use DockerForeignLayer.
lib/registrypush/provider.go:29-34 — KeychainProvider drops the caller’s context. use authn.Resolve(ctx, kc, ref.Context()) so context-aware keychains receive cancellation and deadlines.

Move the blob-store-backed v1.Image reader out of lib/registry into a
new ocicache package so cached images can be reconstructed outside the
inbound push path. Export the registry error classifier as
ClassifyRegistryError for reuse.
Push core for exporting images from the local OCI cache to external
registries. Credentials are resolved through a Provider interface with
keychain, static, and per-host implementations; pushed blobs are
identical to the source so manifest digests are preserved. Registry
errors are classified into the typed hypeman errors.
The classifier substring-matches its input, so embedding the destination
reference in the error before classification let benign target text
(hosts, ports, tags containing 404 or 'not found') turn unrelated
transport failures into images.ErrNotFound.
…er contracts

Classify registry errors by typed transport fields (status/code) instead of
substring-matching the full error text, which embedded the request URL and
misclassified failures when host/port/tag/digest contained "404". Add a
push-side classifier so a 401/403 from the destination surfaces as
ErrUnauthorized instead of ErrNotFound (pull semantics unchanged).

Validate cache digests as 64 lowercase hex chars before using them as path
components, blocking traversal via crafted digests. Make cacheLayer
Uncompressed() actually decompress gzip layers and DiffID() return an error
instead of a lying zero hash. Fix the dead isOCIMediaType condition and
Size() duplication, wrap LayerByDigest errors in ErrNotFound, and share the
duplicated cache-writing test helper via lib/ocicache/testutil.
…shed manifests for GC

cacheImage now parses manifestData once (sync.Once) into gcr's v1.Manifest,
deleting the duplicate internalManifest struct and every v1.NewHash string
round-trip. OCI manifests are returned as-is; Docker v2 manifests are copied
and converted so annotations, subject, and foreign-layer descriptor urls
survive the rewrite instead of being dropped. v1.Hash unmarshal validates
hex/length, so manifest-supplied digests remain safe as cache path
components.

The embedded registry now roots every pushed manifest digest (not just
BuildKit cache tags) in LiveCacheManifestDigests: the Docker manifest blob a
push stores is served by the registry but unreachable from the OCI layout
(which holds the converted manifest under a different digest), so without
this root the OCI cache GC swept it and broke tag/digest pulls and
PushFromCache after MinBlobAge.

Push/PushFromCache docs now state the Docker-v2 conversion digest behavior
and the source-vs-target ErrNotFound phases.
The typed classifier returns a transport error with empty diagnostics (common
for HEAD or non-JSON bodies) unchanged, regressing the old substring match
that turned a 401's "unauthorized" message into ErrNotFound for pull
privacy. Status 401 now maps to ErrNotFound again; the push classifier
handles 401/403 as ErrUnauthorized before reaching this code, so push
semantics are unaffected.
blobHex accepted a bare 64-hex digest, but the cache image stored the caller
string unchanged, so Digest() (v1.NewHash) failed on bare-hex inputs after a
successful push. Store the canonical sha256: form so both inputs work.
@chruffins
chruffins force-pushed the hypeship/registry-push-lib branch from fbc8142 to a80989c Compare August 6, 2026 21:07
Map Docker foreign layers to the OCI restricted-layer media type during
Docker v2 conversion instead of leaking the Docker media type into the
converted manifest; the foreign-layer test now exercises the real
DockerForeignLayer type and asserts the mapping.

Forward the caller's context through KeychainProvider via authn.Resolve
so context-aware keychains receive cancellation and deadlines.
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants