Keep consumer packages small and publish independent runtime releases to R2

.changeset/smaller-markdown-and-vscode-packages.md

@@ -0,0 +1,19 @@
+---
+"@hyperbook/types": minor
+"@hyperbook/fs": minor
+"@hyperbook/markdown": minor
+"hyperbook": patch
+"hyperbook-studio": minor
+---
+
+Keep the Markdown npm package and the VS Code extension small by leaving out large element runtimes.
+
+The Markdown package no longer includes any of the eight large element runtimes; it keeps their integration scripts and styles. With `cdn: true`, Blockflow now loads from `blockflow.openpatch.org` and Excalidraw from its own packages on UNPKG. OpenSCAD, Online IDE and SQL IDE use independent, pinned releases on `cdn.openpatch.org`. Their owning repositories publish immutable runtime files to Cloudflare R2, and Hyperbook verifies the public CDN files before publishing a release.
+
+PyIDE now uses Pyodide's versioned jsDelivr CDN by default because its full local distribution is very large. Builds and asset prefetching skip downloading this runtime unless `elements.pyide.cdn` is explicitly `false`. Set it to `false` for self-hosted and offline books, and prefetch the runtime before an offline build. The documentation explains this default and how to cache runtime downloads in CI and assets on your hosting service.
+
+The VS Code extension no longer includes any of the eight large runtimes. Its preview resolves them like a Hyperbook build: elements with a `cdn` setting load from that CDN, and the others use the runtimes in the Hyperbook CLI's asset cache, which the extension now shares. Runtimes that are not downloaded yet load from their default CDN, and the preview offers to download them. The new commands "Download Element Runtimes..." and "Download All Element Runtimes" work like `hyperbook assets fetch` and `hyperbook assets fetch --all`.
+
+`@hyperbook/fs` now provides `AssetManager` and `assetCacheDir`, which the CLI and the VS Code extension share. `@hyperbook/types` exports `elementRuntimeFiles`, the files that show an element's runtime is available locally.
+
+The built Markdown ESM package also supports direct Node imports while remaining compatible with the CLI and extension bundles.

.github/workflows/changeset-version.yml

@@ -13,7 +13,11 @@ permissions:
   packages: write
 
 jobs:
+  publish-openscad-cdn:
+    uses: ./.github/workflows/publish-openscad-cdn.yml
+    secrets: inherit
   release:
+    needs: publish-openscad-cdn
     name: Release
     runs-on: ubuntu-latest
     steps:
@@ -43,6 +47,11 @@ jobs:
         run: |
           pnpm --filter "hyperbook-studio" vscode:package
 
+      - name: Verify default CDN runtimes before publishing
+        # CDN previews must work before any consumer package can be published.
+        # Validate all files against the locally built runtime bundles.
+        run: node scripts/verify-default-cdns.mjs
+
       - name: Publish CLI asset bundles before npm
         run: node packages/hyperbook/publish-assets.mjs
         env:

.github/workflows/publish-openscad-cdn.yml

@@ -0,0 +1,44 @@
+name: Publish OpenSCAD runtime to R2
+on:
+  workflow_dispatch:
+  workflow_call:
+permissions:
+  contents: read
+concurrency:
+  group: cdn-openscad
+  cancel-in-progress: false
+jobs:
+  publish:
+    runs-on: ubuntu-latest
+    timeout-minutes: 15
+    env:
+      CLOUDFLARE_API_TOKEN: ${{ secrets.CLOUDFLARE_R2_CDN }}
+      CLOUDFLARE_ACCOUNT_ID: ${{ vars.CLOUDFLARE_ACCOUNT_ID || secrets.CLOUDFLARE_ACCOUNT_ID }}
+      WRANGLER_SEND_METRICS: "false"
+    steps:
+      - uses: actions/checkout@v6
+      - uses: pnpm/action-setup@v6
+        with:
+          version: 11.3.0
+      - uses: actions/setup-node@v6
+        with:
+          node-version: "24"
+          cache: pnpm
+      - name: Install runtime downloader dependencies
+        run: pnpm install --frozen-lockfile --filter @hyperbook/markdown...
+      - run: pnpm test:cdn
+      - name: Install Wrangler
+        run: npm install --global wrangler@4.149.0
+      - name: Check Cloudflare configuration
+        run: |
+          test -n "$CLOUDFLARE_API_TOKEN" || { echo 'Missing CLOUDFLARE_R2_CDN Actions secret'; exit 1; }
+          test -n "$CLOUDFLARE_ACCOUNT_ID" || { echo 'Missing CLOUDFLARE_ACCOUNT_ID Actions variable or secret'; exit 1; }
+      - name: Prepare upstream WASM, fonts and libraries
+        run: node scripts/prepare-openscad-cdn.mjs
+      - name: Allow browser access to runtime files
+        run: wrangler r2 bucket cors set cdn --file scripts/cdn-cors.json --force
+      - name: Publish and verify immutable runtime files
+        run: |
+          version=$(node -p 'JSON.parse(require("fs").readFileSync("packages/markdown/openscad-config.json", "utf8")).wasmBuild.cdnVersion')
+          node scripts/publish-cdn.mjs openscad "$version" .cache/cdn-openscad/runtime
+          node scripts/verify-cdn.mjs openscad "$version" .cache/cdn-openscad/runtime

.github/workflows/pull-request.yml

@@ -24,3 +24,5 @@ jobs:
         run: pnpm build
       - name: Test packages
         run: pnpm test
+      - name: Test CDN publishing
+        run: pnpm test:cdn

docs/cdn-releases.md

@@ -0,0 +1,89 @@
+# Runtime releases on cdn.openpatch.org
+
+Cloudflare R2 bucket `cdn` serves immutable runtime releases through the custom
+domain `cdn.openpatch.org`. Each repository owns its release uploads:
+
+| Repository             | Object prefix                      | Trigger                                                       |
+| ---------------------- | ---------------------------------- | ------------------------------------------------------------- |
+| `openpatch/online-ide` | `onlineide/<release-tag>/include/` | Published GitHub release, after attaching `dist-embedded.zip` |
+| `openpatch/sql-ide`    | `sqlide/<release-tag>/include/`    | Published GitHub release, after attaching `dist-embedded.zip` |
+| `openpatch/hyperbook`  | `openscad/<cdnVersion>/`           | Hyperbook release workflow, before publishing packages        |
+
+All workflows use Wrangler and the Cloudflare API token in the organization
+Actions secret `CLOUDFLARE_R2_CDN`. Set `CLOUDFLARE_ACCOUNT_ID` as an organization
+Actions variable or secret. Grant both to the publishing repositories. S3 access
+keys are unnecessary. The token needs **Account → Workers R2 Storage → Edit**
+(the R2 token permission **Admin Read & Write**) for the account holding `cdn`.
+Cloudflare's bucket-scoped **Object Read & Write** permission supports only the
+S3 API, so it cannot authenticate these Wrangler operations. See
+[Cloudflare's permission documentation](https://developers.cloudflare.com/r2/api/tokens/#permissions).
+
+The IDE publishers download the ZIP already attached to the release and verify
+GitHub's SHA-256 digest. HTML examples and source maps are excluded. OpenSCAD is
+prepared directly from the upstream WASM distribution and the pinned font and
+library downloads in `packages/markdown/runtime-assets.json`, with SHA-256
+verification. It does not read a published Markdown package.
+
+Every object gets its correct MIME type and
+`Cache-Control: public, max-age=31536000, immutable`. The publisher retains old
+versions, refuses conflicting files, resumes interrupted uploads and writes
+`manifest.json` last. Every upload is followed by a public-domain check of all
+file checksums, MIME types, cache headers and CORS. A same-version rerun is safe.
+Repository concurrency groups prevent overlapping uploads of the same release;
+keep each prefix owned by its repository.
+
+The workflows apply `scripts/cdn-cors.json` to allow public GET/HEAD requests
+from any origin. This is needed for JavaScript modules, workers, WASM and fonts,
+including VS Code webview previews. There is no public upload permission.
+
+## First upload and retries
+
+In each IDE repository, open **Actions → Release Embedded Build → Run workflow**
+and enter an existing published tag in `cdn_release_tag`. Leaving that input
+empty retains the existing manual build/artifact behavior. The equivalent CLI
+commands are:
+
+```sh
+gh workflow run release-embedded.yml --repo openpatch/online-ide -f cdn_release_tag=v2.2.1-hyperbook.28
+gh workflow run release-embedded.yml --repo openpatch/sql-ide -f cdn_release_tag=v2.0.0-hyperbook.4
+gh workflow run publish-openscad-cdn.yml --repo openpatch/hyperbook
+```
+
+Hyperbook pins these default base URLs. You can also configure them explicitly:
+
+```json
+{
+  "elements": {
+    "onlineide": {
+      "cdn": "https://cdn.openpatch.org/onlineide/v2.2.1-hyperbook.28/"
+    },
+    "sqlide": { "cdn": "https://cdn.openpatch.org/sqlide/v2.0.0-hyperbook.4/" },
+    "openscad": { "cdn": "https://cdn.openpatch.org/openscad/2026.10.08-1/" }
+  }
+}
+```
+
+Keep `include/` below the IDE base URLs. Relative chunks, workers and other
+assets resolve there automatically. Pin a release version; changing a moving
+`latest` alias could break older books.
+
+## Updating OpenSCAD
+
+Update the upstream URL and SHA-256 in `openscad-config.json`, then increment
+`wasmBuild.cdnVersion`. Increment it when changing font or library downloads as
+well. Publish that version before using it as Hyperbook's default. The build
+continues to keep the small Hyperbook integration scripts local.
+
+Update IDE pins in `packages/markdown/ide-releases.json`; local bundle downloads
+and default CDN URLs use the same tags. Hyperbook's release workflow verifies
+every pinned CDN runtime against the local bundle payloads before publishing
+CLI bundles, npm packages or the VS Code extension. A missing or different CDN
+release prevents package publication.
+
+## Cloudflare caching
+
+Let the custom domain respect object Cache-Control headers. Do not cache 404
+responses: a version should not remain missing after its first upload. If the
+hostname already served cached responses before the CORS policy was applied,
+purge those URLs or the hostname once, following the
+[Cloudflare CORS guidance](https://developers.cloudflare.com/r2/buckets/cors/#use-cors-with-a-custom-domain).

package.json

@@ -10,6 +10,7 @@
     "build": "pnpm -r build",
     "build:vercel": "pnpm --filter=!hyperbook-studio build",
     "test": "CI=true pnpm -r test",
+    "test:cdn": "node --test scripts/publish-cdn.test.mjs",
     "lint": "pnpm -r lint",
     "hyperbook:build": "pnpm --filter hyperbook build",
     "version-packages": "changeset version",

packages/fs/package.json

@@ -45,6 +45,7 @@
     "mdast-util-from-markdown": "^2.0.3",
     "micromark-extension-directive": "^4.0.0",
     "mime-types": "^3.0.2",
+    "tar": "7.5.22",
     "unist-util-visit": "^5.1.0",
     "yaml": "^2.9.0"
   },