From 7eedebc17c42ea4d27d643261698e5df5818acec Mon Sep 17 00:00:00 2001 From: Mike Barkmin Date: Sat, 10 Oct 2026 09:57:41 +0200 Subject: [PATCH 1/6] fix: keep Markdown and VS Code packages small with CDN fallbacks The Markdown npm package and the VS Code extension shipped every large element runtime and grew too big to publish. Markdown 0.85.0 never reached npm. - Markdown's npm package leaves out the Pyodide, GeoGebra, Typst, Blockflow and Excalidraw runtimes and keeps their integration files. Blockflow's default CDN is now blockflow.openpatch.org and Excalidraw's is its own packages on UNPKG, so every `cdn: true` default still resolves. OpenSCAD, Online IDE and SQL IDE stay, because their default CDN is this package on UNPKG. A test packs a fixture with npm and enforces this split. - The VS Code extension leaves all eight runtimes out through .vscodeignore. The preview detects a missing runtime and loads it from the element's default CDN, unless the book sets its own CDN URL. A test lists the package with vsce. - @hyperbook/types exports elementRuntimeFiles, now shared by the CLI's asset manager and the VS Code preview. Co-Authored-By: Claude Opus 5.5 Claude-Session: https://claude.ai/code/session_01MsgtLGX2AgDYCZnqCYUHhz --- .../smaller-markdown-and-vscode-packages.md | 14 ++ packages/hyperbook/README.md | 19 ++- packages/hyperbook/helpers/assets.ts | 32 ++--- packages/hyperbook/tests/assets-build.test.ts | 2 +- packages/markdown/README.md | 42 ++++-- packages/markdown/package.json | 16 ++- packages/markdown/src/elementAssets.ts | 14 ++ .../tests/blockflowVscodePreview.test.ts | 130 ++++++++++++++++-- packages/markdown/tests/elementCdn.test.ts | 35 ++++- .../markdown/tests/packageContents.test.ts | 76 ++++++++++ packages/types/src/index.ts | 20 +++ platforms/vscode/.vscodeignore | 27 ++++ platforms/vscode/README.md | 6 + platforms/vscode/src/Preview.ts | 44 +++++- website/de/book/configuration/book.md | 11 +- website/de/book/elements/blockflow.md | 8 +- website/en/book/configuration/book.md | 10 +- website/en/book/elements/blockflow.md | 7 +- 18 files changed, 448 insertions(+), 65 deletions(-) create mode 100644 .changeset/smaller-markdown-and-vscode-packages.md create mode 100644 packages/markdown/tests/packageContents.test.ts diff --git a/.changeset/smaller-markdown-and-vscode-packages.md b/.changeset/smaller-markdown-and-vscode-packages.md new file mode 100644 index 000000000..d9ddc2fef --- /dev/null +++ b/.changeset/smaller-markdown-and-vscode-packages.md @@ -0,0 +1,14 @@ +--- +"@hyperbook/types": minor +"@hyperbook/markdown": minor +"hyperbook": patch +"hyperbook-studio": minor +--- + +Keep the Markdown npm package and the VS Code extension small by leaving out large element runtimes. + +The Markdown package no longer includes the Pyodide, GeoGebra, Typst, Blockflow and Excalidraw runtimes; it keeps their integration scripts and styles. With `cdn: true`, Blockflow now loads from `blockflow.openpatch.org` and Excalidraw from its own packages on UNPKG, so every default CDN keeps working without these files. The OpenSCAD, Online IDE and SQL IDE runtimes stay in the package because their default CDN is this package on UNPKG. + +The VS Code extension no longer includes any of the eight large runtimes. Its preview loads a missing runtime from the element's default CDN, unless the book sets its own CDN URL. + +`@hyperbook/types` exports `elementRuntimeFiles`, the files that show an element's runtime is available locally. diff --git a/packages/hyperbook/README.md b/packages/hyperbook/README.md index b3bac63f9..625f34b8c 100644 --- a/packages/hyperbook/README.md +++ b/packages/hyperbook/README.md @@ -117,8 +117,10 @@ Set `cdn` per element in `hyperbook.json` to load its large runtime from a CDN: ``` Omitting `cdn` or setting it to `false` keeps local assets. `true` selects the -default CDN: jsDelivr for Pyodide and Typst, GeoGebra's server for GeoGebra, and -versioned Hyperbook assets on UNPKG for the other elements. The option supports +default CDN: jsDelivr for Pyodide and Typst, GeoGebra's server for GeoGebra, +`blockflow.openpatch.org` for Blockflow, Excalidraw's own packages on UNPKG for +Excalidraw, and versioned `@hyperbook/markdown` assets on UNPKG for OpenSCAD, +the Online IDE, and the SQL IDE. The option supports `pyide`, `typst`, `geogebra`, `openscad`, `excalidraw`, `onlineide`, `sqlide`, and `blockflow` (both player and editor). @@ -143,10 +145,15 @@ to the CDN. `assets fetch --all` still downloads every bundle. Build the workspace packages before building the CLI. Its postbuild step writes the small npm package to `dist/`, an asset manifest to `dist/asset-manifest.json`, and eight separate archives to `.cache/asset-bundles//`. The Markdown -build and VS Code extension continue to include their complete assets. Markdown's -npm tarball omits the Pyodide distribution to fit npm's upload limit; standalone -users can extract the CLI's `pyide.tar.gz` bundle alongside the other directive -assets. +build keeps its complete assets in `dist/assets`, which the CLI and the VS Code +extension build from. The published packages stay small: + +- Markdown's npm tarball omits the Pyodide, GeoGebra, Typst, Blockflow, and + Excalidraw runtimes, which have upstream CDNs. It keeps OpenSCAD, the Online + IDE, and the SQL IDE, because their default CDN is this package on UNPKG. + `packages/markdown/tests/packageContents.test.ts` enforces this split. +- The VS Code extension omits all eight runtimes through `.vscodeignore`. Its + preview loads a missing runtime from the element's default CDN. Locally built CLIs use these archives directly, including with `--offline`, so workspace builds work before the corresponding release is published. diff --git a/packages/hyperbook/helpers/assets.ts b/packages/hyperbook/helpers/assets.ts index 465cff2bf..c2ed0d9b8 100644 --- a/packages/hyperbook/helpers/assets.ts +++ b/packages/hyperbook/helpers/assets.ts @@ -7,26 +7,11 @@ import { Transform, Readable } from "stream"; import { pipeline } from "stream/promises"; import { extract } from "tar"; import remoteDirectives from "../asset-bundles.json"; -import { elementCoreFiles, DownloadableElement } from "@hyperbook/types"; - -const runtimeFiles: Record = { - blockflow: ["player.html"], - onlineide: ["include/online-ide-embedded.js"], - sqlide: ["include/sql-ide-embedded.js"], - excalidraw: ["hyperbook-excalidraw.umd.js"], - geogebra: ["GeoGebra/deployggb.js"], - pyide: ["pyodide/pyodide.js", "pyodide/pyodide-lock.json"], - typst: [ - "typst-bundle.js", - "typst-compiler.wasm", - "fonts/LibertinusSerif-Regular.otf", - ], - openscad: [ - "openscad.wasm", - "libraries/BOSL2.zip", - "fonts/Roboto-Regular.ttf", - ], -}; +import { + elementCoreFiles, + elementRuntimeFiles, + DownloadableElement, +} from "@hyperbook/types"; export interface AssetBundle { url: string; @@ -157,9 +142,10 @@ export class AssetManager { try { if ((await fs.stat(bundled)).isDirectory()) { await Promise.all( - (cdn ? [] : runtimeFiles[directive] || []).map((file) => - fs.access(path.join(bundled, file)), - ), + (cdn + ? [] + : elementRuntimeFiles[directive as DownloadableElement] || [] + ).map((file) => fs.access(path.join(bundled, file))), ); return bundled; } diff --git a/packages/hyperbook/tests/assets-build.test.ts b/packages/hyperbook/tests/assets-build.test.ts index de85d5df4..62f3e8cee 100644 --- a/packages/hyperbook/tests/assets-build.test.ts +++ b/packages/hyperbook/tests/assets-build.test.ts @@ -131,7 +131,7 @@ it("builds and prefetches CDN elements offline without obtaining their local run ); const out = path.join(root, ".hyperbook", "out"); const html = await fs.readFile(path.join(out, "index.html"), "utf8"); - expect(html).toContain("unpkg.com/@hyperbook/markdown@"); + expect(html).toContain("https://blockflow.openpatch.org/player.html"); expect(html).toContain('data-runtime-url="https://cdn.example.com/python/"'); await expect( fs.stat(path.join(out, "__hyperbook_assets", "directive-blockflow")), diff --git a/packages/markdown/README.md b/packages/markdown/README.md index 4e208b5cc..22435934f 100644 --- a/packages/markdown/README.md +++ b/packages/markdown/README.md @@ -42,16 +42,32 @@ console.log(result.data.headings); // Extracted headings ## Browser assets -Browser scripts and styles are included under `dist/assets`. Pyodide's large -distribution is provided separately to keep the package within npm's upload -limit. When self-hosting PyIDE with this package, download the -[Pyodide asset bundle for Hyperbook 0.113.0](https://github.com/openpatch/hyperbook/releases/download/hyperbook-assets-v0.113.0/pyide.tar.gz), -verify its SHA-256 checksum against the -[CLI asset manifest](https://unpkg.com/hyperbook@0.113.0/dist/asset-manifest.json), -and extract it into the served assets directory, alongside the other -`directive-*` folders. The bundle includes `directive-pyide/pyodide/` and its -complete Python package distribution. - -The Hyperbook CLI downloads this bundle automatically. Hyperbook Studio for -VS Code includes the runtime. Setting `elements.pyide.cdn` to `true` uses the -Pyodide CDN instead of locally served runtime files. +Browser scripts and styles are included under `dist/assets`. To keep the +package small, it leaves out the large runtimes of these elements and keeps only +their integration scripts and styles: + +| Element | Default CDN | +| --- | --- | +| `pyide` | Pyodide on jsDelivr | +| `typst` | Typst packages and fonts on jsDelivr | +| `geogebra` | GeoGebra's server | +| `blockflow` | `blockflow.openpatch.org` | +| `excalidraw` | Excalidraw's own packages on UNPKG | + +The OpenSCAD, Online IDE, and SQL IDE runtimes stay in the package, because +their default CDN is this package on UNPKG. + +To serve a missing runtime, choose one of these options: + +- **Use the CDN.** Set `elements..cdn` to `true` in the configuration + passed to `process`, or to the base URL of your own copy. +- **Download it.** The Hyperbook CLI publishes one verified bundle per element + for each release. For example, the + [PyIDE bundle for Hyperbook 0.113.0](https://github.com/openpatch/hyperbook/releases/download/hyperbook-assets-v0.113.0/pyide.tar.gz) + is listed with its SHA-256 checksum in the + [CLI asset manifest](https://unpkg.com/hyperbook@0.113.0/dist/asset-manifest.json). + Extract a bundle into the served assets directory, alongside the other + `directive-*` folders. + +The Hyperbook CLI downloads these bundles automatically. The Hyperbook extension +for VS Code loads missing runtimes from their default CDN. diff --git a/packages/markdown/package.json b/packages/markdown/package.json index 039e1d50a..75bd56707 100644 --- a/packages/markdown/package.json +++ b/packages/markdown/package.json @@ -14,7 +14,21 @@ }, "files": [ "dist", - "!dist/assets/directive-pyide/pyodide" + "!dist/assets/directive-blockflow/**", + "!dist/assets/directive-excalidraw/**", + "dist/assets/directive-excalidraw/client.js", + "dist/assets/directive-excalidraw/style.css", + "!dist/assets/directive-geogebra/**", + "dist/assets/directive-geogebra/client.js", + "dist/assets/directive-geogebra/geogebra-web-component.js", + "dist/assets/directive-geogebra/style.css", + "!dist/assets/directive-pyide/**", + "dist/assets/directive-pyide/client.js", + "dist/assets/directive-pyide/python-friendly-error-messages.js", + "dist/assets/directive-pyide/style.css", + "!dist/assets/directive-typst/**", + "dist/assets/directive-typst/client.js", + "dist/assets/directive-typst/style.css" ], "publishConfig": { "access": "public" diff --git a/packages/markdown/src/elementAssets.ts b/packages/markdown/src/elementAssets.ts index 0e167c55a..50351ac63 100644 --- a/packages/markdown/src/elementAssets.ts +++ b/packages/markdown/src/elementAssets.ts @@ -4,8 +4,12 @@ import { elementCdn, } from "@hyperbook/types"; import packageJson from "../package.json"; +import excalidrawPackageJson from "../../web-component-excalidraw/package.json"; import runtimeAssets from "../runtime-assets.json"; +/** Hosted Blockflow app; the npm package does not include Blockflow's bundle. */ +const blockflowCdn = "https://blockflow.openpatch.org/"; + /** Resolve payload files; Hyperbook's integration scripts stay local. */ export function elementAssetUrl( ctx: HyperbookContext, @@ -58,5 +62,15 @@ export function elementAssetUrl( .replace(/^5\.0\//, `${version}/`); return `https://www.geogebra.org/apps/${relative}`; } + if (element === "blockflow") return new URL(file, blockflowCdn).href; + if (element === "excalidraw") { + // The npm package omits Excalidraw's fonts and bundle; use their own packages. + if (file === "hyperbook-excalidraw.umd.js") + return `https://unpkg.com/${excalidrawPackageJson.name}@${excalidrawPackageJson.version}/dist/index.umd.js`; + const base = `https://unpkg.com/@excalidraw/excalidraw@${excalidrawPackageJson.dependencies["@excalidraw/excalidraw"]}/dist/prod/`; + return file === "excalidraw.css" + ? `${base}index.css` + : new URL(file, base).href; + } return `https://unpkg.com/@hyperbook/markdown@${packageJson.version}/dist/assets/directive-${element}/${file}`; } diff --git a/packages/markdown/tests/blockflowVscodePreview.test.ts b/packages/markdown/tests/blockflowVscodePreview.test.ts index be5cbbe1a..b051e00a9 100644 --- a/packages/markdown/tests/blockflowVscodePreview.test.ts +++ b/packages/markdown/tests/blockflowVscodePreview.test.ts @@ -1,13 +1,37 @@ -import { readFileSync } from "node:fs"; +import { + mkdirSync, + mkdtempSync, + readFileSync, + rmSync, + writeFileSync, +} from "node:fs"; +import { createRequire } from "node:module"; +import { tmpdir } from "node:os"; import path from "node:path"; import { runInNewContext } from "node:vm"; import ts from "typescript"; -import { HyperbookContext } from "@hyperbook/types"; +import { + DownloadableElement, + HyperbookContext, + HyperbookJson, + elementCoreFiles, + elementRuntimeFiles, +} from "@hyperbook/types"; import { expect, it } from "vitest"; -it("uses valid VS Code resource URLs for the bundle and local projects", async () => { +const vscodePlatform = new URL("../../../platforms/vscode/", import.meta.url); +const elements = Object.keys(elementCoreFiles) as DownloadableElement[]; + +/** Renders the active document with the extension's Preview and a stubbed VS Code. */ +async function renderPreview({ + existing = () => true, + config = { name: "Test" }, +}: { + existing?: (file: string) => boolean; + config?: HyperbookJson; +} = {}): Promise { const source = readFileSync( - new URL("../../../platforms/vscode/src/Preview.ts", import.meta.url), + new URL("src/Preview.ts", vscodePlatform), "utf8", ); const compiled = ts.transpileModule(source, { @@ -61,9 +85,11 @@ it("uses valid VS Code resource URLs for the bundle and local projects", async ( }; const modules: Record = { vscode, - path, + path: path.posix, + fs: { existsSync: existing }, "@hyperbook/fs": filesystem, "@hyperbook/types": { + elementRuntimeFiles, isExternalUrl: (p: string) => /^https?:\/\//.test(p), }, "@hyperbook/markdown": { @@ -79,7 +105,7 @@ it("uses valid VS Code resource URLs for the bundle and local projects", async ( require: (name: string) => modules[name] || {}, }); const preview = Object.create(exports.default.prototype); - preview.context = { extensionUri: "/extension" }; + preview.context = { extensionUri: "/extension", extensionPath: "/extension" }; preview.panel = { webview: { asWebviewUri: (p: string) => ({ @@ -88,18 +114,104 @@ it("uses valid VS Code resource URLs for the bundle and local projects", async ( }, }; preview.checkDocumentIsHyperbookFile = () => true; - preview.getConfig = async () => ({ name: "Test" }); + preview.getConfig = async () => config; preview.publishDiagnostics = () => {}; await preview.handleTextDocumentChange(); + return ctx!; +} - const asset = ctx!.makeUrl(["directive-blockflow", "editor.html"], "assets"); +it("uses valid VS Code resource URLs for the bundle and local projects", async () => { + const ctx = await renderPreview(); + + const asset = ctx.makeUrl(["directive-blockflow", "editor.html"], "assets"); expect(asset).toBe( "https://file+.vscode-resource.vscode-cdn.net/extension/assets/hyperbook/directive-blockflow/editor.html", ); expect( new URL(asset, "https://preview.vscode-webview.net/index.html").hostname, ).toBe("file+.vscode-resource.vscode-cdn.net"); - expect(ctx!.makeUrl("/project.sb3", "public")).toBe( + expect(ctx.makeUrl("/project.sb3", "public")).toBe( "https://file+.vscode-resource.vscode-cdn.net/workspace/public/project.sb3", ); }); + +it("keeps local runtimes when the extension ships them", async () => { + const config = { name: "Test", elements: { pyide: { cdn: false } } }; + const ctx = await renderPreview({ config }); + expect(ctx.config).toEqual(config); +}); + +it("loads runtimes missing from the extension from their CDN", async () => { + const ctx = await renderPreview({ + existing: (file) => !file.includes("/directive-pyide/"), + config: { + name: "Test", + elements: { pyide: { cdn: false }, typst: { cdn: false } }, + }, + }); + expect(ctx.config.elements?.pyide).toEqual({ cdn: true }); + expect(ctx.config.elements?.typst).toEqual({ cdn: false }); +}); + +it("keeps a book's own CDN and element settings", async () => { + const ctx = await renderPreview({ + existing: () => false, + config: { + name: "Test", + elements: { + typst: { cdn: "https://assets.example.com/typst/" }, + onlineide: { height: 400 }, + }, + }, + }); + for (const element of elements) + expect(ctx.config.elements?.[element]?.cdn).toBeTruthy(); + expect(ctx.config.elements?.typst?.cdn).toBe( + "https://assets.example.com/typst/", + ); + expect(ctx.config.elements?.onlineide).toEqual({ height: 400, cdn: true }); +}); + +it("leaves large element runtimes out of the VS Code package", async () => { + const require = createRequire(new URL("package.json", vscodePlatform)); + const vsce = require("@vscode/vsce"); + const fixture = mkdtempSync(path.join(tmpdir(), "hyperbook-vsix-")); + try { + for (const name of ["package.json", ".vscodeignore"]) + writeFileSync( + path.join(fixture, name), + readFileSync(new URL(name, vscodePlatform)), + ); + const write = (file: string) => { + mkdirSync(path.dirname(path.join(fixture, file)), { recursive: true }); + writeFileSync(path.join(fixture, file), ""); + }; + write("out/extension.js"); + write("assets/hyperbook/directive-alert/client.js"); + for (const element of elements) + for (const file of [ + ...elementCoreFiles[element], + ...elementRuntimeFiles[element], + ]) + write(`assets/hyperbook/directive-${element}/${file}`); + + const files: string[] = await vsce.listFiles({ + cwd: fixture, + packageManager: vsce.PackageManager.None, + }); + expect(files).toContain("out/extension.js"); + expect(files).toContain("assets/hyperbook/directive-alert/client.js"); + for (const element of elements) { + for (const file of elementCoreFiles[element]) + expect(files).toContain( + `assets/hyperbook/directive-${element}/${file}`, + ); + for (const file of elementRuntimeFiles[element]) + expect(files).not.toContain( + `assets/hyperbook/directive-${element}/${file}`, + ); + } + } finally { + rmSync(fixture, { recursive: true, force: true }); + } +}, 60_000); diff --git a/packages/markdown/tests/elementCdn.test.ts b/packages/markdown/tests/elementCdn.test.ts index 9a64fc7f5..bd8912bf3 100644 --- a/packages/markdown/tests/elementCdn.test.ts +++ b/packages/markdown/tests/elementCdn.test.ts @@ -3,6 +3,7 @@ import { DownloadableElement, elementCdn } from "@hyperbook/types"; import { process as processMarkdown } from "../src/process"; import { elementAssetUrl } from "../src/elementAssets"; import packageJson from "../package.json"; +import excalidrawPackageJson from "../../web-component-excalidraw/package.json"; import { ctx } from "./mock"; const elements: [DownloadableElement, string][] = [ @@ -49,7 +50,13 @@ describe.each(elements)( pyide: "https://cdn.jsdelivr.net/pyodide/v314.0.7/full/", typst: "https://cdn.jsdelivr.net/npm/@myriaddreamin/", geogebra: "https://www.geogebra.org/apps/", - }[element as "pyide" | "typst" | "geogebra"] || + blockflow: "https://blockflow.openpatch.org/", + excalidraw: + "https://unpkg.com/@hyperbook/web-component-excalidraw@", + }[ + element as + "pyide" | "typst" | "geogebra" | "blockflow" | "excalidraw" + ] || `https://unpkg.com/@hyperbook/markdown@${packageJson.version}/dist/assets/directive-${element}/` : "http://assets.example.com/runtime/", ); @@ -111,3 +118,29 @@ it("uses pinned upstream Typst fonts and GeoGebra codebases", () => { elementAssetUrl(configured, "geogebra", "GeoGebra/HTML5/5.0/web3d/"), ).toBe("https://www.geogebra.org/apps/5.4.931.2/web3d/"); }); + +it("loads Blockflow and Excalidraw from their upstream CDNs", () => { + const configured = { + ...ctx, + config: { + ...ctx.config, + elements: { blockflow: { cdn: true }, excalidraw: { cdn: true } }, + }, + }; + expect(elementAssetUrl(configured, "blockflow", "editor.html")).toBe( + "https://blockflow.openpatch.org/editor.html", + ); + expect( + elementAssetUrl(configured, "excalidraw", "hyperbook-excalidraw.umd.js"), + ).toBe( + `https://unpkg.com/@hyperbook/web-component-excalidraw@${excalidrawPackageJson.version}/dist/index.umd.js`, + ); + const excalidraw = `https://unpkg.com/@excalidraw/excalidraw@${excalidrawPackageJson.dependencies["@excalidraw/excalidraw"]}/dist/prod/`; + expect(elementAssetUrl(configured, "excalidraw", "excalidraw.css")).toBe( + `${excalidraw}index.css`, + ); + // Excalidraw resolves fonts/ beside its asset base. + expect(elementAssetUrl(configured, "excalidraw", "")).toBe(excalidraw); + // An unpkg URL needs an exact version to stay stable. + expect(excalidraw).toMatch(/@\d+\.\d+\.\d+\/dist/); +}); diff --git a/packages/markdown/tests/packageContents.test.ts b/packages/markdown/tests/packageContents.test.ts new file mode 100644 index 000000000..486dc253c --- /dev/null +++ b/packages/markdown/tests/packageContents.test.ts @@ -0,0 +1,76 @@ +import { execFileSync } from "node:child_process"; +import { mkdtempSync, mkdirSync, rmSync, writeFileSync } from "node:fs"; +import { tmpdir } from "node:os"; +import path from "node:path"; +import { + DownloadableElement, + elementCoreFiles, + elementRuntimeFiles, +} from "@hyperbook/types"; +import { expect, it } from "vitest"; +import { elementAssetUrl } from "../src/elementAssets"; +import packageJson from "../package.json"; +import { ctx } from "./mock"; + +const elements = Object.keys(elementCoreFiles) as DownloadableElement[]; + +/** What `npm publish` would upload, given a dist tree with every element. */ +function packedFiles(): string[] { + const fixture = mkdtempSync(path.join(tmpdir(), "hyperbook-markdown-pack-")); + try { + const { name, version, files } = packageJson; + writeFileSync( + path.join(fixture, "package.json"), + JSON.stringify({ name, version, files }), + ); + const write = (file: string) => { + mkdirSync(path.dirname(path.join(fixture, file)), { recursive: true }); + writeFileSync(path.join(fixture, file), ""); + }; + write("dist/index.js"); + write("dist/assets/directive-blockflow-player/client.js"); + for (const element of elements) + for (const file of [ + ...elementCoreFiles[element], + ...elementRuntimeFiles[element], + ]) + write(`dist/assets/directive-${element}/${file}`); + const [pack] = JSON.parse( + execFileSync("npm", ["pack", "--dry-run", "--json", "--ignore-scripts"], { + cwd: fixture, + encoding: "utf8", + stdio: ["ignore", "pipe", "ignore"], + }), + ); + return pack.files.map((file: { path: string }) => file.path); + } finally { + rmSync(fixture, { recursive: true, force: true }); + } +} + +it("publishes only the runtimes its own default CDN serves", () => { + const files = packedFiles(); + expect(files).toContain("dist/index.js"); + expect(files).toContain("dist/assets/directive-blockflow-player/client.js"); + for (const element of elements) { + for (const file of elementCoreFiles[element]) + expect(files).toContain(`dist/assets/directive-${element}/${file}`); + const configured = { + ...ctx, + config: { ...ctx.config, elements: { [element]: { cdn: true } } }, + }; + for (const file of elementRuntimeFiles[element]) { + // Elements whose default CDN is this package on unpkg must ship their + // runtime. Every other runtime stays out to keep the package small. + // Directives resolve nested runtime files through their top folder. + const requested = file.includes("/") ? file.replace(/\/.*/, "/") : file; + const servedByPackage = elementAssetUrl( + configured, + element, + requested, + ).startsWith(`https://unpkg.com/${packageJson.name}@`); + const packed = files.includes(`dist/assets/directive-${element}/${file}`); + expect(packed, `${element}: ${file}`).toBe(servedByPackage); + } + } +}, 60_000); diff --git a/packages/types/src/index.ts b/packages/types/src/index.ts index f5a0abb9b..f8c573fa3 100644 --- a/packages/types/src/index.ts +++ b/packages/types/src/index.ts @@ -21,6 +21,26 @@ export const elementCoreFiles = { export type DownloadableElement = keyof typeof elementCoreFiles; +/** Files whose presence shows that an element's full runtime is available locally. */ +export const elementRuntimeFiles: Record = { + blockflow: ["player.html"], + onlineide: ["include/online-ide-embedded.js"], + sqlide: ["include/sql-ide-embedded.js"], + excalidraw: ["hyperbook-excalidraw.umd.js"], + geogebra: ["GeoGebra/deployggb.js"], + pyide: ["pyodide/pyodide.js", "pyodide/pyodide-lock.json"], + typst: [ + "typst-bundle.js", + "typst-compiler.wasm", + "fonts/LibertinusSerif-Regular.otf", + ], + openscad: [ + "openscad.wasm", + "libraries/BOSL2.zip", + "fonts/Roboto-Regular.ttf", + ], +}; + export function elementCdn( config: HyperbookJson, element: string, diff --git a/platforms/vscode/.vscodeignore b/platforms/vscode/.vscodeignore index aece64b11..421653100 100644 --- a/platforms/vscode/.vscodeignore +++ b/platforms/vscode/.vscodeignore @@ -11,3 +11,30 @@ vsc-extension-quickstart.md **/*.ts .prettierrc screenshots/** + +# Large element runtimes load from their CDNs instead (see src/Preview.ts). +# Keep only the integration files listed in elementCoreFiles (@hyperbook/types). +assets/hyperbook/directive-blockflow/** +assets/hyperbook/directive-onlineide/** +!assets/hyperbook/directive-onlineide/style.css +assets/hyperbook/directive-sqlide/** +!assets/hyperbook/directive-sqlide/client.js +!assets/hyperbook/directive-sqlide/style.css +assets/hyperbook/directive-excalidraw/** +!assets/hyperbook/directive-excalidraw/client.js +!assets/hyperbook/directive-excalidraw/style.css +assets/hyperbook/directive-openscad/** +!assets/hyperbook/directive-openscad/client.js +!assets/hyperbook/directive-openscad/worker.js +!assets/hyperbook/directive-openscad/style.css +assets/hyperbook/directive-geogebra/** +!assets/hyperbook/directive-geogebra/client.js +!assets/hyperbook/directive-geogebra/geogebra-web-component.js +!assets/hyperbook/directive-geogebra/style.css +assets/hyperbook/directive-pyide/** +!assets/hyperbook/directive-pyide/client.js +!assets/hyperbook/directive-pyide/python-friendly-error-messages.js +!assets/hyperbook/directive-pyide/style.css +assets/hyperbook/directive-typst/** +!assets/hyperbook/directive-typst/client.js +!assets/hyperbook/directive-typst/style.css diff --git a/platforms/vscode/README.md b/platforms/vscode/README.md index 2fe8c1e46..45e8d9640 100644 --- a/platforms/vscode/README.md +++ b/platforms/vscode/README.md @@ -19,6 +19,12 @@ You can preview your Hyperbook pages by clicking the preview icon in the top rig ![Preview](https://github.com/openpatch/hyperbook/raw/main/platforms/vscode/screenshots/preview.gif) +To keep the extension small, it does not include the large runtimes of Pyodide, +Typst, GeoGebra, OpenSCAD, Excalidraw, Blockflow, the Online IDE, and the SQL +IDE. The preview loads them from their default CDN, so previewing these +elements needs an internet connection. If your `hyperbook.json` sets `cdn` to +your own URL for an element, the preview uses that URL instead. + ### Hyperbook Config The `hyperbook.json` is validated against a schema, which presents you diff --git a/platforms/vscode/src/Preview.ts b/platforms/vscode/src/Preview.ts index 8604cd0fc..80a91c55b 100644 --- a/platforms/vscode/src/Preview.ts +++ b/platforms/vscode/src/Preview.ts @@ -11,11 +11,14 @@ import { import { process } from "@hyperbook/markdown"; import { disposeAll } from "./utils/dispose"; import path, { posix } from "path"; +import { existsSync } from "fs"; import { + DownloadableElement, HyperbookContext, HyperbookJson, HyperbookPage, Navigation, + elementRuntimeFiles, isExternalUrl, } from "@hyperbook/types"; @@ -48,6 +51,9 @@ export default class Preview { private _resource: vscode.Uri | undefined; private _vfile: VFileBook | VFileGlossary | undefined; + /** Elements whose large runtime is not shipped with the extension. */ + private missingRuntimes: DownloadableElement[] | undefined; + private readonly disposables: vscode.Disposable[] = []; private _disposed: boolean = false; private readonly _onDisposeEmitter = new vscode.EventEmitter(); @@ -124,6 +130,42 @@ export default class Preview { } } + /** + * The published extension leaves out large element runtimes (Pyodide, + * GeoGebra, Typst, ...) to stay small. Elements whose runtime is missing load + * it from their default CDN, unless the book already sets its own CDN URL. + * Development builds keep the full runtimes and stay unaffected. + */ + withCdnFallback(config: HyperbookJson): HyperbookJson { + if (!this.missingRuntimes) { + const assets = path.join( + this.context.extensionPath, + "assets", + "hyperbook", + ); + this.missingRuntimes = ( + Object.keys(elementRuntimeFiles) as DownloadableElement[] + ).filter( + (element) => + !elementRuntimeFiles[element].every((file) => + existsSync( + path.join(assets, `directive-${element}`, ...file.split("/")), + ), + ), + ); + } + if (this.missingRuntimes.length === 0) { + return config; + } + const elements: Record = { ...config.elements }; + for (const element of this.missingRuntimes) { + if (!elements[element]?.cdn) { + elements[element] = { ...elements[element], cdn: true }; + } + } + return { ...config, elements }; + } + async handleTextDocumentChange() { this.hyperbookViewerConfig = vscode.workspace.getConfiguration("hyperbook"); if ( @@ -195,7 +237,7 @@ export default class Preview { ...publicBookFiles, ...publicGlossaryFiles, ]; - const config = await this.getConfig(); + const config = this.withCdnFallback(await this.getConfig()); // Without this a `:::protect{use="..."}` block fails to render in the // preview, even though it builds fine. diff --git a/website/de/book/configuration/book.md b/website/de/book/configuration/book.md index c74716c70..602c43183 100644 --- a/website/de/book/configuration/book.md +++ b/website/de/book/configuration/book.md @@ -136,10 +136,17 @@ Diese Option gibt es für `pyide`, `typst`, `geogebra`, `openscad`, `excalidraw` Player und den Editor. Kiri:Moto verwendet weiterhin seinen externen Dienst. Die Standard-CDNs sind jsDelivr für Pyodide und Typst, der GeoGebra-Server für -GeoGebra und die versionierten Hyperbook-Dateien auf UNPKG für die anderen -Elemente. Die kleinen Integrationsskripte und Styles von Hyperbook bleiben im +GeoGebra, die gehostete Blockflow-App unter `blockflow.openpatch.org` für +Blockflow, die eigenen Pakete von Excalidraw auf UNPKG für Excalidraw und die +versionierten Hyperbook-Dateien auf UNPKG für OpenSCAD, die Online-IDE und die +SQL-IDE. Die kleinen Integrationsskripte und Styles von Hyperbook bleiben im Build. +Die Hyperbook-Erweiterung für VS Code enthält diese Laufzeitumgebungen nicht. +Ihre Vorschau lädt sie vom Standard-CDN, sofern `cdn` nicht auf eine eigene URL +gesetzt ist. Für die Vorschau dieser Elemente ist daher eine Internetverbindung +nötig. + Eine eigene URL muss auf den Inhalt des Verzeichnisses `__hyperbook_assets/directive-/` aus einem lokalen Build zeigen. Behalte die Unterverzeichnisse bei. Bei `pyide` muss die URL direkt auf die diff --git a/website/de/book/elements/blockflow.md b/website/de/book/elements/blockflow.md index ae2ae99aa..3752c6e2c 100644 --- a/website/de/book/elements/blockflow.md +++ b/website/de/book/elements/blockflow.md @@ -7,9 +7,11 @@ lang: de [Blockflow](https://blockflow.openpatch.org) ist ein Fork von Scratch zum Erstellen von geführten Tutorials. Hyperbook bietet zwei Direktiven zum Einbetten von Blockflow: einen **Player** zum Abspielen und einen **Editor** zum Erstellen von Tutorials. -Hyperbook und die VS Code-Erweiterung enthalten ein Blockflow-Browserpaket mit fester Version. -Bücher mit einer dieser Direktiven verwenden eine gemeinsame lokale Kopie und sind für Editor -und Player nicht auf blockflow.openpatch.org angewiesen. Lokale Projekte und Tutorial-Medien +Hyperbook enthält ein Blockflow-Browserpaket mit fester Version. Die CLI lädt es herunter, wenn ein +Build Blockflow zum ersten Mal verwendet. Bücher mit einer dieser Direktiven verwenden eine +gemeinsame lokale Kopie und sind für Editor und Player nicht auf blockflow.openpatch.org +angewiesen. Die VS Code-Erweiterung enthält das Paket nicht; ihre Vorschau lädt Blockflow von +blockflow.openpatch.org. Lokale Projekte und Tutorial-Medien funktionieren offline, wenn das Buch lokal bereitgestellt wird. Externe Projekte, Inhalte aus der Scratch-Bibliothek und Netzwerk-Erweiterungen benötigen weiterhin Internetzugang. Lizenz und Quellcode-Link von Blockflow werden mit den Dateien ausgeliefert. diff --git a/website/en/book/configuration/book.md b/website/en/book/configuration/book.md index c71443ba9..2a66d5344 100644 --- a/website/en/book/configuration/book.md +++ b/website/en/book/configuration/book.md @@ -139,8 +139,14 @@ applies to both the player and editor. Kiri:Moto continues to use its external service. The default CDNs are jsDelivr for Pyodide and Typst, GeoGebra's server for -GeoGebra, and UNPKG's versioned Hyperbook assets for the other elements. -Hyperbook's small integration scripts and styles stay in the build. +GeoGebra, the hosted Blockflow app at `blockflow.openpatch.org` for Blockflow, +Excalidraw's own packages on UNPKG for Excalidraw, and UNPKG's versioned +Hyperbook assets for OpenSCAD, the Online IDE, and the SQL IDE. Hyperbook's +small integration scripts and styles stay in the build. + +The Hyperbook extension for VS Code does not include these runtimes. Its +preview loads them from the default CDN unless `cdn` is set to a custom URL, so +previewing these elements needs an internet connection. A custom URL must point to the contents of the corresponding `__hyperbook_assets/directive-/` directory from a local build. Preserve diff --git a/website/en/book/elements/blockflow.md b/website/en/book/elements/blockflow.md index 099de2a46..e215d5e9a 100644 --- a/website/en/book/elements/blockflow.md +++ b/website/en/book/elements/blockflow.md @@ -7,9 +7,10 @@ permaid: blockflow [Blockflow](https://blockflow.openpatch.org) is a fork of Scratch for building guided tutorials. Hyperbook provides two directives for embedding Blockflow: a **player** for playing tutorials and an **editor** for creating them. -Hyperbook and its VS Code extension include a pinned Blockflow browser bundle. Books that use -either directive serve one shared local copy, so the editor and player do not depend on -blockflow.openpatch.org. Local project files and tutorial media work offline when the book is +Hyperbook includes a pinned Blockflow browser bundle. The CLI downloads it the first time a build +uses Blockflow. Books that use either directive serve one shared local copy, so the editor and +player do not depend on blockflow.openpatch.org. The VS Code extension does not include the bundle; +its preview loads Blockflow from blockflow.openpatch.org. Local project files and tutorial media work offline when the book is served locally; remote projects, Scratch library assets, and network extensions still need internet access. The bundled Blockflow license and source link are included with its assets. From 93ff1be5e989e335ee6a90fd0f40e53adcaa4f2e Mon Sep 17 00:00:00 2001 From: Mike Barkmin Date: Sat, 10 Oct 2026 09:57:42 +0200 Subject: [PATCH 2/6] feat: share the CLI asset cache with the VS Code preview The VS Code preview now resolves element runtimes like a Hyperbook build instead of always falling back to a CDN. - Elements with a `cdn` setting load from that CDN, as in a build. - Other elements use their runtime from the Hyperbook CLI's asset cache. The extension ships the CLI's asset manifest and shares the CLI's content-addressed, SHA-256 verified bundles, so a runtime fetched by either one works for both. - Runtimes that are not downloaded yet load from their default CDN, and the preview offers to download the ones the page uses. - New commands "Download Element Runtimes..." and "Download All Element Runtimes" mirror `hyperbook assets fetch` and `--all`. The asset manager moves from the CLI into @hyperbook/fs so both can use it. The CLI keeps a thin wrapper with its default paths. The shared build script now treats Node built-ins as external, which leaves the other packages' bundles unchanged. Co-Authored-By: Claude Opus 5.5 Claude-Session: https://claude.ai/code/session_01MsgtLGX2AgDYCZnqCYUHhz --- .../smaller-markdown-and-vscode-packages.md | 5 +- packages/fs/package.json | 1 + packages/fs/src/assets.ts | 335 ++++++++++++++++ packages/fs/src/index.ts | 3 + packages/hyperbook/README.md | 7 +- packages/hyperbook/helpers/assets.ts | 356 ++---------------- packages/markdown/README.md | 3 +- .../tests/blockflowVscodePreview.test.ts | 104 +++-- .../tests/vscodeRuntimeAssets.test.ts | 178 +++++++++ platforms/vscode/README.md | 22 +- platforms/vscode/package.json | 11 + platforms/vscode/src/Preview.ts | 110 ++++-- platforms/vscode/src/RuntimeAssets.ts | 129 +++++++ platforms/vscode/src/StatusBarItem.ts | 4 +- platforms/vscode/src/extension.ts | 49 ++- platforms/vscode/webpack.extension.config.js | 18 +- pnpm-lock.yaml | 6 + scripts/buildPackage.mjs | 3 + website/de/book/configuration/book.md | 9 +- website/de/book/elements/blockflow.md | 4 +- website/en/book/configuration/book.md | 9 +- website/en/book/elements/blockflow.md | 5 +- 22 files changed, 961 insertions(+), 410 deletions(-) create mode 100644 packages/fs/src/assets.ts create mode 100644 packages/markdown/tests/vscodeRuntimeAssets.test.ts create mode 100644 platforms/vscode/src/RuntimeAssets.ts diff --git a/.changeset/smaller-markdown-and-vscode-packages.md b/.changeset/smaller-markdown-and-vscode-packages.md index d9ddc2fef..6f3276ef9 100644 --- a/.changeset/smaller-markdown-and-vscode-packages.md +++ b/.changeset/smaller-markdown-and-vscode-packages.md @@ -1,5 +1,6 @@ --- "@hyperbook/types": minor +"@hyperbook/fs": minor "@hyperbook/markdown": minor "hyperbook": patch "hyperbook-studio": minor @@ -9,6 +10,6 @@ Keep the Markdown npm package and the VS Code extension small by leaving out lar The Markdown package no longer includes the Pyodide, GeoGebra, Typst, Blockflow and Excalidraw runtimes; it keeps their integration scripts and styles. With `cdn: true`, Blockflow now loads from `blockflow.openpatch.org` and Excalidraw from its own packages on UNPKG, so every default CDN keeps working without these files. The OpenSCAD, Online IDE and SQL IDE runtimes stay in the package because their default CDN is this package on UNPKG. -The VS Code extension no longer includes any of the eight large runtimes. Its preview loads a missing runtime from the element's default CDN, unless the book sets its own CDN URL. +The VS Code extension no longer includes any of the eight large runtimes. Its preview resolves them like a Hyperbook build: elements with a `cdn` setting load from that CDN, and the others use the runtimes in the Hyperbook CLI's asset cache, which the extension now shares. Runtimes that are not downloaded yet load from their default CDN, and the preview offers to download them. The new commands "Download Element Runtimes..." and "Download All Element Runtimes" work like `hyperbook assets fetch` and `hyperbook assets fetch --all`. -`@hyperbook/types` exports `elementRuntimeFiles`, the files that show an element's runtime is available locally. +`@hyperbook/fs` now provides `AssetManager` and `assetCacheDir`, which the CLI and the VS Code extension share. `@hyperbook/types` exports `elementRuntimeFiles`, the files that show an element's runtime is available locally. diff --git a/packages/fs/package.json b/packages/fs/package.json index aa1d4d7fc..005baa2a3 100644 --- a/packages/fs/package.json +++ b/packages/fs/package.json @@ -45,6 +45,7 @@ "mdast-util-from-markdown": "^2.0.3", "micromark-extension-directive": "^4.0.0", "mime-types": "^3.0.2", + "tar": "7.5.22", "unist-util-visit": "^5.1.0", "yaml": "^2.9.0" }, diff --git a/packages/fs/src/assets.ts b/packages/fs/src/assets.ts new file mode 100644 index 000000000..b30787474 --- /dev/null +++ b/packages/fs/src/assets.ts @@ -0,0 +1,335 @@ +import path from "path"; +import os from "node:os"; +import fs from "fs/promises"; +import { createReadStream, createWriteStream } from "fs"; +import { createHash, randomUUID } from "node:crypto"; +import { Transform, Readable } from "node:stream"; +import { pipeline } from "node:stream/promises"; +import { extract } from "tar"; +import { + elementCoreFiles, + elementRuntimeFiles, + DownloadableElement, +} from "@hyperbook/types"; + +export interface AssetBundle { + url: string; + sha256: string; + bytes: number; + files: Record; +} + +export interface AssetManifest { + format: 1; + version: string; + bundles: Record; +} + +export interface AssetOptions { + /** Use cached and local bundles only; never download. */ + offline?: boolean; + /** Shared cache for downloaded bundles. Defaults to {@link assetCacheDir}. */ + cacheDir?: string; + /** Folder with the `directive-*` assets that ship with the host. */ + assetsPath: string; + /** Asset manifest listing the verified bundle for each element. */ + manifestPath: string; + /** Folder with `/.tar.gz` archives from a workspace build. */ + bundlesPath: string; + /** Elements with downloadable bundles. Defaults to every downloadable element. */ + elements?: readonly string[]; +} + +export function assetCacheDir(): string { + if (process.env.HYPERBOOK_ASSET_CACHE) { + return path.resolve(process.env.HYPERBOOK_ASSET_CACHE); + } + if (process.platform === "win32") { + return path.join( + process.env.LOCALAPPDATA || path.join(os.homedir(), "AppData", "Local"), + "hyperbook", + "assets", + ); + } + if (process.platform === "darwin") { + return path.join(os.homedir(), "Library", "Caches", "hyperbook", "assets"); + } + return path.join( + process.env.XDG_CACHE_HOME || path.join(os.homedir(), ".cache"), + "hyperbook", + "assets", + ); +} + +function safePath(value: string): boolean { + return ( + value.length > 0 && + !value.includes("\\") && + !value.includes("\0") && + !path.posix.isAbsolute(value) && + !/^[a-zA-Z]:/.test(value) && + !value.split("/").some((part) => part === ".." || part === ".") + ); +} + +/** Large assets live outside the installation, so global and npx installs work too. */ +export class AssetManager { + readonly cacheDir: string; + private readonly assetsPath: string; + private readonly manifestPath: string; + private readonly bundlesPath: string; + private readonly offline: boolean; + private readonly remoteDirectives: readonly string[]; + private manifest?: Promise; + private readonly pending = new Map>(); + + constructor(options: AssetOptions) { + this.cacheDir = path.resolve(options.cacheDir || assetCacheDir()); + this.assetsPath = options.assetsPath; + this.manifestPath = options.manifestPath; + this.bundlesPath = options.bundlesPath; + this.offline = options.offline || false; + this.remoteDirectives = options.elements || Object.keys(elementCoreFiles); + } + + async readManifest(): Promise { + if (!this.manifest) { + this.manifest = fs.readFile(this.manifestPath, "utf8").then((text) => { + const manifest: AssetManifest = JSON.parse(text); + if ( + manifest.format !== 1 || + !manifest.bundles || + typeof manifest.version !== "string" || + !safePath(manifest.version) + ) + throw new Error("Invalid Hyperbook asset manifest."); + for (const [name, bundle] of Object.entries(manifest.bundles)) { + if ( + !this.remoteDirectives.includes(name) || + !bundle.url.startsWith("https://") || + !/^[a-f0-9]{64}$/.test(bundle.sha256) || + !Number.isSafeInteger(bundle.bytes) || + bundle.bytes <= 0 || + !bundle.files || + Object.keys(bundle.files).length === 0 || + Object.entries(bundle.files).some( + ([file, size]) => + !safePath(file) || + !file.startsWith(`directive-${name}/`) || + !Number.isSafeInteger(size) || + size < 0, + ) + ) { + throw new Error(`Invalid asset manifest entry: ${name}.`); + } + } + return manifest; + }); + } + return this.manifest; + } + + /** Directives without any assets are valid; missing remote bundles are errors. */ + async ensure(directive: string, cdn = false): Promise { + const bundled = path.join(this.assetsPath, `directive-${directive}`); + try { + if ((await fs.stat(bundled)).isDirectory()) { + await Promise.all( + (cdn + ? [] + : elementRuntimeFiles[directive as DownloadableElement] || [] + ).map((file) => fs.access(path.join(bundled, file))), + ); + return bundled; + } + } catch (error: any) { + if (error.code !== "ENOENT") throw error; + } + if (cdn || !this.remoteDirectives.includes(directive)) return undefined; + const manifest = await this.readManifest(); + const bundle = manifest.bundles[directive]; + if (!bundle) + throw new Error( + `Asset manifest has no bundle for ${directive}. Reinstall Hyperbook.`, + ); + const key = bundle.sha256; + let pending = this.pending.get(key); + if (!pending) { + pending = this.obtain( + directive, + bundle, + path.join(this.bundlesPath, manifest.version, `${directive}.tar.gz`), + ); + this.pending.set(key, pending); + pending.catch(() => this.pending.delete(key)); + } + return pending; + } + + async copyDirective( + directive: string, + assetsOut: string, + cdn = false, + ): Promise { + const source = await this.ensure(directive, cdn); + if (!source) return; + const destination = path.join(assetsOut, `directive-${directive}`); + await fs.mkdir(destination, { recursive: true }); + await fs.cp(source, destination, { + recursive: true, + filter(file) { + if (!cdn) return true; + const relative = path.relative(source, file).split(path.sep).join("/"); + return ( + !relative || + (elementCoreFiles[directive as DownloadableElement]?.includes( + relative, + ) ?? + false) + ); + }, + }); + } + + private async complete( + destination: string, + bundle: AssetBundle, + ): Promise { + try { + if ( + (await fs.readFile(path.join(destination, ".complete"), "utf8")) !== + bundle.sha256 + ) + return false; + for (const [file, size] of Object.entries(bundle.files)) { + const stat = await fs.lstat(path.join(destination, file)); + if (!stat.isFile() || stat.size !== size) return false; + } + return true; + } catch (error: any) { + if (error.code === "ENOENT" || error.code === "ENOTDIR") return false; + throw error; + } + } + + private async obtain( + directive: string, + bundle: AssetBundle, + localArchive: string, + ): Promise { + const destination = path.join(this.cacheDir, bundle.sha256); + // A small pointer selects an immutable generation. Replacing that pointer + // cannot remove files another process is copying, even during cache repair. + try { + const generation = await fs.readFile( + path.join(destination, "current"), + "utf8", + ); + if (/^content-[a-f0-9-]+$/.test(generation)) { + const contents = path.join(destination, generation); + if (await this.complete(contents, bundle)) + return path.join(contents, `directive-${directive}`); + } + } catch (error: any) { + if (error.code !== "ENOENT") throw error; + } + // Workspace builds can use their verified archives before a release exists. + const local = ( + await fs.stat(localArchive).catch((error) => { + if (error.code !== "ENOENT") throw error; + return undefined; + }) + )?.isFile(); + if (this.offline && !local) { + throw new Error( + `Assets for ${directive} are missing from ${this.cacheDir}. Run "hyperbook assets fetch" online before building with --offline.`, + ); + } + + await fs.mkdir(destination, { recursive: true }); + const staging = await fs.mkdtemp(path.join(destination, ".download-")); + const archive = path.join(staging, "bundle.tar.gz"); + const unpacked = path.join(staging, "unpacked"); + try { + let input; + if (local) { + console.log(`[Assets] Using local ${directive} bundle...`); + input = createReadStream(localArchive); + } else { + console.log(`[Assets] Downloading ${directive}...`); + const response = await fetch(bundle.url, { + signal: AbortSignal.timeout(300_000), + }); + if (!response.ok || !response.body) + throw new Error(`HTTP ${response.status} from ${bundle.url}`); + input = Readable.fromWeb(response.body as any); + } + let bytes = 0; + const hash = createHash("sha256"); + const verify = new Transform({ + transform(chunk, _encoding, callback) { + bytes += chunk.length; + if (bytes > bundle.bytes) + return callback( + new Error("Asset download exceeds its expected size."), + ); + hash.update(chunk); + callback(null, chunk); + }, + }); + await pipeline( + input, + verify, + createWriteStream(archive, { flags: "wx" }), + ); + if (bytes !== bundle.bytes || hash.digest("hex") !== bundle.sha256) { + throw new Error("Asset download failed SHA-256 verification."); + } + await fs.mkdir(unpacked); + let invalidEntry: string | undefined; + const seen = new Set(); + await extract({ + file: archive, + cwd: unpacked, + strict: true, + filter(entryPath, entry) { + if (!("type" in entry)) return false; + const normalized = entryPath.replace(/\/$/, ""); + const directory = + entry.type === "Directory" && + (normalized === `directive-${directive}` || + normalized.startsWith(`directive-${directive}/`)); + const file = + entry.type === "File" && + Object.prototype.hasOwnProperty.call(bundle.files, normalized) && + entry.size === bundle.files[normalized] && + !seen.has(normalized); + if (!safePath(normalized) || (!directory && !file)) { + invalidEntry = entryPath; + return false; + } + if (file) seen.add(normalized); + return true; + }, + }); + if (invalidEntry) + throw new Error(`Unexpected asset archive entry: ${invalidEntry}.`); + await fs.writeFile(path.join(unpacked, ".complete"), bundle.sha256); + if (!(await this.complete(unpacked, bundle))) + throw new Error("Asset archive is incomplete."); + const generation = `content-${randomUUID()}`; + const contents = path.join(destination, generation); + await fs.rename(unpacked, contents); + const pointer = path.join(staging, "current"); + await fs.writeFile(pointer, generation); + await fs.rename(pointer, path.join(destination, "current")); + return path.join(contents, `directive-${directive}`); + } catch (error: any) { + throw new Error( + `Could not prepare assets for ${directive}: ${error.message}`, + ); + } finally { + await fs.rm(staging, { recursive: true, force: true }); + } + } +} diff --git a/packages/fs/src/index.ts b/packages/fs/src/index.ts index 695efe2bd..9ffe10331 100644 --- a/packages/fs/src/index.ts +++ b/packages/fs/src/index.ts @@ -28,4 +28,7 @@ export { } from "./passwords"; export type { PasswordRegistry, ResolvedRegistry } from "./passwords"; +export { AssetManager, assetCacheDir } from "./assets"; +export type { AssetBundle, AssetManifest, AssetOptions } from "./assets"; + export { hyperlibrary, hyperbook, hyperproject, vfile, passwords }; diff --git a/packages/hyperbook/README.md b/packages/hyperbook/README.md index 625f34b8c..0d762f3e6 100644 --- a/packages/hyperbook/README.md +++ b/packages/hyperbook/README.md @@ -100,6 +100,9 @@ run `hyperbook assets fetch`, build with `--offline`, and save the cache again. Deleting the cache is safe; the next online build downloads the required bundles again. +The Hyperbook extension for VS Code shares this cache. Its preview uses the +runtimes found there, and its **Download Element Runtimes** commands fill it. + ### Optional CDNs Set `cdn` per element in `hyperbook.json` to load its large runtime from a CDN: @@ -153,7 +156,9 @@ extension build from. The published packages stay small: IDE, and the SQL IDE, because their default CDN is this package on UNPKG. `packages/markdown/tests/packageContents.test.ts` enforces this split. - The VS Code extension omits all eight runtimes through `.vscodeignore`. Its - preview loads a missing runtime from the element's default CDN. + preview uses runtimes from the CLI's asset cache and loads missing ones from + their default CDN. It ships the CLI's `asset-manifest.json`, so its download + commands fetch the same verified bundles into the same cache. Locally built CLIs use these archives directly, including with `--offline`, so workspace builds work before the corresponding release is published. diff --git a/packages/hyperbook/helpers/assets.ts b/packages/hyperbook/helpers/assets.ts index c2ed0d9b8..95c17e998 100644 --- a/packages/hyperbook/helpers/assets.ts +++ b/packages/hyperbook/helpers/assets.ts @@ -1,342 +1,40 @@ import path from "path"; -import os from "os"; -import fs from "fs/promises"; -import { createReadStream, createWriteStream } from "fs"; -import { createHash, randomUUID } from "crypto"; -import { Transform, Readable } from "stream"; -import { pipeline } from "stream/promises"; -import { extract } from "tar"; -import remoteDirectives from "../asset-bundles.json"; import { - elementCoreFiles, - elementRuntimeFiles, - DownloadableElement, -} from "@hyperbook/types"; - -export interface AssetBundle { - url: string; - sha256: string; - bytes: number; - files: Record; -} - -export interface AssetManifest { - format: 1; - version: string; - bundles: Record; -} - -export interface AssetOptions { - offline?: boolean; - cacheDir?: string; - assetsPath?: string; - manifestPath?: string; - bundlesPath?: string; -} + AssetManager as SharedAssetManager, + AssetOptions as SharedAssetOptions, + assetCacheDir, +} from "@hyperbook/fs"; +import remoteDirectives from "../asset-bundles.json"; -export function assetCacheDir(): string { - if (process.env.HYPERBOOK_ASSET_CACHE) { - return path.resolve(process.env.HYPERBOOK_ASSET_CACHE); - } - if (process.platform === "win32") { - return path.join( - process.env.LOCALAPPDATA || path.join(os.homedir(), "AppData", "Local"), - "hyperbook", - "assets", - ); - } - if (process.platform === "darwin") { - return path.join(os.homedir(), "Library", "Caches", "hyperbook", "assets"); - } - return path.join( - process.env.XDG_CACHE_HOME || path.join(os.homedir(), ".cache"), - "hyperbook", - "assets", - ); -} +export { assetCacheDir }; +export type { AssetBundle, AssetManifest } from "@hyperbook/fs"; -function safePath(value: string): boolean { - return ( - value.length > 0 && - !value.includes("\\") && - !value.includes("\0") && - !path.posix.isAbsolute(value) && - !/^[a-zA-Z]:/.test(value) && - !value.split("/").some((part) => part === ".." || part === ".") - ); -} +export type AssetOptions = Partial>; /** Large assets live outside the installation, so global and npx installs work too. */ -export class AssetManager { - readonly cacheDir: string; - private readonly assetsPath: string; - private readonly manifestPath: string; - private readonly bundlesPath: string; - private readonly offline: boolean; - private manifest?: Promise; - private readonly pending = new Map>(); - +export class AssetManager extends SharedAssetManager { constructor(options: AssetOptions = {}) { - this.cacheDir = path.resolve(options.cacheDir || assetCacheDir()); // helpers/ is flattened into dist/ by ncc. Source runs use the package root. const fromSource = path.basename(__dirname) === "helpers"; const packagePath = fromSource ? path.dirname(__dirname) : __dirname; - this.assetsPath = options.assetsPath || path.join(packagePath, "assets"); - this.manifestPath = - options.manifestPath || - path.join( - packagePath, - ...(fromSource ? ["dist"] : []), - "asset-manifest.json", - ); - this.bundlesPath = - options.bundlesPath || - path.join( - fromSource ? packagePath : path.dirname(packagePath), - ".cache", - "asset-bundles", - ); - this.offline = options.offline || false; - } - - async readManifest(): Promise { - if (!this.manifest) { - this.manifest = fs.readFile(this.manifestPath, "utf8").then((text) => { - const manifest: AssetManifest = JSON.parse(text); - if ( - manifest.format !== 1 || - !manifest.bundles || - typeof manifest.version !== "string" || - !safePath(manifest.version) - ) - throw new Error("Invalid Hyperbook asset manifest."); - for (const [name, bundle] of Object.entries(manifest.bundles)) { - if ( - !remoteDirectives.includes(name) || - !bundle.url.startsWith("https://") || - !/^[a-f0-9]{64}$/.test(bundle.sha256) || - !Number.isSafeInteger(bundle.bytes) || - bundle.bytes <= 0 || - !bundle.files || - Object.keys(bundle.files).length === 0 || - Object.entries(bundle.files).some( - ([file, size]) => - !safePath(file) || - !file.startsWith(`directive-${name}/`) || - !Number.isSafeInteger(size) || - size < 0, - ) - ) { - throw new Error(`Invalid asset manifest entry: ${name}.`); - } - } - return manifest; - }); - } - return this.manifest; - } - - /** Directives without any assets are valid; missing remote bundles are errors. */ - async ensure(directive: string, cdn = false): Promise { - const bundled = path.join(this.assetsPath, `directive-${directive}`); - try { - if ((await fs.stat(bundled)).isDirectory()) { - await Promise.all( - (cdn - ? [] - : elementRuntimeFiles[directive as DownloadableElement] || [] - ).map((file) => fs.access(path.join(bundled, file))), - ); - return bundled; - } - } catch (error: any) { - if (error.code !== "ENOENT") throw error; - } - if (cdn || !remoteDirectives.includes(directive)) return undefined; - const manifest = await this.readManifest(); - const bundle = manifest.bundles[directive]; - if (!bundle) - throw new Error( - `Asset manifest has no bundle for ${directive}. Reinstall Hyperbook.`, - ); - const key = bundle.sha256; - let pending = this.pending.get(key); - if (!pending) { - pending = this.obtain( - directive, - bundle, - path.join(this.bundlesPath, manifest.version, `${directive}.tar.gz`), - ); - this.pending.set(key, pending); - pending.catch(() => this.pending.delete(key)); - } - return pending; - } - - async copyDirective( - directive: string, - assetsOut: string, - cdn = false, - ): Promise { - const source = await this.ensure(directive, cdn); - if (!source) return; - const destination = path.join(assetsOut, `directive-${directive}`); - await fs.mkdir(destination, { recursive: true }); - await fs.cp(source, destination, { - recursive: true, - filter(file) { - if (!cdn) return true; - const relative = path.relative(source, file).split(path.sep).join("/"); - return ( - !relative || - (elementCoreFiles[directive as DownloadableElement]?.includes( - relative, - ) ?? - false) - ); - }, + super({ + ...options, + elements: remoteDirectives, + assetsPath: options.assetsPath || path.join(packagePath, "assets"), + manifestPath: + options.manifestPath || + path.join( + packagePath, + ...(fromSource ? ["dist"] : []), + "asset-manifest.json", + ), + bundlesPath: + options.bundlesPath || + path.join( + fromSource ? packagePath : path.dirname(packagePath), + ".cache", + "asset-bundles", + ), }); } - - private async complete( - destination: string, - bundle: AssetBundle, - ): Promise { - try { - if ( - (await fs.readFile(path.join(destination, ".complete"), "utf8")) !== - bundle.sha256 - ) - return false; - for (const [file, size] of Object.entries(bundle.files)) { - const stat = await fs.lstat(path.join(destination, file)); - if (!stat.isFile() || stat.size !== size) return false; - } - return true; - } catch (error: any) { - if (error.code === "ENOENT" || error.code === "ENOTDIR") return false; - throw error; - } - } - - private async obtain( - directive: string, - bundle: AssetBundle, - localArchive: string, - ): Promise { - const destination = path.join(this.cacheDir, bundle.sha256); - // A small pointer selects an immutable generation. Replacing that pointer - // cannot remove files another process is copying, even during cache repair. - try { - const generation = await fs.readFile( - path.join(destination, "current"), - "utf8", - ); - if (/^content-[a-f0-9-]+$/.test(generation)) { - const contents = path.join(destination, generation); - if (await this.complete(contents, bundle)) - return path.join(contents, `directive-${directive}`); - } - } catch (error: any) { - if (error.code !== "ENOENT") throw error; - } - // Workspace builds can use their verified archives before a release exists. - const local = ( - await fs.stat(localArchive).catch((error) => { - if (error.code !== "ENOENT") throw error; - return undefined; - }) - )?.isFile(); - if (this.offline && !local) { - throw new Error( - `Assets for ${directive} are missing from ${this.cacheDir}. Run "hyperbook assets fetch" online before building with --offline.`, - ); - } - - await fs.mkdir(destination, { recursive: true }); - const staging = await fs.mkdtemp(path.join(destination, ".download-")); - const archive = path.join(staging, "bundle.tar.gz"); - const unpacked = path.join(staging, "unpacked"); - try { - let input; - if (local) { - console.log(`[Assets] Using local ${directive} bundle...`); - input = createReadStream(localArchive); - } else { - console.log(`[Assets] Downloading ${directive}...`); - const response = await fetch(bundle.url, { - signal: AbortSignal.timeout(300_000), - }); - if (!response.ok || !response.body) - throw new Error(`HTTP ${response.status} from ${bundle.url}`); - input = Readable.fromWeb(response.body as any); - } - let bytes = 0; - const hash = createHash("sha256"); - const verify = new Transform({ - transform(chunk, _encoding, callback) { - bytes += chunk.length; - if (bytes > bundle.bytes) - return callback( - new Error("Asset download exceeds its expected size."), - ); - hash.update(chunk); - callback(null, chunk); - }, - }); - await pipeline( - input, - verify, - createWriteStream(archive, { flags: "wx" }), - ); - if (bytes !== bundle.bytes || hash.digest("hex") !== bundle.sha256) { - throw new Error("Asset download failed SHA-256 verification."); - } - await fs.mkdir(unpacked); - let invalidEntry: string | undefined; - const seen = new Set(); - await extract({ - file: archive, - cwd: unpacked, - strict: true, - filter(entryPath, entry) { - if (!("type" in entry)) return false; - const normalized = entryPath.replace(/\/$/, ""); - const directory = - entry.type === "Directory" && - (normalized === `directive-${directive}` || - normalized.startsWith(`directive-${directive}/`)); - const file = - entry.type === "File" && - Object.prototype.hasOwnProperty.call(bundle.files, normalized) && - entry.size === bundle.files[normalized] && - !seen.has(normalized); - if (!safePath(normalized) || (!directory && !file)) { - invalidEntry = entryPath; - return false; - } - if (file) seen.add(normalized); - return true; - }, - }); - if (invalidEntry) - throw new Error(`Unexpected asset archive entry: ${invalidEntry}.`); - await fs.writeFile(path.join(unpacked, ".complete"), bundle.sha256); - if (!(await this.complete(unpacked, bundle))) - throw new Error("Asset archive is incomplete."); - const generation = `content-${randomUUID()}`; - const contents = path.join(destination, generation); - await fs.rename(unpacked, contents); - const pointer = path.join(staging, "current"); - await fs.writeFile(pointer, generation); - await fs.rename(pointer, path.join(destination, "current")); - return path.join(contents, `directive-${directive}`); - } catch (error: any) { - throw new Error( - `Could not prepare assets for ${directive}: ${error.message}`, - ); - } finally { - await fs.rm(staging, { recursive: true, force: true }); - } - } } diff --git a/packages/markdown/README.md b/packages/markdown/README.md index 22435934f..4dca9eacb 100644 --- a/packages/markdown/README.md +++ b/packages/markdown/README.md @@ -70,4 +70,5 @@ To serve a missing runtime, choose one of these options: `directive-*` folders. The Hyperbook CLI downloads these bundles automatically. The Hyperbook extension -for VS Code loads missing runtimes from their default CDN. +for VS Code shares the CLI's asset cache and can download them on request. +`AssetManager` from `@hyperbook/fs` implements both. diff --git a/packages/markdown/tests/blockflowVscodePreview.test.ts b/packages/markdown/tests/blockflowVscodePreview.test.ts index b051e00a9..04474c200 100644 --- a/packages/markdown/tests/blockflowVscodePreview.test.ts +++ b/packages/markdown/tests/blockflowVscodePreview.test.ts @@ -22,14 +22,24 @@ import { expect, it } from "vitest"; const vscodePlatform = new URL("../../../platforms/vscode/", import.meta.url); const elements = Object.keys(elementCoreFiles) as DownloadableElement[]; +type RuntimeLocation = + | { kind: "bundled" } + | { kind: "cached"; directory: string } + | { kind: "missing" }; + /** Renders the active document with the extension's Preview and a stubbed VS Code. */ async function renderPreview({ - existing = () => true, + locate = () => ({ kind: "bundled" }), config = { name: "Test" }, + directives = {}, + renders = 1, }: { - existing?: (file: string) => boolean; + locate?: (element: DownloadableElement) => RuntimeLocation; config?: HyperbookJson; -} = {}): Promise { + directives?: Record; + renders?: number; +} = {}): Promise<{ ctx: HyperbookContext; messages: string[] }> { + const messages: string[] = []; const source = readFileSync( new URL("src/Preview.ts", vscodePlatform), "utf8", @@ -49,8 +59,12 @@ async function renderPreview({ }; const vscode = { languages: { createDiagnosticCollection: () => ({}) }, + commands: { executeCommand: () => {} }, workspace: { getConfiguration: () => ({}) }, window: { + showInformationMessage: async (message: string) => { + messages.push(message); + }, activeTextEditor: { document: { uri: { fsPath: "/workspace/book/lesson/index.md" }, @@ -86,16 +100,18 @@ async function renderPreview({ const modules: Record = { vscode, path: path.posix, - fs: { existsSync: existing }, + "./RuntimeAssets": { + runtimeElements: elements, + runtimeLabels: Object.fromEntries(elements.map((e) => [e, `<${e}>`])), + }, "@hyperbook/fs": filesystem, "@hyperbook/types": { - elementRuntimeFiles, isExternalUrl: (p: string) => /^https?:\/\//.test(p), }, "@hyperbook/markdown": { process: async (_content: string, context: HyperbookContext) => { ctx = context; - return ""; + return { data: { directives } }; }, }, }; @@ -105,7 +121,10 @@ async function renderPreview({ require: (name: string) => modules[name] || {}, }); const preview = Object.create(exports.default.prototype); - preview.context = { extensionUri: "/extension", extensionPath: "/extension" }; + preview.context = { extensionUri: "/extension" }; + // Object.create skips the constructor and its field initializers. + preview.offeredDownloads = new Set(); + preview.runtimes = { locate: async (e: DownloadableElement) => locate(e) }; preview.panel = { webview: { asWebviewUri: (p: string) => ({ @@ -116,12 +135,12 @@ async function renderPreview({ preview.checkDocumentIsHyperbookFile = () => true; preview.getConfig = async () => config; preview.publishDiagnostics = () => {}; - await preview.handleTextDocumentChange(); - return ctx!; + for (let i = 0; i < renders; i++) await preview.handleTextDocumentChange(); + return { ctx: ctx!, messages }; } it("uses valid VS Code resource URLs for the bundle and local projects", async () => { - const ctx = await renderPreview(); + const { ctx } = await renderPreview(); const asset = ctx.makeUrl(["directive-blockflow", "editor.html"], "assets"); expect(asset).toBe( @@ -137,39 +156,72 @@ it("uses valid VS Code resource URLs for the bundle and local projects", async ( it("keeps local runtimes when the extension ships them", async () => { const config = { name: "Test", elements: { pyide: { cdn: false } } }; - const ctx = await renderPreview({ config }); + const { ctx } = await renderPreview({ config }); expect(ctx.config).toEqual(config); }); -it("loads runtimes missing from the extension from their CDN", async () => { - const ctx = await renderPreview({ - existing: (file) => !file.includes("/directive-pyide/"), - config: { - name: "Test", - elements: { pyide: { cdn: false }, typst: { cdn: false } }, - }, +it("uses runtimes from the CLI asset cache", async () => { + const { ctx, messages } = await renderPreview({ + locate: (element) => + element === "pyide" || element === "typst" + ? { + kind: "cached", + directory: `/cache/hash/content-1/directive-${element}`, + } + : { kind: "bundled" }, + config: { name: "Test", elements: { pyide: { cdn: false } } }, + directives: { pyide: {}, typst: {} }, + }); + const cache = + "https://file+.vscode-resource.vscode-cdn.net/cache/hash/content-1"; + expect(ctx.config.elements?.pyide).toEqual({ + cdn: `${cache}/directive-pyide/pyodide/`, }); - expect(ctx.config.elements?.pyide).toEqual({ cdn: true }); - expect(ctx.config.elements?.typst).toEqual({ cdn: false }); + expect(ctx.config.elements?.typst).toEqual({ + cdn: `${cache}/directive-typst/`, + }); + expect(ctx.config.elements?.geogebra).toBeUndefined(); + expect(messages).toEqual([]); }); -it("keeps a book's own CDN and element settings", async () => { - const ctx = await renderPreview({ - existing: () => false, +it("loads runtimes that are not downloaded from their CDN and offers a download once", async () => { + const { ctx, messages } = await renderPreview({ + locate: () => ({ kind: "missing" }), + directives: { pyide: {}, "blockflow-player": {}, alert: {} }, + renders: 2, + }); + for (const element of elements) + expect(ctx.config.elements?.[element]).toEqual({ cdn: true }); + expect(messages).toHaveLength(1); + expect(messages[0]).toContain(", "); + expect(messages[0]).not.toContain(""); +}); + +it("keeps a book's own CDN and element settings, like a Hyperbook build", async () => { + const located: DownloadableElement[] = []; + const { ctx, messages } = await renderPreview({ + locate: (element) => { + located.push(element); + return { kind: "missing" }; + }, config: { name: "Test", elements: { typst: { cdn: "https://assets.example.com/typst/" }, + geogebra: { cdn: true }, onlineide: { height: 400 }, }, }, + directives: { typst: {}, geogebra: {} }, }); - for (const element of elements) - expect(ctx.config.elements?.[element]?.cdn).toBeTruthy(); expect(ctx.config.elements?.typst?.cdn).toBe( "https://assets.example.com/typst/", ); + expect(located).not.toContain("typst"); + expect(located).not.toContain("geogebra"); expect(ctx.config.elements?.onlineide).toEqual({ height: 400, cdn: true }); + // The book chose these CDNs, so there is nothing to download. + expect(messages).toEqual([]); }); it("leaves large element runtimes out of the VS Code package", async () => { @@ -187,6 +239,7 @@ it("leaves large element runtimes out of the VS Code package", async () => { writeFileSync(path.join(fixture, file), ""); }; write("out/extension.js"); + write("out/asset-manifest.json"); write("assets/hyperbook/directive-alert/client.js"); for (const element of elements) for (const file of [ @@ -200,6 +253,7 @@ it("leaves large element runtimes out of the VS Code package", async () => { packageManager: vsce.PackageManager.None, }); expect(files).toContain("out/extension.js"); + expect(files).toContain("out/asset-manifest.json"); expect(files).toContain("assets/hyperbook/directive-alert/client.js"); for (const element of elements) { for (const file of elementCoreFiles[element]) diff --git a/packages/markdown/tests/vscodeRuntimeAssets.test.ts b/packages/markdown/tests/vscodeRuntimeAssets.test.ts new file mode 100644 index 000000000..582607f01 --- /dev/null +++ b/packages/markdown/tests/vscodeRuntimeAssets.test.ts @@ -0,0 +1,178 @@ +import { createHash } from "node:crypto"; +import { + mkdirSync, + mkdtempSync, + readFileSync, + rmSync, + statSync, + writeFileSync, +} from "node:fs"; +import { tmpdir } from "node:os"; +import path from "node:path"; +import { runInNewContext } from "node:vm"; +import ts from "typescript"; +import { create } from "tar"; +import * as hyperbookFs from "@hyperbook/fs"; +import * as types from "@hyperbook/types"; +import { afterEach, expect, it, vi } from "vitest"; + +const source = readFileSync( + new URL("../../../platforms/vscode/src/RuntimeAssets.ts", import.meta.url), + "utf8", +); + +let root: string | undefined; +afterEach(() => { + vi.unstubAllEnvs(); + if (root) rmSync(root, { recursive: true, force: true }); + root = undefined; +}); + +/** The extension's RuntimeAssets with a stubbed VS Code API. */ +function loadRuntimeAssets(messages: string[]) { + const vscode = { + EventEmitter: class { + listeners: (() => void)[] = []; + event = (listener: () => void) => this.listeners.push(listener); + fire() { + this.listeners.forEach((listener) => listener()); + } + }, + ProgressLocation: { Notification: 15 }, + window: { + withProgress: (_options: unknown, task: any) => + task({ report: () => {} }), + showInformationMessage: (message: string) => messages.push(message), + showErrorMessage: (message: string) => messages.push(message), + }, + }; + const modules: Record = { + vscode, + path, + "@hyperbook/fs": hyperbookFs, + "@hyperbook/types": types, + }; + const exports: any = {}; + runInNewContext( + ts.transpileModule(source, { + compilerOptions: { + module: ts.ModuleKind.CommonJS, + esModuleInterop: true, + target: ts.ScriptTarget.ES2022, + }, + }).outputText, + { exports, require: (name: string) => modules[name] }, + ); + return exports.default; +} + +it("shares verified runtimes with the CLI's asset cache", async () => { + root = mkdtempSync(path.join(tmpdir(), "hyperbook-vscode-runtimes-")); + const cacheDir = path.join(root, "cache"); + vi.stubEnv("HYPERBOOK_ASSET_CACHE", cacheDir); + + // An installed extension: integration files only, plus the CLI manifest. + const extension = path.join(root, "extension"); + const core = path.join(extension, "assets", "hyperbook", "directive-pyide"); + mkdirSync(core, { recursive: true }); + writeFileSync(path.join(core, "client.js"), "client"); + + // The runtime bundle a CLI release publishes. + const content = path.join(root, "content"); + const runtime = path.join(content, "directive-pyide", "pyodide"); + mkdirSync(runtime, { recursive: true }); + writeFileSync(path.join(runtime, "pyodide.js"), "pyodide"); + writeFileSync(path.join(runtime, "pyodide-lock.json"), "{}"); + const files = { + "directive-pyide/pyodide/pyodide-lock.json": 2, + "directive-pyide/pyodide/pyodide.js": 7, + }; + const archive = path.join(root, "pyide.tar.gz"); + await create( + { file: archive, cwd: content, gzip: true, portable: true, noMtime: true }, + Object.keys(files), + ); + const manifest = { + format: 1, + version: "1.0.0", + bundles: { + pyide: { + url: "https://example.invalid/pyide.tar.gz", + sha256: createHash("sha256") + .update(readFileSync(archive)) + .digest("hex"), + bytes: statSync(archive).size, + files, + }, + }, + }; + mkdirSync(path.join(extension, "out")); + writeFileSync( + path.join(extension, "out", "asset-manifest.json"), + JSON.stringify(manifest), + ); + + const messages: string[] = []; + const RuntimeAssets = loadRuntimeAssets(messages); + const runtimes = new RuntimeAssets(extension); + expect(runtimes.cacheDir).toBe(cacheDir); + expect(await runtimes.locate("pyide")).toEqual({ kind: "missing" }); + expect(await runtimes.locate("typst")).toEqual({ kind: "missing" }); + + // Serve the download from the verified archive instead of the network. + const fetch = vi.fn(async () => new Response(readFileSync(archive))); + vi.stubGlobal("fetch", fetch); + let changed = 0; + runtimes.onDidChange(() => changed++); + expect(await runtimes.download(["pyide"])).toBe(true); + vi.unstubAllGlobals(); + expect(fetch).toHaveBeenCalledWith( + manifest.bundles.pyide.url, + expect.anything(), + ); + expect(changed).toBe(1); + + const location = await runtimes.locate("pyide"); + expect(location.kind).toBe("cached"); + expect(location.directory.startsWith(cacheDir)).toBe(true); + expect( + readFileSync( + path.join(location.directory, "pyodide", "pyodide.js"), + "utf8", + ), + ).toBe("pyodide"); + + // The CLI finds the same bundle offline, so neither downloads it twice. + const cli = new hyperbookFs.AssetManager({ + offline: true, + cacheDir, + assetsPath: path.join(root, "cli-assets"), + manifestPath: path.join(extension, "out", "asset-manifest.json"), + bundlesPath: path.join(root, "cli-bundles"), + }); + expect(await cli.ensure("pyide")).toBe(location.directory); + + // A runtime the extension ships counts as bundled. + const bundled = path.join( + extension, + "assets", + "hyperbook", + "directive-typst", + ); + mkdirSync(path.join(bundled, "fonts"), { recursive: true }); + for (const file of types.elementRuntimeFiles.typst) + writeFileSync(path.join(bundled, file), ""); + expect(await runtimes.locate("typst")).toEqual({ kind: "bundled" }); +}); + +it("reports runtimes it cannot download", async () => { + root = mkdtempSync(path.join(tmpdir(), "hyperbook-vscode-runtimes-")); + vi.stubEnv("HYPERBOOK_ASSET_CACHE", path.join(root, "cache")); + const messages: string[] = []; + const RuntimeAssets = loadRuntimeAssets(messages); + // No manifest: for example a development build without the CLI. + const runtimes = new RuntimeAssets(path.join(root, "extension")); + expect(await runtimes.download(["geogebra"])).toBe(false); + expect(messages.at(-1)).toContain("GeoGebra"); + expect(await runtimes.locate("geogebra")).toEqual({ kind: "missing" }); +}); diff --git a/platforms/vscode/README.md b/platforms/vscode/README.md index 45e8d9640..07417babc 100644 --- a/platforms/vscode/README.md +++ b/platforms/vscode/README.md @@ -19,11 +19,27 @@ You can preview your Hyperbook pages by clicking the preview icon in the top rig ![Preview](https://github.com/openpatch/hyperbook/raw/main/platforms/vscode/screenshots/preview.gif) +### Element runtimes + To keep the extension small, it does not include the large runtimes of Pyodide, Typst, GeoGebra, OpenSCAD, Excalidraw, Blockflow, the Online IDE, and the SQL -IDE. The preview loads them from their default CDN, so previewing these -elements needs an internet connection. If your `hyperbook.json` sets `cdn` to -your own URL for an element, the preview uses that URL instead. +IDE. The preview resolves them like a Hyperbook build: + +- If your `hyperbook.json` sets `cdn` for an element, the preview loads it from + that CDN. +- Otherwise the preview uses the runtime from the Hyperbook CLI's asset cache. + The extension and the CLI share this folder, so a runtime downloaded by + either one works for both. +- A runtime that is not downloaded yet loads from the element's default CDN. + The preview offers to download it. + +To download runtimes for offline use, run **Hyperbook: Download Element +Runtimes...** or **Hyperbook: Download All Element Runtimes**. They work like +`hyperbook assets fetch` and `hyperbook assets fetch --all`. Downloads are +verified with SHA-256 and stored in the CLI's cache folder: `~/.cache/hyperbook/assets` +on Linux, `~/Library/Caches/hyperbook/assets` on macOS, and +`%LOCALAPPDATA%\hyperbook\assets` on Windows. `HYPERBOOK_ASSET_CACHE` and +`XDG_CACHE_HOME` change it, as for the CLI. ### Hyperbook Config diff --git a/platforms/vscode/package.json b/platforms/vscode/package.json index 12d928b17..2500230d3 100644 --- a/platforms/vscode/package.json +++ b/platforms/vscode/package.json @@ -76,6 +76,16 @@ "command": "hyperbook.openDocumentation", "title": "Open Hyperbook Documentation", "category": "Hyperbook" + }, + { + "command": "hyperbook.downloadRuntimes", + "title": "Download Element Runtimes...", + "category": "Hyperbook" + }, + { + "command": "hyperbook.downloadAllRuntimes", + "title": "Download All Element Runtimes", + "category": "Hyperbook" } ], "menus": { @@ -229,6 +239,7 @@ "eslint": "10.8.1", "file-loader": "6.2.0", "glob": "13.0.6", + "hyperbook": "workspace:*", "node-polyfill-webpack-plugin": "^4.1.0", "npm-run-all": "4.1.5", "ovsx": "1.1.1", diff --git a/platforms/vscode/src/Preview.ts b/platforms/vscode/src/Preview.ts index 80a91c55b..9aaa9f0f6 100644 --- a/platforms/vscode/src/Preview.ts +++ b/platforms/vscode/src/Preview.ts @@ -11,16 +11,15 @@ import { import { process } from "@hyperbook/markdown"; import { disposeAll } from "./utils/dispose"; import path, { posix } from "path"; -import { existsSync } from "fs"; import { DownloadableElement, HyperbookContext, HyperbookJson, HyperbookPage, Navigation, - elementRuntimeFiles, isExternalUrl, } from "@hyperbook/types"; +import RuntimeAssets, { runtimeElements, runtimeLabels } from "./RuntimeAssets"; // Helper function to resolve relative paths const resolveRelativePath = (path: string, page: HyperbookPage): string => { @@ -51,8 +50,12 @@ export default class Preview { private _resource: vscode.Uri | undefined; private _vfile: VFileBook | VFileGlossary | undefined; - /** Elements whose large runtime is not shipped with the extension. */ - private missingRuntimes: DownloadableElement[] | undefined; + runtimes: RuntimeAssets; + /** Elements that load from their CDN only because their runtime is missing. */ + private cdnFallbacks = new Set(); + /** Elements the user was already offered a download for. */ + private readonly offeredDownloads = new Set(); + private readonly disposables: vscode.Disposable[] = []; private _disposed: boolean = false; @@ -67,8 +70,11 @@ export default class Preview { private static readonly diagnostics = vscode.languages.createDiagnosticCollection("hyperbook"); - constructor(context: vscode.ExtensionContext) { + constructor(context: vscode.ExtensionContext, runtimes: RuntimeAssets) { this.context = context; + this.runtimes = runtimes; + runtimes.onDidChange(() => this.handleTextDocumentChange()); + vscode.workspace.onDidChangeTextDocument(async (e) => { // Only refresh when hyperbook config files change @@ -131,41 +137,67 @@ export default class Preview { } /** - * The published extension leaves out large element runtimes (Pyodide, - * GeoGebra, Typst, ...) to stay small. Elements whose runtime is missing load - * it from their default CDN, unless the book already sets its own CDN URL. - * Development builds keep the full runtimes and stay unaffected. + * Resolves element runtimes like a Hyperbook build. An element with a CDN in + * the book's config loads from it. Every other element uses its local + * runtime: shipped with a development build of the extension, or downloaded + * into the Hyperbook CLI's asset cache. A runtime that is not downloaded yet + * loads from the element's default CDN, so the preview still works. */ - withCdnFallback(config: HyperbookJson): HyperbookJson { - if (!this.missingRuntimes) { - const assets = path.join( - this.context.extensionPath, - "assets", - "hyperbook", - ); - this.missingRuntimes = ( - Object.keys(elementRuntimeFiles) as DownloadableElement[] - ).filter( - (element) => - !elementRuntimeFiles[element].every((file) => - existsSync( - path.join(assets, `directive-${element}`, ...file.split("/")), - ), - ), - ); - } - if (this.missingRuntimes.length === 0) { - return config; - } + async withRuntimes(config: HyperbookJson): Promise { const elements: Record = { ...config.elements }; - for (const element of this.missingRuntimes) { - if (!elements[element]?.cdn) { - elements[element] = { ...elements[element], cdn: true }; + const fallbacks = new Set(); + for (const element of runtimeElements) { + if (elements[element]?.cdn) { + continue; + } + const location = await this.runtimes.locate(element); + if (location.kind === "bundled") { + continue; } + let cdn: string | true = true; + if (location.kind === "cached" && this.panel) { + // A custom CDN URL points at the contents of directive-/. + // Pyodide's URL points at its distribution folder instead. + cdn = + this.panel.webview + .asWebviewUri(vscode.Uri.file(location.directory)) + .toString() + .replace(/\/?$/, "/") + (element === "pyide" ? "pyodide/" : ""); + } else { + fallbacks.add(element); + } + elements[element] = { ...elements[element], cdn }; } + this.cdnFallbacks = fallbacks; return { ...config, elements }; } + /** Offers to download the missing runtimes a page uses, once per element. */ + private offerDownloads(used: DownloadableElement[]) { + const missing = used.filter( + (element) => + this.cdnFallbacks.has(element) && !this.offeredDownloads.has(element), + ); + if (missing.length === 0) { + return; + } + missing.forEach((element) => this.offeredDownloads.add(element)); + const names = missing.map((element) => runtimeLabels[element]).join(", "); + vscode.window + .showInformationMessage( + `Hyperbook: The preview loads ${names} from the internet. Download the runtime to preview it offline, like a Hyperbook build.`, + "Download", + ) + .then((choice) => { + if (choice === "Download") { + vscode.commands.executeCommand("hyperbook.downloadRuntimes", missing); + } + }); + } + + /** Elements used by the current page, for preselecting downloads. */ + usedElements: DownloadableElement[] = []; + async handleTextDocumentChange() { this.hyperbookViewerConfig = vscode.workspace.getConfiguration("hyperbook"); if ( @@ -237,7 +269,7 @@ export default class Preview { ...publicBookFiles, ...publicGlossaryFiles, ]; - const config = this.withCdnFallback(await this.getConfig()); + const config = await this.withRuntimes(await this.getConfig()); // Without this a `:::protect{use="..."}` block fails to render in the // preview, even though it builds fine. @@ -393,6 +425,15 @@ export default class Preview { const result = await process(this._vfile.markdown.content, ctx); this.publishDiagnostics(result); + // Blockflow registers as blockflow-player and blockflow-editor. + const directives = Object.keys((result.data as any)?.directives || {}); + this.usedElements = runtimeElements.filter((element) => + directives.some( + (name) => name === element || name.startsWith(`${element}-`), + ), + ); + this.offerDownloads(this.usedElements); + this.panel.webview.html = String(result); } @@ -499,6 +540,7 @@ export default class Preview { enableCommandUris: true, localResourceRoots: [ vscode.Uri.joinPath(this.context.extensionUri, "assets"), + vscode.Uri.file(this.runtimes.cacheDir), vscode.Uri.file(vscode.workspace?.rootPath || ""), ], }, diff --git a/platforms/vscode/src/RuntimeAssets.ts b/platforms/vscode/src/RuntimeAssets.ts new file mode 100644 index 000000000..2c7be5582 --- /dev/null +++ b/platforms/vscode/src/RuntimeAssets.ts @@ -0,0 +1,129 @@ +import * as vscode from "vscode"; +import path from "path"; +import { AssetManager, AssetOptions, assetCacheDir } from "@hyperbook/fs"; +import { DownloadableElement, elementCoreFiles } from "@hyperbook/types"; + +export const runtimeElements = Object.keys( + elementCoreFiles, +) as DownloadableElement[]; + +export const runtimeLabels: Record = { + blockflow: "Blockflow", + excalidraw: "Excalidraw", + geogebra: "GeoGebra", + onlineide: "Online IDE", + openscad: "OpenSCAD", + pyide: "PyIDE (Pyodide)", + sqlide: "SQL IDE", + typst: "Typst", +}; + +/** Where an element's runtime is available to the preview. */ +export type RuntimeLocation = + | { kind: "bundled" } + | { kind: "cached"; directory: string } + | { kind: "missing" }; + +/** + * Element runtimes in the Hyperbook CLI's asset cache. + * + * The extension does not ship large runtimes. It shares the CLI's cache + * folder and its verified, content-addressed bundles, so a runtime fetched by + * `hyperbook assets fetch` is used here too, and the other way round. + */ +export default class RuntimeAssets { + readonly cacheDir = assetCacheDir(); + private readonly options: AssetOptions; + private readonly bundled: string; + private readonly found = new Map(); + private readonly changed = new vscode.EventEmitter(); + /** Fires after a download added runtimes to the cache. */ + readonly onDidChange = this.changed.event; + + constructor(extensionPath: string) { + this.bundled = path.join(extensionPath, "assets", "hyperbook"); + this.options = { + cacheDir: this.cacheDir, + assetsPath: this.bundled, + // Copied from the CLI build, which publishes the matching bundles. + manifestPath: path.join(extensionPath, "out", "asset-manifest.json"), + bundlesPath: path.join(extensionPath, "asset-bundles"), + }; + } + + /** Looks the runtime up without downloading it. */ + async locate(element: DownloadableElement): Promise { + const known = this.found.get(element); + if (known) { + return known; + } + let location: RuntimeLocation; + try { + const directory = await new AssetManager({ + ...this.options, + offline: true, + }).ensure(element); + location = + directory === path.join(this.bundled, `directive-${element}`) + ? { kind: "bundled" } + : directory + ? { kind: "cached", directory } + : { kind: "missing" }; + } catch { + location = { kind: "missing" }; + } + // A missing runtime is looked up again, since the CLI may fetch it. + if (location.kind !== "missing") { + this.found.set(element, location); + } + return location; + } + + /** Downloads and verifies runtimes into the shared cache. */ + async download(elements: readonly DownloadableElement[]): Promise { + const pending: DownloadableElement[] = []; + for (const element of elements) { + if ((await this.locate(element)).kind === "missing") { + pending.push(element); + } + } + if (pending.length === 0) { + vscode.window.showInformationMessage( + "Hyperbook: The selected runtimes are already available.", + ); + return true; + } + const failed: string[] = []; + const manager = new AssetManager(this.options); + await vscode.window.withProgress( + { + location: vscode.ProgressLocation.Notification, + title: "Hyperbook: Downloading runtimes", + }, + async (progress) => { + for (const [index, element] of pending.entries()) { + progress.report({ + message: `${runtimeLabels[element]} (${index + 1}/${pending.length})`, + increment: index === 0 ? 0 : 100 / pending.length, + }); + try { + await manager.ensure(element); + } catch (error: any) { + failed.push(`${runtimeLabels[element]}: ${error.message}`); + } + } + }, + ); + this.changed.fire(); + if (failed.length > 0) { + vscode.window.showErrorMessage( + `Hyperbook: Some runtimes could not be downloaded. The preview keeps loading them from their CDN. ${failed.join(" ")}`, + ); + return false; + } + vscode.window.showInformationMessage( + `Hyperbook: Downloaded ${pending.map((element) => runtimeLabels[element]).join(", ")} to ${this.cacheDir}.`, + ); + return true; + } +} diff --git a/platforms/vscode/src/StatusBarItem.ts b/platforms/vscode/src/StatusBarItem.ts index 634c87530..3c2281da3 100644 --- a/platforms/vscode/src/StatusBarItem.ts +++ b/platforms/vscode/src/StatusBarItem.ts @@ -6,14 +6,14 @@ export default class StatusBarItem { statusBarItem: vscode.StatusBarItem; previewUtil: Preview; - constructor(context: vscode.ExtensionContext, utilities?: Preview) { + constructor(context: vscode.ExtensionContext, utilities: Preview) { this.statusBarItem = vscode.window.createStatusBarItem( vscode.StatusBarAlignment.Left ); this.statusBarItem.command = "hyperbook.sidePreview"; this.statusBarItem.tooltip = Constants.ExtensionConstants.STATUS_BAR_HTML_TOOLTIP; - this.previewUtil = (utilities && utilities) || new Preview(context); + this.previewUtil = utilities; } updateStatusbar() { diff --git a/platforms/vscode/src/extension.ts b/platforms/vscode/src/extension.ts index 08a62811f..f25b5fb70 100644 --- a/platforms/vscode/src/extension.ts +++ b/platforms/vscode/src/extension.ts @@ -5,9 +5,12 @@ import StatusBarItem from "./StatusBarItem"; import { hyperbook } from "@hyperbook/fs"; import { createNewHyperbook } from "./NewCommand"; import { HyperbookFoldingProvider } from "./FoldingProvider"; +import RuntimeAssets, { runtimeElements, runtimeLabels } from "./RuntimeAssets"; +import { DownloadableElement } from "@hyperbook/types"; export function activate(context: vscode.ExtensionContext) { - let preview = new Preview(context); + const runtimes = new RuntimeAssets(context.extensionPath); + let preview = new Preview(context, runtimes); let statusBarItem = new StatusBarItem(context, preview); statusBarItem.updateStatusbar(); @@ -169,6 +172,48 @@ Start writing your content here... }, ); + // Like `hyperbook assets fetch`: download element runtimes into the CLI's + // shared asset cache, so the preview works offline. + let disposableDownloadRuntimes = vscode.commands.registerCommand( + "hyperbook.downloadRuntimes", + async (elements?: DownloadableElement[]) => { + if (!elements) { + const items = await Promise.all( + runtimeElements.map(async (element) => { + const location = await runtimes.locate(element); + return { + label: runtimeLabels[element], + element, + description: + location.kind === "missing" ? undefined : "available offline", + picked: + location.kind === "missing" && + preview.usedElements.includes(element), + }; + }), + ); + const picked = await vscode.window.showQuickPick(items, { + canPickMany: true, + title: "Download Hyperbook element runtimes", + placeHolder: `Runtimes are stored in ${runtimes.cacheDir}`, + }); + if (!picked || picked.length === 0) { + return; + } + elements = picked.map((item) => item.element); + } + await runtimes.download(elements); + }, + ); + + // Like `hyperbook assets fetch --all`. + let disposableDownloadAllRuntimes = vscode.commands.registerCommand( + "hyperbook.downloadAllRuntimes", + async () => { + await runtimes.download(runtimeElements); + }, + ); + let disposableNew = vscode.commands.registerCommand( "hyperbook.new", async () => { @@ -184,6 +229,8 @@ Start writing your content here... context.subscriptions.push(disposableOpenIndexMd); context.subscriptions.push(disposableCreateNewPage); context.subscriptions.push(disposableOpenDocumentation); + context.subscriptions.push(disposableDownloadRuntimes); + context.subscriptions.push(disposableDownloadAllRuntimes); context.subscriptions.push(disposableStatusBar); // Completions diff --git a/platforms/vscode/webpack.extension.config.js b/platforms/vscode/webpack.extension.config.js index fdb8a2b7d..891410bbe 100644 --- a/platforms/vscode/webpack.extension.config.js +++ b/platforms/vscode/webpack.extension.config.js @@ -1,7 +1,7 @@ const path = require("path"); const CopyPlugin = require("copy-webpack-plugin"); -module.exports = { +module.exports = (_env, argv) => ({ target: "node", entry: path.join(__dirname, "src", "extension.ts"), resolve: { @@ -31,6 +31,20 @@ module.exports = { ), to: path.resolve(__dirname, "assets", "hyperbook"), }, + { + // Lists the verified runtime bundles of the matching CLI release. + // The preview downloads them into the CLI's shared asset cache. + from: path.resolve( + __dirname, + "node_modules", + "hyperbook", + "dist", + "asset-manifest.json", + ), + to: path.resolve(__dirname, "out", "asset-manifest.json"), + // Releases must ship it; development builds may skip the CLI build. + noErrorOnMissing: argv.mode !== "production", + }, { from: path.resolve( __dirname, @@ -62,4 +76,4 @@ module.exports = { }, ], }, -}; +}); diff --git a/pnpm-lock.yaml b/pnpm-lock.yaml index aba5255ee..e4cf6f6cf 100644 --- a/pnpm-lock.yaml +++ b/pnpm-lock.yaml @@ -114,6 +114,9 @@ importers: mime-types: specifier: ^3.0.2 version: 3.0.2 + tar: + specifier: 7.5.22 + version: 7.5.22 unist-util-visit: specifier: ^5.1.0 version: 5.1.0 @@ -656,6 +659,9 @@ importers: glob: specifier: 13.0.6 version: 13.0.6 + hyperbook: + specifier: workspace:* + version: link:../../packages/hyperbook node-polyfill-webpack-plugin: specifier: ^4.1.0 version: 4.1.0(webpack@5.109.2) diff --git a/scripts/buildPackage.mjs b/scripts/buildPackage.mjs index 722f323a8..66df26423 100644 --- a/scripts/buildPackage.mjs +++ b/scripts/buildPackage.mjs @@ -2,6 +2,7 @@ import chalk from "chalk"; import { build } from "esbuild"; import { existsSync, readFileSync } from "fs"; import { join } from "path"; +import { builtinModules } from "module"; const ignorePackages = []; @@ -48,6 +49,8 @@ export const buildPackage = async (path) => { } external.push("path"); external.push("fs"); + // Node built-ins stay imports, for packages that only run in Node. + external.push(...builtinModules, "node:*"); const platform = JSON.parse(packageJSON)?.platform || "browser"; diff --git a/website/de/book/configuration/book.md b/website/de/book/configuration/book.md index 602c43183..338113c10 100644 --- a/website/de/book/configuration/book.md +++ b/website/de/book/configuration/book.md @@ -143,9 +143,12 @@ SQL-IDE. Die kleinen Integrationsskripte und Styles von Hyperbook bleiben im Build. Die Hyperbook-Erweiterung für VS Code enthält diese Laufzeitumgebungen nicht. -Ihre Vorschau lädt sie vom Standard-CDN, sofern `cdn` nicht auf eine eigene URL -gesetzt ist. Für die Vorschau dieser Elemente ist daher eine Internetverbindung -nötig. +Ihre Vorschau verhält sich wie ein Build: Elemente mit `cdn` laden von diesem +CDN, die anderen verwenden die Laufzeitumgebungen aus dem Asset-Cache der CLI, +den die Erweiterung mitbenutzt. Eine noch nicht heruntergeladene +Laufzeitumgebung lädt vom Standard-CDN. Mit **Hyperbook: Download Element +Runtimes...** oder **Hyperbook: Download All Element Runtimes** lädst du +Laufzeitumgebungen in VS Code für die Offline-Vorschau herunter. Eine eigene URL muss auf den Inhalt des Verzeichnisses `__hyperbook_assets/directive-/` aus einem lokalen Build zeigen. diff --git a/website/de/book/elements/blockflow.md b/website/de/book/elements/blockflow.md index 3752c6e2c..61f2abf4c 100644 --- a/website/de/book/elements/blockflow.md +++ b/website/de/book/elements/blockflow.md @@ -10,8 +10,8 @@ lang: de Hyperbook enthält ein Blockflow-Browserpaket mit fester Version. Die CLI lädt es herunter, wenn ein Build Blockflow zum ersten Mal verwendet. Bücher mit einer dieser Direktiven verwenden eine gemeinsame lokale Kopie und sind für Editor und Player nicht auf blockflow.openpatch.org -angewiesen. Die VS Code-Erweiterung enthält das Paket nicht; ihre Vorschau lädt Blockflow von -blockflow.openpatch.org. Lokale Projekte und Tutorial-Medien +angewiesen. Die VS Code-Erweiterung enthält das Paket nicht. Ihre Vorschau verwendet die von der CLI +heruntergeladene Kopie oder lädt Blockflow bis zum Download von blockflow.openpatch.org. Lokale Projekte und Tutorial-Medien funktionieren offline, wenn das Buch lokal bereitgestellt wird. Externe Projekte, Inhalte aus der Scratch-Bibliothek und Netzwerk-Erweiterungen benötigen weiterhin Internetzugang. Lizenz und Quellcode-Link von Blockflow werden mit den Dateien ausgeliefert. diff --git a/website/en/book/configuration/book.md b/website/en/book/configuration/book.md index 2a66d5344..10532ebbb 100644 --- a/website/en/book/configuration/book.md +++ b/website/en/book/configuration/book.md @@ -144,9 +144,12 @@ Excalidraw's own packages on UNPKG for Excalidraw, and UNPKG's versioned Hyperbook assets for OpenSCAD, the Online IDE, and the SQL IDE. Hyperbook's small integration scripts and styles stay in the build. -The Hyperbook extension for VS Code does not include these runtimes. Its -preview loads them from the default CDN unless `cdn` is set to a custom URL, so -previewing these elements needs an internet connection. +The Hyperbook extension for VS Code does not include these runtimes. Its preview +resolves them like a build: elements with `cdn` load from that CDN, and the +others use the runtimes in the CLI's asset cache, which the extension shares. +A runtime that is not downloaded yet loads from its default CDN. Run +**Hyperbook: Download Element Runtimes...** or **Hyperbook: Download All +Element Runtimes** in VS Code to download runtimes for offline previews. A custom URL must point to the contents of the corresponding `__hyperbook_assets/directive-/` directory from a local build. Preserve diff --git a/website/en/book/elements/blockflow.md b/website/en/book/elements/blockflow.md index e215d5e9a..3ff9fb48d 100644 --- a/website/en/book/elements/blockflow.md +++ b/website/en/book/elements/blockflow.md @@ -9,8 +9,9 @@ permaid: blockflow Hyperbook includes a pinned Blockflow browser bundle. The CLI downloads it the first time a build uses Blockflow. Books that use either directive serve one shared local copy, so the editor and -player do not depend on blockflow.openpatch.org. The VS Code extension does not include the bundle; -its preview loads Blockflow from blockflow.openpatch.org. Local project files and tutorial media work offline when the book is +player do not depend on blockflow.openpatch.org. The VS Code extension does not include the bundle. +Its preview uses the CLI's downloaded copy, or loads Blockflow from blockflow.openpatch.org until it +is downloaded. Local project files and tutorial media work offline when the book is served locally; remote projects, Scratch library assets, and network extensions still need internet access. The bundled Blockflow license and source link are included with its assets. From 960d39ca853988b77a5a05ab2c9e619ff64bef13 Mon Sep 17 00:00:00 2001 From: Mike Barkmin Date: Sat, 10 Oct 2026 10:13:09 +0200 Subject: [PATCH 3/6] fix: exclude every large runtime and verify packaged consumers --- .../smaller-markdown-and-vscode-packages.md | 4 +- .../hyperbook/tests/packageContents.test.ts | 37 +++++ packages/markdown/README.md | 11 +- packages/markdown/package.json | 9 + packages/markdown/postbuild.mjs | 3 +- packages/markdown/src/elementAssets.ts | 8 +- .../tests/blockflowVscodePreview.test.ts | 32 ++++ packages/markdown/tests/elementCdn.test.ts | 16 +- .../markdown/tests/packageContents.test.ts | 45 +++-- platforms/vscode/webpack.extension.config.js | 157 ++++++++++-------- scripts/buildPackage.mjs | 10 ++ 11 files changed, 235 insertions(+), 97 deletions(-) create mode 100644 packages/hyperbook/tests/packageContents.test.ts diff --git a/.changeset/smaller-markdown-and-vscode-packages.md b/.changeset/smaller-markdown-and-vscode-packages.md index 6f3276ef9..607ccb9aa 100644 --- a/.changeset/smaller-markdown-and-vscode-packages.md +++ b/.changeset/smaller-markdown-and-vscode-packages.md @@ -8,8 +8,10 @@ Keep the Markdown npm package and the VS Code extension small by leaving out large element runtimes. -The Markdown package no longer includes the Pyodide, GeoGebra, Typst, Blockflow and Excalidraw runtimes; it keeps their integration scripts and styles. With `cdn: true`, Blockflow now loads from `blockflow.openpatch.org` and Excalidraw from its own packages on UNPKG, so every default CDN keeps working without these files. The OpenSCAD, Online IDE and SQL IDE runtimes stay in the package because their default CDN is this package on UNPKG. +The Markdown package no longer includes any of the eight large element runtimes; it keeps their integration scripts and styles. With `cdn: true`, Blockflow now loads from `blockflow.openpatch.org` and Excalidraw from its own packages on UNPKG. OpenSCAD, Online IDE and SQL IDE use the published `@hyperbook/markdown@0.85.0` runtimes on UNPKG, independently of the renderer version, so their default CDNs keep working after the runtimes are removed from new packages. The VS Code extension no longer includes any of the eight large runtimes. Its preview resolves them like a Hyperbook build: elements with a `cdn` setting load from that CDN, and the others use the runtimes in the Hyperbook CLI's asset cache, which the extension now shares. Runtimes that are not downloaded yet load from their default CDN, and the preview offers to download them. The new commands "Download Element Runtimes..." and "Download All Element Runtimes" work like `hyperbook assets fetch` and `hyperbook assets fetch --all`. `@hyperbook/fs` now provides `AssetManager` and `assetCacheDir`, which the CLI and the VS Code extension share. `@hyperbook/types` exports `elementRuntimeFiles`, the files that show an element's runtime is available locally. + +The built Markdown ESM package also supports direct Node imports while remaining compatible with the CLI and extension bundles. diff --git a/packages/hyperbook/tests/packageContents.test.ts b/packages/hyperbook/tests/packageContents.test.ts new file mode 100644 index 000000000..77d45a4e2 --- /dev/null +++ b/packages/hyperbook/tests/packageContents.test.ts @@ -0,0 +1,37 @@ +import { execFileSync } from "node:child_process"; +import fs from "node:fs/promises"; +import path from "node:path"; +import { + DownloadableElement, + elementCoreFiles, + elementRuntimeFiles, +} from "@hyperbook/types"; +import { expect, it } from "vitest"; + +const dist = path.join(__dirname, "..", "dist"); + +it("starts the bundled CLI with its bundled Markdown renderer", () => { + const help = execFileSync( + process.execPath, + [path.join(dist, "index.js"), "--help"], + { encoding: "utf8" }, + ); + expect(help).toContain("Usage:"); + expect(help).toContain("assets"); +}); + +it("ships integration files and a download manifest without large runtimes", async () => { + const manifest = JSON.parse( + await fs.readFile(path.join(dist, "asset-manifest.json"), "utf8"), + ); + for (const element of Object.keys(elementCoreFiles) as DownloadableElement[]) { + expect(manifest.bundles).toHaveProperty(element); + const directory = path.join(dist, "assets", `directive-${element}`); + for (const file of elementCoreFiles[element]) + expect((await fs.stat(path.join(directory, file))).isFile()).toBe(true); + for (const file of elementRuntimeFiles[element]) + await expect(fs.stat(path.join(directory, file))).rejects.toMatchObject({ + code: "ENOENT", + }); + } +}); diff --git a/packages/markdown/README.md b/packages/markdown/README.md index 4dca9eacb..08cbe01a4 100644 --- a/packages/markdown/README.md +++ b/packages/markdown/README.md @@ -53,9 +53,14 @@ their integration scripts and styles: | `geogebra` | GeoGebra's server | | `blockflow` | `blockflow.openpatch.org` | | `excalidraw` | Excalidraw's own packages on UNPKG | - -The OpenSCAD, Online IDE, and SQL IDE runtimes stay in the package, because -their default CDN is this package on UNPKG. +| `onlineide` | Runtime from `@hyperbook/markdown@0.85.0` on UNPKG | +| `sqlide` | Runtime from `@hyperbook/markdown@0.85.0` on UNPKG | +| `openscad` | Runtime from `@hyperbook/markdown@0.85.0` on UNPKG | + +All eight large runtimes are excluded from the npm package. The OpenSCAD, +Online IDE, and SQL IDE default CDNs are pinned to the last published Markdown +release containing them, independently of the current renderer version. +Workspace builds still prepare the full asset tree for publishing CLI bundles. To serve a missing runtime, choose one of these options: diff --git a/packages/markdown/package.json b/packages/markdown/package.json index 75bd56707..42ac935fb 100644 --- a/packages/markdown/package.json +++ b/packages/markdown/package.json @@ -15,9 +15,18 @@ "files": [ "dist", "!dist/assets/directive-blockflow/**", + "!dist/assets/directive-onlineide/**", + "dist/assets/directive-onlineide/style.css", + "!dist/assets/directive-sqlide/**", + "dist/assets/directive-sqlide/client.js", + "dist/assets/directive-sqlide/style.css", "!dist/assets/directive-excalidraw/**", "dist/assets/directive-excalidraw/client.js", "dist/assets/directive-excalidraw/style.css", + "!dist/assets/directive-openscad/**", + "dist/assets/directive-openscad/client.js", + "dist/assets/directive-openscad/worker.js", + "dist/assets/directive-openscad/style.css", "!dist/assets/directive-geogebra/**", "dist/assets/directive-geogebra/client.js", "dist/assets/directive-geogebra/geogebra-web-component.js", diff --git a/packages/markdown/postbuild.mjs b/packages/markdown/postbuild.mjs index 23c5e04c0..b3af460ac 100644 --- a/packages/markdown/postbuild.mjs +++ b/packages/markdown/postbuild.mjs @@ -140,7 +140,8 @@ async function postbuild() { } // These files are served locally by the elements. The CLI publishes them in - // optional bundles; Markdown and VS Code keep their complete asset trees. + // optional bundles. The Markdown npm package and VS Code extension exclude + // the runtimes, retaining only their small integration scripts and styles. await installRuntimeAssets(JSON.parse(await readFile("runtime-assets.json", "utf8"))); const assets = [ diff --git a/packages/markdown/src/elementAssets.ts b/packages/markdown/src/elementAssets.ts index 50351ac63..5d965b11e 100644 --- a/packages/markdown/src/elementAssets.ts +++ b/packages/markdown/src/elementAssets.ts @@ -3,13 +3,17 @@ import { HyperbookContext, elementCdn, } from "@hyperbook/types"; -import packageJson from "../package.json"; import excalidrawPackageJson from "../../web-component-excalidraw/package.json"; import runtimeAssets from "../runtime-assets.json"; /** Hosted Blockflow app; the npm package does not include Blockflow's bundle. */ const blockflowCdn = "https://blockflow.openpatch.org/"; +// The last published Markdown release that includes these runtimes. Keep this +// independent of the renderer version: new packages omit every large runtime. +const legacyRuntimeCdn = + "https://unpkg.com/@hyperbook/markdown@0.85.0/dist/assets/"; + /** Resolve payload files; Hyperbook's integration scripts stay local. */ export function elementAssetUrl( ctx: HyperbookContext, @@ -72,5 +76,5 @@ export function elementAssetUrl( ? `${base}index.css` : new URL(file, base).href; } - return `https://unpkg.com/@hyperbook/markdown@${packageJson.version}/dist/assets/directive-${element}/${file}`; + return `${legacyRuntimeCdn}directive-${element}/${file}`; } diff --git a/packages/markdown/tests/blockflowVscodePreview.test.ts b/packages/markdown/tests/blockflowVscodePreview.test.ts index 04474c200..458577992 100644 --- a/packages/markdown/tests/blockflowVscodePreview.test.ts +++ b/packages/markdown/tests/blockflowVscodePreview.test.ts @@ -6,9 +6,11 @@ import { writeFileSync, } from "node:fs"; import { createRequire } from "node:module"; +import { execFileSync } from "node:child_process"; import { tmpdir } from "node:os"; import path from "node:path"; import { runInNewContext } from "node:vm"; +import { fileURLToPath } from "node:url"; import ts from "typescript"; import { DownloadableElement, @@ -22,6 +24,36 @@ import { expect, it } from "vitest"; const vscodePlatform = new URL("../../../platforms/vscode/", import.meta.url); const elements = Object.keys(elementCoreFiles) as DownloadableElement[]; +it("loads the compiled extension in a Node extension host", () => { + execFileSync( + process.execPath, + [ + "-e", + ` + const fs = require('node:fs'); + const Module = require('node:module'); + const filename = process.argv[1]; + const extension = new Module(filename); + extension.filename = filename; + extension.paths = Module._nodeModulePaths(require('node:path').dirname(filename)); + const vscode = { languages: { createDiagnosticCollection: () => ({}) } }; + extension.require = name => name === 'vscode' + ? vscode : Module.prototype.require.call(extension, name); + try { + extension._compile(fs.readFileSync(filename, 'utf8'), filename); + if (typeof extension.exports.activate !== 'function') + throw new Error('The compiled extension does not export activate.'); + } catch (error) { + console.error(error.message); + process.exitCode = 1; + } + `, + fileURLToPath(new URL("out/extension.js", vscodePlatform)), + ], + { stdio: "pipe" }, + ); +}); + type RuntimeLocation = | { kind: "bundled" } | { kind: "cached"; directory: string } diff --git a/packages/markdown/tests/elementCdn.test.ts b/packages/markdown/tests/elementCdn.test.ts index bd8912bf3..464a91c2a 100644 --- a/packages/markdown/tests/elementCdn.test.ts +++ b/packages/markdown/tests/elementCdn.test.ts @@ -2,7 +2,6 @@ import { describe, expect, it } from "vitest"; import { DownloadableElement, elementCdn } from "@hyperbook/types"; import { process as processMarkdown } from "../src/process"; import { elementAssetUrl } from "../src/elementAssets"; -import packageJson from "../package.json"; import excalidrawPackageJson from "../../web-component-excalidraw/package.json"; import { ctx } from "./mock"; @@ -57,7 +56,7 @@ describe.each(elements)( element as "pyide" | "typst" | "geogebra" | "blockflow" | "excalidraw" ] || - `https://unpkg.com/@hyperbook/markdown@${packageJson.version}/dist/assets/directive-${element}/` + `https://unpkg.com/@hyperbook/markdown@0.85.0/dist/assets/directive-${element}/` : "http://assets.example.com/runtime/", ); if (element === "blockflow") @@ -144,3 +143,16 @@ it("loads Blockflow and Excalidraw from their upstream CDNs", () => { // An unpkg URL needs an exact version to stay stable. expect(excalidraw).toMatch(/@\d+\.\d+\.\d+\/dist/); }); + +it.each(["onlineide", "sqlide", "openscad"] as const)( + "keeps %s's default CDN pinned to a release containing its runtime", + (element: DownloadableElement) => { + const configured = { + ...ctx, + config: { ...ctx.config, elements: { [element]: { cdn: true } } }, + }; + expect(elementAssetUrl(configured, element, "")).toBe( + `https://unpkg.com/@hyperbook/markdown@0.85.0/dist/assets/directive-${element}/`, + ); + }, +); diff --git a/packages/markdown/tests/packageContents.test.ts b/packages/markdown/tests/packageContents.test.ts index 486dc253c..a42a28dfe 100644 --- a/packages/markdown/tests/packageContents.test.ts +++ b/packages/markdown/tests/packageContents.test.ts @@ -8,9 +8,7 @@ import { elementRuntimeFiles, } from "@hyperbook/types"; import { expect, it } from "vitest"; -import { elementAssetUrl } from "../src/elementAssets"; import packageJson from "../package.json"; -import { ctx } from "./mock"; const elements = Object.keys(elementCoreFiles) as DownloadableElement[]; @@ -48,29 +46,42 @@ function packedFiles(): string[] { } } -it("publishes only the runtimes its own default CDN serves", () => { +it("publishes integration files without any large element runtimes", () => { const files = packedFiles(); expect(files).toContain("dist/index.js"); expect(files).toContain("dist/assets/directive-blockflow-player/client.js"); for (const element of elements) { for (const file of elementCoreFiles[element]) expect(files).toContain(`dist/assets/directive-${element}/${file}`); - const configured = { - ...ctx, - config: { ...ctx.config, elements: { [element]: { cdn: true } } }, - }; for (const file of elementRuntimeFiles[element]) { - // Elements whose default CDN is this package on unpkg must ship their - // runtime. Every other runtime stays out to keep the package small. - // Directives resolve nested runtime files through their top folder. - const requested = file.includes("/") ? file.replace(/\/.*/, "/") : file; - const servedByPackage = elementAssetUrl( - configured, - element, - requested, - ).startsWith(`https://unpkg.com/${packageJson.name}@`); const packed = files.includes(`dist/assets/directive-${element}/${file}`); - expect(packed, `${element}: ${file}`).toBe(servedByPackage); + expect(packed, `${element}: ${file}`).toBe(false); } } }, 60_000); + +it("renders with the built ESM package in plain Node", () => { + execFileSync( + process.execPath, + [ + "--input-type=module", + "-e", + ` + const { process: render } = await import(process.argv[1]); + const result = await render('# Standalone renderer', { + root: process.cwd(), + config: { name: 'Standalone' }, + makeUrl: parts => Array.isArray(parts) ? parts.join('/') : parts, + navigation: { + current: { name: 'Standalone' }, next: null, previous: null, + pages: [], sections: [], glossary: [] + } + }); + if (!String(result.value).includes('Standalone renderer')) + throw new Error('The built package did not render the Markdown.'); + `, + new URL("../dist/index.js", import.meta.url).href, + ], + { stdio: "pipe" }, + ); +}); diff --git a/platforms/vscode/webpack.extension.config.js b/platforms/vscode/webpack.extension.config.js index 891410bbe..fab689ecd 100644 --- a/platforms/vscode/webpack.extension.config.js +++ b/platforms/vscode/webpack.extension.config.js @@ -1,79 +1,94 @@ const path = require("path"); +const fs = require("fs"); const CopyPlugin = require("copy-webpack-plugin"); -module.exports = (_env, argv) => ({ - target: "node", - entry: path.join(__dirname, "src", "extension.ts"), - resolve: { - extensions: [".ts", ".js"], - }, - output: { - filename: "extension.js", - libraryTarget: "commonjs2", - path: path.resolve(__dirname, "out"), - devtoolModuleFilenameTemplate: "../[resource-path]", - }, - devtool: "source-map", - externals: { - vscode: "commonjs vscode", // the vscode-module is created on-the-fly and must be excluded. Add other modules that cannot be webpack'ed, 📖 -> https://webpack.js.org/configuration/externals/ - }, - plugins: [ - new CopyPlugin({ - patterns: [ - { - from: path.resolve( - __dirname, - "node_modules", - "@hyperbook", - "markdown", - "dist", - "assets", - ), - to: path.resolve(__dirname, "assets", "hyperbook"), - }, +module.exports = (_env, argv) => { + // Remove previous build output, which may still contain bundled runtimes. + fs.rmSync(path.resolve(__dirname, "assets", "hyperbook"), { + recursive: true, + force: true, + }); + return { + target: "node", + entry: path.join(__dirname, "src", "extension.ts"), + resolve: { + extensions: [".ts", ".js"], + }, + output: { + filename: "extension.js", + libraryTarget: "commonjs2", + path: path.resolve(__dirname, "out"), + devtoolModuleFilenameTemplate: "../[resource-path]", + }, + devtool: "source-map", + externals: { + vscode: "commonjs vscode", // the vscode-module is created on-the-fly and must be excluded. Add other modules that cannot be webpack'ed, 📖 -> https://webpack.js.org/configuration/externals/ + }, + plugins: [ + new CopyPlugin({ + patterns: [ + { + from: path.resolve( + __dirname, + "node_modules", + // The CLI build retains exactly the small integration files. + "hyperbook", + "dist", + "assets", + ), + to: path.resolve(__dirname, "assets", "hyperbook"), + }, + { + // Lists the verified runtime bundles of the matching CLI release. + // The preview downloads them into the CLI's shared asset cache. + from: path.resolve( + __dirname, + "node_modules", + "hyperbook", + "dist", + "asset-manifest.json", + ), + to: path.resolve(__dirname, "out", "asset-manifest.json"), + // Releases must ship it; development builds may skip the CLI build. + noErrorOnMissing: argv.mode !== "production", + }, + { + from: path.resolve( + __dirname, + "node_modules", + "create-hyperbook", + "dist", + "templates", + ), + to: path.resolve(__dirname, "out", "templates"), + }, + ], + }), + ], + module: { + rules: [ { - // Lists the verified runtime bundles of the matching CLI release. - // The preview downloads them into the CLI's shared asset cache. - from: path.resolve( - __dirname, - "node_modules", - "hyperbook", - "dist", - "asset-manifest.json", - ), - to: path.resolve(__dirname, "out", "asset-manifest.json"), - // Releases must ship it; development builds may skip the CLI build. - noErrorOnMissing: argv.mode !== "production", + test: /[\\/]markdown[\\/]dist[\\/]index\.js$/, + // esbuild already bundled the CommonJS dependencies. Its remaining + // dynamic requires use createRequire for Node built-ins; webpack's + // CommonJS parser would replace them with an empty module context. + parser: { commonjs: false, createRequire: false }, }, { - from: path.resolve( - __dirname, - "node_modules", - "create-hyperbook", - "dist", - "templates", - ), - to: path.resolve(__dirname, "out", "templates"), - }, - ], - }), - ], - module: { - rules: [ - { - test: /\.ts$/, - exclude: /node_modules/, - use: [ - { - loader: "ts-loader", - options: { - compilerOptions: { - module: "es6", // override `tsconfig.json` so that TypeScript emits native JavaScript modules. + test: /\.ts$/, + exclude: /node_modules/, + use: [ + { + loader: "ts-loader", + options: { + compilerOptions: { + module: "es6", // override `tsconfig.json` so that TypeScript emits native JavaScript modules. + }, }, }, - }, - ], - }, - ], - }, -}); + ], + }, + ], + }, + }; +}; diff --git a/scripts/buildPackage.mjs b/scripts/buildPackage.mjs index 66df26423..59b60c0fa 100644 --- a/scripts/buildPackage.mjs +++ b/scripts/buildPackage.mjs @@ -63,6 +63,16 @@ export const buildPackage = async (path) => { bundle: true, platform, external, + // ESM hosts do not provide `require` for bundled CommonJS dependencies. + ...(platform === "node" + ? { + // Downstream CommonJS bundles provide __filename. Using it avoids + // their import.meta.url shim calling a shadowed require during setup. + banner: { + js: 'import { createRequire as __hyperbookCreateRequire } from "node:module"; const require = __hyperbookCreateRequire(typeof __filename === "string" ? __filename : import.meta.url);', + }, + } + : {}), }; // await build({ From ad1572f2bf7db83dedc97215b33dfcde07bcc39d Mon Sep 17 00:00:00 2001 From: Mike Barkmin Date: Sat, 10 Oct 2026 11:06:39 +0200 Subject: [PATCH 4/6] ci: publish versioned OpenSCAD runtime directly to Cloudflare R2 --- .github/workflows/changeset-version.yml | 4 + .github/workflows/publish-openscad-cdn.yml | 48 +++++ .github/workflows/pull-request.yml | 2 + docs/cdn-releases.md | 80 ++++++++ package.json | 1 + packages/markdown/openscad-config.json | 1 + scripts/cdn-cors.json | 13 ++ scripts/prepare-openscad-cdn.mjs | 59 ++++++ scripts/publish-cdn.mjs | 192 ++++++++++++++++++ scripts/publish-cdn.test.mjs | 219 +++++++++++++++++++++ scripts/verify-cdn.mjs | 68 +++++++ 11 files changed, 687 insertions(+) create mode 100644 .github/workflows/publish-openscad-cdn.yml create mode 100644 docs/cdn-releases.md create mode 100644 scripts/cdn-cors.json create mode 100644 scripts/prepare-openscad-cdn.mjs create mode 100644 scripts/publish-cdn.mjs create mode 100644 scripts/publish-cdn.test.mjs create mode 100644 scripts/verify-cdn.mjs diff --git a/.github/workflows/changeset-version.yml b/.github/workflows/changeset-version.yml index 61c7a3444..195a69247 100644 --- a/.github/workflows/changeset-version.yml +++ b/.github/workflows/changeset-version.yml @@ -13,7 +13,11 @@ permissions: packages: write jobs: + publish-openscad-cdn: + uses: ./.github/workflows/publish-openscad-cdn.yml + secrets: inherit release: + needs: publish-openscad-cdn name: Release runs-on: ubuntu-latest steps: diff --git a/.github/workflows/publish-openscad-cdn.yml b/.github/workflows/publish-openscad-cdn.yml new file mode 100644 index 000000000..85d98ccc8 --- /dev/null +++ b/.github/workflows/publish-openscad-cdn.yml @@ -0,0 +1,48 @@ +name: Publish OpenSCAD runtime to R2 +on: + # Bootstrap the first runtime from the migration branch; removed after upload. + push: + branches: [ccr-dc12b6f7-p2rfkj] + paths: [.github/workflows/publish-openscad-cdn.yml] + workflow_dispatch: + workflow_call: +permissions: + contents: read +concurrency: + group: cdn-openscad + cancel-in-progress: false +jobs: + publish: + runs-on: ubuntu-latest + timeout-minutes: 15 + env: + CLOUDFLARE_API_TOKEN: ${{ secrets.CLOUDFLARE_R2_CDN }} + CLOUDFLARE_ACCOUNT_ID: ${{ vars.CLOUDFLARE_ACCOUNT_ID || secrets.CLOUDFLARE_ACCOUNT_ID }} + WRANGLER_SEND_METRICS: "false" + steps: + - uses: actions/checkout@v6 + - uses: pnpm/action-setup@v6 + with: + version: 11.3.0 + - uses: actions/setup-node@v6 + with: + node-version: "24" + cache: pnpm + - name: Install runtime downloader dependencies + run: pnpm install --frozen-lockfile --filter @hyperbook/markdown... + - run: pnpm test:cdn + - name: Install Wrangler + run: npm install --global wrangler@4.149.0 + - name: Check Cloudflare configuration + run: | + test -n "$CLOUDFLARE_API_TOKEN" || { echo 'Missing CLOUDFLARE_R2_CDN Actions secret'; exit 1; } + test -n "$CLOUDFLARE_ACCOUNT_ID" || { echo 'Missing CLOUDFLARE_ACCOUNT_ID Actions variable or secret'; exit 1; } + - name: Prepare upstream WASM, fonts and libraries + run: node scripts/prepare-openscad-cdn.mjs + - name: Allow browser access to runtime files + run: wrangler r2 bucket cors set cdn --file scripts/cdn-cors.json --force + - name: Publish and verify immutable runtime files + run: | + version=$(node -p 'JSON.parse(require("fs").readFileSync("packages/markdown/openscad-config.json", "utf8")).wasmBuild.cdnVersion') + node scripts/publish-cdn.mjs openscad "$version" .cache/cdn-openscad/runtime + node scripts/verify-cdn.mjs openscad "$version" .cache/cdn-openscad/runtime diff --git a/.github/workflows/pull-request.yml b/.github/workflows/pull-request.yml index 6af7fe848..54fb89f80 100644 --- a/.github/workflows/pull-request.yml +++ b/.github/workflows/pull-request.yml @@ -24,3 +24,5 @@ jobs: run: pnpm build - name: Test packages run: pnpm test + - name: Test CDN publishing + run: pnpm test:cdn diff --git a/docs/cdn-releases.md b/docs/cdn-releases.md new file mode 100644 index 000000000..2e7f0c081 --- /dev/null +++ b/docs/cdn-releases.md @@ -0,0 +1,80 @@ +# Runtime releases on cdn.openpatch.org + +Cloudflare R2 bucket `cdn` serves immutable runtime releases through the custom +domain `cdn.openpatch.org`. Each repository owns its release uploads: + +| Repository | Object prefix | Trigger | +| ---------------------- | ---------------------------------- | ------------------------------------------------------------- | +| `openpatch/online-ide` | `onlineide//include/` | Published GitHub release, after attaching `dist-embedded.zip` | +| `openpatch/sql-ide` | `sqlide//include/` | Published GitHub release, after attaching `dist-embedded.zip` | +| `openpatch/hyperbook` | `openscad//` | Hyperbook release workflow, before publishing packages | + +All workflows use Wrangler and the Cloudflare API token in the organization +Actions secret `CLOUDFLARE_R2_CDN`. Set `CLOUDFLARE_ACCOUNT_ID` as an organization +Actions variable or secret. Grant both to the publishing repositories. S3 access +keys are unnecessary. The token needs remote R2 object read/write and bucket +CORS configuration permissions for `cdn`. + +The IDE publishers download the ZIP already attached to the release and verify +GitHub's SHA-256 digest. HTML examples and source maps are excluded. OpenSCAD is +prepared directly from the upstream WASM distribution and the pinned font and +library downloads in `packages/markdown/runtime-assets.json`, with SHA-256 +verification. It does not read a published Markdown package. + +Every object gets its correct MIME type and +`Cache-Control: public, max-age=31536000, immutable`. The publisher retains old +versions, refuses conflicting files, resumes interrupted uploads and writes +`manifest.json` last. Every upload is followed by a public-domain check of all +file checksums, MIME types, cache headers and CORS. A same-version rerun is safe. +Repository concurrency groups prevent overlapping uploads of the same release; +keep each prefix owned by its repository. + +The workflows apply `scripts/cdn-cors.json` to allow public GET/HEAD requests +from any origin. This is needed for JavaScript modules, workers, WASM and fonts, +including VS Code webview previews. There is no public upload permission. + +## First upload and retries + +In each IDE repository, open **Actions → Release Embedded Build → Run workflow** +and enter an existing published tag in `cdn_release_tag`. Leaving that input +empty retains the existing manual build/artifact behavior. The equivalent CLI +commands are: + +```sh +gh workflow run release-embedded.yml --repo openpatch/online-ide -f cdn_release_tag=v2.2.1-hyperbook.28 +gh workflow run release-embedded.yml --repo openpatch/sql-ide -f cdn_release_tag=v2.0.0-hyperbook.4 +gh workflow run publish-openscad-cdn.yml --repo openpatch/hyperbook +``` + +Once the CDN uploads pass verification, Hyperbook can pin these base URLs: + +```json +{ + "elements": { + "onlineide": { + "cdn": "https://cdn.openpatch.org/onlineide/v2.2.1-hyperbook.28/" + }, + "sqlide": { "cdn": "https://cdn.openpatch.org/sqlide/v2.0.0-hyperbook.4/" }, + "openscad": { "cdn": "https://cdn.openpatch.org/openscad/2026.10.08-1/" } + } +} +``` + +Keep `include/` below the IDE base URLs. Relative chunks, workers and other +assets resolve there automatically. Pin a release version; changing a moving +`latest` alias could break older books. + +## Updating OpenSCAD + +Update the upstream URL and SHA-256 in `openscad-config.json`, then increment +`wasmBuild.cdnVersion`. Increment it when changing font or library downloads as +well. Publish that version before using it as Hyperbook's default. The build +continues to keep the small Hyperbook integration scripts local. + +## Cloudflare caching + +Let the custom domain respect object Cache-Control headers. Do not cache 404 +responses: a version should not remain missing after its first upload. If the +hostname already served cached responses before the CORS policy was applied, +purge those URLs or the hostname once, following the +[Cloudflare CORS guidance](https://developers.cloudflare.com/r2/buckets/cors/#use-cors-with-a-custom-domain). diff --git a/package.json b/package.json index 3b82f2a4a..fdcdad007 100644 --- a/package.json +++ b/package.json @@ -10,6 +10,7 @@ "build": "pnpm -r build", "build:vercel": "pnpm --filter=!hyperbook-studio build", "test": "CI=true pnpm -r test", + "test:cdn": "node --test scripts/publish-cdn.test.mjs", "lint": "pnpm -r lint", "hyperbook:build": "pnpm --filter hyperbook build", "version-packages": "changeset version", diff --git a/packages/markdown/openscad-config.json b/packages/markdown/openscad-config.json index 55632e28f..a970f5fbf 100644 --- a/packages/markdown/openscad-config.json +++ b/packages/markdown/openscad-config.json @@ -1,5 +1,6 @@ { "wasmBuild": { + "cdnVersion": "2026.10.08-1", "url": "https://files.openscad.org/snapshots/OpenSCAD-2026.10.08-WebAssembly-web.zip", "sha256": "274c0e725733efea09e99b1186ebafe50ce799b8c8353ad59d9e72a6b9c76ab1", "target": "directive-openscad", diff --git a/scripts/cdn-cors.json b/scripts/cdn-cors.json new file mode 100644 index 000000000..1f31ff5c2 --- /dev/null +++ b/scripts/cdn-cors.json @@ -0,0 +1,13 @@ +{ + "rules": [ + { + "allowed": { + "origins": ["*"], + "methods": ["GET", "HEAD"], + "headers": ["*"] + }, + "exposeHeaders": ["ETag", "Content-Length"], + "maxAgeSeconds": 86400 + } + ] +} diff --git a/scripts/prepare-openscad-cdn.mjs b/scripts/prepare-openscad-cdn.mjs new file mode 100644 index 000000000..d5dcd4856 --- /dev/null +++ b/scripts/prepare-openscad-cdn.mjs @@ -0,0 +1,59 @@ +import fs from "node:fs/promises"; +import path from "node:path"; +import { fileURLToPath } from "node:url"; +import { installRuntimeAssets } from "../packages/markdown/runtime-assets.mjs"; +import { releasePrefix } from "./publish-cdn.mjs"; + +const root = path.resolve(path.dirname(fileURLToPath(import.meta.url)), ".."); + +// Download the upstream WASM distribution, fonts and libraries directly. No +// published Markdown package or compiled Hyperbook integration is needed. +export async function prepareOpenscad( + directory = path.join(root, ".cache/cdn-openscad/runtime"), +) { + const { wasmBuild } = JSON.parse( + await fs.readFile( + path.join(root, "packages/markdown/openscad-config.json"), + "utf8", + ), + ); + releasePrefix("openscad", wasmBuild.cdnVersion); + const assets = JSON.parse( + await fs.readFile( + path.join(root, "packages/markdown/runtime-assets.json"), + "utf8", + ), + ); + const extra = assets + .filter((asset) => asset.target.startsWith(`${wasmBuild.target}/`)) + .map((asset) => ({ + ...asset, + target: asset.target.slice(wasmBuild.target.length + 1), + })); + await fs.rm(directory, { recursive: true, force: true }); + await installRuntimeAssets( + [ + { + url: wasmBuild.url, + sha256: wasmBuild.sha256, + format: "zip", + target: ".", + }, + ...extra, + ], + directory, + path.join(root, "packages/markdown/.cache/runtime-assets"), + ); + for (const file of wasmBuild.files) + await fs.access(path.join(directory, file)); + console.log( + `Prepared OpenSCAD ${wasmBuild.cdnVersion} from checksum-verified upstream assets`, + ); + return wasmBuild.cdnVersion; +} + +if ( + process.argv[1] && + path.resolve(process.argv[1]) === fileURLToPath(import.meta.url) +) + await prepareOpenscad(); diff --git a/scripts/publish-cdn.mjs b/scripts/publish-cdn.mjs new file mode 100644 index 000000000..7b15df322 --- /dev/null +++ b/scripts/publish-cdn.mjs @@ -0,0 +1,192 @@ +import { createHash } from "node:crypto"; +import { execFile } from "node:child_process"; +import fs from "node:fs/promises"; +import os from "node:os"; +import path from "node:path"; +import { fileURLToPath } from "node:url"; +import { promisify } from "node:util"; + +// Kept in each publishing repository so releases do not depend on another repo. +const run = promisify(execFile); +const hash = (bytes) => createHash("sha256").update(bytes).digest("hex"); +const cacheControl = "public, max-age=31536000, immutable"; +const contentTypes = { + ".js": "text/javascript", + ".mjs": "text/javascript", + ".css": "text/css", + ".wasm": "application/wasm", + ".json": "application/json", + ".zip": "application/zip", + ".svg": "image/svg+xml", + ".png": "image/png", + ".jpg": "image/jpeg", + ".jpeg": "image/jpeg", + ".gif": "image/gif", + ".webp": "image/webp", + ".ico": "image/x-icon", + ".woff": "font/woff", + ".woff2": "font/woff2", + ".ttf": "font/ttf", + ".otf": "font/otf", + ".mp3": "audio/mpeg", + ".ogg": "audio/ogg", + ".wav": "audio/wav", + ".txt": "text/plain", +}; + +export function releasePrefix(element, version) { + if ( + !["onlineide", "sqlide", "openscad"].includes(element) || + !/^[a-zA-Z0-9][a-zA-Z0-9._-]*$/.test(version) || + version === "latest" + ) { + throw new Error( + "Expected a supported element and an immutable release version", + ); + } + return `${element}/${version}`; +} + +export async function runtimeManifest(directory) { + const files = {}; + async function visit(folder, prefix = "") { + const entries = await fs.readdir(folder, { withFileTypes: true }); + entries.sort((a, b) => a.name.localeCompare(b.name, "en")); + for (const entry of entries) { + const relative = `${prefix}${entry.name}`; + if (entry.isSymbolicLink()) + throw new Error(`Runtime contains a symlink: ${relative}`); + if (entry.isDirectory()) + await visit(path.join(folder, entry.name), `${relative}/`); + else if (entry.isFile() && !/\.(map|html)$/i.test(relative)) { + if (relative === "manifest.json") + throw new Error("manifest.json is reserved for publication metadata"); + const bytes = await fs.readFile(path.join(folder, entry.name)); + files[relative] = { + sha256: hash(bytes), + bytes: bytes.length, + contentType: + contentTypes[path.extname(relative).toLowerCase()] || + "application/octet-stream", + }; + } + } + } + await visit(directory); + if (!Object.keys(files).length) + throw new Error("No runtime files to publish"); + return { schema: 1, files }; +} + +export function wranglerStore(bucket = "cdn", runImpl = run) { + return { + async get(key) { + const temporary = await fs.mkdtemp(path.join(os.tmpdir(), "cdn-read-")); + const file = path.join(temporary, "object"); + try { + await runImpl("wrangler", [ + "r2", + "object", + "get", + `${bucket}/${key}`, + "--remote", + "--file", + file, + ]); + return await fs.readFile(file); + } catch (error) { + if ( + /The specified key does not exist/.test( + `${error.stdout}\n${error.stderr}`, + ) + ) + return null; + throw error; + } finally { + await fs.rm(temporary, { recursive: true, force: true }); + } + }, + async put(key, file, contentType) { + await runImpl("wrangler", [ + "r2", + "object", + "put", + `${bucket}/${key}`, + "--remote", + "--file", + file, + "--content-type", + contentType, + "--cache-control", + cacheControl, + ]); + }, + }; +} + +export async function publishCdn({ + directory, + element, + version, + store = wranglerStore(), +}) { + const prefix = releasePrefix(element, version); + const manifest = await runtimeManifest(directory); + const bytes = Buffer.from(JSON.stringify(manifest, null, 2) + "\n"); + const manifestKey = `${prefix}/manifest.json`; + const previous = await store.get(manifestKey); + if (previous) { + if (!bytes.equals(previous)) + throw new Error(`Published runtime is immutable: ${prefix}`); + console.log(`${prefix} is already published and matches the release`); + return manifest; + } + // A failed upload can be resumed. Existing objects must match before reuse. + // The manifest is written last, after all payload objects have been uploaded. + const files = Object.entries(manifest.files); + let next = 0; + const uploads = await Promise.allSettled( + Array.from({ length: Math.min(4, files.length) }, async () => { + while (next < files.length) { + const [relative, metadata] = files[next++]; + const key = `${prefix}/${relative}`; + const previous = await store.get(key); + if (previous) { + if (hash(previous) !== metadata.sha256) + throw new Error(`Existing runtime object differs: ${key}`); + } else { + const file = path.join(directory, relative); + if (hash(await fs.readFile(file)) !== metadata.sha256) + throw new Error(`Runtime changed during upload: ${relative}`); + await store.put(key, file, metadata.contentType); + } + } + }), + ); + const failed = uploads.find((result) => result.status === "rejected"); + if (failed) throw failed.reason; + const temporary = await fs.mkdtemp(path.join(os.tmpdir(), "cdn-manifest-")); + try { + const file = path.join(temporary, "manifest.json"); + await fs.writeFile(file, bytes); + await store.put(manifestKey, file, "application/json"); + } finally { + await fs.rm(temporary, { recursive: true, force: true }); + } + console.log( + `Published ${files.length} files at https://cdn.openpatch.org/${prefix}/`, + ); + return manifest; +} + +if ( + process.argv[1] && + path.resolve(process.argv[1]) === fileURLToPath(import.meta.url) +) { + const [element, version, directory] = process.argv.slice(2); + if (!directory) + throw new Error( + "Usage: node scripts/publish-cdn.mjs ", + ); + await publishCdn({ directory, element, version }); +} diff --git a/scripts/publish-cdn.test.mjs b/scripts/publish-cdn.test.mjs new file mode 100644 index 000000000..2fa0d82ff --- /dev/null +++ b/scripts/publish-cdn.test.mjs @@ -0,0 +1,219 @@ +import assert from "node:assert/strict"; +import { createHash } from "node:crypto"; +import fs from "node:fs/promises"; +import os from "node:os"; +import path from "node:path"; +import { test } from "node:test"; +import { + publishCdn, + releasePrefix, + runtimeManifest, + wranglerStore, +} from "./publish-cdn.mjs"; +import { verifyCdn } from "./verify-cdn.mjs"; + +async function fixture(t) { + const directory = await fs.mkdtemp(path.join(os.tmpdir(), "cdn-test-")); + t.after(() => fs.rm(directory, { recursive: true, force: true })); + await fs.mkdir(path.join(directory, "include", "assets"), { + recursive: true, + }); + await fs.writeFile( + path.join(directory, "include", "online-ide-embedded.js"), + "export const version = 1;", + ); + await fs.writeFile( + path.join(directory, "include", "assets", "runtime.wasm"), + Buffer.from([0, 97, 115, 109]), + ); + await fs.writeFile( + path.join(directory, "include", "example.html"), + "example", + ); + await fs.writeFile( + path.join(directory, "include", "bundle.js.map"), + "source map", + ); + const objects = new Map([ + ["sqlide/v1/unchanged.js", Buffer.from("existing SQL release")], + ]); + const writes = []; + const store = { + get: async (key) => objects.get(key) || null, + async put(key, file, type) { + writes.push({ key, type }); + objects.set(key, await fs.readFile(file)); + }, + }; + return { + directory, + objects, + writes, + store, + element: "onlineide", + version: "v1", + }; +} + +test("publishes workers/WASM with correct MIME types, skips maps and HTML, and marks completion last", async (t) => { + const f = await fixture(t); + const manifest = await publishCdn(f); + assert.equal(f.writes.at(-1).key, "onlineide/v1/manifest.json"); + assert.equal( + f.writes.find((write) => write.key.endsWith(".js")).type, + "text/javascript", + ); + assert.equal( + f.writes.find((write) => write.key.endsWith(".wasm")).type, + "application/wasm", + ); + assert.equal(Object.keys(manifest.files).length, 2); + assert.equal( + manifest.files["include/assets/runtime.wasm"].sha256, + createHash("sha256") + .update(Buffer.from([0, 97, 115, 109])) + .digest("hex"), + ); + assert.equal( + f.objects.get("sqlide/v1/unchanged.js").toString(), + "existing SQL release", + ); +}); + +test("retrying a completed release performs no writes; changed releases cannot replace it", async (t) => { + const f = await fixture(t); + await publishCdn(f); + const writes = f.writes.length; + await publishCdn(f); + assert.equal(f.writes.length, writes); + await fs.writeFile( + path.join(f.directory, "include", "online-ide-embedded.js"), + "changed", + ); + await assert.rejects(publishCdn(f), /immutable/); + assert.equal(f.writes.length, writes); +}); + +test("failed publication leaves no completion marker and can resume existing payload files", async (t) => { + const f = await fixture(t); + const put = f.store.put; + f.store.put = async (key, ...args) => { + if (key.endsWith(".wasm")) throw new Error("Upload failed"); + await put(key, ...args); + }; + await assert.rejects(publishCdn(f), /Upload failed/); + assert.equal(f.objects.has("onlineide/v1/manifest.json"), false); + f.store.put = put; + await publishCdn(f); + assert.equal(f.objects.has("onlineide/v1/manifest.json"), true); +}); + +test("a conflicting object in a partial release is never overwritten", async (t) => { + const f = await fixture(t); + f.objects.set( + "onlineide/v1/include/assets/runtime.wasm", + Buffer.from("different"), + ); + await assert.rejects(publishCdn(f), /Existing runtime object differs/); + assert.equal( + f.objects.get("onlineide/v1/include/assets/runtime.wasm").toString(), + "different", + ); + assert.equal(f.objects.has("onlineide/v1/manifest.json"), false); +}); + +test("new versions retain all objects from prior versions", async (t) => { + const f = await fixture(t); + await publishCdn(f); + const previous = new Map(f.objects); + await publishCdn({ ...f, version: "v2" }); + for (const [key, value] of previous) + assert.deepEqual(f.objects.get(key), value); +}); + +test("unsafe namespaces, empty runtimes, reserved metadata and symlinks are rejected", async (t) => { + for (const version of ["../x", "/x", "..", "latest", "v1?x", "v1/x"]) { + assert.throws(() => releasePrefix("onlineide", version)); + } + assert.throws(() => releasePrefix("unknown", "v1")); + const f = await fixture(t); + const empty = path.join(f.directory, "empty"); + await fs.mkdir(empty); + await assert.rejects(runtimeManifest(empty), /No runtime files/); + await fs.writeFile(path.join(f.directory, "manifest.json"), "reserved"); + await assert.rejects(runtimeManifest(f.directory), /reserved/); + await fs.rm(path.join(f.directory, "manifest.json")); + await fs.symlink( + "include/assets/runtime.wasm", + path.join(f.directory, "link"), + ); + await assert.rejects(runtimeManifest(f.directory), /symlink/); +}); + +test("Wrangler uploads to remote R2 with immutable caching and treats only missing keys as absent", async () => { + const calls = []; + const store = wranglerStore("cdn", async (command, args) => { + calls.push({ command, args }); + }); + await store.put( + "onlineide/v1/include/worker.js", + "/tmp/worker.js", + "text/javascript", + ); + assert.equal(calls[0].command, "wrangler"); + assert.deepEqual(calls[0].args, [ + "r2", + "object", + "put", + "cdn/onlineide/v1/include/worker.js", + "--remote", + "--file", + "/tmp/worker.js", + "--content-type", + "text/javascript", + "--cache-control", + "public, max-age=31536000, immutable", + ]); + const missing = wranglerStore("cdn", async () => { + throw { stderr: "The specified key does not exist." }; + }); + assert.equal(await missing.get("absent"), null); + const denied = wranglerStore("cdn", async () => { + throw new Error("Authentication failed"); + }); + await assert.rejects(denied.get("absent"), /Authentication failed/); +}); + +test("public verification checks all payloads and fails for bad CORS, cache, MIME types and checksums", async (t) => { + const f = await fixture(t); + const manifest = await publishCdn(f); + let fault; + const fetchImpl = async (url, options) => { + assert.equal(options.headers.Origin, "https://hyperbook.openpatch.org"); + const key = new URL(url).pathname.slice(1); + const relative = key.slice("onlineide/v1/".length); + const headers = { + "Access-Control-Allow-Origin": "*", + "Cache-Control": "public, max-age=31536000, immutable", + "Content-Type": + manifest.files[relative]?.contentType || "application/json", + }; + if (fault === "CORS") delete headers["Access-Control-Allow-Origin"]; + if (fault === "Cache-Control") delete headers["Cache-Control"]; + if (fault === "Content-Type" && relative !== "manifest.json") + headers["Content-Type"] = "text/plain"; + return new Response( + fault === "checksum" && relative !== "manifest.json" + ? "corrupt" + : f.objects.get(key), + { headers }, + ); + }; + await verifyCdn({ ...f, fetchImpl }); + for (fault of ["CORS", "Cache-Control", "Content-Type", "checksum"]) { + await assert.rejects( + verifyCdn({ ...f, fetchImpl }), + /CORS|Cache-Control|Content-Type|checksum/, + ); + } +}); diff --git a/scripts/verify-cdn.mjs b/scripts/verify-cdn.mjs new file mode 100644 index 000000000..6ede7d074 --- /dev/null +++ b/scripts/verify-cdn.mjs @@ -0,0 +1,68 @@ +import { createHash } from "node:crypto"; +import path from "node:path"; +import { fileURLToPath } from "node:url"; +import { releasePrefix, runtimeManifest } from "./publish-cdn.mjs"; + +// Read through the public hostname, exercising the path browsers actually use. +export async function verifyCdn({ + element, + version, + directory, + fetchImpl = fetch, +}) { + const base = `https://cdn.openpatch.org/${releasePrefix(element, version)}/`; + const expected = await runtimeManifest(directory); + async function get(relative) { + const response = await fetchImpl(new URL(relative, base), { + headers: { Origin: "https://hyperbook.openpatch.org" }, + signal: AbortSignal.timeout(120_000), + }); + if (!response.ok) throw new Error(`${relative}: HTTP ${response.status}`); + if (response.headers.get("access-control-allow-origin") !== "*") { + throw new Error(`${relative}: missing public CORS headers`); + } + if (!response.headers.get("cache-control")?.includes("immutable")) { + throw new Error(`${relative}: missing immutable Cache-Control`); + } + return response; + } + const manifest = await (await get("manifest.json")).json(); + if (JSON.stringify(manifest) !== JSON.stringify(expected)) + throw new Error("Public CDN manifest differs from the release"); + const files = Object.entries(expected.files); + let next = 0; + const results = await Promise.allSettled( + Array.from({ length: Math.min(4, files.length) }, async () => { + while (next < files.length) { + const [relative, metadata] = files[next++]; + const response = await get(relative); + if ( + response.headers.get("content-type")?.split(";")[0] !== + metadata.contentType + ) { + throw new Error(`${relative}: incorrect Content-Type`); + } + const bytes = Buffer.from(await response.arrayBuffer()); + if ( + bytes.length !== metadata.bytes || + createHash("sha256").update(bytes).digest("hex") !== metadata.sha256 + ) { + throw new Error(`${relative}: public CDN checksum mismatch`); + } + } + }), + ); + const failed = results.find((result) => result.status === "rejected"); + if (failed) throw failed.reason; + console.log( + `Verified ${files.length} public CDN files, CORS, MIME types and immutable caching at ${base}`, + ); +} + +if ( + process.argv[1] && + path.resolve(process.argv[1]) === fileURLToPath(import.meta.url) +) { + const [element, version, directory] = process.argv.slice(2); + await verifyCdn({ element, version, directory }); +} From fa27cad0431987bdfb3c2ee2ed9f27790f82d38a Mon Sep 17 00:00:00 2001 From: Mike Barkmin Date: Sat, 10 Oct 2026 11:19:29 +0200 Subject: [PATCH 5/6] feat: use independent versioned R2 runtimes and verify them before release --- .github/workflows/changeset-version.yml | 5 +++ .github/workflows/publish-openscad-cdn.yml | 4 -- docs/cdn-releases.md | 15 +++++-- packages/markdown/ide-releases.json | 4 ++ packages/markdown/postbuild.mjs | 35 ++++++++++++---- packages/markdown/src/elementAssets.ts | 21 +++++++--- packages/markdown/tests/elementCdn.test.ts | 25 +++++++---- scripts/publish-cdn.test.mjs | 1 + scripts/verify-cdn.mjs | 3 +- scripts/verify-default-cdns.mjs | 48 ++++++++++++++++++++++ website/en/book/configuration/book.md | 5 ++- 11 files changed, 133 insertions(+), 33 deletions(-) create mode 100644 packages/markdown/ide-releases.json create mode 100644 scripts/verify-default-cdns.mjs diff --git a/.github/workflows/changeset-version.yml b/.github/workflows/changeset-version.yml index 195a69247..f45aa04ad 100644 --- a/.github/workflows/changeset-version.yml +++ b/.github/workflows/changeset-version.yml @@ -47,6 +47,11 @@ jobs: run: | pnpm --filter "hyperbook-studio" vscode:package + - name: Verify default CDN runtimes before publishing + # CDN previews must work before any consumer package can be published. + # Validate all files against the locally built runtime bundles. + run: node scripts/verify-default-cdns.mjs + - name: Publish CLI asset bundles before npm run: node packages/hyperbook/publish-assets.mjs env: diff --git a/.github/workflows/publish-openscad-cdn.yml b/.github/workflows/publish-openscad-cdn.yml index 85d98ccc8..4fd3eea5a 100644 --- a/.github/workflows/publish-openscad-cdn.yml +++ b/.github/workflows/publish-openscad-cdn.yml @@ -1,9 +1,5 @@ name: Publish OpenSCAD runtime to R2 on: - # Bootstrap the first runtime from the migration branch; removed after upload. - push: - branches: [ccr-dc12b6f7-p2rfkj] - paths: [.github/workflows/publish-openscad-cdn.yml] workflow_dispatch: workflow_call: permissions: diff --git a/docs/cdn-releases.md b/docs/cdn-releases.md index 2e7f0c081..c8654454e 100644 --- a/docs/cdn-releases.md +++ b/docs/cdn-releases.md @@ -12,8 +12,11 @@ domain `cdn.openpatch.org`. Each repository owns its release uploads: All workflows use Wrangler and the Cloudflare API token in the organization Actions secret `CLOUDFLARE_R2_CDN`. Set `CLOUDFLARE_ACCOUNT_ID` as an organization Actions variable or secret. Grant both to the publishing repositories. S3 access -keys are unnecessary. The token needs remote R2 object read/write and bucket -CORS configuration permissions for `cdn`. +keys are unnecessary. The token needs **Account → Workers R2 Storage → Edit** +(the R2 token permission **Admin Read & Write**) for the account holding `cdn`. +Cloudflare's bucket-scoped **Object Read & Write** permission supports only the +S3 API, so it cannot authenticate these Wrangler operations. See +[Cloudflare's permission documentation](https://developers.cloudflare.com/r2/api/tokens/#permissions). The IDE publishers download the ZIP already attached to the release and verify GitHub's SHA-256 digest. HTML examples and source maps are excluded. OpenSCAD is @@ -46,7 +49,7 @@ gh workflow run release-embedded.yml --repo openpatch/sql-ide -f cdn_release_tag gh workflow run publish-openscad-cdn.yml --repo openpatch/hyperbook ``` -Once the CDN uploads pass verification, Hyperbook can pin these base URLs: +Hyperbook pins these default base URLs. You can also configure them explicitly: ```json { @@ -71,6 +74,12 @@ Update the upstream URL and SHA-256 in `openscad-config.json`, then increment well. Publish that version before using it as Hyperbook's default. The build continues to keep the small Hyperbook integration scripts local. +Update IDE pins in `packages/markdown/ide-releases.json`; local bundle downloads +and default CDN URLs use the same tags. Hyperbook's release workflow verifies +every pinned CDN runtime against the local bundle payloads before publishing +CLI bundles, npm packages or the VS Code extension. A missing or different CDN +release prevents package publication. + ## Cloudflare caching Let the custom domain respect object Cache-Control headers. Do not cache 404 diff --git a/packages/markdown/ide-releases.json b/packages/markdown/ide-releases.json new file mode 100644 index 000000000..33a337a79 --- /dev/null +++ b/packages/markdown/ide-releases.json @@ -0,0 +1,4 @@ +{ + "onlineide": "v2.2.1-hyperbook.28", + "sqlide": "v2.0.0-hyperbook.4" +} diff --git a/packages/markdown/postbuild.mjs b/packages/markdown/postbuild.mjs index b3af460ac..248ee12ef 100644 --- a/packages/markdown/postbuild.mjs +++ b/packages/markdown/postbuild.mjs @@ -1,5 +1,13 @@ import path from "path"; -import { cp, readFile, writeFile, mkdir, access, rm, readdir } from "fs/promises"; +import { + cp, + readFile, + writeFile, + mkdir, + access, + rm, + readdir, +} from "fs/promises"; import { minify } from "terser"; import https from "https"; import { Extract } from "unzipper"; @@ -111,7 +119,10 @@ async function downloadAndExtractZip(url, destination, sha256) { async function postbuild() { // Read openscad-config.json for WASM asset configuration - const openscadConfig = JSON.parse(await readFile("openscad-config.json", "utf-8")); + const openscadConfig = JSON.parse( + await readFile("openscad-config.json", "utf-8"), + ); + const ideReleases = JSON.parse(await readFile("ide-releases.json", "utf8")); // Download and extract zips const zipFiles = [ @@ -120,11 +131,11 @@ async function postbuild() { dst: path.join("./dist", "assets", "directive-blockflow"), }, { - url: "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/openpatch/sql-ide/releases/download/v2.0.0-hyperbook.4/dist-embedded.zip", + url: `https://github.com/openpatch/sql-ide/releases/download/${ideReleases.sqlide}/dist-embedded.zip`, dst: path.join("./dist", "assets", "directive-sqlide", "include"), }, { - url: "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/openpatch/online-ide/releases/download/v2.2.1-hyperbook.28/dist-embedded.zip", + url: `https://github.com/openpatch/online-ide/releases/download/${ideReleases.onlineide}/dist-embedded.zip`, dst: path.join("./dist", "assets", "directive-onlineide", "include"), }, { @@ -142,7 +153,9 @@ async function postbuild() { // These files are served locally by the elements. The CLI publishes them in // optional bundles. The Markdown npm package and VS Code extension exclude // the runtimes, retaining only their small integration scripts and styles. - await installRuntimeAssets(JSON.parse(await readFile("runtime-assets.json", "utf8"))); + await installRuntimeAssets( + JSON.parse(await readFile("runtime-assets.json", "utf8")), + ); const assets = [ { @@ -414,14 +427,18 @@ async function postbuild() { await cp("locales", "./dist/locales", { recursive: true }); await esbuild({ - entryPoints: ["./node_modules/@raspberrypifoundation/python-friendly-error-messages/dist/index.browser.js"], + entryPoints: [ + "./node_modules/@raspberrypifoundation/python-friendly-error-messages/dist/index.browser.js", + ], bundle: false, format: "iife", globalName: "PythonFriendlyErrorMessages", outfile: "./dist/assets/directive-pyide/python-friendly-error-messages.js", minify: true, }); - console.log("Built python-friendly-error-messages → dist/assets/directive-pyide/python-friendly-error-messages.js"); + console.log( + "Built python-friendly-error-messages → dist/assets/directive-pyide/python-friendly-error-messages.js", + ); // Bundle any directive that ships a src/index.js into dist/assets//client.js. // Each src/ directory is removed from dist after bundling — only the bundle is served. @@ -479,6 +496,8 @@ async function postbuild() { outfile: "./dist/assets/codemirror/codemirror.bundle.js", minify: true, }); - console.log("Built CodeMirror bundle → dist/assets/codemirror/codemirror.bundle.js"); + console.log( + "Built CodeMirror bundle → dist/assets/codemirror/codemirror.bundle.js", + ); } postbuild(); diff --git a/packages/markdown/src/elementAssets.ts b/packages/markdown/src/elementAssets.ts index 5d965b11e..4b5bd2598 100644 --- a/packages/markdown/src/elementAssets.ts +++ b/packages/markdown/src/elementAssets.ts @@ -5,15 +5,12 @@ import { } from "@hyperbook/types"; import excalidrawPackageJson from "../../web-component-excalidraw/package.json"; import runtimeAssets from "../runtime-assets.json"; +import ideReleases from "../ide-releases.json"; +import openscadConfig from "../openscad-config.json"; /** Hosted Blockflow app; the npm package does not include Blockflow's bundle. */ const blockflowCdn = "https://blockflow.openpatch.org/"; -// The last published Markdown release that includes these runtimes. Keep this -// independent of the renderer version: new packages omit every large runtime. -const legacyRuntimeCdn = - "https://unpkg.com/@hyperbook/markdown@0.85.0/dist/assets/"; - /** Resolve payload files; Hyperbook's integration scripts stay local. */ export function elementAssetUrl( ctx: HyperbookContext, @@ -37,6 +34,18 @@ export function elementAssetUrl( cdn, ).href; } + if (element === "onlineide" || element === "sqlide") { + return new URL( + file, + `https://cdn.openpatch.org/${element}/${ideReleases[element]}/`, + ).href; + } + if (element === "openscad") { + return new URL( + file, + `https://cdn.openpatch.org/openscad/${openscadConfig.wasmBuild.cdnVersion}/`, + ).href; + } if (element === "pyide") { const version = runtimeAssets .find((asset) => asset.target === "directive-pyide")! @@ -76,5 +85,5 @@ export function elementAssetUrl( ? `${base}index.css` : new URL(file, base).href; } - return `${legacyRuntimeCdn}directive-${element}/${file}`; + throw new Error(`No default CDN asset for ${element}/${file}`); } diff --git a/packages/markdown/tests/elementCdn.test.ts b/packages/markdown/tests/elementCdn.test.ts index 464a91c2a..244775bca 100644 --- a/packages/markdown/tests/elementCdn.test.ts +++ b/packages/markdown/tests/elementCdn.test.ts @@ -52,11 +52,11 @@ describe.each(elements)( blockflow: "https://blockflow.openpatch.org/", excalidraw: "https://unpkg.com/@hyperbook/web-component-excalidraw@", - }[ - element as - "pyide" | "typst" | "geogebra" | "blockflow" | "excalidraw" - ] || - `https://unpkg.com/@hyperbook/markdown@0.85.0/dist/assets/directive-${element}/` + onlineide: + "https://cdn.openpatch.org/onlineide/v2.2.1-hyperbook.28/", + sqlide: "https://cdn.openpatch.org/sqlide/v2.0.0-hyperbook.4/", + openscad: "https://cdn.openpatch.org/openscad/2026.10.08-1/", + }[element] : "http://assets.example.com/runtime/", ); if (element === "blockflow") @@ -144,15 +144,22 @@ it("loads Blockflow and Excalidraw from their upstream CDNs", () => { expect(excalidraw).toMatch(/@\d+\.\d+\.\d+\/dist/); }); -it.each(["onlineide", "sqlide", "openscad"] as const)( - "keeps %s's default CDN pinned to a release containing its runtime", - (element: DownloadableElement) => { +it.each([ + ["onlineide", "v2.2.1-hyperbook.28", "include/online-ide-embedded.js"], + ["sqlide", "v2.0.0-hyperbook.4", "include/sql-ide-embedded.js"], + ["openscad", "2026.10.08-1", "openscad.wasm"], +] as const)( + "pins %s's default CDN independently of the Markdown package", + (element: DownloadableElement, version: string, file: string) => { const configured = { ...ctx, config: { ...ctx.config, elements: { [element]: { cdn: true } } }, }; expect(elementAssetUrl(configured, element, "")).toBe( - `https://unpkg.com/@hyperbook/markdown@0.85.0/dist/assets/directive-${element}/`, + `https://cdn.openpatch.org/${element}/${version}/`, + ); + expect(elementAssetUrl(configured, element, file)).toBe( + `https://cdn.openpatch.org/${element}/${version}/${file}`, ); }, ); diff --git a/scripts/publish-cdn.test.mjs b/scripts/publish-cdn.test.mjs index 2fa0d82ff..0f74a0bec 100644 --- a/scripts/publish-cdn.test.mjs +++ b/scripts/publish-cdn.test.mjs @@ -210,6 +210,7 @@ test("public verification checks all payloads and fails for bad CORS, cache, MIM ); }; await verifyCdn({ ...f, fetchImpl }); + await verifyCdn({ ...f, directory: undefined, manifest, fetchImpl }); for (fault of ["CORS", "Cache-Control", "Content-Type", "checksum"]) { await assert.rejects( verifyCdn({ ...f, fetchImpl }), diff --git a/scripts/verify-cdn.mjs b/scripts/verify-cdn.mjs index 6ede7d074..693ca7e74 100644 --- a/scripts/verify-cdn.mjs +++ b/scripts/verify-cdn.mjs @@ -8,10 +8,11 @@ export async function verifyCdn({ element, version, directory, + manifest: expectedManifest, fetchImpl = fetch, }) { const base = `https://cdn.openpatch.org/${releasePrefix(element, version)}/`; - const expected = await runtimeManifest(directory); + const expected = expectedManifest || (await runtimeManifest(directory)); async function get(relative) { const response = await fetchImpl(new URL(relative, base), { headers: { Origin: "https://hyperbook.openpatch.org" }, diff --git a/scripts/verify-default-cdns.mjs b/scripts/verify-default-cdns.mjs new file mode 100644 index 000000000..cd6ee215d --- /dev/null +++ b/scripts/verify-default-cdns.mjs @@ -0,0 +1,48 @@ +import fs from "node:fs/promises"; +import path from "node:path"; +import { fileURLToPath } from "node:url"; +import { runtimeManifest } from "./publish-cdn.mjs"; +import { verifyCdn } from "./verify-cdn.mjs"; + +const root = path.resolve(path.dirname(fileURLToPath(import.meta.url)), ".."); +const markdown = path.join(root, "packages/markdown"); +const readJson = async (file) => + JSON.parse(await fs.readFile(path.join(markdown, file), "utf8")); +const ideReleases = await readJson("ide-releases.json"); +const { wasmBuild } = await readJson("openscad-config.json"); +const assets = await readJson("runtime-assets.json"); +const openscadFiles = new Set([ + ...wasmBuild.files, + ...assets + .filter((asset) => asset.target.startsWith("directive-openscad/")) + .map((asset) => asset.target.slice("directive-openscad/".length)), +]); + +// Gate package publication on the exact runtimes built into the CLI bundles. +// The small Hyperbook integration scripts remain local, outside CDN manifests. +const results = await Promise.allSettled( + Object.entries({ ...ideReleases, openscad: wasmBuild.cdnVersion }).map( + async ([element, version]) => { + const directory = path.join( + markdown, + "dist/assets", + `directive-${element}`, + ); + const manifest = await runtimeManifest(directory); + manifest.files = Object.fromEntries( + Object.entries(manifest.files).filter(([file]) => + element === "openscad" + ? openscadFiles.has(file) + : file.startsWith("include/"), + ), + ); + await verifyCdn({ element, version, manifest }); + }, + ), +); +const failed = results.filter((result) => result.status === "rejected"); +if (failed.length) + throw new AggregateError( + failed.map((result) => result.reason), + "Default CDN runtimes must be published before Hyperbook packages", + ); diff --git a/website/en/book/configuration/book.md b/website/en/book/configuration/book.md index 10532ebbb..8acbb4091 100644 --- a/website/en/book/configuration/book.md +++ b/website/en/book/configuration/book.md @@ -140,8 +140,9 @@ service. The default CDNs are jsDelivr for Pyodide and Typst, GeoGebra's server for GeoGebra, the hosted Blockflow app at `blockflow.openpatch.org` for Blockflow, -Excalidraw's own packages on UNPKG for Excalidraw, and UNPKG's versioned -Hyperbook assets for OpenSCAD, the Online IDE, and the SQL IDE. Hyperbook's +Excalidraw's own packages on UNPKG for Excalidraw, and `cdn.openpatch.org` for +OpenSCAD, the Online IDE, and the SQL IDE. Each Openpatch runtime uses a pinned +release, published directly from its owning repository. Hyperbook's small integration scripts and styles stay in the build. The Hyperbook extension for VS Code does not include these runtimes. Its preview From 6ac3fab8883d5ef15a095bcaa7b955e68aa81a0b Mon Sep 17 00:00:00 2001 From: Mike Barkmin Date: Sat, 10 Oct 2026 11:50:21 +0200 Subject: [PATCH 6/6] feat: default Pyodide to its CDN and document hosting caches --- .../smaller-markdown-and-vscode-packages.md | 4 +- packages/hyperbook/tests/assets-build.test.ts | 102 ++++++++++++++++++ .../remarkDirectivePyide.test.ts.snap | 8 +- .../tests/blockflowVscodePreview.test.ts | 24 ++++- packages/markdown/tests/elementCdn.test.ts | 30 +++++- packages/types/src/index.ts | 5 +- .../vscode/schemas/hyperbook.schema.json | 8 +- platforms/vscode/src/Preview.ts | 3 +- website/de/book/configuration/book.md | 26 +++-- website/de/book/elements/pyide.md | 17 ++- website/de/book/hosting/caching.md | 88 +++++++++++++++ website/de/book/hosting/custom.md | 3 + website/de/book/hosting/ghpages.md | 3 + website/de/book/hosting/glpages.md | 3 + website/de/book/hosting/index.md | 15 +++ website/de/book/hosting/vercel.md | 3 + website/en/book/configuration/book.md | 15 ++- website/en/book/elements/pyide.md | 17 ++- website/en/book/hosting/caching.md | 84 +++++++++++++++ website/en/book/hosting/custom.md | 3 + website/en/book/hosting/ghpages.md | 3 + website/en/book/hosting/glpages.md | 3 + website/en/book/hosting/index.md | 14 +++ website/en/book/hosting/vercel.md | 3 + 24 files changed, 450 insertions(+), 34 deletions(-) create mode 100644 website/de/book/hosting/caching.md create mode 100644 website/en/book/hosting/caching.md diff --git a/.changeset/smaller-markdown-and-vscode-packages.md b/.changeset/smaller-markdown-and-vscode-packages.md index 607ccb9aa..5e4c108ac 100644 --- a/.changeset/smaller-markdown-and-vscode-packages.md +++ b/.changeset/smaller-markdown-and-vscode-packages.md @@ -8,7 +8,9 @@ Keep the Markdown npm package and the VS Code extension small by leaving out large element runtimes. -The Markdown package no longer includes any of the eight large element runtimes; it keeps their integration scripts and styles. With `cdn: true`, Blockflow now loads from `blockflow.openpatch.org` and Excalidraw from its own packages on UNPKG. OpenSCAD, Online IDE and SQL IDE use the published `@hyperbook/markdown@0.85.0` runtimes on UNPKG, independently of the renderer version, so their default CDNs keep working after the runtimes are removed from new packages. +The Markdown package no longer includes any of the eight large element runtimes; it keeps their integration scripts and styles. With `cdn: true`, Blockflow now loads from `blockflow.openpatch.org` and Excalidraw from its own packages on UNPKG. OpenSCAD, Online IDE and SQL IDE use independent, pinned releases on `cdn.openpatch.org`. Their owning repositories publish immutable runtime files to Cloudflare R2, and Hyperbook verifies the public CDN files before publishing a release. + +PyIDE now uses Pyodide's versioned jsDelivr CDN by default because its full local distribution is very large. Builds and asset prefetching skip downloading this runtime unless `elements.pyide.cdn` is explicitly `false`. Set it to `false` for self-hosted and offline books, and prefetch the runtime before an offline build. The documentation explains this default and how to cache runtime downloads in CI and assets on your hosting service. The VS Code extension no longer includes any of the eight large runtimes. Its preview resolves them like a Hyperbook build: elements with a `cdn` setting load from that CDN, and the others use the runtimes in the Hyperbook CLI's asset cache, which the extension now shares. Runtimes that are not downloaded yet load from their default CDN, and the preview offers to download them. The new commands "Download Element Runtimes..." and "Download All Element Runtimes" work like `hyperbook assets fetch` and `hyperbook assets fetch --all`. diff --git a/packages/hyperbook/tests/assets-build.test.ts b/packages/hyperbook/tests/assets-build.test.ts index 62f3e8cee..56b5c48f9 100644 --- a/packages/hyperbook/tests/assets-build.test.ts +++ b/packages/hyperbook/tests/assets-build.test.ts @@ -167,6 +167,108 @@ it("honors CDN configuration for a directive first added during incremental deve expect(download).not.toHaveBeenCalled(); }); +it("defaults PyIDE to its CDN during prefetching and offline builds", async () => { + await fs.writeFile(path.join(root, "book", "index.md"), "::pyide"); + await fs.mkdir(path.join(options.assetsPath!, "directive-pyide")); + await fs.writeFile( + path.join(options.assetsPath!, "directive-pyide", "client.js"), + "client", + ); + const manager = new AssetManager({ ...options, offline: true }); + await fetchAssets(project, manager); + await runBuildProject( + project, + project, + undefined, + undefined, + undefined, + manager, + ); + const out = path.join(root, ".hyperbook", "out"); + expect(await fs.readFile(path.join(out, "index.html"), "utf8")).toContain( + 'data-runtime-url="https://cdn.jsdelivr.net/pyodide/v314.0.7/full/"', + ); + expect( + await fs.readFile( + path.join(out, "__hyperbook_assets", "directive-pyide", "client.js"), + "utf8", + ), + ).toBe("client"); + await expect( + fs.stat(path.join(out, "__hyperbook_assets", "directive-pyide", "pyodide")), + ).rejects.toMatchObject({ code: "ENOENT" }); + expect(download).not.toHaveBeenCalled(); + + await fs.writeFile( + path.join(root, "hyperbook.json"), + JSON.stringify({ name: "Assets", elements: { pyide: { cdn: false } } }), + ); + expect(await projectDirectives(project)).toContain("pyide"); + const input = path.join(root, "input"); + await fs.mkdir(path.join(input, "directive-pyide", "pyodide"), { + recursive: true, + }); + await fs.writeFile( + path.join(input, "directive-pyide", "pyodide", "pyodide.js"), + "python", + ); + await fs.writeFile( + path.join(input, "directive-pyide", "pyodide", "pyodide-lock.json"), + "{}", + ); + const archivePath = path.join(root, "python.tar.gz"); + await create({ file: archivePath, cwd: input, gzip: true }, [ + "directive-pyide", + ]); + const archive = await fs.readFile(archivePath); + await fs.writeFile( + options.manifestPath!, + JSON.stringify({ + format: 1, + version: "test", + bundles: { + pyide: { + url: "https://example.com/python.tar.gz", + sha256: createHash("sha256").update(archive).digest("hex"), + bytes: archive.length, + files: { + "directive-pyide/pyodide/pyodide.js": 6, + "directive-pyide/pyodide/pyodide-lock.json": 2, + }, + }, + }, + }), + ); + download.mockImplementation( + async () => new Response(new Uint8Array(archive)), + ); + await fetchAssets(project, new AssetManager(options)); + await runBuildProject( + project, + project, + undefined, + undefined, + undefined, + new AssetManager({ ...options, offline: true }), + ); + expect(await fs.readFile(path.join(out, "index.html"), "utf8")).not.toContain( + "https://cdn.jsdelivr.net/pyodide/", + ); + expect( + await fs.readFile( + path.join( + out, + "__hyperbook_assets", + "directive-pyide", + "pyodide", + "pyodide.js", + ), + "utf8", + ), + ).toBe("python"); + expect(download).toHaveBeenCalledOnce(); +}); + it("downloads a directive first introduced during an incremental build", async () => { const builder = new IncrementalBuilder( root, diff --git a/packages/markdown/tests/__snapshots__/remarkDirectivePyide.test.ts.snap b/packages/markdown/tests/__snapshots__/remarkDirectivePyide.test.ts.snap index 49e17271d..25a7dcdc6 100644 --- a/packages/markdown/tests/__snapshots__/remarkDirectivePyide.test.ts.snap +++ b/packages/markdown/tests/__snapshots__/remarkDirectivePyide.test.ts.snap @@ -2,7 +2,7 @@ exports[`remarkDirectivePyide > should transform basic pyide 1`] = ` " -
+

@@ -35,7 +35,7 @@ exports[`remarkDirectivePyide > should transform basic pyide 1`] = `
 
 exports[`remarkDirectivePyide > should transform pyide with canvas and packages attributes 1`] = `
 "
-

+

   

     

     
@@ -74,7 +74,7 @@ exports[`remarkDirectivePyide > should transform pyide with canvas and packages
 
 exports[`remarkDirectivePyide > should transform pyide with canvas attribute 1`] = `
 "
-

+

   

     

     
@@ -114,7 +114,7 @@ pygame.init()

 
 exports[`remarkDirectivePyide > should transform pyide with packages attribute 1`] = `
 "
-

+

   

     

     

diff --git a/packages/markdown/tests/blockflowVscodePreview.test.ts b/packages/markdown/tests/blockflowVscodePreview.test.ts
index 458577992..78bdd6812 100644
--- a/packages/markdown/tests/blockflowVscodePreview.test.ts
+++ b/packages/markdown/tests/blockflowVscodePreview.test.ts
@@ -17,6 +17,7 @@ import {
   HyperbookContext,
   HyperbookJson,
   elementCoreFiles,
+  elementCdn,
   elementRuntimeFiles,
 } from "@hyperbook/types";
 import { expect, it } from "vitest";
@@ -138,6 +139,7 @@ async function renderPreview({
     },
     "@hyperbook/fs": filesystem,
     "@hyperbook/types": {
+      elementCdn,
       isExternalUrl: (p: string) => /^https?:\/\//.test(p),
     },
     "@hyperbook/markdown": {
@@ -192,6 +194,23 @@ it("keeps local runtimes when the extension ships them", async () => {
   expect(ctx.config).toEqual(config);
 });
 
+it("uses Pyodide's default CDN even when its local runtime is cached", async () => {
+  const located: DownloadableElement[] = [];
+  const { ctx, messages } = await renderPreview({
+    locate: (element) => {
+      located.push(element);
+      return {
+        kind: "cached",
+        directory: `/cache/hash/content-1/directive-${element}`,
+      };
+    },
+    directives: { pyide: {} },
+  });
+  expect(located).not.toContain("pyide");
+  expect(elementCdn(ctx.config, "pyide")).toBe(true);
+  expect(messages).toEqual([]);
+});
+
 it("uses runtimes from the CLI asset cache", async () => {
   const { ctx, messages } = await renderPreview({
     locate: (element) =>
@@ -223,9 +242,10 @@ it("loads runtimes that are not downloaded from their CDN and offers a download
     renders: 2,
   });
   for (const element of elements)
-    expect(ctx.config.elements?.[element]).toEqual({ cdn: true });
+    expect(elementCdn(ctx.config, element)).toBe(true);
   expect(messages).toHaveLength(1);
-  expect(messages[0]).toContain(", ");
+  expect(messages[0]).toContain("");
+  expect(messages[0]).not.toContain("");
   expect(messages[0]).not.toContain("");
 });
 
diff --git a/packages/markdown/tests/elementCdn.test.ts b/packages/markdown/tests/elementCdn.test.ts
index 244775bca..b9c434342 100644
--- a/packages/markdown/tests/elementCdn.test.ts
+++ b/packages/markdown/tests/elementCdn.test.ts
@@ -19,8 +19,26 @@ const elements: [DownloadableElement, string][] = [
 describe.each(elements)(
   "%s CDN configuration",
   (element: DownloadableElement, markdown: string) => {
-    it("keeps local assets as the default and accepts explicit false", async () => {
-      const local = await processMarkdown(markdown, ctx);
+    it("defaults to CDN for PyIDE and local assets for other elements", async () => {
+      const defaults = await processMarkdown(markdown, ctx);
+      const configured = await processMarkdown(markdown, {
+        ...ctx,
+        config: {
+          ...ctx.config,
+          elements: {
+            ...ctx.config.elements,
+            [element]: {
+              ...ctx.config.elements?.[element],
+              cdn: element === "pyide",
+            },
+          },
+        },
+      });
+      expect(defaults.value).toBe(configured.value);
+      expect(defaults.data.directives).toEqual(configured.data.directives);
+    });
+
+    it("accepts explicit false for local assets", async () => {
       const disabled = await processMarkdown(markdown, {
         ...ctx,
         config: {
@@ -31,8 +49,12 @@ describe.each(elements)(
           },
         },
       });
-      expect(disabled.value).toBe(local.value);
-      expect(disabled.data.directives).toEqual(local.data.directives);
+      const output =
+        String(disabled.value) + JSON.stringify(disabled.data.directives);
+      expect(output).not.toContain('data-runtime-url="https://');
+      expect(output).not.toContain("https://cdn.");
+      expect(output).not.toContain("http://assets.example.com/");
+      expect(disabled.data.directives).toHaveProperty(element);
     });
 
     it("uses the default CDN and a custom HTTP base URL", async () => {
diff --git a/packages/types/src/index.ts b/packages/types/src/index.ts
index f8c573fa3..ee5f82a59 100644
--- a/packages/types/src/index.ts
+++ b/packages/types/src/index.ts
@@ -3,7 +3,7 @@ type ElementConfig = {
 };
 
 export type ElementAssetConfig = ElementConfig & {
-  /** Use the default CDN (true), a custom HTTP(S) asset base URL, or local assets (false/default). */
+  /** Use the default CDN (true), a custom HTTP(S) asset base URL, or local assets (false). Defaults to true for PyIDE and false for other elements. */
   cdn?: boolean | string;
 };
 
@@ -48,7 +48,8 @@ export function elementCdn(
   if (!Object.prototype.hasOwnProperty.call(elementCoreFiles, element))
     return false;
   const value = config.elements?.[element as DownloadableElement]?.cdn;
-  if (value === undefined || value === false) return false;
+  if (value === undefined) return element === "pyide";
+  if (value === false) return false;
   if (value === true) return true;
   if (typeof value === "string") {
     try {
diff --git a/platforms/vscode/schemas/hyperbook.schema.json b/platforms/vscode/schemas/hyperbook.schema.json
index b951a0f9a..ed152e1f8 100644
--- a/platforms/vscode/schemas/hyperbook.schema.json
+++ b/platforms/vscode/schemas/hyperbook.schema.json
@@ -14,7 +14,7 @@
                 {
                     "properties": {
                         "cdn": {
-                            "description": "Use the default CDN (true), a custom HTTP(S) asset base URL, or local assets (false/default).",
+                            "description": "Use the default CDN (true), a custom HTTP(S) asset base URL, or local assets (false). Defaults to true for PyIDE and false for other elements.",
                             "type": [
                                 "string",
                                 "boolean"
@@ -267,7 +267,7 @@
                         {
                             "properties": {
                                 "cdn": {
-                                    "description": "Use the default CDN (true), a custom HTTP(S) asset base URL, or local assets (false/default).",
+                                    "description": "Use the default CDN (true), a custom HTTP(S) asset base URL, or local assets (false). Defaults to true for PyIDE and false for other elements.",
                                     "type": [
                                         "string",
                                         "boolean"
@@ -334,7 +334,7 @@
                         {
                             "properties": {
                                 "cdn": {
-                                    "description": "Use the default CDN (true), a custom HTTP(S) asset base URL, or local assets (false/default).",
+                                    "description": "Use the default CDN (true), a custom HTTP(S) asset base URL, or local assets (false). Defaults to true for PyIDE and false for other elements.",
                                     "type": [
                                         "string",
                                         "boolean"
@@ -378,7 +378,7 @@
                         {
                             "properties": {
                                 "cdn": {
-                                    "description": "Use the default CDN (true), a custom HTTP(S) asset base URL, or local assets (false/default).",
+                                    "description": "Use the default CDN (true), a custom HTTP(S) asset base URL, or local assets (false). Defaults to true for PyIDE and false for other elements.",
                                     "type": [
                                         "string",
                                         "boolean"
diff --git a/platforms/vscode/src/Preview.ts b/platforms/vscode/src/Preview.ts
index 9aaa9f0f6..edf794021 100644
--- a/platforms/vscode/src/Preview.ts
+++ b/platforms/vscode/src/Preview.ts
@@ -17,6 +17,7 @@ import {
   HyperbookJson,
   HyperbookPage,
   Navigation,
+  elementCdn,
   isExternalUrl,
 } from "@hyperbook/types";
 import RuntimeAssets, { runtimeElements, runtimeLabels } from "./RuntimeAssets";
@@ -147,7 +148,7 @@ export default class Preview {
     const elements: Record = { ...config.elements };
     const fallbacks = new Set();
     for (const element of runtimeElements) {
-      if (elements[element]?.cdn) {
+      if (elementCdn(config, element)) {
         continue;
       }
       const location = await this.runtimes.locate(element);
diff --git a/website/de/book/configuration/book.md b/website/de/book/configuration/book.md
index 338113c10..674fe834d 100644
--- a/website/de/book/configuration/book.md
+++ b/website/de/book/configuration/book.md
@@ -109,15 +109,16 @@ Hier ist eine Beispielkonfiguration:
 
 ## Lokale Dateien und CDNs
 
-Hyperbook lädt große Laufzeitumgebungen beim ersten Bedarf herunter und kopiert
-sie in den Build. Mit `cdn` in der Elementkonfiguration kannst du sie stattdessen
-von einem CDN laden:
+PyIDE verwendet standardmäßig das Pyodide-CDN, weil die vollständige Distribution
+sehr groß ist. Andere Elemente laden ihre großen Laufzeitumgebungen beim ersten
+Bedarf herunter und kopieren sie in den Build. Mit `cdn` kannst du dieses
+Verhalten ausdrücklich festlegen:
 
 ```json
 {
   "name": "Mein Hyperbook",
   "elements": {
-    "pyide": { "cdn": true },
+    "pyide": { "cdn": false },
     "typst": { "cdn": "https://assets.example.com/directive-typst/" },
     "geogebra": { "cdn": true },
     "openscad": { "cdn": false }
@@ -127,7 +128,8 @@ von einem CDN laden:
 
 | Wert | Verhalten |
 | --- | --- |
-| Nicht gesetzt oder `false` | Die Laufzeitumgebung herunterladen und in den Build kopieren. Dies ist der Standard. |
+| Nicht gesetzt | Bei `pyide` das Standard-CDN verwenden; bei anderen Elementen die Laufzeitumgebung herunterladen und in den Build kopieren. |
+| `false` | Die Laufzeitumgebung herunterladen und in den Build kopieren, auch bei `pyide`. |
 | `true` | Die Laufzeitumgebung vom Standard-CDN laden. |
 | Eine HTTP(S)-Basis-URL | Die Laufzeitumgebung vom eigenen Server oder CDN laden. |
 
@@ -137,10 +139,11 @@ Player und den Editor. Kiri:Moto verwendet weiterhin seinen externen Dienst.
 
 Die Standard-CDNs sind jsDelivr für Pyodide und Typst, der GeoGebra-Server für
 GeoGebra, die gehostete Blockflow-App unter `blockflow.openpatch.org` für
-Blockflow, die eigenen Pakete von Excalidraw auf UNPKG für Excalidraw und die
-versionierten Hyperbook-Dateien auf UNPKG für OpenSCAD, die Online-IDE und die
-SQL-IDE. Die kleinen Integrationsskripte und Styles von Hyperbook bleiben im
-Build.
+Blockflow, die eigenen Pakete von Excalidraw auf UNPKG für Excalidraw und
+`cdn.openpatch.org` für OpenSCAD, die Online-IDE und die SQL-IDE. Diese
+Openpatch-Laufzeitumgebungen verwenden festgelegte Releases, die direkt aus
+ihrem jeweiligen Repository veröffentlicht werden. Die kleinen
+Integrationsskripte und Styles von Hyperbook bleiben im Build.
 
 Die Hyperbook-Erweiterung für VS Code enthält diese Laufzeitumgebungen nicht.
 Ihre Vorschau verhält sich wie ein Build: Elemente mit `cdn` laden von diesem
@@ -149,6 +152,8 @@ den die Erweiterung mitbenutzt. Eine noch nicht heruntergeladene
 Laufzeitumgebung lädt vom Standard-CDN. Mit **Hyperbook: Download Element
 Runtimes...** oder **Hyperbook: Download All Element Runtimes** lädst du
 Laufzeitumgebungen in VS Code für die Offline-Vorschau herunter.
+Setze bei PyIDE zusätzlich `elements.pyide.cdn` auf `false`, um die
+heruntergeladene Laufzeitumgebung zu verwenden.
 
 Eine eigene URL muss auf den Inhalt des Verzeichnisses
 `__hyperbook_assets/directive-/` aus einem lokalen Build zeigen.
@@ -168,3 +173,6 @@ Für `hyperbook build --offline` müssen diese nicht im Cache liegen. Zum Lesen
 des Buchs ist weiterhin Zugriff auf das konfigurierte CDN nötig.
 `hyperbook assets fetch --all` lädt unabhängig von der Elementkonfiguration
 alle Laufzeitumgebungen herunter.
+
+Siehe [Hosting und Caching](/hosting/caching), um Downloads zwischen CI-Builds
+aufzubewahren und Cache-Header für das exportierte Buch festzulegen.
diff --git a/website/de/book/elements/pyide.md b/website/de/book/elements/pyide.md
index af5b49c83..66184694c 100644
--- a/website/de/book/elements/pyide.md
+++ b/website/de/book/elements/pyide.md
@@ -4,7 +4,22 @@ permaid: pyide
 lang: de
 ---
 
-Die Pyodide-Laufzeitumgebung und ihre mitgelieferten Pakete sind lokal im exportierten Hyperbook enthalten. Die CLI lädt sie beim ersten Build mit diesem Element herunter und speichert sie im Cache. Mitgelieferte Pakete können ohne CDN-Verbindung importiert werden. Pakete, die über das Attribut `packages` von PyPI installiert werden, benötigen weiterhin eine Netzwerkverbindung.
+PyIDE lädt Pyodide standardmäßig vom versionierten jsDelivr-CDN. Die vollständige Laufzeitumgebung mit ihren mitgelieferten Paketen ist sehr groß und bleibt deshalb außerhalb des exportierten Hyperbooks. Leser benötigen eine Internetverbindung. Ihr Browser lädt die Laufzeitumgebung und angeforderte Pakete und kann diese Dateien für spätere Besuche zwischenspeichern.
+
+Für selbst gehostete oder offline nutzbare Bücher kannst du Pyodide lokal einbinden. Setze dafür `elements.pyide.cdn` in `hyperbook.json` auf `false`:
+
+```json
+{
+  "name": "Mein Offline-Hyperbook",
+  "elements": {
+    "pyide": { "cdn": false }
+  }
+}
+```
+
+Die CLI lädt dann beim ersten Build mit PyIDE die vollständige Distribution herunter und speichert sie im Cache. Führe für einen Offline-Build zunächst mit Internetverbindung `npx hyperbook assets fetch` und anschließend `npx hyperbook build --offline` aus. Führe in VS Code zusätzlich **Hyperbook: Download Element Runtimes...** aus und wähle PyIDE. Mitgelieferte Pakete funktionieren dann ohne CDN-Verbindung. Pakete, die über das Attribut `packages` von PyPI installiert werden, benötigen weiterhin eine Netzwerkverbindung.
+
+Siehe [CDN-Konfiguration](/configuration/book#lokale-dateien-und-cdns) für eigene Server und [Hosting und Caching](/hosting/caching) für Browser- und Build-Caches.
 
 Das `pyide`-Element repräsentiert eine Python-Integrated-Development-Environment (IDE)-Komponente.
 Es wird verwendet, um eine Python-Coding-Umgebung in die Hyperbook-Website einzubetten.
diff --git a/website/de/book/hosting/caching.md b/website/de/book/hosting/caching.md
new file mode 100644
index 000000000..5c4754eac
--- /dev/null
+++ b/website/de/book/hosting/caching.md
@@ -0,0 +1,88 @@
+---
+name: Caching
+lang: de
+---
+
+# Caching
+
+Der Build-Cache von Hyperbook speichert heruntergeladene Laufzeitumgebungen auf
+dem Rechner der CLI. HTTP-Caches speichern Dateien, die Leser von deinem Host
+oder einem CDN abrufen.
+
+## Downloads zwischen Builds aufbewahren
+
+CLI und VS Code teilen sich einen Asset-Cache. Builds verwenden bereits
+heruntergeladene Laufzeitumgebungen erneut. Mit `HYPERBOOK_ASSET_CACHE` kannst
+du das Verzeichnis festlegen. Standardmäßig wird dieser Pfad verwendet:
+
+| System | Verzeichnis |
+| --- | --- |
+| Linux | `$XDG_CACHE_HOME/hyperbook/assets` oder `~/.cache/hyperbook/assets` |
+| macOS | `~/Library/Caches/hyperbook/assets` |
+| Windows | `%LOCALAPPDATA%\hyperbook\assets` |
+
+CI-Runner starten normalerweise mit einem leeren Dateisystem. Stelle den
+Asset-Cache vor dem Build wieder her und speichere ihn danach. Füge zum
+Beispiel diese Schritte nach Checkout und Node.js-Einrichtung in deinen
+GitHub-Pages-Workflow ein:
+
+```yaml
+- name: Cache Hyperbook runtimes
+  uses: actions/cache@v4
+  with:
+    path: .cache/hyperbook-assets
+    key: ${{ runner.os }}-hyperbook-assets-${{ hashFiles('package-lock.json', 'pnpm-lock.yaml', 'hyperbook.json') }}-${{ github.sha }}
+    restore-keys: |
+      ${{ runner.os }}-hyperbook-assets-
+
+- name: Build
+  env:
+    HYPERBOOK_ASSET_CACHE: .cache/hyperbook-assets
+  run: npx hyperbook build
+```
+
+Der Cache-Schlüssel speichert neu benötigte Laufzeitumgebungen, wenn sich das
+Buch ändert. Mit `restore-keys` verwenden spätere Builds frühere Downloads
+erneut. Hyperbook prüft die Dateien anhand seines Laufzeitmanifests und lädt
+fehlende oder geänderte Pakete herunter. Verwende den normalen Online-Build,
+damit ein fehlender Cache automatisch ergänzt wird. Siehe die
+[GitHub-Dokumentation zum Caching](https://docs.github.com/en/actions/reference/workflows-and-actions/dependency-caching).
+
+Bei GitLab kannst du `.cache/hyperbook-assets` zwischenspeichern und dieselbe
+Variable `HYPERBOOK_ASSET_CACHE` im Pages-Job setzen. Veröffentliche dieses
+Verzeichnis nicht, sondern nur `.hyperbook/out`.
+
+Elemente mit aktiviertem CDN benötigen keinen lokalen Download. PyIDE verwendet
+standardmäßig ein CDN. Setze für ein Offline-Buch `elements.pyide.cdn` auf
+`false`, führe mit Internetverbindung `npx hyperbook assets fetch` aus und baue
+anschließend mit `npx hyperbook build --offline`. `--offline` steuert die
+Downloads beim Build. Ein Buch mit CDN-URLs wird dadurch nicht offline nutzbar.
+
+## Dateien im Browser zwischenspeichern
+
+Wenn dein Host eigene Antwort-Header erlaubt, kannst du für öffentliche Bücher
+mit diesen Einstellungen beginnen:
+
+| Dateien | `Cache-Control` | Wirkung |
+| --- | --- | --- |
+| HTML und Dateien, die unter derselben URL ersetzt werden | `no-cache` | Vor der Wiederverwendung auf Aktualisierungen prüfen. |
+| Dateien mit Inhalts-Hash oder unveränderlicher Release-Version in der URL | `public, max-age=31536000, immutable` | Ein Jahr wiederverwenden; Änderungen unter einer neuen URL veröffentlichen. |
+
+Aktiviere auf deinem Server `ETag` oder `Last-Modified`, damit unveränderte
+Dateien ohne erneuten Download ihrer Inhalte geprüft werden können. Für stabile
+URLs ist auch eine kurze Dauer wie `public, max-age=3600` möglich, wenn bis zu
+eine Stunde Verzögerung nach Änderungen akzeptabel ist. Diese Einstellungen
+folgen dem [HTTP-Caching-Standard](https://httpwg.org/specs/rfc9111.html#field.cache-control).
+
+Die lokal exportierten Laufzeitumgebungen verwenden stabile Pfade unter
+`__hyperbook_assets/directive-*/`. Gib nicht dem gesamten Verzeichnis eine
+unveränderliche Cache-Dauer: Ein Hyperbook-Update kann Dateien unter derselben
+URL ersetzen. Das gilt auch für Dateien aus deinem `public`-Verzeichnis. Setze
+die Header beim Hosting-Dienst oder Webserver. Ein Eintrag in `hyperbook.json`
+konfiguriert kein HTTP-Caching.
+
+Die versionierten Laufzeit-Releases auf `cdn.openpatch.org` senden bereits
+`Cache-Control: public, max-age=31536000, immutable`. Auch die versionierte
+Pyodide-Distribution auf jsDelivr wird im Browser zwischengespeichert; siehe die
+[Pyodide-Dokumentation](https://pyodide.org/en/stable/usage/downloading-and-deploying.html).
+Die Hosting-Einstellungen deines Buchs ändern die Header dieser CDNs nicht.
diff --git a/website/de/book/hosting/custom.md b/website/de/book/hosting/custom.md
index 7e3ce819b..4c408a9d2 100644
--- a/website/de/book/hosting/custom.md
+++ b/website/de/book/hosting/custom.md
@@ -5,6 +5,9 @@ lang: de
 
 # Auf eigenen Server bereitstellen
 
+Siehe [Caching](/hosting/caching#dateien-im-browser-zwischenspeichern) für
+empfohlene HTTP-Header bei der Konfiguration deines Servers.
+
 Du muss nur den Ausgabeordner auf deinen Server kopieren, nachdem du den build Befehl ausgeführt hast.
 
 ```
diff --git a/website/de/book/hosting/ghpages.md b/website/de/book/hosting/ghpages.md
index 94861ba23..0ee5a73c5 100644
--- a/website/de/book/hosting/ghpages.md
+++ b/website/de/book/hosting/ghpages.md
@@ -5,6 +5,9 @@ lang: de
 
 # Auf GitHub Pages bereitstellen
 
+Siehe [Caching](/hosting/caching#downloads-zwischen-builds-aufbewahren), um
+heruntergeladene Laufzeitumgebungen zwischen Workflow-Läufen wiederzuverwenden.
+
 :::alert{warn}
 Denke daran den basePath in deiner [Konfiguration](/configuration/book) zu
 setzen, when du dein Hyperbook auf GitHub bereitstellst.
diff --git a/website/de/book/hosting/glpages.md b/website/de/book/hosting/glpages.md
index fba15f972..106e07db7 100644
--- a/website/de/book/hosting/glpages.md
+++ b/website/de/book/hosting/glpages.md
@@ -5,6 +5,9 @@ lang: de
 
 # Deploy on GitLab Pages
 
+Siehe [Caching](/hosting/caching#downloads-zwischen-builds-aufbewahren), um
+heruntergeladene Laufzeitumgebungen zwischen Pages-Builds wiederzuverwenden.
+
 :::alert{warn}
 Denke daran den basePath in deiner [Konfiguration](/configuration/book) zu
 setzen, when du dein Hyperbook auf GitLab bereitstellst.
diff --git a/website/de/book/hosting/index.md b/website/de/book/hosting/index.md
index cec07a566..caa427b6a 100644
--- a/website/de/book/hosting/index.md
+++ b/website/de/book/hosting/index.md
@@ -1,3 +1,18 @@
 ---
 name: Hosting
 ---
+
+# Hosting
+
+Baue dein Hyperbook mit `npx hyperbook build` und veröffentliche `.hyperbook/out`
+auf einem statischen Hosting-Dienst. Wähle [GitHub Pages](/hosting/ghpages),
+[GitLab Pages](/hosting/glpages), [Vercel](/hosting/vercel) oder
+[einen eigenen Server](/hosting/custom).
+
+PyIDE verwendet standardmäßig das Pyodide-CDN, damit das exportierte Buch klein
+bleibt. Leser benötigen Zugriff auf dieses CDN. Setze `elements.pyide.cdn` auf
+`false`, um die vollständige Laufzeitumgebung selbst zu hosten; siehe
+[PyIDE](/elements/pyide).
+
+Unter [Caching](/hosting/caching) erfährst du, wie CI-Builds schneller werden und
+Leser große Dateien wiederverwenden, während sie aktualisierte Seiten erhalten.
diff --git a/website/de/book/hosting/vercel.md b/website/de/book/hosting/vercel.md
index db82b99fc..eceec44c2 100644
--- a/website/de/book/hosting/vercel.md
+++ b/website/de/book/hosting/vercel.md
@@ -5,6 +5,9 @@ lang: de
 
 # Deploy on Vercel
 
+Siehe [Caching](/hosting/caching#dateien-im-browser-zwischenspeichern) für
+Cache-Header bei der Konfiguration deines Deployments.
+
 Zuerst musst du eine `vercel.json`-Datei erstellen mit dem folgenden Inhalt:
 
 ```json
diff --git a/website/en/book/configuration/book.md b/website/en/book/configuration/book.md
index 8acbb4091..d2d947cfc 100644
--- a/website/en/book/configuration/book.md
+++ b/website/en/book/configuration/book.md
@@ -111,15 +111,15 @@ Here is an example configuration:
 
 ## Local assets and CDNs
 
-Hyperbook downloads large element runtimes when they are first needed and copies
-them into the build. To load a runtime from a CDN instead, set `cdn` in its
-element configuration:
+PyIDE uses Pyodide's CDN by default because its full distribution is very large.
+Other elements download their large runtimes when first needed and copy them
+into the build. Use `cdn` in an element's configuration to choose explicitly:
 
 ```json
 {
   "name": "My Hyperbook",
   "elements": {
-    "pyide": { "cdn": true },
+    "pyide": { "cdn": false },
     "typst": { "cdn": "https://assets.example.com/directive-typst/" },
     "geogebra": { "cdn": true },
     "openscad": { "cdn": false }
@@ -129,7 +129,8 @@ element configuration:
 
 | Value | Behavior |
 | --- | --- |
-| Omitted or `false` | Download and include the runtime in the build. This is the default. |
+| Omitted | Use the default CDN for `pyide`; download and include the runtime for other elements. |
+| `false` | Download and include the runtime in the build, including for `pyide`. |
 | `true` | Load the runtime from its default CDN. |
 | An HTTP(S) base URL | Load the runtime from your own server or CDN. |
 
@@ -151,6 +152,7 @@ others use the runtimes in the CLI's asset cache, which the extension shares.
 A runtime that is not downloaded yet loads from its default CDN. Run
 **Hyperbook: Download Element Runtimes...** or **Hyperbook: Download All
 Element Runtimes** in VS Code to download runtimes for offline previews.
+For PyIDE, also set `elements.pyide.cdn` to `false` to use the downloaded runtime.
 
 A custom URL must point to the contents of the corresponding
 `__hyperbook_assets/directive-/` directory from a local build. Preserve
@@ -168,3 +170,6 @@ CDN-enabled runtimes are skipped by `hyperbook assets fetch` and do not need to
 be cached for `hyperbook build --offline`. Readers still need access to the
 configured CDN. `hyperbook assets fetch --all` downloads every runtime,
 regardless of the element configuration.
+
+See [hosting and caching](/hosting/caching) for keeping downloads between CI
+builds and choosing cache headers for your exported book.
diff --git a/website/en/book/elements/pyide.md b/website/en/book/elements/pyide.md
index 907465a29..21c6b8d30 100644
--- a/website/en/book/elements/pyide.md
+++ b/website/en/book/elements/pyide.md
@@ -5,7 +5,22 @@ permaid: pyide
 
 # PyIDE
 
-The Pyodide runtime and its bundled packages are included locally in the exported book. The CLI downloads and caches them the first time a build uses this element. Imports of bundled packages work without a CDN connection. Packages installed from PyPI through the `packages` attribute still need a network connection.
+PyIDE loads Pyodide from its versioned jsDelivr CDN by default. The full runtime distribution and its bundled packages are very large, so Hyperbook keeps them out of the exported book. Readers need an internet connection; their browsers download the runtime and requested packages and can cache these files for later visits.
+
+To include Pyodide locally for self-hosted or offline books, set `elements.pyide.cdn` to `false` in `hyperbook.json`:
+
+```json
+{
+  "name": "My offline Hyperbook",
+  "elements": {
+    "pyide": { "cdn": false }
+  }
+}
+```
+
+The CLI then downloads and caches the full distribution the first time a build uses PyIDE. To prepare an offline build, run `npx hyperbook assets fetch` while online, then `npx hyperbook build --offline`. In VS Code, also run **Hyperbook: Download Element Runtimes...** and select PyIDE. Bundled packages then work without a CDN connection. Packages installed from PyPI through the `packages` attribute still need a network connection.
+
+See [CDN configuration](/configuration/book#local-assets-and-cdns) for custom asset servers and [hosting and caching](/hosting/caching) for browser and build caches.
 
 The `pyide` element represents a Python Integrated Development Environment (IDE) component.
 It is used to embed a Python coding environment within the hyperbook website.
diff --git a/website/en/book/hosting/caching.md b/website/en/book/hosting/caching.md
new file mode 100644
index 000000000..cb58dc525
--- /dev/null
+++ b/website/en/book/hosting/caching.md
@@ -0,0 +1,84 @@
+---
+name: Caching
+---
+
+# Caching
+
+Hyperbook's build cache stores downloaded runtimes on the machine running the
+CLI. HTTP caches store the files readers request from your host or a CDN.
+
+## Cache runtime downloads between builds
+
+The CLI and VS Code share an asset cache. Builds reuse downloaded runtimes
+instead of fetching them again. You can set `HYPERBOOK_ASSET_CACHE` to choose
+its directory; the default is:
+
+| System | Directory |
+| --- | --- |
+| Linux | `$XDG_CACHE_HOME/hyperbook/assets`, or `~/.cache/hyperbook/assets` |
+| macOS | `~/Library/Caches/hyperbook/assets` |
+| Windows | `%LOCALAPPDATA%\hyperbook\assets` |
+
+CI runners usually start with an empty filesystem. Restore the asset cache
+before building and save it afterwards. For example, add these steps after
+checkout and Node.js setup in your GitHub Pages workflow:
+
+```yaml
+- name: Cache Hyperbook runtimes
+  uses: actions/cache@v4
+  with:
+    path: .cache/hyperbook-assets
+    key: ${{ runner.os }}-hyperbook-assets-${{ hashFiles('package-lock.json', 'pnpm-lock.yaml', 'hyperbook.json') }}-${{ github.sha }}
+    restore-keys: |
+      ${{ runner.os }}-hyperbook-assets-
+
+- name: Build
+  env:
+    HYPERBOOK_ASSET_CACHE: .cache/hyperbook-assets
+  run: npx hyperbook build
+```
+
+The cache key saves newly needed runtimes when the book changes, and
+`restore-keys` allows later builds to reuse earlier downloads. Hyperbook checks
+cached files against its runtime manifest and downloads missing or changed
+bundles. Keep the normal online build so a cache miss can recover. See
+[GitHub's caching documentation](https://docs.github.com/en/actions/reference/workflows-and-actions/dependency-caching).
+
+On GitLab, cache `.cache/hyperbook-assets` and set the same
+`HYPERBOOK_ASSET_CACHE` variable in your Pages job. Do not publish this directory;
+publish only `.hyperbook/out`.
+
+CDN-enabled elements need no local runtime download. PyIDE uses a CDN by
+default. For an offline book, set `elements.pyide.cdn` to `false`, run
+`npx hyperbook assets fetch` while online, and then build with
+`npx hyperbook build --offline`. The `--offline` flag controls the build's
+downloads; it does not make a book with CDN URLs usable offline.
+
+## Cache files for readers
+
+If your host lets you configure response headers, use the following policies
+as a starting point for public books:
+
+| Files | `Cache-Control` | Effect |
+| --- | --- | --- |
+| HTML and files replaced at the same URL | `no-cache` | Check for updates before reusing the cached response. |
+| Files with a content hash or immutable release version in their URL | `public, max-age=31536000, immutable` | Reuse the file for a year; publish changes at a new URL. |
+
+Enable `ETag` or `Last-Modified` on your server so unchanged files can be
+revalidated without downloading their contents. For files at stable URLs, a
+short lifetime such as `public, max-age=3600` is also possible if a delay of up
+to one hour after updates is acceptable. These settings follow the
+[HTTP caching standard](https://httpwg.org/specs/rfc9111.html#field.cache-control).
+
+Hyperbook's locally exported runtimes use stable paths under
+`__hyperbook_assets/directive-*/`. Do not give that entire directory an
+immutable lifetime: upgrading Hyperbook can replace files at the same URL.
+Apply the same rule to files copied from your `public` directory. Set headers
+on your hosting service or web server; adding them to `hyperbook.json` does not
+configure HTTP caching.
+
+Openpatch's versioned runtime releases on `cdn.openpatch.org` already send
+`Cache-Control: public, max-age=31536000, immutable`. Pyodide's versioned
+jsDelivr distribution is also cached by browsers; see the
+[Pyodide deployment documentation](https://pyodide.org/en/stable/usage/downloading-and-deploying.html).
+Your book's hosting settings do not change headers sent by these CDNs.
diff --git a/website/en/book/hosting/custom.md b/website/en/book/hosting/custom.md
index 2259a9d23..0f816539d 100644
--- a/website/en/book/hosting/custom.md
+++ b/website/en/book/hosting/custom.md
@@ -4,6 +4,9 @@ name: Custom
 
 # Deploy on Your Server
 
+See [Caching](/hosting/caching#cache-files-for-readers) for recommended HTTP
+headers when configuring your server.
+
 You just need to copy the output folder after running build command.
 
 ```
diff --git a/website/en/book/hosting/ghpages.md b/website/en/book/hosting/ghpages.md
index f6f64c431..e6f9d536c 100644
--- a/website/en/book/hosting/ghpages.md
+++ b/website/en/book/hosting/ghpages.md
@@ -4,6 +4,9 @@ name: GitHub Pages
 
 # Deploy on GitHub Pages
 
+See [Caching](/hosting/caching#cache-runtime-downloads-between-builds) to reuse
+downloaded runtimes between workflow runs.
+
 :::alert{warn}
 Remember to add a basePath to your [configuration](/configuration/book), when deploying to GitHub pages.
 :::
diff --git a/website/en/book/hosting/glpages.md b/website/en/book/hosting/glpages.md
index 05795c035..2d5c95db3 100644
--- a/website/en/book/hosting/glpages.md
+++ b/website/en/book/hosting/glpages.md
@@ -4,6 +4,9 @@ name: GitLab Pages
 
 # Deploy on GitLab Pages
 
+See [Caching](/hosting/caching#cache-runtime-downloads-between-builds) to reuse
+downloaded runtimes between Pages builds.
+
 :::alert{warn}
 Remember to add a basePath to your [configuration](/configuration/book), when deploying to GitLab pages.
 :::
diff --git a/website/en/book/hosting/index.md b/website/en/book/hosting/index.md
index 854114741..87ec52a72 100644
--- a/website/en/book/hosting/index.md
+++ b/website/en/book/hosting/index.md
@@ -2,3 +2,17 @@
 name: Hosting
 index: 5
 ---
+
+# Hosting
+
+Build your Hyperbook with `npx hyperbook build` and publish `.hyperbook/out` on
+a static hosting service. Choose [GitHub Pages](/hosting/ghpages),
+[GitLab Pages](/hosting/glpages), [Vercel](/hosting/vercel), or
+[your own server](/hosting/custom).
+
+PyIDE uses Pyodide's CDN by default to keep the exported book small. Readers
+need access to that CDN. Set `elements.pyide.cdn` to `false` to host the full
+runtime yourself; see [PyIDE](/elements/pyide).
+
+See [Caching](/hosting/caching) for faster CI builds and cache headers that let
+readers reuse large runtime files while receiving updated pages.
diff --git a/website/en/book/hosting/vercel.md b/website/en/book/hosting/vercel.md
index 7c1fad80e..dc9a8b96d 100644
--- a/website/en/book/hosting/vercel.md
+++ b/website/en/book/hosting/vercel.md
@@ -4,6 +4,9 @@ name: Vercel
 
 # Deploy on Vercel
 
+See [Caching](/hosting/caching#cache-files-for-readers) for cache headers when
+configuring your deployment.
+
 First you need to create a `vercel.json` file at the root of your
 project with the following content.