Skip to content

docs(reference): give module loading an owner in reference - #664

Open
Ethan-Arrowood wants to merge 1 commit into
mainfrom
module-loading-ref
Open

docs(reference): give module loading an owner in reference#664
Ethan-Arrowood wants to merge 1 commit into
mainfrom
module-loading-ref

Conversation

@Ethan-Arrowood

Copy link
Copy Markdown
Member

Problem

Harper's v5 application module loading is documented almost entirely in the release notes, and the reference pages that should own it point outward at them.

  • reference/components/javascript-environment has a ## Module Loading heading, but it covers ESM vs CJS, importing from harper, and Vite SSR externals. The string moduleLoader never appears in that section. It's the heading you'd click, and it's about something else.
  • reference/configuration/options.md#applications documents lockdown and moduleLoader by linking to /release-notes/v5-lincoln/v5-migration#.... That's an evergreen reference page depending on a frozen point-in-time migration doc — if moduleLoader gains a mode, the only full description of the modes lives in a document that by definition describes v4→v5.
  • The only per-mode behavior matrix anywhere in reference is buried inside ## Child Processes. You have to be researching sidecar processes to find the clearest statement of what each loader does to your imports.
  • allowedDirectory and allowedBuiltInModules were documented only in the migration guide, and were missing from the configuration reference entirely — even though allowedDirectory: app ships in core's static/defaultConfig.yaml.

Three separate pages also asserted the default loader's behavior as though it were the only behavior, which made javascript-environment ("seeded from the same process globals") read as a flat contradiction of 5.0.md ("its own global object"). They describe different modes; nothing said so.

Changes

New pagereference/components/module-loading.md, in the Components sidebar after JavaScript Environment. Covers the four moduleLoader modes with a comparison table (module cache / intrinsics / global object / application context / constrained child_process), dependencyLoader, lockdown, allowedDirectory, allowedBuiltInModules, the constrained fetch, and a symptom-driven "Choosing a Mode" section.

javascript-environment.md — renamed ## Module Loading to ## Module Formats so the heading stops colliding; scoped the intro and the process-globals claim to the loaders they actually describe; the child-process substitution table now links to the loader reference instead of re-explaining modes.

configuration/options.md — documented allowedDirectory and allowedBuiltInModules; repointed lockdown and moduleLoader at the new page.

v5-migration.md — trimmed from 66 lines to 38. Kept what breaks and how to cope; moved the mode-by-mode detail, lockdown mode list, allowedBuiltinModules block, and dependencyLoader section into reference. Deliberately kept #module-loader-modes and #intrinsic-lockdown as real sections so those anchors survive — they were the two linked from elsewhere.

5.0.md — fixed applications.allowedShellCommands. That option does not exist; the real name is allowedSpawnCommands. Anyone who followed that release note wrote a key Harper silently ignores, then had their spawn blocked with no explanation.

learn/developers/multiple-applications.mdx — notes the VM loader is the default and configurable.

Verification

Everything on the new page is checked against harper origin/main rather than carried over on faith:

  • dev-mode allowedDirectory: any override — utility/install/installer.ts:67
  • realpath-then-prefix-match, and the node: strip + first-segment match for built-ins — security/jsLoader.ts:1202-1216
  • REPLACED_BUILTIN_MODULES contains only child_process, so an allowlisted built-in still gets the substitute — security/jsLoader.ts:986
  • allowedShellCommands returns zero hits anywhere in core; allowedSpawnCommands is in static/defaultConfig.yaml:32 and utility/hdbTerms.ts:519

One inherited claim corrected along the way: the migration guide said the https-only fetch under lockdown: ses applies "only in vm mode". It's installed on the custom global object (security/jsLoader.ts:856), which both vm and compartment build, so compartment gets it too.

npm run build is clean (onBrokenLinks: 'throw', no anchor warnings) and format:check passes.

Note for reviewers

The trim drops two anchors from the migration guide: #allowed-built-in-modules and #dependency-loading. Nothing in-repo linked to them, but external links (support threads, blog posts) would break silently. Happy to add stub headings if that's a concern.

sent with Claude Opus 5

Application module loading was documented only in the v5 release notes.
The reference page that should own it (`components/javascript-environment`)
used `## Module Loading` for an unrelated topic (ESM vs CJS, importing from
`harper`), the configuration reference linked outward to the migration guide
for `lockdown` and `moduleLoader`, and the only per-mode behavior matrix was
buried under `## Child Processes`. Two options -- `allowedDirectory` and
`allowedBuiltInModules` -- were absent from the configuration reference
entirely.

- Add `reference/components/module-loading.md` covering the `moduleLoader`
  modes, dependency loading, intrinsic lockdown, allowed directory, allowed
  built-in modules, and how to choose a mode.
- Rename `javascript-environment`'s `## Module Loading` to `## Module Formats`
  so the heading stops colliding, and scope its intro and process-globals
  claim to the loaders they actually describe.
- Document `allowedDirectory` and `allowedBuiltInModules` in the configuration
  reference, and repoint `lockdown`/`moduleLoader` from the release notes to
  the new page.
- Trim the migration guide to migration-grade content, keeping the
  `#module-loader-modes` and `#intrinsic-lockdown` anchors.
- Fix `applications.allowedShellCommands` in the 5.0 notes; that option does
  not exist, the real name is `allowedSpawnCommands`.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
@Ethan-Arrowood
Ethan-Arrowood requested a review from a team as a code owner September 2, 2026 21:55
@github-actions

github-actions Bot commented Sep 2, 2026

Copy link
Copy Markdown

🚀 Preview Deployment

Your preview deployment is ready!

🔗 Preview URL: https://preview.harper-documentation.harperfabric.com/pr-664

This preview will update automatically when you push new commits.

@gemini-code-assist gemini-code-assist Bot left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Code Review

This pull request introduces a new 'Module Loading' documentation page, updates the 'JavaScript Environment' and 'Configuration Options' pages, and refactors the v5 migration guide to detail the module loading configurations and modes in Harper v5.0.0. The review feedback suggests correcting the configuration file name to harper-config.yaml and formatting a critical security warning regarding the compartment mode's bypass of the constrained child_process using a caution block for better visibility.


The loader is also what makes application context work. It gives each application a `harper` module scoped to that application: the `logger` it exports is tagged with the application name, and `config` reflects that application's own configuration. Under the VM loaders it additionally substitutes a constrained [`child_process`](./javascript-environment.md#child-processes) module.

Everything on this page is controlled by the `applications` section of `harperdb-config.yaml`:

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

medium

The configuration file name is referred to as harperdb-config.yaml here, but it has been rebranded to harper-config.yaml in v5. Please update this reference to ensure consistency with the rest of the documentation.


SES `Compartment`-based loading, using the [`ses`](https://www.npmjs.com/package/ses) implementation of the proposed Compartment API. One compartment per application, created on demand because it is considerably heavier than the other modes.

Advanced; only needed for specialized sandboxing requirements. Note that compartments resolve built-in modules through Node directly, so Harper's constrained `child_process` is bypassed entirely under this mode — the spawn allowlist, the mandatory `name` option, the single-process lock, and the `execSync` block all disappear together.

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

security-medium medium

This critical security warning regarding the bypass of the constrained child_process under compartment mode is currently combined with other concepts. To make it prominent and easily scannable, please format it using a :::caution block and present the warning in separate, distinct sentences.

References
  1. When documenting critical caveats, potential runtime errors, or limitations, format them using a :::caution block (or appropriate admonition) to ensure they are prominent and visible, while maintaining consistency with the document's existing formatting patterns.
  2. Ensure critical security warnings, such as unauthorized access risks or fallback behaviors, are presented in separate, distinct sentences rather than being combined with other concepts or buried behind semicolons, so that readers scanning the documentation can easily find them.

@kriszyp kriszyp left a comment

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This is really good! Excellent documentation! I included codex suggestions, but I'm not sure we even should follow them; as the edge cases and bugs it names aren't necessarily worth the doc complexity/confusion of adding them (in particular, I think we should file bugs, not document them).
🤖 Reviewed with Codex


`allowedDirectory` restricts where application modules may be loaded from.

- `app` (default) — an application may only load modules from within its own directory tree. Loading from outside it throws `Can not load module at <path> outside of allowed path <path>`.

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Please avoid presenting this as a complete directory boundary in current Harper releases. A VM-loaded CommonJS component can require() a file outside its application without invoking the path check; native and ordinary compartment file loads also bypass it. Even checked ESM paths use a raw string-prefix comparison, so an application rooted at /components/foo can load /components/foo-other/file.js. The root fix belongs in Harper core: apply a separator-aware path.relative() containment check on every file-loading path. Until supported releases contain that fix, document the exact enforcement matrix and these bypasses instead of promising access only within the application's tree.


## Allowed Built-in Modules

`allowedBuiltInModules` restricts which Node.js built-ins applications may import. If it is omitted, all built-ins are allowed — which is the default.

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This allowlist is not currently application-wide. Under the default VM loader, CommonJS require('fs') returns Node's module before the allowlist check; moduleLoader: native skips the check entirely; and packages selected for native loading by dependencyLoader: auto can import any built-in. Consequently, allowedBuiltInModules: [path] does not prevent application code from reaching fs. If this is intended as a security boundary, enforce it consistently in Harper core. Otherwise, qualify this as applying only to imports handled by the application loader and include the bypass matrix here.


### Constrained `fetch`

Under `lockdown: ses`, the modes that build a custom global object (`vm` and `compartment`) also install an https-only `fetch` in that global. Under `vm-current-context` and `native`, application code uses the standard global `fetch`.

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

The constrained fetch is not usable with the standard string form currently described. Shipped core derives the URL with typeof resource === 'string' || resource.url; for fetch('https://example.com') that produces boolean true, and new URL(true) throws TypeError: Invalid URL before checking the protocol. A URL object also produces undefined; only a Request-like object with .url works. Please fix and test the expression in Harper core, or document the current input limitation until that fix ships.

```yaml
applications:
lockdown: freeze-after-load # freeze-after-load (default) | freeze | ses | none
moduleLoader: vm-current-context # vm-current-context (default) | vm | native | compartment

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

The v5.0.0 page badge makes these default labels appear valid throughout v5, but the tagged core configurations differ: v5.0.0 defaulted to lockdown: freeze and moduleLoader: vm; freeze-after-load became the default in v5.0.2; allowedDirectory appeared in v5.0.4; and vm-current-context became the default in v5.1.0. Please add the required changed-version annotation for v5.1.0 and state the earlier defaults and patch-level availability explicitly, so v5.0 users do not reason from the wrong isolation model.


The loader is also what makes application context work. It gives each application a `harper` module scoped to that application: the `logger` it exports is tagged with the application name, and `config` reflects that application's own configuration. Under the VM loaders it additionally substitutes a constrained [`child_process`](./javascript-environment.md#child-processes) module.

Everything on this page is controlled by the `applications` section of `harperdb-config.yaml`:

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Please use the canonical v5 filename, harper-config.yaml. origin/main now standardizes the v5 reference on that name and explains separately that upgraded installations may still point settings_path at legacy harperdb-config.yaml. This new page otherwise reintroduces the legacy name as if it were canonical; align it while rebasing onto current main.

Harper runs as a single process. Every co-located application shares that process and its worker threads, so it is worth being precise about what is isolated between applications and what is not.

- **Module contexts are isolated.** Harper loads each application's JavaScript in its own module context using Node.js's VM module loader, giving every application a distinct module cache. One application's modules, imports, and module-scoped state are not visible to another, so two applications can depend on different packages—or different versions of the same package—without colliding.
- **Module contexts are isolated.** Harper loads each application's JavaScript in its own module context using Node.js's VM module loader, giving every application a distinct module cache. One application's modules, imports, and module-scoped state are not visible to another, so two applications can depend on different packages—or different versions of the same package—without colliding. This is the default (`moduleLoader: vm-current-context`) and it is configurable—see [Module Loading](/reference/v5/components/module-loading) for the other modes, including `native`, which drops the per-application module cache entirely.

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

High — Module-cache isolation is overstated. With the default dependencyLoader: auto, packages that do not depend on harper use Node's native loader; when two applications resolve the same package file, they share Node's cache and singleton state, contrary to this unconditional guarantee. Qualify the claim to modules handled by the application loader and note that native-loaded dependencies may share cache and state.


Reviewed f6a97c3

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants