[Kinflou]

Step A3 of the generator-protocol track (core#8): the CLI runs generators outside its binary, and the built-ins go through the same protocol.

Re-pin

• core 98d64fa, comline-rust 8bbce6d, comline-typescript be8bc79.
• generation is pinned at 2d922ce, not e8b8a73: both generators pin 2d922ce, and Cargo only unifies identical rev strings. codegen/ is the same at both revs (git diff 2d922ce e8b8a73 -- codegen is empty).
• Generated output is unchanged: chat_project, simple_project and validated_project give byte-identical trees in code and lib mode from main and from this branch.

Built-ins

external::builtin encodes Request::generate(..) and answers it with protocol::respond(&registry, &bytes), so the built-ins go through the same Request/Response types and the same response checks as an external generator. The cost is well under a millisecond: about 0.05–0.6 ms on chat_project in a release build, next to about 0.15–0.3 ms for generation itself.

generator = "<path>"

[[generate.target]]
generator = "tools/gen.wasm"   # relative to comline.toml's directory
language  = "rust"             # optional when the generator offers one language
timeout   = 60                 # seconds per call

• .wasm: runs under wasmtime =49.0.3 with wasmtime-wasi p1, behind cargo feature gen-wasm (on by default).
  • stdin is the encoded request.
  • stdout and stderr go to in-memory capped buffers.
  • Sandbox: no preopened directories, no environment, argv is [file name] only, and TCP, UDP and name lookup are off.
  • Memory is limited to 1 GiB through StoreLimits, with one instance.
  • The time limit uses epoch interruption. The guest also runs on its own thread, so a blocking host call can't hold the limit up.
• Native: runs as a process with env_clear(), with stdin, stdout and stderr piped. PATH isn't kept: a generator needs no other programs. On Windows only, SYSTEMROOT is kept, because a process can't load system DLLs without it. The process is killed when it times out or overflows stdout.
• Flow: Describe comes first.
  • The CLI refuses a generator whose protocols range doesn't include PROTOCOL_VERSION, or whose ir_formats doesn't include IR_FORMAT. The message names both sides.
  • It also refuses a language or version the generator doesn't offer. When several entries cover the language, it picks the newest floor at or below the target's version, as the registry does.
  • Then Generate. A Response::Error becomes generator : <message>.
  • The dependency-minimum check from lang_targets applies to external generators too.
• Limits: a 60 s default wall time, a 256 MiB stdout cap, a 256 MiB cap on total generated bytes, and stderr shown up to 64 KiB, each line prefixed [<file>].
• Paths: every FileOut.path must be non-empty and relative. It may not contain \, NUL, : or empty components, and no component may be . or ... A response with any bad path is refused whole, before anything is written. This applies to the built-ins too.

Tests

tests/fixtures/generators is a small crate built by the tests, natively and for wasm32-wasip1, into CARGO_TARGET_TMPDIR:

• real is the official Rust generator over serve_stdio, the same as comline-gen-rust. Its output is byte-identical to the built-in's, in code and lib mode, both native and wasm.
• hostile misbehaves according to its file name. Each case runs native and wasm, fails with a message naming the generator, and leaves nothing written:
  • dotdot (../)
  • absolute
  • oversized
  • timeout (an infinite loop with timeout = 2)
  • mismatch (protocol 7)
  • stderr (prefixed and capped)
  • future (version not offered)
  • readfs, wasm only: reading the filesystem fails.

CI installs wasm32-wasip1, and adds a clippy run without gen-wasm and a fmt check of the fixture crate.

Binary size (release, x86_64 Linux)

	unstripped	stripped
without gen-wasm	12.6 MB	9.5 MB
with gen-wasm (wasmtime with cranelift, runtime and std only)	31.2 MB	22.6 MB

With wasmtime's default features it was 37.6 MB unstripped and 27.0 MB stripped.

Left for later

• encodings is Default::default() until B3.
• The docs repo's reference/generator-protocol.md.