Skip to content

Generate the module wrapper from clang's JSON AST, with a declarative export policy - #1

Merged
ZheFeng7110 merged 7 commits into
mcpplibs:mainfrom
ZheFeng7110:change-gen
Nov 6, 2026
Merged

Generate the module wrapper from clang's JSON AST, with a declarative export policy#1
ZheFeng7110 merged 7 commits into
mcpplibs:mainfrom
ZheFeng7110:change-gen

Conversation

@ZheFeng7110

@ZheFeng7110 ZheFeng7110 commented Nov 5, 2026

Copy link
Copy Markdown
Collaborator

Summary

Applies the mwrap generation principle (.agents/docs/2026-10-28-mwrap-module-wrapper-generation.md): the compiler computes the public surface, the generator only lists it, and nothing edits the result afterwards. Design record: .agents/docs/2026-11-05-json-ast-generation.md.

  • AST from clang++, no libclang. tools/ast_json.py compiles each bundle TU once with clang++ -fsyntax-only -Xclang -ast-dump=json -Xclang -ast-dump-filter=boost and streams the dump into a flat declaration table. Geometry's filtered dump is 6.8 GB; the reader handles it in about 45 s at roughly 300 MB peak memory. It tracks clang's delta-encoded loc.file and drops implicit instantiations, aliasing them back to their template. That one compile is also the syntax gate.
  • Generated files are written only by the generator. The 35 module-side patches, patches/final/, the programmatic .inc guards in reapply_hand_edits.py, and tools/curated/ are replaced by tools/export_policy.toml: guard / drop / extra / add / gmf / facade-includes / inline-namespaces / deps rules, each with a written reason. generated/meta/skipped.txt lists every name a rule touches. generated/meta/api-surface.json records the generator identity (clang release, target, defines) plus per-library GMF, export count and edges.
  • gen_exports.py --check regenerates into a scratch copy, compares byte for byte, and verifies that each facade's GMF equals the bundle it was generated from. CI uses it instead of a git diff oracle.
  • Facades are hand-maintained. The generator never rewrites src/*.cppm. tools/apply_patches.py replays only the vendored upstream/boost patches.
  • CI uses the clang++ pinned in .xlings.json (llvm 23.1.3, installed by xlings) instead of downloading the LLVM release archive. The tests.yml mcpp pin is aligned to 2026.11.5.1.

Behavior changes

  • Macro-spelled using-declarations are now visible. boost::filesystem::{absolute,canonical,equivalent,weakly_canonical} are exported under their documented spelling (previously missing). The same applies to mpl_::* from BOOST_MPL_AUX_ADL_BARRIER_DECL.
  • Entities claimed by another library are no longer re-exported by non-owner modules. As a result .deps goes from 608 to 633 edges (no import cycles), and the mcpp.toml default closure gains bind and function.
  • flyweight.inc, algorithm.inc/.deps and version.deps are generated from policy rules instead of hand-maintained final forms.
  • Facade export import blocks are unchanged.

Validation (local, x86_64 Linux)

  • gen_exports.py --check clean; gen_features.py --check clean
  • gcc 16.1: mcpp build --features all-modules OK; mcpp test --features all-modules 144/144
  • LLVM 23.1.3: mcpp test with CI groups A and B, 144/144 each (the whole closure hits clang's known source-location limit)
  • tool unit tests 19/19 (new test_ast_json.py)
  • macOS and Windows were not run locally and rely on CI

🤖 Generated with Claude Code

@ZheFeng7110
Add upstream compat design doc
1711634
@ZheFeng7110
Generate the module wrapper from clang's JSON AST, with a declarative…
cc882e2 
… export policy

Follow the mwrap generation principle (.agents/docs/2026-10-28-mwrap-module-
wrapper-generation.md): the compiler computes the public surface, the
generator only lists it, and nothing edits the result afterwards.

- tools/ast_json.py: compile each bundle TU once with
  `clang++ -fsyntax-only -Xclang -ast-dump=json -Xclang -ast-dump-filter=boost`
  and stream the dump into a flat declaration table (delta-encoded loc.file
  tracking, implicit instantiations dropped and aliased to their template).
  The same compile is the syntax gate; libclang is gone.
- tools/gen_exports.py: rebuilt on the JSON model; `--check` regenerates into a
  scratch copy and compares byte for byte; writes generated/meta/
  api-surface.json and skipped.txt; verifies each facade's GMF against its
  bundle; cache entries survive pass-2 edge merges; --libs runs replay claims.
- tools/export_policy.toml replaces the module-side patches, patches/final/,
  the programmatic .inc guards, tools/curated/ and the GMF/define overrides;
  every rule carries a reason.
- src/*.cppm are hand-maintained facades; tools/apply_patches.py replays only
  the vendored upstream/boost patches (replaces reapply_hand_edits.py).
- CI uses the clang++ pinned in .xlings.json (llvm 23.1.3) instead of a full
  LLVM archive; mcpp pin aligned to 2026.11.5.1.

Validated locally: gen_exports --check clean; gcc 16.1 all-modules build and
mcpp test 144/144; LLVM 23.1.3 groups A/B 144/144 each.
@ZheFeng7110

ZheFeng7110 commented Nov 6, 2026 •
edited
Loading

Copy link
Copy Markdown
Collaborator Author

代码审查报告（high 级别）

共发现 10 个问题，按严重程度排序。审查期间没有构建或运行代码；导出丢失问题已通过对比重新生成的 generated/exports/*.inc 与 origin/main 及上游 Boost 头文件确认。

修复某一项后，请勾选该项前面的复选框表示已解决。

第 1–4 项已在 e1402ed 中修复（含回归测试）。

概要

  • gen_exports.py 的导出缺陷： boost::token_compress_on/off、algorithm 模块的 boost::join、boost::range::formatted_range 不再被导出，分别由处理 using 声明时的三个不同缺陷导致。
  • 门面文件过期： src/algorithm.cppm 少了约 130 个名称（包括 boost::shared_ptr、boost::bind），另有约 50 个门面文件存在同样缺口；check_facade 不检查导入，CI 发现不了。
  • 增量缓存： 可能复用过期输出，CI 中 gen_exports.py --check 可能在不该通过时通过。
  • 其余：ast_json.py 别名解析缺陷、缺少 git 时执行不到的 patch 回退路径，以及 CI 开销问题。

问题列表

  • 1. tools/gen_exports.py:723：非限定作用域枚举的 using 注入被静默丢弃

    • 问题： qualified_name() 返回的名称带有枚举名（boost::algorithm::token_compress_mode_type::token_compress_on），但注入目标按命名空间级写法（boost::algorithm::token_compress_on）查找，qmap.get(target) 和 qmap.get(q) 都查不到，这条注入被跳过。
    • 失败场景： boost/algorithm/string/constants.hpp 中有 namespace boost { using algorithm::token_compress_on; }。main 分支的 algorithm.inc 导出了 using boost::token_compress_on;，本 PR 之后没有任何 .inc 导出它。import mcpplibs.mwrap.boost.algorithm; 之后调用 boost::split(v, s, boost::is_any_of(","), boost::token_compress_on) 会编译失败。

  • 2. tools/gen_exports.py:716：先到先得的注入去重导致 algorithm 丢失 boost::join

    • 问题： using 注入的去重（claimed_inject）只以注入后的限定名为键。两个库在同一名称下注入不同重载集时，后处理的库的注入被丢弃，但先处理的模块的 using 声明只包含其自身 GMF 中可见的重载。
    • 失败场景： range.inc 先认领了 boost::join（来自 range::join），algorithm 的 namespace boost { using algorithm::join; } 被跳过（main 分支曾导出 using boost::join;）。algorithm 模块使用者调用 boost::join(std::vector<std::string>{...}, ",") 只能找到 range::join，编译失败。

  • 3. tools/gen_exports.py:693：带 boost:: 前缀的 using 声明以目标名导出

    • 问题： 对以 boost:: 前缀书写的 using 声明，_using_target_name 返回 (rel, rel)，注入名变成了目标名，而不是 <外层命名空间>::<最后一段>，与 ::boost:: 分支不一致，using 声明引入的公开名称从未被导出。
    • 失败场景： boost/range/adaptor/formatted.hpp 中有 namespace boost { namespace range { using boost::range_detail::formatted_range; } }。range.inc 只有 using boost::range_detail::formatted_range;，模块使用者看不到 boost::range::formatted_range<...>。

  • 4. src/algorithm.cppm:51：门面文件未随新的 .deps 依赖更新，约 130 个名称丢失

    • 问题： 重新生成的 .inc 去掉了已被其他模块认领的实体，默认它们经由所属模块的 export import 提供；但手写门面文件没有按新的 .deps 更新，而 check_facade（gen_exports.py:1148）明确不校验导入。
    • 失败场景： algorithm.inc 少了约 130 个名称（boost::shared_ptr、weak_ptr、bind、_bi::*、enable_shared_from_this 等），现归 smart_ptr、bind、function 所有，但 src/algorithm.cppm 仍只 export import 了 core、iterator、range、tuple、type_traits。import mcpplibs.mwrap.boost.algorithm; 之后使用 boost::shared_ptr 无法编译。另有约 50 个门面文件存在同样缺口（function→bind、dll→io、signals2→bind 等）。

  • 5. tools/gen_exports.py:1340：跳过缓存的键忽略了更早处理的库的认领

    • 问题： 增量跳过缓存的键只覆盖本库 include 图中的依赖，但先到先得的认领（claimed、claimed_inject）取决于拓扑序中所有更早处理的库。无关的更早的库改变实体归属时，缓存的 .inc/.deps 不会失效。
    • 失败场景： type_erasure 和 poly_collection 都引用 boost::mpl::item_by_order_impl，按 export_policy.toml 说明其归属随处理顺序变化。若 poly_collection 的改动使其认领或释放该实体，type_erasure 的缓存 .inc 仍被复用，造成重复导出或漏导出；CI 恢复缓存后 gen_exports.py --check 照样通过，而不用缓存运行时结果不同。

  • 6. tools/ast_json.py:274：lookups 数组中的 id 被错误别名到模板

    • 问题： 跳过隐式实例化时，所有以 { 开头的行之后的 "id" 都被别名到该模板，这也会匹配 lookups 数组元素，而这些 id 是被引用的声明（通常是 std:: 中的），不是实例化内部的节点。
    • 失败场景： 某实例化模式中有 UnresolvedLookupExpr，其 lookups 包含 std::swap（id 为 X），于是 alias[X] = <某 boost 模板>。之后任何引用 std::swap 的实体都会解析到该模板，collect_body_edges 会添加错误的 .deps 依赖（或一个被跳过的环）。

  • 7. tools/apply_patches.py:84：没有 git 时 patch -p1 回退路径执行不到

    • 问题： 文档说明 Git 不可用时回退到 patch -p1，但 _run('git', ...) 使用 subprocess.run，git 未安装时抛出 FileNotFoundError 而不是返回非零退出码。
    • 失败场景： 在没有 git 的主机上从源码包运行 python3 tools/apply_patches.py，处理第一个补丁时即因 FileNotFoundError 崩溃，vendored 源码树没有打上补丁。

  • 8. tools/gen_exports.py:1258：闭包哈希遗漏相对路径 include 和宏 include

    • 问题： 负责让 gate 缓存和跳过缓存失效的 _closure_hash 只追踪 #include <boost/...> 和 "boost/..." 写法，不追踪相对路径引号 include 和宏计算出的 include（BOOST_PP_ITERATE、经 BOOST_PP_STRINGIZE 引入的 mpl 预处理头文件）。
    • 失败场景： 修改或打补丁的头文件若只通过宏 include 引入（如 item_by_order_impl 背后的 mpl 预处理 map 头文件），闭包哈希不变，过期的 .inc 和裁剪后的 GMF 被复用，--check 仍报告"已是最新"。

  • 9. .xlings.json:4：仅生成器需要的 LLVM 23 被每个 CI 任务安装

    • 进度： 9a（测试任务只安装 mcpp）已修复；9b（gen_audit 重复转储 AST）留待后续。
    • 问题： 仅生成器使用的 LLVM 23.1.3 被加入共享工作区清单，而 tests.yml 在每个构建任务中都运行 xlings install -y，每个任务都多装一整套用不到的 LLVM；gen_audit（gen_audit.py:192）还会在 gen_exports --check 后再次执行数 GB 的 AST 转储。
    • 失败场景： 测试矩阵中约 10 个任务（使用 llvm@22.1.8 或 gcc）都要额外安装 LLVM 23。verify-scripts 中即使 gen_exports 命中缓存，gen_audit 仍重新转储所有 bundle（仅 geometry 就有数 GB），生成器 CI 时间约翻倍。

  • 10. tools/gen_exports.py:915：重复构建 by_key 索引；跳过缓存的键哈希了错误参数

    • 问题： collect_body_edges 重新构建了 closure_from 在第 860 行已构建的同一 by_key 索引，每个库多一整遍计算。另外 _skip_key 哈希的是 bc.CLANG_ARGS，而 gate 键哈希的是 aj.clang_command(...)，切换 CXX/CLANG_NO_DEFAULT_CONFIG 不会让跳过缓存失效。
    • 失败场景： 对声明数以百万计的实例化密集 bundle，重复构建索引使该阶段耗时翻倍（应在 Model 上构建一次后共享）。另外按 CI 方式运行过一次后，本地未设置 CLANG_NO_DEFAULT_CONFIG 运行时，可能复用在不同驱动配置下生成的 .inc 文件。

更新： 5～8 已在 4ffac21 处修复。

@ZheFeng7110
Export the names using-declarations publish; facades import every .de…
e1402ed 
…ps edge

Generator (tools/gen_exports.py):
- A using-declaration written `boost::...` injects its last segment into the
  enclosing namespace, like the `::` and `std::` forms
  (boost::range::formatted_range).
- Enumerators of an unscoped enum are also indexed under the enclosing scope,
  so `using algorithm::token_compress_on;` resolves (boost::token_compress_on).
- Injection dedup is keyed by (name, target): range's boost::join no longer
  swallows algorithm's boost::join, a different overload set.
- check_facade requires each facade's `export import` block to cover its
  .deps edges; an edge a facade cannot take is declared in the policy's
  [<lib>.facade] omit-imports with a reason (range -> regex, scope -> core,
  sort -> range).

Facades: add the 44 facades' missing .deps edges (algorithm regains
boost::shared_ptr, boost::bind, ... through smart_ptr/bind/function).
Policy: drop the atomic BOOST_HAS_INT128 guards the corrected injected names
no longer match. Regenerated generated/.

Tests: unit tests for the three generator fixes and the facade check;
consumer assertions in tests/algorithm.cpp and tests/range.cpp (they fail on
the previous commit).
@ZheFeng7110 ZheFeng7110 mentioned this pull request Nov 6, 2026
Open
4 tasks
@ZheFeng7110
Validate generator caches against what clang read and the ownership s…
4ffac21 
…een; check for cache drift nightly

- apply_patches: a missing git is a failed run, so the patch -p1 fallback
  is reachable; failures report both tools.
- ast_json: while skipping an implicit instantiation, alias only nodes of
  its "inner" arrays, not the declarations listed in lookups.
- gen_exports: gate and skip entries record every file clang read (its -MD
  dependency list) and are valid only while those files are unchanged;
  skip entries also record the first-wins ownership they were computed
  under and are valid only while it still holds.
- verify-scripts: nightly cache-drift job compares cached, cold and
  committed generation, uploads the evidence, saves the cold cache and
  reports to a cache-drift issue; toolchain setup moves to a shared
  composite action.
@ZheFeng7110
Install only mcpp on the build legs; share the export index; key skip…
8e1a7b2 
…s on the driver line

- tests.yml: install `mcpp@$MCPP_VERSION` instead of the whole .xlings.json
  manifest; its llvm pin is the generator's clang++, which the build legs
  never use.
- gen_exports: the {stable key: decl} export index is built once per Model
  (export_index) and shared by closure_from and collect_body_edges; the
  closure's extra targets stay local to closure_from, so the output is
  unchanged (cold --check up to date).
- gen_exports: the skip key hashes the clang driver line, as the gate key
  does, so CXX and CLANG_NO_DEFAULT_CONFIG invalidate it.
@ZheFeng7110
Cache linkage audit reports; key them on the compile gate
f91817f 
- gen_exports: the gate key is computed by gate_key(lib, headers, policy),
  shared by parse_bundle and gen_audit; its value is unchanged.
- gen_audit: each library's report and counts are cached in
  target/gen/exports-cache/<lib>.audit.json, keyed on the gate key, the
  generator's code and gen_audit.py itself, and valid only while every file
  clang read for it is unchanged. A hit skips the AST dump, so the audit no
  longer re-dumps every library after a fully skipped gen_exports --check.
@ZheFeng7110
Check the audit cache for drift nightly
9169427 
- check_cache_drift: both passes also run gen_audit.py, and the linkage
  reports of the cached and cold runs must match. The reports are not
  committed, so a stale replay has nothing to be caught against on pull
  requests. Drift is reported with its diff and the affected .audit.json
  entries, and saves the cold cache like export drift does.
- verify-scripts: comments name the audit cache.
@ZheFeng7110
ZheFeng7110 merged commit d1d2504 into mcpplibs:main Nov 6, 2026
20 checks passed
@ZheFeng7110
ZheFeng7110 deleted the change-gen branch November 6, 2026 14:12
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Reviewers

No reviews

Assignees

No one assigned

Labels

None yet

Projects

None yet

Milestone

No milestone

Development

Successfully merging this pull request may close these issues.

1 participant

@ZheFeng7110