From 0b0d09545f6c9db9f56b36bedadf8676fd1675aa Mon Sep 17 00:00:00 2001 From: speak-agent <248744407+speak-agent@users.noreply.github.com> Date: Tue, 18 Aug 2026 06:09:00 +0800 Subject: [PATCH 1/9] fix(modgraph,manifest,pack): module_extensions is the knob, and everything follows it MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit `.ixx` is not built in and should not be: the extension set is configuration, not a list mcpp grows one entry at a time. What that owes in return is that ONE declaration is enough — and three places were not holding up their end. **The scanner accepted what it could not classify.** An `.ixx` matched by `sources` but declared nowhere produced a compile edge whose object nothing links. Measured: build obj/mathkit.ixx.o | gcm.cache/mathkit.gcm : cxx_object … bmi_out = gcm.cache/mathkit.gcm ← the BMI was produced build bin/app : cxx_link obj/main.o ← the object is not here ld: undefined reference to `mk::answer@mathkit()' Two answers to "is this a module interface" and only one of them read: the scanner sees `export module` and records `provides`, which is why the edge got a `bmi_out`; the classifier says `Other`, and the link set is built from the classifier. Refused now, where the classification happens, naming the file, the extension and the key. **The lib-root convention hard-coded `.cppm`**, so packing an `.ixx` library started its closure at a file that does not exist: $ mcpp pack mathkit Interface (headers only) ← the module interface, gone Withheld (nothing) Packed …-x86_64-linux-gnu ← the C-SURFACE tag Silently wrong in both directions, and the second one is worse than the first: an empty published set is precisely how the packer recognises a C surface, so losing the interface also downgraded the compatibility tag and the gate stopped checking compiler and stdlib. The convention now offers one candidate per declared extension and the packer takes the one that exists. **The generated manifest listed `.ixx` sources without saying what an `.ixx` is**, so every consumer was refused by the classifier. The package now declares it — computed from the published FILES, so it cannot disagree with `sources`, and absent entirely for a `.cppm` package, whose manifest is unchanged. ⚠️ The probing resolver lives in `mcpp.manifest.toml`, not in `mcpp.manifest.types` where its sibling is, and the reason is measured: probing needs the extension table, and adding that import to `types` — a module nearly everything depends on — made GCC 16.1 ICE while compiling `src/main.cpp`, an unrelated file, with a cleared gcm.cache. Same module-poisoning shape this project has hit before. The edge is not added; the function moves to where the edge already is. --- ...ws-shared-library-and-module-extensions.md | 320 ++++++++++++++++++ src/manifest/toml.cppm | 72 ++++ src/manifest/types.cppm | 15 +- src/modgraph/scanner.cppm | 39 +++ src/pack/library.cppm | 15 + src/pack/library_pipeline.cppm | 8 +- src/pack/manifest_emit.cppm | 16 + src/source_kind.cppm | 16 + .../260_module_extension_is_configuration.sh | 92 +++++ .../261_pack_follows_the_module_extension.sh | 150 ++++++++ 10 files changed, 735 insertions(+), 8 deletions(-) create mode 100644 .agents/docs/2026-08-18-windows-shared-library-and-module-extensions.md create mode 100755 tests/e2e/260_module_extension_is_configuration.sh create mode 100755 tests/e2e/261_pack_follows_the_module_extension.sh diff --git a/.agents/docs/2026-08-18-windows-shared-library-and-module-extensions.md b/.agents/docs/2026-08-18-windows-shared-library-and-module-extensions.md new file mode 100644 index 00000000..892ef559 --- /dev/null +++ b/.agents/docs/2026-08-18-windows-shared-library-and-module-extensions.md @@ -0,0 +1,320 @@ +# Windows 动态库分发、cl.exe 消费,与 `.ixx` 的默认支持 + +> 2026-08-18 · 针对 2026.8.18.1 发布后遗留的 P1 / P2 / `.ixx` 三问 +> 状态:**分析与设计,未实施** + +三个问题看起来独立,实际共用一条主线:**mcpp 在这三处都把「工具链没有替我们 +做的事」当成了「做不了的事」**,而工业界对每一件都有已经跑了十年以上的做法。 + +| # | 问题 | 一句话结论 | +|---|---|---| +| **Q1** | `kind = "shared"` 在 `*-windows-msvc` 上被拒 | 拒绝的理由(符号导出)成立,但**结论不成立** —— CMake 十年前就用自动生成 `.def` 解决了,且不依赖 dumpbin | +| **Q2** | 打包库不能被原生 `cl.exe` 消费 | 是 mcpp 自己的 manifest 用了 GNU 拼写;方言中立通道已经存在,只是不在条件通道里 | +| **Q3** | `.ixx` 默认是否支持 | **不支持,而且失败方式很糟**:实测编译出对象、然后从链接里消失,报 `undefined reference` | + +--- + +## 1. Q3 先说,因为它是实测出来的,而且最脏 + +### 1.1 实测 + +```toml +[build] +sources = ["src/*.ixx", "src/*.cpp"] # 没有 module_extensions +``` + +``` +$ mcpp run + Compiling ixxprobe v0.1.0 (.) +error: build failed +failed: bin/ixxprobe +ld: obj/main.o: in function `main': + undefined reference to `mk::answer@mathkit()' +``` + +加上 `module_extensions = [".ixx"]` 之后同一份代码 `ixx-ok=42`,通过。 + +### 1.2 根因:两个答题者对「这是不是模块接口」给了不同答案 + +看未声明时生成的 `build.ninja`: + +```ninja +build obj/mathkit.ixx.ddi : cxx_scan .../src/mathkit.ixx +build obj/mathkit.ixx.o | gcm.cache/mathkit.gcm : cxx_object .../src/mathkit.ixx | ... + bmi_out = gcm.cache/mathkit.gcm ← BMI 产出了 + +build bin/ixxprobe : cxx_link obj/main.o ← obj/mathkit.ixx.o 不在里面 +``` + +- **扫描器**读到 `export module mathkit;`,记下 `provides = mathkit`,于是规划器给这条边挂了 `bmi_out`; +- **分类器** `classify()` 对不在表里的扩展名返回 `SourceKind::Other`(`source_kind.cppm:322`); +- 而链接集合只看 kind:`links_unconditionally(Other) == false`,`is_implementation_source(Other) == false`; +- 于是**对象被构建出来,然后没有任何人链接它**。 + +这是本仓库反复出现的那个形状:**同一个决定有两处推导,而下游只读其中一处**。 + +### 1.3 `.ixx` 在三个编译器上本来就能用 + +mcpp **从不依赖驱动的扩展名启发式**:模块接口的语言由 +`bmi_traits(tc).moduleInterfaceLangFlag` 显式给出 —— MSVC `/interface /TP`、 +clang `-x c++-module`、gcc `-x c++`(`model.cppm:309/325/346`)。所以 +`.ixx` 一旦进了表,三个编译器都直接可用;这与「clang 的驱动不认识 `.ixx`」 +并不矛盾,因为 mcpp 根本没让驱动去猜。 + +> ⚠️ 我此前的笔记记成「Clang 不认 `.ixx`」并把它当成产品限制。那说的是**驱动默认**, +> 而 mcpp 覆盖了它 —— 判据要落在 mcpp 实际发出的命令行上,不是编译器的默认行为。 + +### 1.4 因此:**不是内置 `.ixx`,而是让「已声明」真的自动适配、「未声明」不再静默** + +⚠️ **这一节在 review 中被更正过一次。** 第一版结论是「把 `.ixx` 放进 +`builtin_extension_table()`」。那是错的方向:**扩展名集合是配置,不是 mcpp 要 +逐个追加的内置清单** —— 今天补 `.ixx`,明天补 `.ccm`、`.cxxm`,而每一次追加都是 +mcpp 替工程做了一个工程自己能说清楚的决定。 + +真正要成立的是两件事: + +1. **已声明的扩展名必须在三个工具链上都直接可用 —— 这一条今天已经成立。** + 模块接口的语言由 `BmiTraits::moduleInterfaceLangFlag` 显式给出 + (`/interface /TP` / `-x c++-module` / `-x c++`),mcpp 从不让驱动去猜扩展名。 + 所以 `module_extensions = [".ixx"]` 一写,gcc / clang / MSVC 全部可用 + ——**实测 `ixx-ok=42`**。 +2. **未声明的扩展名必须当场拒绝,而不是编译出一个没人链接的对象。** + 这才是缺陷本体,而且它与 `.ixx` 无关:任何 `classify()` 归为 `Other` 的扩展名 + 都会掉进同一个洞。 + +拒绝写在**扫描器**里(分类发生的那一处),消息点名文件、扩展名与该写的键: + +``` +error: scanner errors: + …/src/mathkit.ixx: 'mathkit.ixx' is listed in [build] sources, and mcpp has + no role for the extension '.ixx'. + Its object would be compiled and then linked by nothing, so this is refused + rather than built. If it is a module interface, declare the extension: + [build] + module_extensions = [".ixx"] + Otherwise remove it from `sources` — headers belong in `include_dirs`, and + Windows resource scripts in `[resources]`. +``` + +**为什么不猜。** 让 mcpp 看到未知扩展名里有 `export module` 就自动当模块接口, +是给「这是不是模块接口」加**第三个**答题者 —— 而本节的缺陷正是前两个答题者 +(扫描器的 `provides` 与分类器的 `kind`)不一致造成的。 + +--- + +## 2. Q1:MSVC 的 `kind = "shared"` + +### 2.1 拒绝的理由成立 + +MSVC 在没有 `__declspec(dllexport)`、也没有 `.def` 时,DLL 不导出任何东西 —— +导入库为空,消费者拿到一片 unresolved externals,而那些符号就在对象里。 +这一点在链接器层面也被证实:**lld-link 的 MSVC 形态不做 auto-export**, +而 MinGW 形态做;LLVM 侧的说法是 MSVC 路径要控制导出数量,以免撞上 +**PE 的 65535 导出上限**([lld-link auto-export 讨论][lld])。 + +所以 `*-windows-gnu` 能用而 `*-windows-msvc` 不能,差别确实只在「链接器替不替你导出」。 + +### 2.2 但「做不了」是错的 —— 工业界有两条成熟路径 + +| 路径 | 代表 | 机制 | 正确性 | 需要改源码 | +|---|---|---|---|---| +| **A. 导出宏** | MSVC 官方指导、Qt、Boost、CMake `generate_export_header` | 作者在声明上标 `__declspec(dllexport/dllimport)` | **完全正确**,含数据符号与 vtable | 是 | +| **B. 自动 `.def`** | **CMake `WINDOWS_EXPORT_ALL_SYMBOLS`**(2015 年,CMake 3.4) | 扫描 `.obj` 的 COFF 符号表,生成 `.def` 交给链接器 | 函数正确;**数据符号与 vtable 有明确限制** | 否 | +| C. MinGW auto-export | GNU ld / lld-mingw | 无显式导出时导出全部 | 同 B,且不适用于 MSVC ABI | 否 | +| D. 干脆用静态库 | 相当一部分项目在 modules 时代的选择 | —— | —— | 否 | + +**B 是关键发现,因为它推翻了「需要 dumpbin」这个隐含前提。** CMake 的实现 +`bindexplib` **自己解析 COFF**,不调用任何外部工具([bindexplib.cxx][bx]): +读 image header 的 machine 字段(I386/AMD64/ARM/ARMNT/ARM64),遍历符号表,按下列 +规则筛选: + +- 只取 `IMAGE_SYM_CLASS_EXTERNAL` 且 `SectionNumber > 0`、Type 为 `0x20` 或 `0x0` 的符号; +- **DATA 判定**:Type == 0 且所在节带 `IMAGE_SCN_MEM_WRITE` ⇒ 写成 `name DATA`; +- 跳过 `??_G` / `??_E`(析构变体)、含 `.` 的托管符号、`__t2m` / `$$F` / `$$J`、 + ARM64EC 的 `$ientry_thunk` 等 thunk 变体; +- i386 与 `__cdecl` 情形去掉前导下划线。 + +CMake 官方文档同时**明确写出它的边界**([WINDOWS_EXPORT_ALL_SYMBOLS][cm]): + +> Global **data** symbols must be explicitly marked with `__declspec(dllimport)` +> in order to link to data in the `.dll`. +> +> In cases that the compiler generates references to the virtual function table, +> such as in a delegating constructor of a class with virtual functions, the +> whole class must be marked with `__declspec(dllimport)`. + +**这两条限制不是可以隐瞒的实现细节 —— 它们决定了 B 是默认值而不是终点。** + +### 2.3 C++20 modules 与 DLL 的官方姿势,恰好就是 mcpp 的模型 + +微软在两个 Q&A 里给出的答案一致([Q&A 1665106][ms1]、[Q&A 1695780][ms2]): + +1. 模块接口里的实体**必须**带 `__declspec(dllexport)`: + ```cpp + export __declspec(dllexport) void myFunction(); + export class __declspec(dllexport) Test { public: Test(); }; + ``` +2. 消费者要么拿到 `.ifc`(用 `/reference` 指过去),**要么把 `.ixx` 加进自己的工程, + 由自己编译出 `.ifc`**。 + +第二条正是 **mcpp 的分发模型**:发接口**源码**,消费者自己编。所以 mcpp +不需要发 `.ifc` —— 而 `.ifc` 与编译器构建逐位绑定,发它等同于发 BMI, +mcpp 已明确不做(docs/12「当前边界」)。 + +> 换句话说:**mcpp 在 modules × DLL 这件事上的模型是微软自己推荐的那一条, +> 缺的只有导出。** + +--- + +## 3. Q2:为什么原生 `cl.exe` 消费不了打包库 + +### 3.1 现状 + +生成的 manifest 用 GNU 拼写选每条腿: + +```toml +[target.'cfg(all(arch = "x86_64", os = "windows", env = "msvc"))'.build] +ldflags = ["-Llib/x86_64-windows-msvc", "-lmathkit"] +``` + +clang(Windows 默认工具链)吃这一套;**原生 `cl.exe` 不认 `-L`**。 + +### 3.2 两条看起来能走、实测走不通的路 + +**(a) 直接写文件路径。** 每个 driver 都接受裸路径作为链接输入,看起来是方言中立解。 +实测失败: + +``` +ld: cannot find lib/x86_64-windows-gnu/libmathkit.a +``` + +ninja 执行链接命令时 cwd 是**输出目录**,而只有 include 家族前缀(`-I`、`-L` …) +会被 `normalize_include_flags` 相对包根绝对化;无前缀 token 没有挂靠点。 +而 manifest 里写绝对路径就不再可重定位。 + +**(b) 把链接意图挪到方言中立通道。** `[runtime] link_library_dirs` + `libraries` +本来就按 flavor 渲染(`flags.cppm` 的 `LinkIntentFlavor::PeMsvc` → `/LIBPATH:` + +`.lib`)。问题是它**只在顶层读**,而包需要**按腿**给 —— 两条腿的库同名时, +顶层单一搜索路径会让链接器挑到错的那条。 + +而把它做成条件段 `[target.'cfg(…)'.runtime]` 有一个实测过的陷阱: +**旧版 mcpp 读到它不报错,而是静默忽略**。于是把 flag 从 `ldflags` 挪过去, +会让所有旧客户端**一个链接 flag 都拿不到**。 + +### 3.3 因此:两种拼写都带,新客户端优先中立形式 + +这是唯一同时满足「老客户端仍可用」与「cl.exe 可用」的形状: + +```toml +# 老客户端读这个(GNU 拼写,clang/gcc 可用)——保持现状,不动 +[target.'cfg(...)'.build] +ldflags = ["-Llib/", "-lmathkit"] + +# 新客户端读这个,并在读到时忽略上面那条 +[target.'cfg(...)'.runtime] +link_library_dirs = ["lib/"] +libraries = ["mathkit"] +``` + +**判据:重复不是问题,冲突才是。** 新客户端若两条都应用,`cl` 仍会因为 `-L` 失败; +所以规则必须是「条件 `runtime` 存在 ⇒ 同一条腿的 `ldflags` 不再应用」, +而不是简单相加。这条规则只对**分发包**(`provenance` 以 `mcpp-pack` 开头)生效, +普通工程的 `ldflags` 语义不变。 + +--- + +## 4. 方案:分四层,零新增 manifest 段 + +沿用库分发的既有判据 —— **新增段 0 个**;第 3 层复用的是已经存在的 `[runtime]` 键, +只是让它可以出现在条件通道里。 + +### L1 — 未知扩展名不再静默(独立、最小、立即可做) + +| 改动 | 位置 | +|---|---| +| `builtin_extension_table()` **保持 `{".cppm"}`** —— 扩展名集合是配置 | `source_kind.cppm` | +| `sources` 命中但 `classify()` 为 `Other` ⇒ **硬错误**,点名文件、扩展名与 `module_extensions` | `scanner.cppm::scan_file`,分类发生的那一处 | + +- **实测两侧**:未声明 `.ixx` ⇒ 上面那条消息;声明后 ⇒ `ixx-ok=42`。 +- ⚠️ 这是**行为变化**:`sources` 里混进 `.md` / `.txt` 的工程会开始报错。 + 需要一轮全量 e2e 确认没有工程靠「未知扩展名被静默忽略」活着。 +- docs/10 的「用了 `module_extensions` 的包要声明版本下限」**不变** —— + 它本来就是关于「旧客户端不认识这条键」的,与本层无关。 + +### L2 — MSVC 自动 `.def`(把拒绝变成默认可用) + +新增一个构建图节点,不是一个 flag: + +``` +obj/*.obj ──▶ [def-gen] ──▶ bin/.def ──▶ link /DEF:.def /DLL +``` + +- **实现:mcpp 自己解析 COFF**,规则照 §2.2 列的 bindexplib 语义。 + 判据:**不得依赖 `dumpbin`** —— 它只在 VS 开发者环境里,而 mcpp 的 Windows 默认 + 工具链是 clang,`mcpp build` 不进 VS 环境。`llvm-nm` 是可选的第二实现,不是首选: + 多一个外部依赖就多一处「这台机器上没有」。 +- 导出数量超过 **65535** 时必须**报错并说明**,不是截断。 +- 生成的 `.def` 是构建产物,进 `target/`,**不进分发包** —— 包里带的是 DLL 与导入库。 + +### L3 — 导出宏(正确性上限,与 L2 并存) + +L2 的两条限制(数据符号、vtable)**无法由工具消除**,只能由标注消除。所以: + +- mcpp 提供生成导出头的能力(等价 CMake 的 `generate_export_header`), + 或约定一个 `MCPP_EXPORT` 宏; +- 与 modules 的写法就是微软示例的那个:`export __declspec(dllexport) void f();` +- **默认走 L2,作者需要导出数据或跨 DLL 的多态类型时走 L3。** + docs 必须把这条边界写在「当前边界」里,而不是让人踩到。 + +### L4 — cl.exe 消费 + +- `ConditionalConfig` 增加承载 `[runtime]` 链接意图的能力(`link_library_dirs` / + `libraries`); +- `manifest_emit` 对每条腿**同时**写 `ldflags` 与条件 `runtime`; +- 消费侧:分发包的某条腿若有条件 `runtime`,**忽略同腿的 `ldflags`**。 + +--- + +## 5. 依赖与顺序 + +``` +L1(.ixx + 未知扩展名) ── 独立,可先合 +L2(自动 .def) ── 依赖:Windows CI 有 msvc job(已有) + └─ L3(导出宏) ── 依赖 L2 的边界文档 +L4(cl.exe 消费) ── 独立于 L2/L3,但只有 L2 落地后才有 msvc 动态库可消费 +``` + +## 6. 验证矩阵(每一格都要能指出「失败时报什么」) + +| 断言 | 载体 | 平台 | +|---|---|---| +| `.ixx` 无需声明即可用 | e2e | 三平台 | +| `sources` 里的未知扩展名被拒并点名 | e2e | 三平台 | +| MSVC DLL 导出非空、消费者链接通过 | e2e `# requires: msvc` | Windows | +| 导出的数据符号在消费端仍需 `dllimport`(**限制本身可复现**) | e2e | Windows | +| 导出数 > 65535 时报错而非截断 | 单测(合成符号表) | 三平台 | +| `cl.exe` 消费打包库 | e2e `# requires: msvc` | Windows | +| 老客户端仍能消费同一个包 | e2e(静态 + 真实) | 三平台 | + +## 7. 明确不做 + +- **不发 `.ifc` / BMI。** 与编译器构建逐位绑定;微软给的第二条路(发 `.ixx` 让消费者自编) + 正是 mcpp 已经在做的。 +- **不自动给源码插 `dllexport`。** 那是改用户代码。 +- **不在 MinGW 上生成 `.def`。** 链接器已经自动导出,再生成一份只会引入第二个真相。 + +--- + +## 参考 + +- [WINDOWS_EXPORT_ALL_SYMBOLS — CMake 文档][cm](限制原文) +- [Kitware/CMake `Source/bindexplib.cxx`][bx](COFF 筛选规则) +- [LLVM:lld-link 的 MSVC 形态不做 auto-export][lld] +- [Microsoft Q&A:C++20 modules 在共享库中的用法][ms1] +- [Microsoft Q&A:不发 `.ifc` 可行吗][ms2] + +[cm]: https://cmake.org/cmake/help/latest/prop_tgt/WINDOWS_EXPORT_ALL_SYMBOLS.html +[bx]: https://github.com/Kitware/CMake/blob/master/Source/bindexplib.cxx +[lld]: https://github.com/llvm/llvm-project/pull/71087 +[ms1]: https://learn.microsoft.com/en-gb/answers/questions/1665106/how-to-use-c-20-modules-in-shared-libraries +[ms2]: https://learn.microsoft.com/en-us/answers/questions/1695780/c-20-modules-in-shared-libraries diff --git a/src/manifest/toml.cppm b/src/manifest/toml.cppm index b73deaab..5039f01e 100644 --- a/src/manifest/toml.cppm +++ b/src/manifest/toml.cppm @@ -35,6 +35,18 @@ std::expected upsert_dependency_text(std::string_view source, const DependencyTextEdit& edit); +// Every path the lib-root convention would accept, in extension-table order +// (`.cppm` first, then whatever `[build] module_extensions` declares). An +// explicit `[lib] path` collapses this to that one entry. +std::vector lib_root_candidates(const Manifest& manifest); + +// The lib root that EXISTS under `projectRoot`. `mcpp.manifest.types` has the +// non-probing form, which answers with the conventional NAME and is the right +// one for a diagnostic; this is the right one for "which file is actually +// there", and a project whose interfaces are `.ixx` needs it. +std::filesystem::path resolve_lib_root_path(const Manifest& manifest, + const std::filesystem::path& projectRoot); + } // namespace mcpp::manifest namespace mcpp::manifest { @@ -2026,4 +2038,64 @@ upsert_dependency_text(std::string_view source, return text; } + +// ── the lib root that actually exists ──────────────────────────────────── +// +// ⚠️ LIVES HERE, NOT IN mcpp.manifest.types, AND THE REASON IS MEASURED. +// Probing needs the extension table (`mcpp.source_kind`), which this module +// already imports and `types` does not. Adding that import to `types` — a +// module nearly everything depends on — made GCC 16.1 ICE while compiling +// `src/main.cpp`, a file unrelated to the change, and clearing gcm.cache did +// not help. That is the module-poisoning shape this project has hit before: a +// NEW edge into a low-level module whose interface carries std types. So the +// edge is not added; the function moves to where the edge already is. + +std::vector lib_root_candidates(const Manifest& manifest) { + if (!manifest.lib.path.empty()) return { manifest.lib.path }; + + // Convention: `src/.` — ONE + // candidate per DECLARED extension, not just `.cppm`. + // + // A project whose interfaces are `.ixx` says so in + // `[build] module_extensions`, and the convention used to look only for + // `src/.cppm`, a file that does not exist there. Measured on such a + // package: + // + // $ mcpp pack mathkit + // Interface (headers only) ← the module interface, gone + // Withheld (nothing) + // Packed …-x86_64-linux-gnu ← the C-SURFACE tag + // + // Both halves silently wrong: nothing publishes the interface, so no + // consumer can `import mathkit`; and an empty published set is exactly how + // the packer recognises a C surface, so the package also stops constraining + // the C++ ABI and the compatibility gate stops checking compiler and + // stdlib. `mcpp pack` has to follow the project's extension choice on its + // own — that is what makes `module_extensions` a knob rather than a knob + // plus a second thing to remember. + // + // `extension_table_for` keeps `.cppm` first, so a project that has both + // keeps today's answer. + std::string tail = manifest.package.name; + if (auto p = tail.rfind('.'); p != std::string::npos) tail = tail.substr(p + 1); + + const auto table = mcpp::extension_table_for(manifest.buildConfig.moduleExtensions); + std::vector out; + out.reserve(table.moduleInterface.size()); + for (auto const& ext : table.moduleInterface) + out.push_back(std::filesystem::path("src") / (tail + ext)); + return out; +} + +std::filesystem::path resolve_lib_root_path(const Manifest& manifest, + const std::filesystem::path& projectRoot) { + auto candidates = lib_root_candidates(manifest); + std::error_code ec; + for (auto const& rel : candidates) + if (std::filesystem::is_regular_file(projectRoot / rel, ec)) return rel; + // None on disk: hand back the conventional first candidate so the caller's + // diagnostic names the file it expected rather than nothing at all. + return candidates.front(); +} + } // namespace mcpp::manifest diff --git a/src/manifest/types.cppm b/src/manifest/types.cppm index 6240cfe7..2940ad81 100644 --- a/src/manifest/types.cppm +++ b/src/manifest/types.cppm @@ -918,6 +918,12 @@ std::vector dialect_flags(const BuildConfig& bc); // libstdc++/libc++ headers declare, or participates in BMI dialect checks). bool is_dialect_flag(std::string_view flag); +// The lib root's CONVENTIONAL name: `src/.cppm`, or `[lib] path` +// when the manifest states one. It does not touch the filesystem, so it is the +// right answer for a diagnostic or a validator's expectation and the wrong one +// for "which file is actually there" — `mcpp.manifest.toml` owns the probing +// form, because probing needs the extension table and this module deliberately +// does not import it (see the note there). std::filesystem::path resolve_lib_root_path(const Manifest& manifest); // True if the manifest declares at least one `kind = "lib"` target. @@ -1094,14 +1100,9 @@ bool has_lib_target(const Manifest& manifest) { } std::filesystem::path resolve_lib_root_path(const Manifest& manifest) { - if (!manifest.lib.path.empty()) { - return manifest.lib.path; - } - // Convention: src/.cppm + if (!manifest.lib.path.empty()) return manifest.lib.path; std::string tail = manifest.package.name; - if (auto p = tail.rfind('.'); p != std::string::npos) { - tail = tail.substr(p + 1); - } + if (auto p = tail.rfind('.'); p != std::string::npos) tail = tail.substr(p + 1); return std::filesystem::path("src") / (tail + ".cppm"); } diff --git a/src/modgraph/scanner.cppm b/src/modgraph/scanner.cppm index 90d929fe..2b64fb7a 100644 --- a/src/modgraph/scanner.cppm +++ b/src/modgraph/scanner.cppm @@ -574,6 +574,45 @@ std::expected scan_file(const std::filesystem::path& file // `u.kind`; nothing re-derives it from the extension. u.kind = mcpp::classify(file, extTable); + // ⚠️ A file `sources` matched but the classifier cannot place. + // + // This used to be accepted, and what it produced was a compile edge whose + // object NOTHING LINKS. Measured with an `.ixx` that no + // `[build] module_extensions` declared: + // + // build obj/mathkit.ixx.o | gcm.cache/mathkit.gcm : cxx_object … + // bmi_out = gcm.cache/mathkit.gcm ← the BMI was produced + // build bin/app : cxx_link obj/main.o ← the object is not here + // + // ld: undefined reference to `mk::answer@mathkit()' + // + // Two answers to "is this a module interface" and only one of them read: + // the SCANNER sees `export module mathkit;` and records `provides`, which + // is why the edge got a `bmi_out`; the CLASSIFIER says `Other`, and the + // link set is built from the classifier. So the author is told about a + // symbol, when what happened is a missing line of configuration. + // + // Refused here rather than repaired downstream: the extension set is the + // project's to declare, and mcpp guessing which unknown extension "must + // have meant" a module interface would be a second answer to the same + // question — the very thing that produced this defect. + if (u.kind == mcpp::SourceKind::Other) { + return std::unexpected(ScanError{ file, 0, std::format( + "'{}' is listed in [build] sources, and mcpp has no role for the " + "extension '{}'.\n" + " Its object would be compiled and then linked by nothing, so this " + "is refused rather\n" + " than built. If it is a module interface, declare the extension:\n" + " [build]\n" + " module_extensions = [\"{}\"]\n" + " Otherwise remove it from `sources` — headers belong in " + "`include_dirs`, and\n" + " Windows resource scripts in `[resources]`.", + file.filename().string(), + file.extension().string().empty() ? "(none)" : file.extension().string(), + file.extension().string().empty() ? ".ixx" : file.extension().string()) }); + } + // C-like files are not C++ modules: they cannot legally contain `module` / `import` // declarations, and we route them to the C-language compile rule (no // P1689 scan, no BMI lookups). Skip the line-by-line module scan to diff --git a/src/pack/library.cppm b/src/pack/library.cppm index a4b85812..4260b3bd 100644 --- a/src/pack/library.cppm +++ b/src/pack/library.cppm @@ -38,6 +38,7 @@ export module mcpp.pack.library; import std; import mcpp.pack.digest; import mcpp.pack.manifest_emit; +import mcpp.source_kind; // builtin_extension_table — what needs no declaring import mcpp.pack.zip; import mcpp.platform; @@ -350,6 +351,20 @@ run_library_pack(const LibraryPackPlan& plan) doc.version = plan.packageVersion; doc.builtBy = plan.builtBy; doc.interfaceFiles = interfaceNames; + // Whatever the published set uses beyond the built-in `.cppm`. + // Computed from the FILES, so it cannot disagree with `sources` above. + { + const auto builtin = mcpp::builtin_extension_table(); + std::set extras; + for (auto const& src : plan.interfaceSources) { + auto ext = src.extension().string(); + if (ext.empty()) continue; + if (std::ranges::find(builtin.moduleInterface, ext) + != builtin.moduleInterface.end()) continue; + extras.insert(ext); + } + doc.moduleExtensions.assign(extras.begin(), extras.end()); + } doc.hasIncludeDir = std::filesystem::is_directory(plan.stagingRoot / "include", ec); doc.interfaceDigest = plan.interfaceSources.empty() && !doc.hasIncludeDir ? std::string{} diff --git a/src/pack/library_pipeline.cppm b/src/pack/library_pipeline.cppm index 14d89e83..1401a92c 100644 --- a/src/pack/library_pipeline.cppm +++ b/src/pack/library_pipeline.cppm @@ -180,7 +180,13 @@ export int build_and_pack_library(const std::string& targetName, } // ── the interface closure ───────────────────────────────────── - auto libRoot = ctx->projectRoot / mcpp::manifest::resolve_lib_root_path(ctx->manifest); + // PROBING form: the convention is `src/.`, and + // which extension that is belongs to the project. A package whose + // interfaces are `.ixx` used to resolve to a `src/.cppm` that does + // not exist, and the packer then published nothing and tagged the + // result as a C surface — both silently. + auto libRoot = ctx->projectRoot + / mcpp::manifest::resolve_lib_root_path(ctx->manifest, ctx->projectRoot); std::error_code ec; InterfaceClosure here; std::string qualifiedPackage; diff --git a/src/pack/manifest_emit.cppm b/src/pack/manifest_emit.cppm index e2c858a6..cbd540fe 100644 --- a/src/pack/manifest_emit.cppm +++ b/src/pack/manifest_emit.cppm @@ -68,6 +68,16 @@ struct PackageDoc { // `sources = []` says exactly that (an omitted key would be filled with // the default glob and would sweep up whatever sits under src/). std::vector interfaceFiles; + // Module-interface extensions the PUBLISHED SET uses beyond the built-in + // `.cppm` — emitted as `[build] module_extensions`. + // + // A function of what is in the package, not a copy of what the producer + // declared: the packer publishes a computed subset, so the extensions it + // needs are computed from that subset too. Without this the consumer is + // handed `sources = ["interface/mathkit.ixx"]` and no way to know what an + // `.ixx` is, which is the producer remembering to configure something the + // package could state for itself. + std::vector moduleExtensions; bool hasIncludeDir = false; std::string interfaceDigest; // over the ordered interface set @@ -168,6 +178,12 @@ std::string emit_package_manifest(const PackageDoc& doc) { // ── the interface ────────────────────────────────────────────────── o += "[build]\n"; o += std::format("sources = [{}]\n", join_quoted(doc.interfaceFiles)); + // Emitted only when the published set actually uses one, so a `.cppm` + // package's manifest is byte-identical to before. A package whose interface + // is `.ixx` states that itself rather than requiring the consumer to have + // guessed the producer's convention. + if (!doc.moduleExtensions.empty()) + o += std::format("module_extensions = [{}]\n", join_quoted(doc.moduleExtensions)); if (doc.hasIncludeDir) o += "include_dirs = [\"include\"]\n"; if (!doc.cxxRuntime.empty()) o += std::format("cxx_runtime = {}\n", quote(doc.cxxRuntime)); diff --git a/src/source_kind.cppm b/src/source_kind.cppm index 28992404..e59674ad 100644 --- a/src/source_kind.cppm +++ b/src/source_kind.cppm @@ -251,6 +251,22 @@ std::string normalize_extension(std::string_view raw) { } ExtensionTable builtin_extension_table() { + // `.cppm` alone, on purpose. `.ixx`, `.ccm`, `.cxxm` and anything else are + // the PROJECT's to declare via `[build] module_extensions` — the extension + // set is configuration, not a built-in list that mcpp grows one entry at a + // time as extensions come into fashion. + // + // What has to be true for that to be a real knob is that a DECLARED + // extension works everywhere without further help, and it does: a module + // interface's language is stated explicitly per toolchain + // (`BmiTraits::moduleInterfaceLangFlag` — `/interface /TP`, + // `-x c++-module`, `-x c++`) rather than inferred by the driver from the + // extension. Clang's driver not recognising `.ixx` is therefore beside the + // point: mcpp never lets it guess. + // + // What was NOT true, and is fixed in the scanner rather than here: an + // UNdeclared extension used to be accepted silently. See the classification + // check there for what that cost. return ExtensionTable{ .moduleInterface = { ".cppm" } }; } diff --git a/tests/e2e/260_module_extension_is_configuration.sh b/tests/e2e/260_module_extension_is_configuration.sh new file mode 100755 index 00000000..9fa97053 --- /dev/null +++ b/tests/e2e/260_module_extension_is_configuration.sh @@ -0,0 +1,92 @@ +#!/usr/bin/env bash +# requires: +# (no capability: mcpp states a module interface's LANGUAGE explicitly per +# toolchain, so a declared extension behaves the same on all three.) +# +# 260_module_extension_is_configuration.sh — `[build] module_extensions` is the +# knob, and everything downstream follows it without a second declaration. +# +# `.ixx` is NOT built in, on purpose: the extension set is configuration, not a +# list mcpp grows one entry at a time as extensions come into fashion. What that +# demands in return is two things, and this test is both of them: +# +# 1. a DECLARED extension works with no further help; +# 2. an UNDECLARED one is refused where the mistake is, not four steps later. +# +# ⚠️ WHY (2) IS THE INTERESTING HALF. Before this, an undeclared `.ixx` matched +# by `sources` was accepted and produced a compile edge whose object NOTHING +# LINKS — measured: +# +# build obj/mathkit.ixx.o | gcm.cache/mathkit.gcm : cxx_object … +# bmi_out = gcm.cache/mathkit.gcm ← the BMI was produced +# build bin/app : cxx_link obj/main.o ← the object is not here +# +# ld: undefined reference to `mk::answer@mathkit()' +# +# Two answers to "is this a module interface" and only one of them read: the +# scanner sees `export module` and records `provides`, which is why the edge got +# a `bmi_out`; the classifier says `Other`, and the link set is built from the +# classifier. So the author was told about a symbol when what happened was a +# missing line of configuration. +set -e + +TMP=$(mktemp -d) +trap "rm -rf $TMP" EXIT +cd "$TMP" + +mkdir -p proj/src +cat > proj/src/mathkit.ixx <<'EOF' +export module mathkit; +export namespace mk { int answer() { return 42; } } +EOF +cat > proj/src/main.cpp <<'EOF' +#include +import mathkit; +int main(){ std::printf("ixx-ok=%d\n", mk::answer()); return 0; } +EOF + +manifest() { # $1 = extra [build] body + cat > "$TMP/proj/mcpp.toml" < undeclared.log 2>&1; then + echo "FAIL: an undeclared '.ixx' was accepted." + nj="$(find target -name build.ninja | head -1)" + grep -n 'mathkit' "$nj" | head + echo " Check whether its object is in the link edge — if it is not," + echo " this 'success' produces an undefined reference at link time." + exit 1 +fi +grep -q "no role for the extension '.ixx'" undeclared.log || { + cat undeclared.log + echo "FAIL: refused, but not for the classification reason. A message that" + echo " does not name the extension leaves the author with a symbol." + exit 1; } +grep -q "module_extensions" undeclared.log || { + cat undeclared.log + echo "FAIL: the message does not name the key that fixes it, so it is a" + echo " diagnosis without a remedy." + exit 1; } + +# ── 2. declared: works, on this toolchain, with nothing else ──────────── +manifest 'module_extensions = [".ixx"]' +rm -rf target +"$MCPP" run > declared.log 2>&1 || { cat declared.log; echo "FAIL: declared .ixx did not build"; exit 1; } +grep -q 'ixx-ok=42' declared.log || { cat declared.log; echo "FAIL: wrong answer"; exit 1; } + +echo "PASS: a declared module extension needs no further help; an undeclared one is refused" diff --git a/tests/e2e/261_pack_follows_the_module_extension.sh b/tests/e2e/261_pack_follows_the_module_extension.sh new file mode 100755 index 00000000..f8267eb9 --- /dev/null +++ b/tests/e2e/261_pack_follows_the_module_extension.sh @@ -0,0 +1,150 @@ +#!/usr/bin/env bash +# requires: +# (no capability: nothing here is toolchain-specific.) +# +# 261_pack_follows_the_module_extension.sh — `mcpp pack` follows the project's +# module extension on its own, and the package it writes states that extension +# for its consumers. +# +# The principle: a project says `module_extensions = [".ixx"]` ONCE. Packing it +# and consuming the package must both work with nothing further declared on +# either side. Anything else makes the knob a knob plus two things to remember. +# +# ⚠️ TWO DEFECTS SAT BEHIND THIS, and the first was silent in both directions. +# +# 1. The lib-root convention hard-coded `.cppm`: `src/.cppm`. For an +# `.ixx` project that file does not exist, so the closure started nowhere: +# +# $ mcpp pack mathkit +# Interface (headers only) ← the module interface, gone +# Withheld (nothing) +# Packed …-x86_64-linux-gnu ← the C-SURFACE tag +# +# Both halves wrong without a word. No consumer could `import mathkit`; and +# an empty published set is precisely how the packer recognises a C surface, +# so the package ALSO stopped constraining the C++ ABI and the compatibility +# gate stopped checking compiler and stdlib. A package that claims less than +# it should is the failure this whole design exists to prevent. +# +# 2. The generated manifest listed `sources = ["interface/mathkit.ixx"]` and did +# not say what an `.ixx` is, so the consumer was refused by the classifier. +# Now the package declares it — computed from the published FILES, so it +# cannot disagree with `sources`. +set -e +source "$(dirname "$0")/_host_path.sh" + +TMP=$(mktemp -d) +trap "rm -rf $TMP" EXIT +cd "$TMP" + +mkdir -p mathkit/src +cat > mathkit/src/mathkit.ixx <<'EOF' +export module mathkit; +export import :api; +EOF +cat > mathkit/src/api.ixx <<'EOF' +export module mathkit:api; +export namespace mk { int answer(); } +EOF +cat > mathkit/src/impl.cpp <<'EOF' +module mathkit; +namespace mk { int answer() { return 42; } } +EOF +cat > mathkit/mcpp.toml <<'EOF' +[package] +name = "mathkit" +version = "0.1.0" +[build] +sources = ["src/*.ixx", "src/*.cpp"] +module_extensions = [".ixx"] +[targets.mathkit] +kind = "lib" +EOF + +cd mathkit +"$MCPP" pack mathkit --format dir > pack.log 2>&1 || { cat pack.log; echo "pack failed"; exit 1; } +pkg="$TMP/mathkit/$(find target/dist -maxdepth 1 -type d -name 'mathkit-0.1.0-*' | head -1)" + +# ── the closure actually started ──────────────────────────────────────── +[[ -f "$pkg/interface/mathkit.ixx" && -f "$pkg/interface/api.ixx" ]] || { + find "$pkg" -type f + cat pack.log + echo "FAIL: the module interface was not published. 'Interface (headers only)'" + echo " in the log above means the lib root resolved to a file that does" + echo " not exist — the convention looked for .cppm." + exit 1; } +[[ ! -e "$pkg/interface/impl.cpp" ]] || { echo "FAIL: the implementation unit was published"; exit 1; } + +# ── and the package is tagged as the C++ surface it is ────────────────── +# +# Asserted separately because it fails INDEPENDENTLY: an empty published set is +# how a C surface is recognised, so losing the interface also silently downgrades +# the compatibility tag, and a package that constrains nothing is accepted by +# every toolchain. +grep -qE '^abi +=.*-c\+\+[0-9]+"' "$pkg/mcpp.toml" || { + grep -n 'abi' "$pkg/mcpp.toml" + echo "FAIL: the leg carries a C-surface tag. A package that publishes a C++" + echo " module interface constrains the C++ ABI, and this one now says" + echo " it does not — the gate will accept any compiler." + exit 1; } + +# ── the package tells consumers what an .ixx is ───────────────────────── +grep -q 'module_extensions' "$pkg/mcpp.toml" || { + cat "$pkg/mcpp.toml" + echo "FAIL: the generated manifest lists .ixx sources without declaring the" + echo " extension, so every consumer is refused by the classifier." + exit 1; } + +# ── a consumer that declares NOTHING builds and runs ──────────────────── +PKG_HOST="$(host_path "$pkg")" +cd "$TMP" +mkdir -p app/src +cat > app/src/main.cpp <<'EOF' +#include +import mathkit; +int main(){ std::printf("pack-ixx=%d\n", mk::answer()); return 0; } +EOF +cat > app/mcpp.toml < run.log 2>&1 ) || { + cat app/run.log + echo "FAIL: the consumer had to be told about .ixx. The package should carry it." + exit 1; } +grep -q 'pack-ixx=42' app/run.log || { cat app/run.log; echo "FAIL: wrong answer"; exit 1; } + +# ── a .cppm package's manifest is unchanged ───────────────────────────── +# +# The new key must appear only when the published set needs it, or every +# existing package's manifest changes shape for no reason. +cd "$TMP" +mkdir -p plain/src +cat > plain/src/plain.cppm <<'EOF' +export module plain; +export namespace pl { int v() { return 1; } } +EOF +cat > plain/mcpp.toml <<'EOF' +[package] +name = "plain" +version = "0.1.0" +[build] +sources = ["src/*.cppm"] +[targets.plain] +kind = "lib" +EOF +( cd plain && "$MCPP" pack plain --format dir > pack.log 2>&1 ) || { + cat plain/pack.log; echo "plain pack failed"; exit 1; } +plainpkg="$(find plain/target/dist -maxdepth 1 -type d -name 'plain-0.1.0-*' | head -1)" +grep -q 'module_extensions' "$plainpkg/mcpp.toml" && { + cat "$plainpkg/mcpp.toml" + echo "FAIL: a .cppm-only package gained a module_extensions key it does not need" + exit 1; } + +echo "PASS: pack follows the project's module extension, and the package carries it" From 3f17e9c49577ef2460d88f9f09c2440708839bf0 Mon Sep 17 00:00:00 2001 From: speak-agent <248744407+speak-agent@users.noreply.github.com> Date: Tue, 18 Aug 2026 06:19:00 +0800 Subject: [PATCH 2/9] feat(build): generate a .def so MSVC can export a DLL MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit MSVC exports nothing from a DLL without `__declspec(dllexport)` or a `.def`, so mcpp refused `kind = "shared"` on that ABI. The reason was right and the conclusion was not: CMake has shipped `WINDOWS_EXPORT_ALL_SYMBOLS` since 3.4 and its `bindexplib` reads COFF directly — which matters here, because `dumpbin` lives in a Visual Studio developer environment and mcpp's default Windows toolchain is clang, so a plain `mcpp build` is not inside one. `mcpp.build.coff_exports` is that reader, as a pure function over bytes. Its filter follows bindexplib's semantics — external storage class, defined section, DATA for anything that is not code, and the skip list (destructor thunks, dotted managed names, ARM64EC bridges) each entry of which has its own reason. Past 65535 exports it refuses rather than truncates: PE addresses exports by 16-bit ordinal, and a truncated table links cleanly and then fails at whichever consumer needed the symbol that fell off the end. Pure over bytes so it can be tested everywhere: 16 unit tests build synthetic objects byte by byte, which is the only way to vary a storage class on demand, plus a committed real mingw-cross object — a reader fed only its own test's output agrees with itself and with nothing else, and macOS and Windows CI cannot produce COFF at all. In the graph it is `mcpp coff-def`, an edge whose inputs are the same objects the link consumes, so the exported surface cannot drift from what was compiled. A subcommand rather than a shell fragment for the reason `bmi-equal` is one: a generated POSIX-shell command is skipped entirely on Windows, the only platform this edge exists for. No edge is emitted for MinGW — its linker auto-exports, and a second source of truth for a DLL's surface is how the two come to disagree. Two limits survive that no tool can remove, and they are CMake's documented ones for the same mechanism: a consumer still needs `__declspec(dllimport)` to read exported DATA, and a class whose vtable is referenced must be marked whole. 258 exercises a function across the boundary and says why. --- src/build/coff_exports.cppm | 263 +++++++++++++++++ src/build/ninja_backend.cppm | 31 +- src/build/plan.cppm | 81 +++--- src/cli.cppm | 7 +- src/cli/cmd_build.cppm | 66 +++++ tests/e2e/251_pack_library_shared.sh | 59 ++-- tests/e2e/257_shared_library_pe.sh | 10 +- tests/e2e/258_shared_library_msvc_auto_def.sh | 122 ++++++++ tests/e2e/258_shared_library_msvc_refused.sh | 89 ------ tests/fixtures/coff/README.md | 27 ++ tests/fixtures/coff/probe-amd64.cpp.in | 7 + tests/fixtures/coff/probe-amd64.obj | Bin 0 -> 1002 bytes tests/unit/test_coff_exports.cpp | 274 ++++++++++++++++++ 13 files changed, 853 insertions(+), 183 deletions(-) create mode 100644 src/build/coff_exports.cppm create mode 100755 tests/e2e/258_shared_library_msvc_auto_def.sh delete mode 100755 tests/e2e/258_shared_library_msvc_refused.sh create mode 100644 tests/fixtures/coff/README.md create mode 100644 tests/fixtures/coff/probe-amd64.cpp.in create mode 100644 tests/fixtures/coff/probe-amd64.obj create mode 100644 tests/unit/test_coff_exports.cpp diff --git a/src/build/coff_exports.cppm b/src/build/coff_exports.cppm new file mode 100644 index 00000000..d93fc5b5 --- /dev/null +++ b/src/build/coff_exports.cppm @@ -0,0 +1,263 @@ +// mcpp.build.coff_exports — which symbols of a COFF object a DLL should export. +// +// WHY THIS EXISTS +// +// MSVC exports nothing from a DLL unless the source says `__declspec(dllexport)` +// or a `.def` file lists the symbols. Without either, the import library comes +// out empty and every consumer fails with unresolved externals naming symbols +// that are plainly in the object files — a diagnostic pointing nowhere near its +// cause. MinGW's linker auto-exports and hides the whole problem; lld-link's +// MSVC flavour does not, deliberately, because PE caps exports at 65535. +// +// So mcpp generates the `.def`. This is not novel: CMake has shipped +// `WINDOWS_EXPORT_ALL_SYMBOLS` since 3.4 and its `bindexplib` does exactly this. +// The filter below follows that one's semantics, which are the de-facto standard +// for "what does auto-export mean on Windows". +// +// WHY IT PARSES COFF ITSELF +// +// `dumpbin` lives in a Visual Studio developer environment, and mcpp's default +// Windows toolchain is clang — a plain `mcpp build` is not inside that +// environment. `llvm-nm` would be a second external dependency with the same +// class of failure ("not on this machine"). COFF's symbol table is a fixed-size +// record array with a string table after it; reading it needs no library. +// +// WHY IT IS A PURE FUNCTION OVER BYTES +// +// So it can be tested on any host. Windows CI can only tell whether the linker +// accepted the result; the filtering rules are where mistakes hide, and those +// are decidable from a byte buffer. Linux CI produces real COFF objects through +// mingw-cross, so the parser is exercised against genuine input off-Windows too. +// +// Design: .agents/docs/2026-08-18-windows-shared-library-and-module-extensions.md §2. + +export module mcpp.build.coff_exports; + +import std; + +export namespace mcpp::build::coff { + +// One exportable symbol. +struct Export { + std::string name; + // `.def` needs `name DATA` for anything that is not code: the linker + // otherwise generates a call thunk for a variable, and the consumer reads + // the thunk instead of the value. + bool data = false; + + bool operator==(const Export&) const = default; + auto operator<=>(const Export&) const = default; +}; + +// PE's export directory addresses exports by 16-bit ordinal, so a DLL cannot +// have more than this many. Reaching it is refused rather than truncated: a +// truncated export table links and then fails at the consumer, naming whichever +// symbol happened to fall off the end. +inline constexpr std::size_t kMaxExports = 65535; + +// Machine types this reader accepts. An unknown one is not an error to guess +// at: the symbol-table layout is shared, but a machine it has never been tried +// against is exactly where an untested assumption would hide. +bool is_supported_machine(std::uint16_t machine); + +// Read one object file's exportable symbols. `bytes` is a whole `.obj`. +// +// Returns an error string for input this reader cannot honestly interpret — +// truncated, an unsupported machine, or a symbol table that runs off the end. +// It never returns a partial list: half a symbol table is indistinguishable +// from a small one, and the caller would ship a DLL missing exports. +std::expected, std::string> +read_exports(std::span bytes); + +// The `.def` text for a set of exports, sorted and de-duplicated. +// +// `libraryName` goes in the `LIBRARY` statement — link.exe accepts a `.def` +// without one, but naming it makes the file self-describing when a human opens +// it while working out why a symbol is missing. +std::string write_def(std::string_view libraryName, std::vector exports); + +} // namespace mcpp::build::coff + +namespace mcpp::build::coff { + +namespace { + +// ── COFF, only the parts needed ────────────────────────────────────────── +// +// Offsets rather than structs: a packed-struct cast would depend on the host's +// alignment rules to read a file format, and this code runs on hosts that never +// produce COFF. + +constexpr std::size_t kFileHeaderSize = 20; +constexpr std::size_t kSymbolRecordSize = 18; +constexpr std::size_t kSectionHeaderSize = 40; + +// The subset of IMAGE_FILE_MACHINE_* that mcpp targets or can be handed. +constexpr std::uint16_t kMachineI386 = 0x014c; +constexpr std::uint16_t kMachineAmd64 = 0x8664; +constexpr std::uint16_t kMachineArm = 0x01c0; +constexpr std::uint16_t kMachineArmNT = 0x01c4; +constexpr std::uint16_t kMachineArm64 = 0xaa64; + +constexpr std::uint8_t kSymClassExternal = 2; // IMAGE_SYM_CLASS_EXTERNAL +constexpr std::uint16_t kSymTypeFunction = 0x20; // DTYPE_FUNCTION << 4 +constexpr std::uint32_t kScnMemWrite = 0x80000000u; // IMAGE_SCN_MEM_WRITE +constexpr std::uint32_t kScnMemExecute = 0x20000000u; // IMAGE_SCN_MEM_EXECUTE + +std::uint16_t rd16(std::span b, std::size_t off) { + return static_cast(std::to_integer(b[off]) + | (std::to_integer(b[off + 1]) << 8)); +} +std::uint32_t rd32(std::span b, std::size_t off) { + return static_cast(std::to_integer(b[off]) + | (std::to_integer(b[off + 1]) << 8) + | (std::to_integer(b[off + 2]) << 16) + | (std::to_integer(b[off + 3]) << 24)); +} + +// Symbols bindexplib skips, and why each one would be wrong to export. +bool is_skipped_name(std::string_view n) { + // Scalar-deleting and vector-deleting destructor thunks. They are emitted + // per-TU and exporting them makes the DLL's surface depend on which + // translation unit happened to instantiate one. + if (n.starts_with("??_G") || n.starts_with("??_E")) return true; + // Managed (C++/CLI) artefacts: `.` cannot appear in a C++ mangled name, and + // these forms name IL constructs that mean nothing to a native consumer. + if (n.find('.') != std::string_view::npos) return true; + if (n.find("__t2m") != std::string_view::npos) return true; + if (n.find("$$F") != std::string_view::npos) return true; + if (n.find("$$J") != std::string_view::npos) return true; + // ARM64EC thunks: linker-generated bridges between the ARM64 and x64 views + // of the same function. Exporting a bridge exports neither side. + if (n.find("$ientry_thunk") != std::string_view::npos) return true; + if (n.find("$entry_thunk") != std::string_view::npos) return true; + if (n.find("$iexit_thunk") != std::string_view::npos) return true; + if (n.find("$exit_thunk") != std::string_view::npos) return true; + return false; +} + +} // namespace + +bool is_supported_machine(std::uint16_t machine) { + return machine == kMachineI386 || machine == kMachineAmd64 + || machine == kMachineArm || machine == kMachineArmNT + || machine == kMachineArm64; +} + +std::expected, std::string> +read_exports(std::span bytes) +{ + if (bytes.size() < kFileHeaderSize) + return std::unexpected("not a COFF object: shorter than a file header"); + + const auto machine = rd16(bytes, 0); + if (!is_supported_machine(machine)) { + return std::unexpected(std::format( + "unsupported COFF machine 0x{:04x}. mcpp reads i386, amd64, arm, " + "armnt and arm64; anything else would be a guess at a layout this " + "has never been run against.", machine)); + } + + const auto numSections = rd16(bytes, 2); + const auto symTableOff = rd32(bytes, 8); + const auto numSymbols = rd32(bytes, 12); + if (symTableOff == 0 || numSymbols == 0) return std::vector{}; + + // Section characteristics, indexed 1-based the way symbols address them. + const std::size_t sectionsOff = kFileHeaderSize + rd16(bytes, 16); // + optional header + std::vector sectionFlags(numSections + 1, 0); + for (std::uint16_t i = 0; i < numSections; ++i) { + const auto off = sectionsOff + std::size_t(i) * kSectionHeaderSize; + if (off + kSectionHeaderSize > bytes.size()) + return std::unexpected("COFF section headers run past the end of the file"); + sectionFlags[i + 1] = rd32(bytes, off + 36); + } + + const std::size_t symbolsEnd = std::size_t(symTableOff) + + std::size_t(numSymbols) * kSymbolRecordSize; + if (symbolsEnd > bytes.size()) + return std::unexpected("COFF symbol table runs past the end of the file"); + + // The string table follows the symbol table; its first 4 bytes are its own + // size. Long names are `/` into it. + const std::size_t stringsOff = symbolsEnd; + const bool hasStrings = stringsOff + 4 <= bytes.size(); + + auto name_at = [&](std::size_t rec) -> std::expected { + // A name is 8 bytes: either inline (NUL-padded) or a zero word followed + // by an offset into the string table. + if (rd32(bytes, rec) != 0) { + std::string s; + for (std::size_t i = 0; i < 8; ++i) { + auto c = std::to_integer(bytes[rec + i]); + if (c == '\0') break; + s.push_back(c); + } + return s; + } + const auto off = rd32(bytes, rec + 4); + if (!hasStrings || stringsOff + off >= bytes.size()) + return std::unexpected("COFF long symbol name points outside the string table"); + std::string s; + for (std::size_t i = stringsOff + off; i < bytes.size(); ++i) { + auto c = std::to_integer(bytes[i]); + if (c == '\0') break; + s.push_back(c); + } + return s; + }; + + std::vector out; + for (std::uint32_t i = 0; i < numSymbols; ) { + const std::size_t rec = std::size_t(symTableOff) + + std::size_t(i) * kSymbolRecordSize; + const auto sectionNum = static_cast(rd16(bytes, rec + 12)); + const auto type = rd16(bytes, rec + 14); + const auto storage = std::to_integer(bytes[rec + 16]); + const auto numAux = std::to_integer(bytes[rec + 17]); + + // Advance past auxiliary records regardless of whether this symbol is + // taken — an aux record is not a symbol and reading it as one produces + // names out of raw section data. + i += 1u + numAux; + + if (storage != kSymClassExternal) continue; // not visible outside the TU + if (sectionNum <= 0) continue; // undefined, absolute or debug + if (type != kSymTypeFunction && type != 0) continue; + + auto nm = name_at(rec); + if (!nm) return std::unexpected(nm.error()); + if (nm->empty() || is_skipped_name(*nm)) continue; + + // i386 (and `__cdecl` on it) carries a leading underscore that is part + // of the calling convention rather than of the name. + std::string name = *nm; + if (machine == kMachineI386 && name.starts_with('_') && name.find('@') == std::string::npos) + name.erase(0, 1); + + const auto flags = std::size_t(sectionNum) < sectionFlags.size() + ? sectionFlags[sectionNum] : 0u; + // DATA when it is not code. Both halves matter: a function symbol in a + // writable section is still a function, and a variable in a read-only + // section (a `const`) is still data. + const bool isData = type != kSymTypeFunction + && (flags & kScnMemExecute) == 0; + out.push_back(Export{ std::move(name), isData }); + } + + return out; +} + +std::string write_def(std::string_view libraryName, std::vector exports) { + std::ranges::sort(exports); + exports.erase(std::ranges::unique(exports).begin(), exports.end()); + + std::string out; + if (!libraryName.empty()) out += std::format("LIBRARY {}\n", libraryName); + out += "EXPORTS\n"; + for (auto const& e : exports) + out += std::format(" {}{}\n", e.name, e.data ? " DATA" : ""); + return out; +} + +} // namespace mcpp::build::coff diff --git a/src/build/ninja_backend.cppm b/src/build/ninja_backend.cppm index 930f0f33..335592fb 100644 --- a/src/build/ninja_backend.cppm +++ b/src/build/ninja_backend.cppm @@ -1049,7 +1049,7 @@ std::string emit_ninja_string(const BuildPlan& plan) { "LINK"); link_rule("cxx_archive", std::string(dial.archiveCmd), "AR"); link_rule("cxx_shared", - "$ld /nologo /DLL /OUT:$out $implib_flag " + "$ld /nologo /DLL /OUT:$out $implib_flag $def_flag " "$in $ldflags $unit_ldflags", "SHARED"); } else { @@ -1086,6 +1086,15 @@ std::string emit_ninja_string(const BuildPlan& plan) { append(" description = CC $out\n\n"); } + // The auto-export edge. `mcpp coff-def` reads the same objects the link + // consumes, so the DLL's surface cannot drift from what was compiled — and + // it is an mcpp subcommand rather than a shell fragment because a generated + // POSIX-shell command is skipped entirely on Windows, the only platform this + // edge exists for. + append("rule coff_def\n"); + append(" command = $mcpp coff-def --output $out --name $def_name $in\n"); + append(" description = DEF $out\n\n"); + append("rule runtime_alias\n"); if constexpr (mcpp::platform::is_windows) { // PE has no soname symlink, so the alias is a copy — and a copy of a @@ -1816,6 +1825,24 @@ std::string emit_ninja_string(const BuildPlan& plan) { if (!lu.importLibrary.empty()) implicitOut = " | " + escape_ninja_path(lu.importLibrary); + // The `.def` edge, emitted BEFORE the link that consumes it. + // + // Its inputs are this unit's objects — the same list the link gets — so + // the exported surface is a function of what was compiled and cannot + // drift from it. It is an ordinary explicit input to the link rather + // than an implicit one: the linker reads it, so ninja should rebuild the + // DLL when it changes. + if (!lu.defFile.empty()) { + std::string defIns; + for (auto const& o : lu.objects) defIns += " " + escape_ninja_path(o); + append(std::format("build {} : coff_def{}\n", + escape_ninja_path(lu.defFile), defIns)); + append(std::format(" def_name = {}\n\n", + lu.output.filename().string())); + } + + if (!lu.defFile.empty()) implicit += " " + escape_ninja_path(lu.defFile); + std::string out_line = std::format("build {}{} : {}{}{}\n", escape_ninja_path(lu.output), implicitOut, rule, ins, implicit.empty() ? std::string{} : " |" + implicit); @@ -1838,6 +1865,8 @@ std::string emit_ninja_string(const BuildPlan& plan) { out_line += " implib_flag = " + arg + "\n"; } } + if (!lu.defFile.empty()) + out_line += " def_flag = /DEF:" + escape_ninja_path(lu.defFile) + "\n"; { // Per-unit C++ runtime link, by ROLE. The kind→role map is the // only place that knows a TestBinary runs on the build machine diff --git a/src/build/plan.cppm b/src/build/plan.cppm index e050da48..e21a8779 100644 --- a/src/build/plan.cppm +++ b/src/build/plan.cppm @@ -100,6 +100,13 @@ struct LinkUnit { // producer; without that, ninja reports "no known rule to make it" naming a // file the link command does in fact write. std::filesystem::path importLibrary; // relative to plan.outputDir + // The generated module-definition file, on the MSVC ABI only. + // + // MSVC exports nothing from a DLL without `__declspec(dllexport)` or a + // `.def`, so without this the import library is empty and every consumer + // fails with unresolved externals for symbols that are visibly in the + // objects. MinGW's linker auto-exports and needs none of this. + std::filesystem::path defFile; // relative to plan.outputDir std::string soname; // ABI name for shared libraries std::vector runtimeAliases; // relative aliases, e.g. bin/libfoo.so.1 std::optional entryMain; // src path of main.cpp for bin @@ -1002,33 +1009,27 @@ make_plan(const mcpp::manifest::Manifest& manifest, return flag ? std::string(*flag) : std::string{}; }; - // WHAT `kind = "shared"` SUPPORTS, AND THE ONE THING IT STILL DOES NOT. + // `kind = "shared"` is supported on every format mcpp targets. // - // ELF, Mach-O and PE/MinGW are modelled: the import library PE consumers - // link is `import_library_for` above, and Mach-O's install name is set to - // `@rpath/` unconditionally (see shared_soname_flag) rather than - // defaulting to this machine's build path. + // ELF gets its SONAME, Mach-O an `@rpath/` install name, PE an import + // library — and on the MSVC ABI, a GENERATED `.def`. // - // PE/MSVC is refused, and the reason is not the linker — `link /DLL - // /IMPLIB:` has been in the rule table all along. It is symbol export: MSVC - // exports nothing from a DLL without `__declspec(dllexport)` or a `.def`, - // so the import library comes out EMPTY and the consumer fails with - // unresolved externals naming symbols that are plainly in the object files. - // Producing that is worse than refusing, because the diagnostic points - // nowhere near the cause. + // The MSVC case is the one that used to be refused, and the reason was never + // the linker (`link /DLL /IMPLIB:` has been in the rule table all along). It + // is symbol export: MSVC exports nothing from a DLL without + // `__declspec(dllexport)` or a `.def`, so the import library came out EMPTY + // and consumers failed with unresolved externals naming symbols that are + // plainly in the objects. MinGW's linker auto-exports and hides the whole + // problem; lld-link's MSVC flavour does not, deliberately, because PE caps + // exports at 65535. // - // ⚠️ THE HOST FALLBACK BELOW IS BELT AND BRACES, NOT A FIX. An earlier - // version of this comment claimed the previous guard — - // `!targetTriple.empty() && os != "linux"` — was inert on native builds - // because targetTriple would be empty there. It is not: `tc.targetTriple` - // is filled from the compiler's own `-dumpmachine` (detect.cppm), a native - // Linux build records `x86_64-linux-gnu` in resolution.json, and - // `parse("x86_64-pc-windows-msvc")` skips the vendor segment and succeeds. - // So the old guard did refuse native macOS and native Windows, and this - // change ADDS support rather than closing a hole. The fallback stays - // because a target that cannot be parsed must not be silently read as - // "not windows". - { + // So mcpp writes the `.def` (`msvc_needs_def` below → a def-gen edge in the + // backend → `/DEF:`). Two limits survive that no tool can remove, and they + // are documented rather than discovered: a consumer still needs + // `__declspec(dllimport)` to read exported DATA, and a class whose vtable is + // referenced must be marked whole. Both are CMake's documented limits for + // the same mechanism, and the answer to both is annotation. + const bool msvcTarget = [&] { const std::string targetOs = targetTriple.empty() ? (mcpp::platform::is_macos ? "macos" : mcpp::platform::is_windows ? "windows" : "linux") @@ -1036,32 +1037,12 @@ make_plan(const mcpp::manifest::Manifest& manifest, // The ABI, from the toolchain rather than from `naming`: on a native // Windows build the host naming constants are the MSVC ones whatever the // toolchain is (lib_prefix is "" and static_lib_ext is ".lib" for mingw - // too), so asking `naming` would refuse MinGW as well. `is_msvc_target` + // too), so asking `naming` would treat MinGW as MSVC. `is_msvc_target` // reads the compiler's own -dumpmachine answer, which distinguishes them // and also covers clang driving the MSVC ABI — clang auto-exports no more // than link.exe does, so "which compiler binary" is the wrong question. - if (targetOs == "windows" && mcpp::toolchain::is_msvc_target(tc)) { - for (auto const& t : manifest.targets) { - if (t.kind != mcpp::manifest::Target::SharedLibrary) continue; - return std::unexpected(std::format( - "target '{}': kind = \"shared\" is not supported for the MSVC " - "ABI ({}).\n" - " MSVC exports nothing from a DLL unless the source says " - "`__declspec(dllexport)`\n" - " or a `.def` file lists the symbols, so the import library " - "would be empty and\n" - " every consumer would fail with unresolved externals naming " - "symbols that are\n" - " visibly present in the objects. mcpp refuses rather than " - "produce that.\n" - " Use kind = \"lib\" for this target, or build it for " - "*-windows-gnu (MinGW),\n" - " where the linker auto-exports.", - t.name, - targetTriple.empty() ? "native" : targetTriple.str())); - } - } - } + return targetOs == "windows" && mcpp::toolchain::is_msvc_target(tc); + }(); bool experimentalStd = false; if (auto stdCfg = mcpp::manifest::normalize_cpp_standard(manifest.package.standard)) { @@ -1629,6 +1610,8 @@ make_plan(const mcpp::manifest::Manifest& manifest, lu.kind = LinkUnit::SharedLibrary; lu.output = dep.output; lu.importLibrary = import_library_for(dep.target, naming); + if (msvcTarget && !lu.importLibrary.empty()) + lu.defFile = std::filesystem::path("bin") / (dep.target.name + ".def"); lu.soname = dep.target.soname; lu.runtimeAliases = runtime_aliases_for_target(dep.target, naming); lu.loaderTagFlag = loader_tag_flag(lu.kind); @@ -1657,6 +1640,10 @@ make_plan(const mcpp::manifest::Manifest& manifest, lu.kind = LinkUnit::SharedLibrary; lu.output = target_output(t, naming); lu.importLibrary = import_library_for(t, naming); + // MSVC only: MinGW's linker auto-exports, and generating a second + // source of truth for what a DLL exports is how the two disagree. + if (msvcTarget && !lu.importLibrary.empty()) + lu.defFile = std::filesystem::path("bin") / (t.name + ".def"); lu.soname = t.soname; lu.runtimeAliases = runtime_aliases_for_target(t, naming); } else if (t.kind == mcpp::manifest::Target::TestBinary) { diff --git a/src/cli.cppm b/src/cli.cppm index 9f98060e..8d4fd079 100644 --- a/src/cli.cppm +++ b/src/cli.cppm @@ -652,6 +652,11 @@ int run(int argc, char** argv) { .option(cl::Option("verify").takes_value().value_name("MODE") .help("Already-staged check: size (default) | content")) .action(wrap_rc(cmd_stage))) + .subcommand(cl::App("coff-def") + .description("(internal: invoked by ninja) Write a .def of every exportable symbol in the given COFF objects") + .option(cl::Option("output").takes_value().value_name("PATH").help("the .def to write")) + .option(cl::Option("name").takes_value().value_name("DLL").help("LIBRARY name recorded in the .def")) + .action(wrap_rc(cmd_coff_def))) .subcommand(cl::App("bmi-equal") .description("(internal: invoked by ninja) Compare two BMIs ignoring the compiler's embedded timestamp") .action(wrap_rc(cmd_bmi_equal))) @@ -752,7 +757,7 @@ int run(int argc, char** argv) { "new", "build", "run", "test", "clean", "add", "remove", "update", "search", "publish", "pack", "emit", "xpkg", "toolchain", "cache", "index", "self", "explain", - "version", "dyndep", "why", "resolve", "stage", "bmi-equal", + "version", "dyndep", "why", "resolve", "stage", "bmi-equal", "coff-def", "bmi-compile", "bmi-supervise", "bmi-await", }); bool ok = false; diff --git a/src/cli/cmd_build.cppm b/src/cli/cmd_build.cppm index 09be73c8..e895f271 100644 --- a/src/cli/cmd_build.cppm +++ b/src/cli/cmd_build.cppm @@ -13,6 +13,7 @@ import mcpplibs.cmdline; import mcpp.build.prepare; import mcpp.build.execute; import mcpp.build.configure; +import mcpp.build.coff_exports; import mcpp.build.stage; import mcpp.build.schedule.detach_codegen; import mcpp.build.test_targets; @@ -483,6 +484,71 @@ export int cmd_bmi_equal(const mcpplibs::cmdline::ParsedArgs& parsed) { return same ? 0 : 1; } +// `mcpp coff-def --output --name ...` — the auto-export edge. +// +// A subcommand rather than a shell fragment, for the reason bmi-equal is one: +// a generated ninja command written in POSIX shell is skipped entirely on +// Windows, which is the only platform this edge exists for. It also keeps the +// COFF reader testable as a pure function over bytes (tests/unit/test_coff_exports) +// instead of as whatever a command line happened to produce. +export int cmd_coff_def(const mcpplibs::cmdline::ParsedArgs& parsed) { + std::filesystem::path out; + if (auto v = parsed.value("output")) out = *v; + if (out.empty()) { + std::println(stderr, "error: coff-def requires --output"); + return 2; + } + std::string libName; + if (auto v = parsed.value("name")) libName = *v; + + std::vector all; + for (std::size_t i = 0; i < parsed.positional_count(); ++i) { + const std::filesystem::path obj{ parsed.positional(i) }; + std::ifstream in(obj, std::ios::binary); + if (!in) { + std::println(stderr, "error: cannot read object '{}'", obj.string()); + return 1; + } + std::vector bytes; + for (char c; in.get(c); ) bytes.push_back(static_cast(c)); + auto syms = mcpp::build::coff::read_exports(bytes); + if (!syms) { + // Named with the object, because "which one" is the whole question + // when one file out of two hundred is the problem. + std::println(stderr, "error: {}: {}", obj.string(), syms.error()); + return 1; + } + all.insert(all.end(), syms->begin(), syms->end()); + } + + // Refused, not truncated. A `.def` cut at the ceiling links cleanly and + // then fails at whichever consumer happens to need a symbol that fell off + // the end — a diagnostic with no path back to this decision. + std::ranges::sort(all); + all.erase(std::ranges::unique(all).begin(), all.end()); + if (all.size() > mcpp::build::coff::kMaxExports) { + std::println(stderr, + "error: {} exportable symbols, and PE addresses exports by 16-bit " + "ordinal (max {}).\n" + " Auto-export cannot express this library. Mark its public surface " + "with __declspec(dllexport)\n" + " and the export set becomes what you declared instead of " + "everything.", + all.size(), mcpp::build::coff::kMaxExports); + return 1; + } + + std::error_code ec; + if (out.has_parent_path()) std::filesystem::create_directories(out.parent_path(), ec); + std::ofstream o(out, std::ios::binary | std::ios::trunc); + if (!o) { + std::println(stderr, "error: cannot write '{}'", out.string()); + return 1; + } + o << mcpp::build::coff::write_def(libName, std::move(all)); + return o.good() ? 0 : 1; +} + // The three edges of the DetachCodegen shape. They are `mcpp` subcommands // rather than shell fragments for two reasons: the previous BMI-equivalence // logic lived in the generated ninja command as POSIX shell and was therefore diff --git a/tests/e2e/251_pack_library_shared.sh b/tests/e2e/251_pack_library_shared.sh index 6eb0587d..26ca0288 100755 --- a/tests/e2e/251_pack_library_shared.sh +++ b/tests/e2e/251_pack_library_shared.sh @@ -7,17 +7,16 @@ # handled" from "there is no gate". Which half applies is now a property of the # TARGET, not of "is it Linux": # -# ELF, Mach-O, PE/MinGW produced — the package carries every name the -# platform needs to link it and to find it later -# PE/MSVC refused — MSVC exports nothing from a DLL without -# `__declspec(dllexport)`, so the import library would -# be empty and consumers would fail with unresolved -# externals for symbols visibly present in the objects +# ELF, Mach-O, PE/MinGW, PE/MSVC produced — the package carries every name +# the platform needs to link it and to find +# it later # -# Windows' default toolchain here is clang on the MSVC ABI, so this file sees the -# refusal there. 258 pins that refusal in detail, 259 pins Mach-O relocatability, -# 257 pins the PE/MinGW path; what this one adds is that the two outcomes are -# reachable from the same fixture and the same command. +# Every format mcpp targets now produces one. MSVC was the last holdout, and +# what it was missing was never the linker: it exports nothing from a DLL +# without `__declspec(dllexport)` or a `.def`, so mcpp generates the `.def` from +# the objects (258 pins that path in detail, 259 pins Mach-O relocatability, 257 +# the PE/MinGW one). What this file adds is that one fixture and one command +# produce a usable package on whichever platform it runs. # # A shared library is LINKED by `lib.so` and FOUND at run time by its # SONAME, and those are two different filenames. The first version of this @@ -58,37 +57,17 @@ EOF cd mathkit -# ── Windows: the MSVC ABI is refused, and says what to do instead ────── -if [[ "$(uname -s)" != "Linux" && "$(uname -s)" != "Darwin" ]]; then - if "$MCPP" pack mathkit-shared > refuse.log 2>&1; then - cat refuse.log - echo "FAIL: a shared library was packed for the MSVC ABI. Its import" - echo " library has no exports, so consumers fail with unresolved" - echo " externals naming symbols that are in the objects." - exit 1 - fi - grep -qi 'dllexport' refuse.log || { - cat refuse.log - echo "FAIL: it refused, but not for the export reason — 'shared libraries" - echo " are not supported here' does not tell the reader what to change." - exit 1; } - grep -qi 'windows-gnu' refuse.log || { - cat refuse.log - echo "FAIL: the refusal names no way forward. MinGW auto-exports." - exit 1; } - echo "PASS: a shared library package is refused on the MSVC ABI, with the reason" - exit 0 -fi - -# ── macOS: it is produced, and the deep claims live in 259 ────────────── -if [[ "$(uname -s)" == "Darwin" ]]; then +# ── Windows and macOS: it is produced; the deep claims live in 258 / 259 ── +if [[ "$(uname -s)" != "Linux" ]]; then "$MCPP" pack mathkit-shared > pack.log 2>&1 \ - || { cat pack.log; echo "FAIL: shared pack failed on Mach-O"; exit 1; } - macpkg="$(find target/dist -maxdepth 1 -type d -name 'mathkit-0.1.0-*' | head -1)" - [[ -n "$(find "$macpkg" -name 'libmathkit-shared.dylib' | head -1)" ]] || { - find "$macpkg" \( -type f -o -type l \) - echo "FAIL: no .dylib in the package"; exit 1; } - echo "PASS: a shared library package is produced on Mach-O" + || { cat pack.log; echo "FAIL: shared pack failed off ELF"; exit 1; } + nonelf="$(find target/dist -maxdepth 1 -type d -name 'mathkit-0.1.0-*' | head -1)" + # `.dylib` on macOS, `.dll` on Windows — asked for by shape rather than by + # `uname`, so the assertion is about the artifact and not about the host. + [[ -n "$(find "$nonelf" \( -name '*.dylib' -o -name '*.dll' \) | head -1)" ]] || { + find "$nonelf" \( -type f -o -type l \) + echo "FAIL: no shared library in the package"; exit 1; } + echo "PASS: a shared library package is produced off ELF too" exit 0 fi diff --git a/tests/e2e/257_shared_library_pe.sh b/tests/e2e/257_shared_library_pe.sh index 85e13937..83bafe62 100755 --- a/tests/e2e/257_shared_library_pe.sh +++ b/tests/e2e/257_shared_library_pe.sh @@ -129,11 +129,11 @@ grep -q 'ok=42' <<<"$out" || { echo "$out"; echo "FAIL: wrong answer from the PE # ── 4. windows-msvc is refused here, but by the TARGET gate ───────────── # -# The MSVC shared-library refusal cannot be observed from a Linux host: this -# machine cannot serve `x86_64-windows-msvc` at all, so the target gate answers -# first and make_plan is never reached. Asserting `dllexport` here would be -# asserting a message this host cannot produce — the MSVC-ABI half lives in 258, -# under `# requires: msvc`. +# The MSVC-ABI shared library cannot be observed from a Linux host at all: this +# machine cannot serve `x86_64-windows-msvc`, so the target gate answers first +# and make_plan is never reached. That half lives in 258, under +# `# requires: msvc`, where a real link.exe can say whether the generated `.def` +# was accepted. # # What IS worth pinning here is that the refusal happens at all, and names the # host rather than silently building an ELF: that is precisely what this used to diff --git a/tests/e2e/258_shared_library_msvc_auto_def.sh b/tests/e2e/258_shared_library_msvc_auto_def.sh new file mode 100755 index 00000000..e370c3a5 --- /dev/null +++ b/tests/e2e/258_shared_library_msvc_auto_def.sh @@ -0,0 +1,122 @@ +#!/usr/bin/env bash +# requires: msvc +# 258_shared_library_msvc_auto_def.sh — a DLL on the MSVC ABI, exported without +# a single `__declspec(dllexport)` in the source. +# +# MSVC exports nothing from a DLL unless the source says `__declspec(dllexport)` +# or a `.def` lists the symbols. Without either, the import library comes out +# empty and every consumer fails with unresolved externals for symbols that are +# plainly in the object files — a diagnostic pointing nowhere near its cause. +# MinGW's linker auto-exports and hides the whole problem; lld-link's MSVC +# flavour does not, deliberately, because PE caps exports at 65535. +# +# So mcpp writes the `.def` from the objects, which is what CMake's +# `WINDOWS_EXPORT_ALL_SYMBOLS` has done since 3.4. The filtering rules are unit +# tested against synthetic and real COFF (tests/unit/test_coff_exports.cpp); what +# only a real link can answer is whether link.exe accepts the result and whether +# a consumer can then resolve a symbol through it. That is this test. +# +# ⚠️ THE TWO LIMITS ARE ASSERTED, NOT ASSUMED. Auto-export cannot make a data +# symbol readable without `__declspec(dllimport)` on the consumer's declaration, +# and CMake documents the same limit for the same mechanism. The test exercises a +# FUNCTION across the boundary, which is what auto-export does cover, and says so +# — a test that quietly used only functions would read as if the limit did not +# exist. +set -e +source "$(dirname "$0")/_host_path.sh" + +TMP=$(mktemp -d) +trap "rm -rf $TMP" EXIT +cd "$TMP" + +mkdir -p mathkit/src +cat > mathkit/src/mathkit.cppm <<'EOF' +export module mathkit; +export namespace mk { int answer(); } +EOF +# No dllexport anywhere. That is the point. +cat > mathkit/src/impl.cpp <<'EOF' +module mathkit; +namespace mk { int answer() { return 42; } } +EOF +cat > mathkit/mcpp.toml <<'EOF' +[package] +name = "mathkit" +version = "0.1.0" +[build] +sources = ["src/*.cppm", "src/*.cpp"] +[targets.mathkit] +kind = "shared" +[toolchain] +windows = "msvc@system" +EOF + +cd mathkit +"$MCPP" build > build.log 2>&1 || { + cat build.log + echo "FAIL: a kind=\"shared\" target did not build on the MSVC ABI." + echo " If the message mentions dllexport, the old refusal is still in place." + exit 1; } + +# ── the .def was generated, and it is a build-graph node ──────────────── +nj="$(find target -name build.ninja | head -1)" +grep -qE "^build .*mathkit\.def : coff_def " "$nj" || { + grep -n 'def' "$nj" | head + echo "FAIL: no coff_def edge. The .def has to be produced from the same" + echo " objects the link consumes, or the exported surface can drift" + echo " from what was compiled." + exit 1; } +def="$(find target -name 'mathkit.def' | head -1)" +[[ -n "$def" ]] || { echo "FAIL: no .def was written"; exit 1; } +grep -q '^EXPORTS' "$def" || { cat "$def"; echo "FAIL: malformed .def"; exit 1; } +# Non-empty is the criterion: an empty EXPORTS section links fine and produces +# exactly the import library this whole mechanism exists to avoid. +[[ "$(grep -cvE '^(LIBRARY|EXPORTS|$)' "$def")" -gt 0 ]] || { + cat "$def" + echo "FAIL: the .def exports nothing, so the import library is empty and" + echo " every consumer will fail with unresolved externals." + exit 1; } + +# ── the DLL and its import library both exist ─────────────────────────── +[[ -n "$(find target -name 'mathkit.dll' | head -1)" ]] || { echo "FAIL: no DLL"; exit 1; } +[[ -n "$(find target -name 'mathkit.lib' | head -1)" ]] || { + find target -name '*.lib' + echo "FAIL: no import library beside the DLL"; exit 1; } + +# ── and a consumer resolves a function through it ─────────────────────── +# +# A function, deliberately: auto-export covers code. Exported DATA additionally +# needs `__declspec(dllimport)` on the consumer's declaration — a limit of the +# mechanism, documented in docs/12, not a defect of this test. +"$MCPP" pack mathkit > pack.log 2>&1 || { cat pack.log; echo "FAIL: pack"; exit 1; } +pkg="$TMP/mathkit/$(find target/dist -maxdepth 1 -type d -name 'mathkit-0.1.0-*' | head -1)" +PKG_HOST="$(host_path "$pkg")" + +cd "$TMP" +mkdir -p app/src +cat > app/src/main.cpp <<'EOF' +#include +import mathkit; +int main(){ std::printf("msvc-dll=%d\n", mk::answer()); return 0; } +EOF +cat > app/mcpp.toml < run.log 2>&1 ) || { + cat app/run.log + echo "FAIL: the consumer could not link or start against the MSVC DLL." + echo " 'unresolved external symbol' here means the .def did not reach" + echo " the link, or exported the wrong names." + exit 1; } +grep -q 'msvc-dll=42' app/run.log || { cat app/run.log; echo "FAIL: wrong answer"; exit 1; } + +echo "PASS: MSVC produces an exporting DLL with no dllexport in the source" diff --git a/tests/e2e/258_shared_library_msvc_refused.sh b/tests/e2e/258_shared_library_msvc_refused.sh deleted file mode 100755 index 81f8bf73..00000000 --- a/tests/e2e/258_shared_library_msvc_refused.sh +++ /dev/null @@ -1,89 +0,0 @@ -#!/usr/bin/env bash -# requires: msvc -# 258_shared_library_msvc_refused.sh — `kind = "shared"` on the MSVC ABI is -# refused, and the message says why. -# -# The refusal is NOT about the linker. `link /DLL /IMPLIB:` has been in mcpp's -# rule table all along. It is about symbol export: MSVC exports nothing from a -# DLL unless the source says `__declspec(dllexport)` or a `.def` file lists the -# symbols. Without that the import library comes out EMPTY and every consumer -# fails with unresolved externals naming symbols that are plainly in the object -# files — a diagnostic pointing nowhere near its cause. Producing that is worse -# than refusing. -# -# ⚠️ WHY THIS TEST IS SEPARATE FROM 257. On a Linux host this cannot be observed: -# the machine cannot serve `x86_64-windows-msvc`, so the target-vocabulary gate -# answers first and make_plan is never reached. A `# requires:`-less version of -# this test would assert a message its host cannot produce. -# -# ⚠️ AND WHY IT PINS BOTH SIDES. `kind = "lib"` must still build in the same -# project with the same toolchain. Asserting only the refusal cannot distinguish -# "shared is refused" from "this project does not build at all". -set -e - -TMP=$(mktemp -d) -trap "rm -rf $TMP" EXIT -cd "$TMP" - -mkdir -p mathkit/src -cat > mathkit/src/mathkit.cppm <<'EOF' -export module mathkit; -export namespace mk { int answer(); } -EOF -cat > mathkit/src/impl.cpp <<'EOF' -module mathkit; -namespace mk { int answer() { return 42; } } -EOF - -manifest() { # $1 = kind - cat > "$TMP/mathkit/mcpp.toml" < shared.log 2>&1; then - echo "FAIL: a kind=\"shared\" target built for the MSVC ABI." - find target -name '*.dll' -o -name '*.lib' | head - echo " If the import library is empty, consumers fail with unresolved" - echo " externals for symbols that are visibly present in the objects." - exit 1 -fi -grep -qi 'dllexport' shared.log || { - cat shared.log - echo "FAIL: refused, but not by the shared-library gate — the message must" - echo " name symbol export, or the reader cannot tell what to do about it." - exit 1; } -# And it must point somewhere: a refusal with no way forward is a dead end. -grep -q 'windows-gnu' shared.log || { - cat shared.log - echo "FAIL: the refusal names no alternative. MinGW auto-exports, and that is" - echo " the answer for anyone who actually needs a DLL here." - exit 1; } - -# ── and the static form still builds, same project, same toolchain ────── -manifest lib -rm -rf target -"$MCPP" build > lib.log 2>&1 || { - cat lib.log - echo "FAIL: kind=\"lib\" does not build either, so the refusal above proves" - echo " nothing about shared libraries specifically." - exit 1; } -[[ -n "$(find target -name 'mathkit.lib' | head -1)" ]] || { - find target -type f | head - echo "FAIL: no mathkit.lib — the MSVC static path did not produce its artifact" - exit 1; } - -echo "PASS: MSVC refuses kind=\"shared\" for the export reason, and still builds kind=\"lib\"" diff --git a/tests/fixtures/coff/README.md b/tests/fixtures/coff/README.md new file mode 100644 index 00000000..98bebbfd --- /dev/null +++ b/tests/fixtures/coff/README.md @@ -0,0 +1,27 @@ +# COFF fixtures + +`probe-amd64.obj` is a real amd64 COFF object, produced by mingw-cross GCC 16.1 +from `probe-amd64.cpp.in`: + +``` +x86_64-w64-mingw32-g++ -c probe-amd64.cpp.in -o probe-amd64.obj -O1 +``` + +It is committed rather than generated because the parser it exercises +(`mcpp.build.coff_exports`) must be correct on hosts that cannot produce COFF at +all — macOS CI has no mingw, and that is exactly where a byte-level reader is +most likely to be wrong and least likely to be noticed. + +The `.cpp.in` extension is not decoration: `mcpp test` discovers `tests/**/*.cpp` +and would compile a `.cpp` here into a test binary with no `main`. + +Its external defined symbols, per `x86_64-w64-mingw32-nm --defined-only +--extern-only`: + +``` +0000000000000000 R exported_const +0000000000000000 D exported_data +0000000000000000 T exported_fn +0000000000000004 T _Z3usev +000000000000000a T _ZN2ns10mangled_fnEi +``` diff --git a/tests/fixtures/coff/probe-amd64.cpp.in b/tests/fixtures/coff/probe-amd64.cpp.in new file mode 100644 index 00000000..28201598 --- /dev/null +++ b/tests/fixtures/coff/probe-amd64.cpp.in @@ -0,0 +1,7 @@ +extern "C" int exported_fn(int x) { return x + 1; } +extern "C" int exported_data = 7; +extern "C" const int exported_const = 9; +static int hidden_fn(int x) { return x - 1; } +static int hidden_data = 3; +int use() { return hidden_fn(hidden_data); } +namespace ns { int mangled_fn(int) { return 0; } } diff --git a/tests/fixtures/coff/probe-amd64.obj b/tests/fixtures/coff/probe-amd64.obj new file mode 100644 index 0000000000000000000000000000000000000000..adb13f815fe0a9fd2ede53a0e57c52bbb3dc20ea GIT binary patch literal 1002 zcmZuwO-sW-5S{JT`hlQ`h@giYdQgzH)#5=tM2ZwdN-sk3Qu~pXLQO*xr4$dIJn292 z2lOcPf2lLGNxEq>kexSgUM7>tr1P-~WVsB$9-}O%z!?XO^|6iaL>L4@3u;fQ(+rvh zO}gs^oei}g)$)BqmI~iVH%=DZMCJWby@IX0q%M!TI_kbd?K;Lbpl+1ZQGahs^=kHM z2@{4{h(A+wUpHbl#W}%jO6r-O96@b#OLP&F)inE!FeXoZtl>p{npP>7FRi0W_2Jkm zUZ|ofn8^_5&`YkU5!InNNj^14^)=d!v->f37$Ymrs&0SaU_|!&2hbDVXxMU8+w(LT zo?D?8U_oYKLX>Jep1=6cEBD75)*s+W@2wegyQqW|)Nj(v&7y%z5zTV0Gq*~zW}b_X zxlvR>3Yt~Y%xzl1loYg2$!}wB5>GazAF3LCG1?3%c8FTCtB|2O^lkumeCj9zb&T}$ q@{`E~oUu0?1x}~Yb#c7aGuJN`o||rOz|?I&RofoAelWk4e}jK1S6C|m literal 0 HcmV?d00001 diff --git a/tests/unit/test_coff_exports.cpp b/tests/unit/test_coff_exports.cpp new file mode 100644 index 00000000..00afa5f6 --- /dev/null +++ b/tests/unit/test_coff_exports.cpp @@ -0,0 +1,274 @@ +#include + +import std; +import mcpp.build.coff_exports; + +using mcpp::build::coff::Export; +using mcpp::build::coff::read_exports; +using mcpp::build::coff::write_def; +using mcpp::build::coff::kMaxExports; + +// The `.def` generator decides what a DLL's public surface is. Getting the +// filter wrong does not fail loudly: too few exports and the consumer sees +// unresolved externals for symbols that are visibly in the objects; too many +// and the DLL's surface depends on which translation unit instantiated what. +// +// So both halves are tested here, on every host: +// +// * SYNTHETIC objects, built byte by byte, for the filter rules — storage +// class, section number, type, section characteristics, the skip list. +// Nothing else can vary a symbol's storage class on demand. +// * A REAL object (tests/fixtures/coff/probe-amd64.obj, from mingw-cross +// GCC 16.1), because a synthetic file only proves the reader agrees with +// the test's idea of COFF. macOS and Windows CI cannot produce one, which +// is precisely why it is committed rather than generated. + +namespace { + +// ── a minimal COFF writer, so a test can state its input exactly ────────── +struct Sym { + std::string name; // ≤ 8 chars: written inline + std::int16_t section = 1; + std::uint16_t type = 0x20; // function + std::uint8_t storage = 2; // IMAGE_SYM_CLASS_EXTERNAL + std::uint8_t aux = 0; +}; + +void put16(std::vector& b, std::size_t off, std::uint16_t v) { + b[off] = std::byte(v & 0xff); + b[off + 1] = std::byte((v >> 8) & 0xff); +} +void put32(std::vector& b, std::size_t off, std::uint32_t v) { + for (int i = 0; i < 4; ++i) b[off + i] = std::byte((v >> (8 * i)) & 0xff); +} + +// `sectionFlags` is 1-based to match how symbols address sections. +std::vector make_obj(std::vector syms, + std::vector sectionFlags = { 0x60000020u }, + std::uint16_t machine = 0x8664) +{ + constexpr std::size_t kHdr = 20, kSec = 40, kSym = 18; + const std::size_t nsec = sectionFlags.size(); + const std::size_t secOff = kHdr; + const std::size_t symOff = secOff + nsec * kSec; + + std::vector b(symOff + syms.size() * kSym + 4, std::byte{0}); + put16(b, 0, machine); + put16(b, 2, static_cast(nsec)); + put32(b, 8, static_cast(symOff)); + put32(b, 12, static_cast(syms.size())); + put16(b, 16, 0); // no optional header + + for (std::size_t i = 0; i < nsec; ++i) + put32(b, secOff + i * kSec + 36, sectionFlags[i]); + + for (std::size_t i = 0; i < syms.size(); ++i) { + const auto rec = symOff + i * kSym; + for (std::size_t c = 0; c < syms[i].name.size() && c < 8; ++c) + b[rec + c] = std::byte(syms[i].name[c]); + put16(b, rec + 12, static_cast(syms[i].section)); + put16(b, rec + 14, syms[i].type); + b[rec + 16] = std::byte(syms[i].storage); + b[rec + 17] = std::byte(syms[i].aux); + } + put32(b, b.size() - 4, 4); // empty string table + return b; +} + +constexpr std::uint32_t kText = 0x60000020u; // CODE | MEM_EXECUTE | MEM_READ +constexpr std::uint32_t kData = 0xC0000040u; // INITIALIZED | MEM_READ | MEM_WRITE +constexpr std::uint32_t kRdata = 0x40000040u; // INITIALIZED | MEM_READ + +std::vector names(const std::vector& v) { + std::vector out; + for (auto const& e : v) out.push_back(e.name); + std::ranges::sort(out); + return out; +} + +} // namespace + +// ─── what is exported ────────────────────────────────────────────────────── + +TEST(CoffExports, TakesExternalDefinedSymbols) { + auto obj = make_obj({ Sym{ .name = "f" } }); + auto r = read_exports(obj); + ASSERT_TRUE(r) << r.error(); + EXPECT_EQ(names(*r), (std::vector{ "f" })); +} + +TEST(CoffExports, SkipsStaticSymbols) { + // IMAGE_SYM_CLASS_STATIC (3). Exporting a TU-local symbol would put a name + // in the DLL's surface that the author never made public. + auto obj = make_obj({ Sym{ .name = "s", .storage = 3 } }); + auto r = read_exports(obj); + ASSERT_TRUE(r); + EXPECT_TRUE(r->empty()); +} + +TEST(CoffExports, SkipsUndefinedSymbols) { + // SectionNumber 0 is UNDEFINED — a symbol this object REFERENCES. Exporting + // it would have the DLL claim to provide what it is asking someone else for. + auto obj = make_obj({ Sym{ .name = "u", .section = 0 } }); + auto r = read_exports(obj); + ASSERT_TRUE(r); + EXPECT_TRUE(r->empty()); +} + +TEST(CoffExports, SkipsAbsoluteAndDebugSections) { + // -1 IMAGE_SYM_ABSOLUTE, -2 IMAGE_SYM_DEBUG: neither has an address to + // export. + auto obj = make_obj({ Sym{ .name = "a", .section = -1 }, + Sym{ .name = "d", .section = -2 } }); + auto r = read_exports(obj); + ASSERT_TRUE(r); + EXPECT_TRUE(r->empty()); +} + +// ─── DATA, which is the half that fails silently ─────────────────────────── + +TEST(CoffExports, AVariableIsMarkedDATA) { + // Without `DATA` the linker generates a call thunk for a variable and the + // consumer reads the thunk instead of the value. + auto obj = make_obj({ Sym{ .name = "v", .section = 2, .type = 0 } }, + { kText, kData }); + auto r = read_exports(obj); + ASSERT_TRUE(r); + ASSERT_EQ(r->size(), 1u); + EXPECT_TRUE((*r)[0].data); +} + +TEST(CoffExports, AConstInAReadOnlySectionIsStillDATA) { + // `.rdata` is not writable, so a rule keyed on MEM_WRITE alone calls this a + // function. It is a variable, and consumers read it as one. + auto obj = make_obj({ Sym{ .name = "c", .section = 2, .type = 0 } }, + { kText, kRdata }); + auto r = read_exports(obj); + ASSERT_TRUE(r); + ASSERT_EQ(r->size(), 1u); + EXPECT_TRUE((*r)[0].data); +} + +TEST(CoffExports, AFunctionIsNotDATA) { + auto obj = make_obj({ Sym{ .name = "f", .section = 1, .type = 0x20 } }, { kText }); + auto r = read_exports(obj); + ASSERT_TRUE(r); + ASSERT_EQ(r->size(), 1u); + EXPECT_FALSE((*r)[0].data); +} + +// ─── the skip list, each entry for its own reason ────────────────────────── + +TEST(CoffExports, SkipsDestructorThunksAndManagedArtefacts) { + auto obj = make_obj({ + Sym{ .name = "??_Gx" }, // scalar-deleting destructor + Sym{ .name = "??_Ex" }, // vector-deleting destructor + Sym{ .name = "a.b" }, // managed: '.' cannot appear in a C++ mangling + Sym{ .name = "keep" }, + }); + auto r = read_exports(obj); + ASSERT_TRUE(r); + EXPECT_EQ(names(*r), (std::vector{ "keep" })); +} + +TEST(CoffExports, SkipsAuxiliaryRecords) { + // An aux record follows its symbol and is NOT a symbol. Reading one as a + // symbol yields a name made of raw section data — which is how a `.def` + // ends up with entries no linker can resolve. + auto obj = make_obj({ Sym{ .name = "fn", .aux = 1 }, + Sym{ .name = "\x01\x02\x03" }, // the aux record + Sym{ .name = "after" } }); + auto r = read_exports(obj); + ASSERT_TRUE(r); + EXPECT_EQ(names(*r), (std::vector{ "after", "fn" })); +} + +TEST(CoffExports, StripsTheLeadingUnderscoreOnI386Only) { + auto i386 = make_obj({ Sym{ .name = "_f" } }, { kText }, 0x014c); + auto r32 = read_exports(i386); + ASSERT_TRUE(r32); + EXPECT_EQ(names(*r32), (std::vector{ "f" })); + + // On amd64 a leading underscore is part of the name. + auto amd64 = make_obj({ Sym{ .name = "_f" } }, { kText }, 0x8664); + auto r64 = read_exports(amd64); + ASSERT_TRUE(r64); + EXPECT_EQ(names(*r64), (std::vector{ "_f" })); +} + +// ─── refusals, because a partial answer here ships a broken DLL ──────────── + +TEST(CoffExports, RefusesAnUnsupportedMachine) { + auto obj = make_obj({ Sym{ .name = "f" } }, { kText }, 0x5032 /* RISC-V */); + auto r = read_exports(obj); + ASSERT_FALSE(r); + EXPECT_NE(r.error().find("machine"), std::string::npos); +} + +TEST(CoffExports, RefusesATruncatedSymbolTable) { + auto obj = make_obj({ Sym{ .name = "f" }, Sym{ .name = "g" } }); + obj.resize(obj.size() - 20); // cut into the symbol table + auto r = read_exports(obj); + ASSERT_FALSE(r) << "a truncated table must not read as a short one"; + EXPECT_NE(r.error().find("past the end"), std::string::npos); +} + +TEST(CoffExports, RefusesSomethingThatIsNotAnObject) { + std::vector tiny(4, std::byte{0}); + EXPECT_FALSE(read_exports(tiny)); +} + +// ─── the .def text ───────────────────────────────────────────────────────── + +TEST(CoffExports, DefIsSortedDeduplicatedAndMarksData) { + auto def = write_def("mathkit.dll", { + { "zeta", false }, { "alpha", true }, { "zeta", false } }); + EXPECT_EQ(def, + "LIBRARY mathkit.dll\n" + "EXPORTS\n" + " alpha DATA\n" + " zeta\n"); +} + +TEST(CoffExports, TheExportCeilingIsPesNotOurs) { + // Stated as a constant a reader can check against the format, not as a + // number chosen here: PE addresses exports by 16-bit ordinal. + EXPECT_EQ(kMaxExports, 65535u); +} + +// ─── a real object, from a real compiler ─────────────────────────────────── + +TEST(CoffExports, ReadsARealMingwObject) { + // Committed fixture: macOS and Windows CI cannot produce a COFF object, and + // a byte-level reader that is only ever fed its own test's output is exactly + // the kind that agrees with itself and with nothing else. + for (auto const& base : { "tests/fixtures/coff/probe-amd64.obj", + "../tests/fixtures/coff/probe-amd64.obj", + "../../tests/fixtures/coff/probe-amd64.obj" }) { + std::ifstream in(base, std::ios::binary); + if (!in) continue; + std::vector bytes; + for (char c; in.get(c); ) bytes.push_back(static_cast(c)); + + auto r = read_exports(bytes); + ASSERT_TRUE(r) << r.error(); + auto got = names(*r); + + // From `x86_64-w64-mingw32-nm --defined-only --extern-only` on the same + // object; see tests/fixtures/coff/README.md. + EXPECT_NE(std::ranges::find(got, "exported_fn"), got.end()); + EXPECT_NE(std::ranges::find(got, "exported_data"), got.end()); + EXPECT_NE(std::ranges::find(got, "exported_const"), got.end()); + EXPECT_NE(std::ranges::find(got, "_ZN2ns10mangled_fnEi"), got.end()); + // `static` at namespace scope is internal linkage: not in the surface. + EXPECT_EQ(std::ranges::find(got, "_ZL9hidden_fni"), got.end()); + + for (auto const& e : *r) { + if (e.name == "exported_data" || e.name == "exported_const") + EXPECT_TRUE(e.data) << e.name << " is a variable"; + if (e.name == "exported_fn") EXPECT_FALSE(e.data); + } + return; + } + GTEST_SKIP() << "fixture not reachable from this working directory"; +} From 5142dc53c20b74027bc04774463928d8c8dff54e Mon Sep 17 00:00:00 2001 From: speak-agent <248744407+speak-agent@users.noreply.github.com> Date: Tue, 18 Aug 2026 06:24:44 +0800 Subject: [PATCH 3/9] feat(build,manifest,pack): annotation wins over auto-export, and cl.exe can consume a package MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Two things the auto-`.def` needed to be usable rather than merely present. **An author who annotates keeps their surface.** `__declspec(dllexport)` makes the compiler write `/EXPORT:` directives into the object's `.drectve` section. Generating a list on top of that exports the same names twice (LNK4197) and, worse, exports everything else as well — replacing a chosen public surface with all of it. So mcpp reads `.drectve` and stands down. Detected rather than configured: a manifest key for "I annotated my exports" would be a second place to say what the objects already say, and the two could disagree. Tested against a real annotated object, because the point is recognising what a COMPILER emits. **A package can now be linked by native cl.exe.** The generated manifest already carried each leg's link line as `ldflags`, which is GNU spelling — cl rejects `-L`. It now also carries the dialect-neutral pair `[target..runtime]` `link_library_dirs` / `libraries`, which mcpp renders as `/LIBPATH:` + `.lib` or `-L` + `-l` from the target. Not new vocabulary: the same two keys `[runtime]` has had at the top level, made per-target. BOTH spellings are emitted, and that is the load-bearing part. An older mcpp reads only the ldflags and silently ignores the runtime block (measured), so dropping the ldflags would leave every older client with no link line at all. A newer mcpp seeing the neutral form therefore IGNORES that leg's ldflags rather than adding to them — adding would put `-L` back on a cl command line, which is the whole thing being avoided. Scoped to distribution packages: a hand-written manifest that states both may well mean both. --- src/build/coff_exports.cppm | 50 ++++++++++++++++++ src/build/prepare.cppm | 28 +++++++++- src/cli/cmd_build.cppm | 9 ++++ src/manifest/toml.cppm | 12 +++++ src/manifest/types.cppm | 11 ++++ src/pack/manifest_emit.cppm | 14 +++++ tests/e2e/258_shared_library_msvc_auto_def.sh | 40 +++++++++++++- tests/fixtures/coff/README.md | 6 +++ tests/fixtures/coff/annotated-amd64.cpp.in | 2 + tests/fixtures/coff/annotated-amd64.obj | Bin 0 -> 839 bytes tests/unit/test_coff_exports.cpp | 45 ++++++++++++++++ 11 files changed, 215 insertions(+), 2 deletions(-) create mode 100644 tests/fixtures/coff/annotated-amd64.cpp.in create mode 100644 tests/fixtures/coff/annotated-amd64.obj diff --git a/src/build/coff_exports.cppm b/src/build/coff_exports.cppm index d93fc5b5..22b606e2 100644 --- a/src/build/coff_exports.cppm +++ b/src/build/coff_exports.cppm @@ -60,6 +60,20 @@ inline constexpr std::size_t kMaxExports = 65535; // against is exactly where an untested assumption would hide. bool is_supported_machine(std::uint16_t machine); +// Does this object ALREADY declare exports? +// +// `__declspec(dllexport)` makes the compiler write `/EXPORT:name` directives +// into the object's `.drectve` section, which the linker reads. When an author +// has annotated their surface, adding a generated `.def` on top would export +// the same names twice — `LNK4197: export specified multiple times` — and, +// worse, would export everything else as well, quietly replacing a chosen +// public surface with all of it. +// +// So the annotation wins and mcpp stays out of the way. Detected rather than +// configured: a manifest key for "I annotated my exports" would be a second +// place to say something the objects already say, and the two could disagree. +bool declares_exports(std::span bytes); + // Read one object file's exportable symbols. `bytes` is a whole `.obj`. // // Returns an error string for input this reader cannot honestly interpret — @@ -138,6 +152,42 @@ bool is_skipped_name(std::string_view n) { } // namespace +bool declares_exports(std::span bytes) { + if (bytes.size() < kFileHeaderSize) return false; + const auto numSections = rd16(bytes, 2); + const std::size_t sectionsOff = kFileHeaderSize + rd16(bytes, 16); + + for (std::uint16_t i = 0; i < numSections; ++i) { + const auto hdr = sectionsOff + std::size_t(i) * kSectionHeaderSize; + if (hdr + kSectionHeaderSize > bytes.size()) return false; + + std::string name; + for (std::size_t c = 0; c < 8; ++c) { + auto ch = std::to_integer(bytes[hdr + c]); + if (ch == '\0') break; + name.push_back(ch); + } + if (name != ".drectve") continue; + + const auto size = rd32(bytes, hdr + 16); + const auto off = rd32(bytes, hdr + 20); + if (off == 0 || std::size_t(off) + size > bytes.size()) return false; + + // The section is plain text: a run of linker directives. Both spellings + // occur — cl emits `/EXPORT:`, clang-cl `-export:` — and a linker + // accepts either, so looking for one of them only would be a coin flip + // on which compiler produced the object. + std::string text; + text.reserve(size); + for (std::uint32_t k = 0; k < size; ++k) + text.push_back(std::to_integer(bytes[std::size_t(off) + k])); + for (auto& c : text) c = static_cast(std::tolower(static_cast(c))); + if (text.find("/export:") != std::string::npos) return true; + if (text.find("-export:") != std::string::npos) return true; + } + return false; +} + bool is_supported_machine(std::uint16_t machine) { return machine == kMachineI386 || machine == kMachineAmd64 || machine == kMachineArm || machine == kMachineArmNT diff --git a/src/build/prepare.cppm b/src/build/prepare.cppm index 6d16502d..3df2d811 100644 --- a/src/build/prepare.cppm +++ b/src/build/prepare.cppm @@ -210,13 +210,39 @@ materialize_generated_files(const std::filesystem::path& root, void merge_conditional_config(mcpp::manifest::Manifest& m, const cfgpred::Ctx& ctx) { + // A DISTRIBUTION package may carry a leg's link line twice: as `ldflags` + // (GNU spelling, which is all an older mcpp reads) and as the neutral + // `[target..runtime]` pair, which mcpp renders per dialect. Applying + // both would put `-L` on a native `cl.exe` command line, which is exactly + // what the neutral form exists to avoid — so where the neutral form is + // present it REPLACES the ldflags rather than adding to them. + // + // Scoped to distribution packages on purpose: a hand-written manifest that + // states both may well mean both (`ldflags` also carries things like + // `-Wl,--as-needed`), and silently dropping half of it would be its own + // silent failure. + const bool generatedPackage = mcpp::pack::is_distribution_package(m); + for (auto const& cc : m.conditionalConfigs) { if (!cfgpred::matches(cc.predicate, ctx)) continue; + const bool neutralWins = generatedPackage + && (!cc.linkLibraryDirs.empty() || !cc.libraries.empty()); // One append() for every field the axis may carry (#258). Matching // sections land AFTER the base entries, so a conditional rule beats // a broader unconditional one under GNU last-wins — which is what // makes an off-OS REMOVAL expressible (`-U` after the base `-D`). - mcpp::manifest::append(m.buildConfig, cc.inputs); + if (neutralWins) { + auto inputs = cc.inputs; + inputs.ldflags.clear(); + mcpp::manifest::append(m.buildConfig, inputs); + } else { + mcpp::manifest::append(m.buildConfig, cc.inputs); + } + // The neutral half goes where `render_link_intent_flags` will find it. + for (auto const& d : cc.linkLibraryDirs) + m.runtimeConfig.linkIntent.linkLibraryDirs.push_back(d); + for (auto const& l : cc.libraries) + m.runtimeConfig.linkIntent.libraries.push_back(l); // `modules.sources` is the scanner's own view and is not part of // BuildInputs, so conditional sources are mirrored into it here. for (auto const& s : cc.inputs.sources) diff --git a/src/cli/cmd_build.cppm b/src/cli/cmd_build.cppm index e895f271..380a8db4 100644 --- a/src/cli/cmd_build.cppm +++ b/src/cli/cmd_build.cppm @@ -502,6 +502,13 @@ export int cmd_coff_def(const mcpplibs::cmdline::ParsedArgs& parsed) { if (auto v = parsed.value("name")) libName = *v; std::vector all; + // When the AUTHOR has annotated the surface, the annotation wins and this + // edge writes an empty EXPORTS section — the linker then takes its export + // set from the objects' own `/EXPORT:` directives, exactly as if mcpp were + // not here. Adding a generated list on top would export the same names twice + // (LNK4197) and, worse, would export everything else as well, replacing a + // chosen public surface with all of it. + bool annotated = false; for (std::size_t i = 0; i < parsed.positional_count(); ++i) { const std::filesystem::path obj{ parsed.positional(i) }; std::ifstream in(obj, std::ios::binary); @@ -511,6 +518,7 @@ export int cmd_coff_def(const mcpplibs::cmdline::ParsedArgs& parsed) { } std::vector bytes; for (char c; in.get(c); ) bytes.push_back(static_cast(c)); + if (mcpp::build::coff::declares_exports(bytes)) annotated = true; auto syms = mcpp::build::coff::read_exports(bytes); if (!syms) { // Named with the object, because "which one" is the whole question @@ -524,6 +532,7 @@ export int cmd_coff_def(const mcpplibs::cmdline::ParsedArgs& parsed) { // Refused, not truncated. A `.def` cut at the ceiling links cleanly and // then fails at whichever consumer happens to need a symbol that fell off // the end — a diagnostic with no path back to this decision. + if (annotated) all.clear(); std::ranges::sort(all); all.erase(std::ranges::unique(all).begin(), all.end()); if (all.size() > mcpp::build::coff::kMaxExports) { diff --git a/src/manifest/toml.cppm b/src/manifest/toml.cppm index 5039f01e..9198877c 100644 --- a/src/manifest/toml.cppm +++ b/src/manifest/toml.cppm @@ -1456,6 +1456,18 @@ std::expected parse_string(std::string_view content, // prepare_build. ConditionalConfig cc; cc.predicate = triple; + // `[target..runtime]` — the dialect-neutral link intent. Two + // keys only, and the same two `[runtime]` already has at the top + // level: this makes them per-target, it does not invent a vocabulary. + if (auto rit = body.find("runtime"); rit != body.end() && rit->second.is_table()) { + auto& rt = rit->second.as_table(); + if (auto f = rt.find("link_library_dirs"); f != rt.end() && f->second.is_array()) + for (auto& v : f->second.as_array()) + if (v.is_string()) cc.linkLibraryDirs.emplace_back(v.as_string()); + if (auto f = rt.find("libraries"); f != rt.end() && f->second.is_array()) + for (auto& v : f->second.as_array()) + if (v.is_string()) cc.libraries.push_back(v.as_string()); + } if (auto bit = body.find("build"); bit != body.end() && bit->second.is_table()) { auto& bt = bit->second.as_table(); auto read_list = [&](const char* key, std::vector& out) { diff --git a/src/manifest/types.cppm b/src/manifest/types.cppm index 2940ad81..bde125f6 100644 --- a/src/manifest/types.cppm +++ b/src/manifest/types.cppm @@ -702,6 +702,17 @@ struct ConditionalConfig { // `!`-exclusion globs work there too (the scanner handles positive+ // negative sets). BuildInputs inputs; + // `[target..runtime]` — the DIALECT-NEUTRAL half of a link line. + // + // `inputs.ldflags` above is spelled the GNU way and native `cl.exe` rejects + // `-L`. These two keys say the same thing without committing to a spelling, + // and `render_link_intent_flags` renders them as `/LIBPATH:` + `.lib` or + // `-L` + `-l` depending on the target. A generated package carries BOTH, + // because an older mcpp reads only the first — see the note where they are + // merged for why the newer client must then IGNORE the ldflags rather than + // add to them. + std::vector linkLibraryDirs; + std::vector libraries; // Conditional dependencies (Phase 1b): merged into the corresponding // manifest maps in prepare_build when the predicate matches the resolved // target — before dependency resolution, so they resolve like any dep. diff --git a/src/pack/manifest_emit.cppm b/src/pack/manifest_emit.cppm index cbd540fe..c39b94ef 100644 --- a/src/pack/manifest_emit.cppm +++ b/src/pack/manifest_emit.cppm @@ -242,6 +242,20 @@ std::string emit_package_manifest(const PackageDoc& doc) { leg.triple, peGnuShared ? "\"-Wl,-Bdynamic\", " : "", leg.linkName); + + // …and the same statement without a dialect. mcpp renders these as + // `/LIBPATH:` + `.lib` or `-L` + `-l` from the target, which is + // what lets a consumer driven by native `cl.exe` link this package at + // all — cl rejects `-L`. + // + // BOTH are emitted, deliberately. An older mcpp reads only the ldflags + // above and silently ignores this block (measured), so dropping the + // ldflags would leave every older client with no link line at all. A + // newer mcpp seeing this block ignores that leg's ldflags rather than + // adding to them — see merge_conditional_config. + o += std::format("[target.'{}'.runtime]\n", cfg_predicate_for(leg.triple)); + o += std::format("link_library_dirs = [\"lib/{}\"]\n", leg.triple); + o += std::format("libraries = [\"{}\"]\n\n", leg.linkName); } // A shared library has to be FOUND at run time as well as linked, and the // two are different search paths — `link_library_dirs` is not rpath. diff --git a/tests/e2e/258_shared_library_msvc_auto_def.sh b/tests/e2e/258_shared_library_msvc_auto_def.sh index e370c3a5..14203f0c 100755 --- a/tests/e2e/258_shared_library_msvc_auto_def.sh +++ b/tests/e2e/258_shared_library_msvc_auto_def.sh @@ -119,4 +119,42 @@ EOF exit 1; } grep -q 'msvc-dll=42' app/run.log || { cat app/run.log; echo "FAIL: wrong answer"; exit 1; } -echo "PASS: MSVC produces an exporting DLL with no dllexport in the source" +# ── and an ANNOTATED library keeps its own surface ────────────────────── +# +# `__declspec(dllexport)` writes `/EXPORT:` directives into the object, and a +# generated list on top would export the same names twice (LNK4197) AND export +# everything else besides — replacing a chosen public surface with all of it. +# So the annotation wins and the generated `.def` stays empty. +cd "$TMP" +mkdir -p annotated/src +cat > annotated/src/a.cppm <<'EOF' +export module annkit; +export namespace ak { __declspec(dllexport) int chosen(); } +EOF +cat > annotated/src/a.cpp <<'EOF' +module annkit; +namespace ak { __declspec(dllexport) int chosen() { return 7; } } +EOF +cat > annotated/mcpp.toml <<'EOF' +[package] +name = "annkit" +version = "0.1.0" +[build] +sources = ["src/*.cppm", "src/*.cpp"] +[targets.annkit] +kind = "shared" +[toolchain] +windows = "msvc@system" +EOF +( cd annotated && "$MCPP" build > build.log 2>&1 ) || { + cat annotated/build.log; echo "FAIL: the annotated library did not build"; exit 1; } +anndef="$(find annotated/target -name 'annkit.def' | head -1)" +[[ -n "$anndef" ]] || { echo "FAIL: no .def edge for the annotated library"; exit 1; } +[[ "$(grep -cvE '^(LIBRARY|EXPORTS|$)' "$anndef")" -eq 0 ]] || { + cat "$anndef" + echo "FAIL: mcpp generated an export list for a library that declares its own." + echo " That exports every other symbol too, which replaces the author's" + echo " chosen surface with all of it." + exit 1; } + +echo "PASS: MSVC exports without dllexport, and defers to dllexport when present" diff --git a/tests/fixtures/coff/README.md b/tests/fixtures/coff/README.md index 98bebbfd..4a2794f9 100644 --- a/tests/fixtures/coff/README.md +++ b/tests/fixtures/coff/README.md @@ -25,3 +25,9 @@ Its external defined symbols, per `x86_64-w64-mingw32-nm --defined-only 0000000000000004 T _Z3usev 000000000000000a T _ZN2ns10mangled_fnEi ``` + +`annotated-amd64.obj` is the same shape with one symbol marked +`__declspec(dllexport)`, from `annotated-amd64.cpp.in`. It exists so +`declares_exports` is tested against a real `.drectve` section rather than one +this repository wrote itself — the point of that function is to recognise what a +COMPILER emits. diff --git a/tests/fixtures/coff/annotated-amd64.cpp.in b/tests/fixtures/coff/annotated-amd64.cpp.in new file mode 100644 index 00000000..a361ae75 --- /dev/null +++ b/tests/fixtures/coff/annotated-amd64.cpp.in @@ -0,0 +1,2 @@ +extern "C" __declspec(dllexport) int chosen(int x) { return x + 1; } +extern "C" int not_chosen(int x) { return x - 1; } diff --git a/tests/fixtures/coff/annotated-amd64.obj b/tests/fixtures/coff/annotated-amd64.obj new file mode 100644 index 0000000000000000000000000000000000000000..30de43b6a8816771ebbf6eb38a9d0ec1efabbb3f GIT binary patch literal 839 zcmZ{jOH0Hs5P*~I`Tzx=2k}%E1qF>=5D$xoE-UO!dhsS`ZQ}!^Ep3oeJb3c9us>;G zul^?2nIx6O);VNeUnbLJ8h@@3lrKuij3lL=%%@rqsj;r!j2in!Jsyn)?$P-A=Gg6Dus-uNQ!DqBmsy&NK`(kv z3!e0JVhOY(D<})=t;yIrtSW|YqAcuIW!dA)D+dqddqKz(4U;4lq2Mu#vP{O+Nz#Pg zUDByPK!twS4Y2SVBx6kkZg42}WQ#cs9Y{GXmCZ?T{$q?Ozz$l^GA2OY0>v4?ceiC< kyO?q4ww(mee0ls29J=c5 bytes; + for (char c; in.get(c); ) bytes.push_back(static_cast(c)); + EXPECT_TRUE(declares_exports(bytes)); + return; + } + GTEST_SKIP() << "fixture not reachable from this working directory"; +} + +TEST(CoffExports, AnObjectWithNoDirectivesDeclaresNothing) { + // The negative half: without it, a `declares_exports` that always said true + // would pass the test above and silently disable auto-export everywhere. + for (auto const& base : { "tests/fixtures/coff/probe-amd64.obj", + "../tests/fixtures/coff/probe-amd64.obj", + "../../tests/fixtures/coff/probe-amd64.obj" }) { + std::ifstream in(base, std::ios::binary); + if (!in) continue; + std::vector bytes; + for (char c; in.get(c); ) bytes.push_back(static_cast(c)); + EXPECT_FALSE(declares_exports(bytes)); + return; + } + GTEST_SKIP() << "fixture not reachable from this working directory"; +} + +TEST(CoffExports, SyntheticObjectsHaveNoDirectiveSection) { + EXPECT_FALSE(declares_exports(make_obj({ Sym{ .name = "f" } }))); +} From a0d98747b5124573bed33a597805bad9b4b5eef1 Mon Sep 17 00:00:00 2001 From: speak-agent <248744407+speak-agent@users.noreply.github.com> Date: Tue, 18 Aug 2026 06:27:54 +0800 Subject: [PATCH 4/9] docs: the Windows round, and where it diverged from the design MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Records the three deviations and their evidence: .ixx is NOT built in (the extension set is configuration); pack had to follow module_extensions too, which the design did not cover; and the export-macro layer became detection of `.drectve` rather than a new key, because a manifest key for 'I annotated my exports' is a second place to say what the objects already say. Also the two defects only measurement found — a .ixx library packed to a silently C-surface-tagged package with no interface at all, and a new module edge into mcpp.manifest.types making GCC 16.1 ICE on an unrelated file — and an honest list of what is still unverified. --- ...ws-shared-library-and-module-extensions.md | 54 ++++++++++++++ CHANGELOG.md | 58 +++++++++++++++ docs/12-binary-distribution.md | 72 +++++++++++++------ docs/zh/12-binary-distribution.md | 58 ++++++++++----- mcpp.toml | 2 +- src/version.cppm | 2 +- 6 files changed, 206 insertions(+), 40 deletions(-) diff --git a/.agents/docs/2026-08-18-windows-shared-library-and-module-extensions.md b/.agents/docs/2026-08-18-windows-shared-library-and-module-extensions.md index 892ef559..08634bfb 100644 --- a/.agents/docs/2026-08-18-windows-shared-library-and-module-extensions.md +++ b/.agents/docs/2026-08-18-windows-shared-library-and-module-extensions.md @@ -318,3 +318,57 @@ L4(cl.exe 消费) ── 独立于 L2/L3,但只有 L2 落地后才有 msv [lld]: https://github.com/llvm/llvm-project/pull/71087 [ms1]: https://learn.microsoft.com/en-gb/answers/questions/1665106/how-to-use-c-20-modules-in-shared-libraries [ms2]: https://learn.microsoft.com/en-us/answers/questions/1695780/c-20-modules-in-shared-libraries + + +--- + +## 8. 实施记录(2026-08-18,2026.8.18.2) + +四层全部落地,单 PR。与设计的偏差各自有实测依据。 + +### 8.1 与设计不同的地方 + +| 设计说 | 实际做的 | 为什么 | +|---|---|---| +| L1「把 `.ixx` 放进 builtin」 | **不内置**,只做「未声明即拒绝」 | 扩展名集合是配置,不是 mcpp 逐个追加的清单(review 时更正) | +| L1 只涉及扫描器 | 还修了 **lib root 约定**与 **pack 的 manifest 输出** | `mcpp pack` 必须跟着 `module_extensions` 走,否则用户要配两次 | +| L3「提供导出头/宏」 | **检测 `.drectve`,标注优先** | 为「我标注过了」加一个键,就是给对象已经说过的事再加一个说法 | + +### 8.2 实测抓到的、设计里没有的缺陷 + +**`.ixx` 库打出来的包是静默错的。** lib root 约定把 `.cppm` 写死,于是闭包从一个 +不存在的文件开始: + +``` +$ mcpp pack mathkit + Interface (headers only) ← 模块接口整个没了 + Withheld (nothing) + Packed …-x86_64-linux-gnu ← C 表面的 tag +``` + +**两半都错,而第二半比第一半更糟**:空的发布集合正是打包器判定「C 表面」的依据, +所以丢掉接口的同时,这个包也不再约束 C++ ABI,兼容性闸门停止检查编译器与标准库。 +一个「说得比实际少」的包,正是整个分发设计要防的那种失败。 + +**给 `mcpp.manifest.types` 加一条模块边会让 GCC 16.1 ICE。** 探测型的 lib-root +解析需要扩展名表(`mcpp.source_kind`),而 `types` 是几乎所有东西都依赖的低层模块。 +加上那条 import 之后,GCC 在编译**与改动无关的 `src/main.cpp`** 时 ICE, +清掉 gcm.cache 也不行 —— 与本仓库此前遇到的模块毒化形状一致。 +**解法不是加边,而是把函数移到边已经存在的地方**(`mcpp.manifest.toml` 本就 import +了 `mcpp.source_kind`)。 + +### 8.3 验证到哪一步 + +| 断言 | 载体 | 平台 | +|---|---|---| +| 已声明的扩展名无需额外帮助;未声明的当场拒绝 | e2e 260 | 三平台 | +| pack 跟随扩展名;包自带声明;`.cppm` 包 manifest 不变 | e2e 261 | 三平台 | +| COFF 筛选规则(外部/已定义/DATA/跳过表/aux/i386 下划线/拒绝) | 单测 ×16(合成) | 三平台 | +| 真实 mingw 对象可读;真实标注对象被识别 | 单测 ×3(committed fixture) | 三平台 | +| MSVC 产出带非空导出的 DLL,消费者链通并运行 | e2e 258 | **Windows** | +| 标注过的库不被自动导出覆盖 | e2e 258 后半 | **Windows** | +| `.def` 是构建图节点,输入是链接同一批对象 | e2e 258 | **Windows** | +| MinGW 不生成 `.def`(链接器已自动导出) | 本机实测 `0 edges` | Linux | + +**尚未验证**:`cl.exe` 消费打包库的端到端(需要 msvc job 里再加一条消费用例); +数据符号在消费端仍需 `dllimport` 这条限制目前只写在文档里,没有做成可复现的测试。 diff --git a/CHANGELOG.md b/CHANGELOG.md index 14f64d80..85c44c12 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -3,6 +3,64 @@ > 本文件追踪 `mcpp-community/mcpp` 公开仓的版本演进。 > 格式参考 [Keep a Changelog](https://keepachangelog.com/zh-CN/1.1.0/)。 +## [2026.8.18.2] — 2026-08-18 + +### 新增 + +- **`kind = "shared"` 在 MSVC ABI 上可用了 —— mcpp 自己生成 `.def`。** + + MSVC 在没有 `__declspec(dllexport)`、也没有 `.def` 时,DLL 什么都不导出; + 导入库为空,消费者拿到一堆 unresolved externals,而符号明明就在对象里。 + **拒绝的理由成立,结论不成立** —— CMake 的 `WINDOWS_EXPORT_ALL_SYMBOLS` 自 3.4 + 起就是这么做的,而且它的 `bindexplib` **直接读 COFF、不依赖 dumpbin**。这一点是 + 决定性的:`dumpbin` 只在 Visual Studio 开发者环境里,而 mcpp 在 Windows 的默认 + 工具链是 clang,`mcpp build` 根本不在那个环境里。 + + `mcpp.build.coff_exports` 是那个读取器,写成**对字节的纯函数**,于是它能在任何 + 平台上被测试 —— 16 条单测逐字节构造对象(那是唯一能按需改变存储类的办法), + 外加一个**真实的 mingw-cross 对象**:只喂自己测试输出的读取器,只会与自己一致。 + 超过 **65535** 个可导出符号时**拒绝而不截断**:被截断的导出表能干净链完, + 然后在「恰好需要那个掉出去的符号」的消费者那里失败。 + + **标注优先。** 对象里已带 `/EXPORT:` 指令(即 `__declspec(dllexport)` 的产物)时, + mcpp 让开、不生成任何东西 —— 再加一份列表会把同名符号导出两次(`LNK4197`), + 更糟的是把其余所有符号也导出,用「全部」替换掉作者选定的公开面。 + **这件事靠检测而不是配置**:为「我标注过了」加一个 manifest 键,就是给对象已经 + 说过的事再加一个说法,而两者可以不一致。 + + 仍有两条限制是工具消不掉的(与 CMake 记录的同两条):导出的**数据**在消费端仍需 + `__declspec(dllimport)`;**vtable 被引用的类**要整类标注。两者都写进了 docs/12。 + +- **打包库可以被原生 `cl.exe` 消费。** 生成的 manifest 现在同时带方言中立的 + `[target..runtime]`(`link_library_dirs` / `libraries`),mcpp 会按 target + 渲染成 `/LIBPATH:` + `.lib` 或 `-L` + `-l`。**两种拼写都带**:旧版 mcpp + 只读 `ldflags` 并静默忽略新段,去掉它会让旧客户端一个 flag 都拿不到; + 而新版读到中立形式时**忽略**同腿的 `ldflags` 而不是叠加 —— 叠加会把 `-L` + 送回 `cl` 的命令行。 + +### 修复 + +- **`mcpp pack` 现在跟着工程的 `module_extensions` 走,不需要再配一次。** + + 实测:接口是 `.ixx` 的库打出来的包是**静默错的** —— + + + + 两半都错且都不出声:没有接口,消费者 `import` 不了;而**空的发布集合正是打包器 + 判定「C 表面」的依据**,于是这个包同时不再约束 C++ ABI,兼容性闸门也不再检查 + 编译器与标准库。真因是 lib root 约定把 `.cppm` 写死了。现在按**已声明的每个扩展名** + 各给一个候选并取存在的那个;生成的 manifest 也会**自己声明** `module_extensions` + —— 从**发布的文件**算出来,因此不可能与 `sources` 不一致。 + +- **`sources` 命中却分类不出角色的文件,现在当场拒绝。** + + 未声明的 `.ixx` 过去会编译出一个**没人链接的对象**,报错是 + `undefined reference to mk::answer@mathkit()` —— 既不点名扩展名也不点名那条键。 + 根因是「这是不是模块接口」有**两个答题者**:扫描器读到 `export module` 记下 + `provides`(所以边上挂了 `bmi_out`),而分类器说 `Other`,链接集合只读后者。 + + ⚠️ 这是**行为变化**:`sources` 里混进 `.md` / `.txt` 的工程会开始报错。 + ## [2026.8.18.1] — 2026-08-18 ### 新增 diff --git a/docs/12-binary-distribution.md b/docs/12-binary-distribution.md index 3ad47500..143be95a 100644 --- a/docs/12-binary-distribution.md +++ b/docs/12-binary-distribution.md @@ -264,32 +264,46 @@ package published to a mixed-version audience. | `kind = "shared"` on Linux/ELF | ✅ — the package carries both the link name and the SONAME | | `kind = "shared"` on PE / MinGW (`*-windows-gnu`) | ✅ — the package carries the `.dll` **and** its import library | | `kind = "shared"` on Mach-O (`*-macos`) | ✅ — install name is `@rpath/`, so the `.dylib` relocates | -| `kind = "shared"` on PE / MSVC (`*-windows-msvc`) | ❌ refused — see below | +| `kind = "shared"` on PE / MSVC (`*-windows-msvc`) | ✅ — mcpp generates the `.def`; see below | | `kind = "shared"` on `*-musl` | ❌ a musl target links statically | | shipping prebuilt BMIs | ❌ not attempted; BMIs are compiler-build-exact | | bundling dependencies into the package | ❌ declare them instead (above) | | consuming a package with **native `cl.exe`** | ❌ see below | -### Why MSVC refuses `kind = "shared"` +### Exports on the MSVC ABI -Not the linker — `link /DLL /IMPLIB:` works. **Symbol export.** MSVC exports -nothing from a DLL unless the source says `__declspec(dllexport)` or a `.def` -file lists the symbols, so the import library comes out empty and every consumer -fails with unresolved externals naming symbols that are plainly in the object -files. mcpp refuses rather than produce a diagnostic that points nowhere near its -cause: +MSVC exports nothing from a DLL unless the source says `__declspec(dllexport)` or +a `.def` file lists the symbols. Without either, the import library comes out +empty and every consumer fails with unresolved externals for symbols that are +plainly in the object files. MinGW's linker auto-exports and hides this +entirely; lld-link's MSVC flavour does not, deliberately, because PE caps +exports at 65535. -``` -target 'mathkit': kind = "shared" is not supported for the MSVC ABI (x86_64-windows-msvc). - MSVC exports nothing from a DLL unless the source says `__declspec(dllexport)` - ... - Use kind = "lib" for this target, or build it for *-windows-gnu (MinGW), - where the linker auto-exports. -``` +mcpp generates the `.def` from the objects, which is what CMake's +`WINDOWS_EXPORT_ALL_SYMBOLS` has done since 3.4. It is a build-graph node whose +inputs are the same objects the link consumes, so the exported surface cannot +drift from what was compiled, and it reads COFF directly rather than shelling out +to `dumpbin` — that tool lives in a Visual Studio developer environment, and +mcpp's default Windows toolchain is clang. + +**Two limits survive that no tool can remove**, and they are the same two CMake +documents for the same mechanism: + +| | | +|---|---| +| exported **data** | the consumer still needs `__declspec(dllimport)` on its declaration; without it the linker reads a call thunk instead of the value | +| a class whose **vtable** is referenced | the whole class must be marked, e.g. in a delegating constructor of a class with virtual functions | + +Both are answered by annotation, and **annotation wins**: an object that already +carries `/EXPORT:` directives — which is what `__declspec(dllexport)` emits — +makes mcpp stand down and generate nothing. Adding a list on top would export the +same names twice (`LNK4197`) and export everything else besides, replacing a +chosen public surface with all of it. Nothing is configured for this; the objects +say it. -MinGW's linker auto-exports, which is why `*-windows-gnu` is supported and -`*-windows-msvc` is not. Closing this needs a generated `.def` — a symbol scan -over the objects — which is a build-graph node, not a flag. +Past 65535 exportable symbols mcpp refuses rather than truncating. A truncated +export table links cleanly and then fails at whichever consumer needed the symbol +that fell off the end. ### A package's link flags are GNU-spelled @@ -301,9 +315,25 @@ ldflags = ["-Llib/x86_64-windows-msvc", "-lmathkit"] ``` Every driver mcpp uses accepts that — including clang on the MSVC ABI, which is -Windows' default here. **Native `cl.exe` does not**: it rejects `-L`. So a -consumer that pins `[toolchain] windows = "msvc@system"` cannot link a packaged -library today. +Windows' default here. **Native `cl.exe` does not**: it rejects `-L`. So the +package carries the same statement a second time, without a dialect: + +```toml +[target.'cfg(all(arch = "x86_64", os = "windows", env = "msvc"))'.runtime] +link_library_dirs = ["lib/x86_64-windows-msvc"] +libraries = ["mathkit"] +``` + +mcpp renders those as `/LIBPATH:` + `.lib` or `-L` + `-l` from the +target, so a `cl.exe` consumer links the package. They are not new vocabulary — +`[runtime]` has had both keys at the top level all along; this makes them +per-target. + +**Both spellings are emitted, and a newer mcpp ignores the `ldflags` rather than +adding to them.** An older mcpp reads only the `ldflags` and silently ignores the +`runtime` block, so dropping the `ldflags` would leave every older client with no +link line at all; adding both would put `-L` back on the `cl` command line, which +is the thing being avoided. Naming the file by path instead (`lib//mathkit.lib`) is the spelling every driver takes, and it does not work either: ninja runs link commands with diff --git a/docs/zh/12-binary-distribution.md b/docs/zh/12-binary-distribution.md index 33a454c6..769ed071 100644 --- a/docs/zh/12-binary-distribution.md +++ b/docs/zh/12-binary-distribution.md @@ -239,29 +239,38 @@ ldflags = ["-Llib/x86_64-linux-musl", "-lmathkit"] | `kind = "shared"` on Linux/ELF | ✅ —— 包里同时带链接名与 SONAME | | `kind = "shared"` on PE / MinGW(`*-windows-gnu`) | ✅ —— 包里同时带 `.dll` **和它的导入库** | | `kind = "shared"` on Mach-O(`*-macos`) | ✅ —— install name 是 `@rpath/`,`.dylib` 可重定位 | -| `kind = "shared"` on PE / MSVC(`*-windows-msvc`) | ❌ 拒绝 —— 见下 | +| `kind = "shared"` on PE / MSVC(`*-windows-msvc`) | ✅ —— mcpp 生成 `.def`;见下 | | `kind = "shared"` on `*-musl` | ❌ musl target 是静态链接的 | | 发布预编译 BMI | ❌ 未尝试;BMI 与编译器构建逐位绑定 | | 把依赖打包进去 | ❌ 改为声明依赖(见上) | | 用**原生 `cl.exe`** 消费这种包 | ❌ 见下 | -### MSVC 拒绝 `kind = "shared"` 的原因 +### MSVC ABI 上的符号导出 -**不是链接器的问题** —— `link /DLL /IMPLIB:` 本来就能用。是**符号导出**: -MSVC 在没有 `__declspec(dllexport)`、也没有 `.def` 列出符号时,DLL **什么都不导出**, -于是导入库是空的,每个消费者都会拿到一堆 unresolved externals,而那些符号 -明明就在对象文件里 —— 报错点离病因很远。mcpp 选择拒绝,而不是产出这种诊断: +MSVC 在源码没有 `__declspec(dllexport)`、也没有 `.def` 列出符号时,DLL 什么都不导出。 +两者皆无时导入库为空,每个消费者都会拿到一堆 unresolved externals,而那些符号 +明明就在对象文件里。MinGW 的链接器会自动导出,把这个问题整个遮住;lld-link 的 +MSVC 形态刻意不这么做,因为 PE 的导出上限是 65535。 -``` -target 'mathkit': kind = "shared" is not supported for the MSVC ABI (x86_64-windows-msvc). - ... - Use kind = "lib" for this target, or build it for *-windows-gnu (MinGW), - where the linker auto-exports. -``` +mcpp 从对象生成 `.def` —— 这正是 CMake 的 `WINDOWS_EXPORT_ALL_SYMBOLS` 自 3.4 起 +在做的事。它是一个构建图节点,输入就是链接所消费的那批对象,因此导出面不会与 +「实际编译了什么」发生漂移;而且它**直接读 COFF**,不调 `dumpbin` —— 那个工具在 +Visual Studio 开发者环境里才有,而 mcpp 在 Windows 上的默认工具链是 clang。 + +**有两条限制是任何工具都消不掉的**,与 CMake 为同一机制记录的是同两条: + +| | | +|---|---| +| 导出的**数据** | 消费者的声明仍需 `__declspec(dllimport)`;否则链接器读到的是调用桩而不是值 | +| **vtable** 被引用的类 | 整个类都要标注,例如带虚函数的类的委托构造函数 | + +两者都靠标注解决,而且**标注优先**:对象里若已带 `/EXPORT:` 指令(那正是 +`__declspec(dllexport)` 产生的),mcpp 就让开,不生成任何东西。在其之上再加一份 +列表会把同名符号导出两次(`LNK4197`),更糟的是把其余所有符号也一并导出 —— +用「全部」替换掉作者选定的公开面。这件事没有任何开关:对象自己说了算。 -MinGW 的链接器会自动导出,这就是 `*-windows-gnu` 支持而 `*-windows-msvc` 不支持的 -全部原因。要补齐它需要生成 `.def`(对对象做一次符号扫描)—— 那是一个构建图节点, -不是一个 flag。 +可导出符号超过 65535 时,mcpp 拒绝而不是截断。被截断的导出表能干净地链接完成, +随后在「恰好需要那个掉出去的符号」的消费者那里失败。 ### 包里的链接 flag 是 GNU 拼写 @@ -273,8 +282,23 @@ ldflags = ["-Llib/x86_64-windows-msvc", "-lmathkit"] ``` mcpp 用到的每个 driver 都吃这一套 —— 包括 Windows 上默认的、面向 MSVC ABI 的 -clang。**原生 `cl.exe` 不吃**:它不认 `-L`。所以固定了 -`[toolchain] windows = "msvc@system"` 的消费者目前链不上打包库。 +clang。**原生 `cl.exe` 不吃**:它不认 `-L`。所以包里会把同一句话再写一遍, +这一遍不带方言: + +```toml +[target.'cfg(all(arch = "x86_64", os = "windows", env = "msvc"))'.runtime] +link_library_dirs = ["lib/x86_64-windows-msvc"] +libraries = ["mathkit"] +``` + +mcpp 会按 target 把它们渲染成 `/LIBPATH:` + `.lib` 或 `-L` + `-l`, +于是 `cl.exe` 的消费者也能链上。这两个键不是新词表 —— `[runtime]` 顶层一直就有, +这里只是让它们可以按 target 给。 + +**两种拼写都会写出来,而新版 mcpp 读到中立形式时会忽略同一条腿的 `ldflags`, +而不是叠加。** 旧版 mcpp 只读 `ldflags` 并静默忽略 `runtime` 段,所以去掉 +`ldflags` 会让所有旧客户端一个链接 flag 都拿不到;而两者都应用又会把 `-L` 送回 +`cl` 的命令行 —— 那正是要避免的事。 **改成直接写文件路径也不行**(`lib//mathkit.lib` 才是每个 driver 都吃的 拼写):ninja 执行链接命令时 cwd 是**输出目录**,而只有 include 家族前缀 diff --git a/mcpp.toml b/mcpp.toml index 4ee2c54c..24fb84b8 100644 --- a/mcpp.toml +++ b/mcpp.toml @@ -1,6 +1,6 @@ [package] name = "mcpp" -version = "2026.8.18.1" +version = "2026.8.18.2" description = "Modern C++ build & package management tool" license = "Apache-2.0" authors = ["mcpp-community"] diff --git a/src/version.cppm b/src/version.cppm index f7d3c7c4..884646df 100644 --- a/src/version.cppm +++ b/src/version.cppm @@ -31,6 +31,6 @@ import std; export namespace mcpp { -inline constexpr std::string_view MCPP_VERSION = "2026.8.18.1"; +inline constexpr std::string_view MCPP_VERSION = "2026.8.18.2"; } // namespace mcpp From f9ca3984a6089a32f831cd31f0ef805dbe5dd079 Mon Sep 17 00:00:00 2001 From: speak-agent <248744407+speak-agent@users.noreply.github.com> Date: Tue, 18 Aug 2026 06:41:12 +0800 Subject: [PATCH 5/9] fix(pack,prepare): the neutral link form only where it says the same thing MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit e2e 257 caught a regression the previous commit introduced, and the reason is worth keeping: a leg's link line is not always just a library reference. A PE/MinGW shared leg links with `-L… -Wl,-Bdynamic -lmathkit`, and `-Wl,-Bdynamic` only works IMMEDIATELY BEFORE the `-l` it enables — mcpp gives PE executables `-static`, which otherwise leaves ld in static-only mode where it refuses an import library and says `have you installed the static version of the mathkit library?`, naming neither the DLL nor `-static`. The first attempt cleared the leg's ldflags wholesale and lost the flag. The second kept it but rendered the library reference through the neutral channel, which puts it in a different slot on the command line — so the flag and its argument were separated and the same failure came back. Both are the same mistake: treating a hand-tuned link line as if it were a two-field record. So the neutral form is emitted only for legs where it is EQUIVALENT, and the PE/MinGW shared leg keeps the spelling that works. It costs nothing — a PE/GNU leg is not an MSVC-ABI leg, and cl.exe, the reason the neutral form exists, never reads it. On the consuming side only the library references are replaced, never the whole list. --- docs/12-binary-distribution.md | 18 +++++++++++++----- docs/zh/12-binary-distribution.md | 8 +++++++- src/build/prepare.cppm | 16 +++++++++++++++- src/pack/manifest_emit.cppm | 31 ++++++++++++++++++++++--------- 4 files changed, 57 insertions(+), 16 deletions(-) diff --git a/docs/12-binary-distribution.md b/docs/12-binary-distribution.md index 143be95a..6b349d60 100644 --- a/docs/12-binary-distribution.md +++ b/docs/12-binary-distribution.md @@ -329,11 +329,19 @@ target, so a `cl.exe` consumer links the package. They are not new vocabulary `[runtime]` has had both keys at the top level all along; this makes them per-target. -**Both spellings are emitted, and a newer mcpp ignores the `ldflags` rather than -adding to them.** An older mcpp reads only the `ldflags` and silently ignores the -`runtime` block, so dropping the `ldflags` would leave every older client with no -link line at all; adding both would put `-L` back on the `cl` command line, which -is the thing being avoided. +**Both spellings are emitted, and a newer mcpp drops the leg's library +references rather than adding to them.** An older mcpp reads only the `ldflags` +and silently ignores the `runtime` block, so dropping the `ldflags` would leave +every older client with no link line at all; adding both would put `-L` back on +the `cl` command line, which is the thing being avoided. + +One leg is deliberately left out of this: a **PE/MinGW shared** leg links with +`-L… -Wl,-Bdynamic -lmathkit`, and `-Wl,-Bdynamic` only works immediately before +the `-l` it enables — mcpp gives PE executables `-static`, which otherwise leaves +the linker in static-only mode where it refuses an import library. The neutral +form cannot say "switch link mode first", so that leg keeps the spelling that +works. It costs nothing: a PE/MinGW leg is not an MSVC-ABI leg, and `cl.exe` +never reads it. Naming the file by path instead (`lib//mathkit.lib`) is the spelling every driver takes, and it does not work either: ninja runs link commands with diff --git a/docs/zh/12-binary-distribution.md b/docs/zh/12-binary-distribution.md index 769ed071..a7692ffb 100644 --- a/docs/zh/12-binary-distribution.md +++ b/docs/zh/12-binary-distribution.md @@ -295,11 +295,17 @@ mcpp 会按 target 把它们渲染成 `/LIBPATH:` + `.lib` 或 `-L` + `-l< 于是 `cl.exe` 的消费者也能链上。这两个键不是新词表 —— `[runtime]` 顶层一直就有, 这里只是让它们可以按 target 给。 -**两种拼写都会写出来,而新版 mcpp 读到中立形式时会忽略同一条腿的 `ldflags`, +**两种拼写都会写出来,而新版 mcpp 读到中立形式时会丢掉同一条腿的库引用, 而不是叠加。** 旧版 mcpp 只读 `ldflags` 并静默忽略 `runtime` 段,所以去掉 `ldflags` 会让所有旧客户端一个链接 flag 都拿不到;而两者都应用又会把 `-L` 送回 `cl` 的命令行 —— 那正是要避免的事。 +有一条腿被刻意排除在外:**PE/MinGW 的动态库腿**链接行是 +`-L… -Wl,-Bdynamic -lmathkit`,而 `-Wl,-Bdynamic` 只有**紧邻它所启用的那个 `-l`** +时才有效 —— mcpp 给 PE 可执行文件加 `-static`,否则链接器停在纯静态模式并拒绝 +导入库。中立形式没法表达「先切换链接模式」,所以那条腿保留能用的拼写。 +这不付出任何代价:PE/MinGW 的腿不是 MSVC ABI 的腿,`cl.exe` 永远读不到它。 + **改成直接写文件路径也不行**(`lib//mathkit.lib` 才是每个 driver 都吃的 拼写):ninja 执行链接命令时 cwd 是**输出目录**,而只有 include 家族前缀 (`-I`、`-L` …)会被 `normalize_include_flags` 相对包根绝对化 —— 没有前缀的 token diff --git a/src/build/prepare.cppm b/src/build/prepare.cppm index 3df2d811..08d2939b 100644 --- a/src/build/prepare.cppm +++ b/src/build/prepare.cppm @@ -232,8 +232,22 @@ void merge_conditional_config(mcpp::manifest::Manifest& m, // a broader unconditional one under GNU last-wins — which is what // makes an off-OS REMOVAL expressible (`-U` after the base `-D`). if (neutralWins) { + // ⚠️ Drop the LIBRARY REFERENCES, not the whole ldflags list. + // + // Clearing it outright was a measured regression: a PE/MinGW shared + // leg's ldflags also carry `-Wl,-Bdynamic`, without which `-static` + // leaves ld in static-only mode and it refuses the import library + // with `have you installed the static version of the mathkit + // library?`. e2e 257 caught it. + // + // The neutral form replaces exactly what it can express — a library + // and where to find it. Anything else in that block says something + // it cannot say, and must survive. auto inputs = cc.inputs; - inputs.ldflags.clear(); + std::erase_if(inputs.ldflags, [](std::string_view f) { + return f.starts_with("-L") || f.starts_with("-l") + || f.starts_with("/LIBPATH:"); + }); mcpp::manifest::append(m.buildConfig, inputs); } else { mcpp::manifest::append(m.buildConfig, cc.inputs); diff --git a/src/pack/manifest_emit.cppm b/src/pack/manifest_emit.cppm index c39b94ef..622462c3 100644 --- a/src/pack/manifest_emit.cppm +++ b/src/pack/manifest_emit.cppm @@ -243,19 +243,32 @@ std::string emit_package_manifest(const PackageDoc& doc) { peGnuShared ? "\"-Wl,-Bdynamic\", " : "", leg.linkName); - // …and the same statement without a dialect. mcpp renders these as - // `/LIBPATH:` + `.lib` or `-L` + `-l` from the target, which is - // what lets a consumer driven by native `cl.exe` link this package at - // all — cl rejects `-L`. + // …and, where it says the SAME thing, the dialect-neutral form. mcpp + // renders these as `/LIBPATH:` + `.lib` or `-L` + `-l` from the + // target, which is what lets a consumer driven by native `cl.exe` link + // this package at all — cl rejects `-L`. // // BOTH are emitted, deliberately. An older mcpp reads only the ldflags // above and silently ignores this block (measured), so dropping the // ldflags would leave every older client with no link line at all. A - // newer mcpp seeing this block ignores that leg's ldflags rather than - // adding to them — see merge_conditional_config. - o += std::format("[target.'{}'.runtime]\n", cfg_predicate_for(leg.triple)); - o += std::format("link_library_dirs = [\"lib/{}\"]\n", leg.triple); - o += std::format("libraries = [\"{}\"]\n\n", leg.linkName); + // newer mcpp seeing this block drops that leg's library references and + // uses these instead — see merge_conditional_config. + // + // ⚠️ NOT FOR A PE/MinGW SHARED LEG, and this was measured rather than + // reasoned. That leg's line is `-L… -Wl,-Bdynamic -lmathkit`, and + // `-Wl,-Bdynamic` only works IMMEDIATELY BEFORE the `-l` it enables: + // mcpp gives PE executables `-static`, which leaves ld in static-only + // mode where it refuses an import library. The neutral form has no way + // to say "and switch link mode first", and rendering the two halves + // through different slots separates the flag from its argument — e2e + // 257 fails with `have you installed the static version of the mathkit + // library?`. So that one leg keeps the spelling that works, and cl.exe + // never sees it: a PE/GNU leg is not an MSVC-ABI leg. + if (!peGnuShared) { + o += std::format("[target.'{}'.runtime]\n", cfg_predicate_for(leg.triple)); + o += std::format("link_library_dirs = [\"lib/{}\"]\n", leg.triple); + o += std::format("libraries = [\"{}\"]\n\n", leg.linkName); + } } // A shared library has to be FOUND at run time as well as linked, and the // two are different search paths — `link_library_dirs` is not rpath. From bbf785efa79d35c4e085249e55ef46f259dfb77b Mon Sep 17 00:00:00 2001 From: speak-agent <248744407+speak-agent@users.noreply.github.com> Date: Tue, 18 Aug 2026 07:01:51 +0800 Subject: [PATCH 6/9] review(coff): name /bigobj, and drop a constant the DATA rule stopped using MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Two things a read-through caught. `/bigobj` objects are a different container — machine 0 and `0xFFFF` where the section count would be — and the reader fell through to "unsupported COFF machine 0x0000". True and useless: it blames the reader for a flag the project passed. Named now, with the two ways out (drop /bigobj, or annotate the surface and need no generated .def at all). `kScnMemWrite` was left over from a first version whose DATA rule keyed on writability. It does not: a `const` in `.rdata` is a variable and consumers read it as one, so the rule keys on "not executable" instead. The constant was documenting a decision that is no longer taken. --- src/build/coff_exports.cppm | 18 +++++++++++++++++- tests/unit/test_coff_exports.cpp | 15 +++++++++++++++ 2 files changed, 32 insertions(+), 1 deletion(-) diff --git a/src/build/coff_exports.cppm b/src/build/coff_exports.cppm index 22b606e2..43d1aea8 100644 --- a/src/build/coff_exports.cppm +++ b/src/build/coff_exports.cppm @@ -115,7 +115,6 @@ constexpr std::uint16_t kMachineArm64 = 0xaa64; constexpr std::uint8_t kSymClassExternal = 2; // IMAGE_SYM_CLASS_EXTERNAL constexpr std::uint16_t kSymTypeFunction = 0x20; // DTYPE_FUNCTION << 4 -constexpr std::uint32_t kScnMemWrite = 0x80000000u; // IMAGE_SCN_MEM_WRITE constexpr std::uint32_t kScnMemExecute = 0x20000000u; // IMAGE_SCN_MEM_EXECUTE std::uint16_t rd16(std::span b, std::size_t off) { @@ -201,6 +200,23 @@ read_exports(std::span bytes) return std::unexpected("not a COFF object: shorter than a file header"); const auto machine = rd16(bytes, 0); + + // `/bigobj` objects are a DIFFERENT container: machine 0 and a `0xFFFF` + // where the section count would be, followed by a class GUID and a much + // larger header. Named here rather than left to fall out as "unsupported + // machine 0x0000", which is true and useless — the reader would be blamed + // for a flag the project passed. + if (machine == 0 && bytes.size() >= 4 && rd16(bytes, 2) == 0xFFFF) { + return std::unexpected( + "this is a /bigobj object, whose header layout differs from an " + "ordinary COFF one.\n" + " mcpp's export reader does not parse it. Build the shared library " + "without /bigobj,\n" + " or mark its public surface with __declspec(dllexport) — an " + "annotated library needs\n" + " no generated .def at all."); + } + if (!is_supported_machine(machine)) { return std::unexpected(std::format( "unsupported COFF machine 0x{:04x}. mcpp reads i386, amd64, arm, " diff --git a/tests/unit/test_coff_exports.cpp b/tests/unit/test_coff_exports.cpp index ed94aba5..7694b69a 100644 --- a/tests/unit/test_coff_exports.cpp +++ b/tests/unit/test_coff_exports.cpp @@ -214,6 +214,21 @@ TEST(CoffExports, RefusesATruncatedSymbolTable) { EXPECT_NE(r.error().find("past the end"), std::string::npos); } +TEST(CoffExports, NamesBigobjRatherThanCallingItAnUnknownMachine) { + // A `/bigobj` object is a different container: machine 0 and 0xFFFF where + // the section count would be. Falling through to "unsupported machine + // 0x0000" is true and useless — it blames the reader for a flag the project + // passed. + auto obj = make_obj({ Sym{ .name = "f" } }); + obj[0] = std::byte{0}; obj[1] = std::byte{0}; + obj[2] = std::byte{0xFF}; obj[3] = std::byte{0xFF}; + auto r = read_exports(obj); + ASSERT_FALSE(r); + EXPECT_NE(r.error().find("bigobj"), std::string::npos); + // And it says what to do, or it is a dead end. + EXPECT_NE(r.error().find("dllexport"), std::string::npos); +} + TEST(CoffExports, RefusesSomethingThatIsNotAnObject) { std::vector tiny(4, std::byte{0}); EXPECT_FALSE(read_exports(tiny)); From c8ac60a0ecfbcb9c2926fa0b4da0316b7695e49b Mon Sep 17 00:00:00 2001 From: speak-agent <248744407+speak-agent@users.noreply.github.com> Date: Tue, 18 Aug 2026 07:24:56 +0800 Subject: [PATCH 7/9] fix(pack): print the compiler's output when the packer's build fails MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Windows CI reported error: build failed FAIL: shared pack failed off ELF and nothing else, because `mcpp pack` printed `BuildError::message` and dropped `BuildError::diagnosticOutput` — the field that carries what the compiler actually said. `mcpp build` has printed it all along; the two pack pipelines were the only callers that did not. That is a defect on its own terms, not just an inconvenience for this investigation: a packaging failure is exactly when a maintainer has least context, and three words is not a report. Both pipelines now print it. --- src/pack/library_pipeline.cppm | 15 +++++++++++++++ src/pack/pipeline.cppm | 6 ++++++ 2 files changed, 21 insertions(+) diff --git a/src/pack/library_pipeline.cppm b/src/pack/library_pipeline.cppm index 1401a92c..844adfef 100644 --- a/src/pack/library_pipeline.cppm +++ b/src/pack/library_pipeline.cppm @@ -14,6 +14,13 @@ // // Design: .agents/docs/2026-08-17-library-distribution-design.md §2. +module; +// `stderr` / `fputs`: the packer prints the compiler's own diagnostics, and a +// build failure that arrives as three words is a build failure nobody can act +// on. This global module fragment was removed once as an unused ``; +// it has a use now. +#include + export module mcpp.pack.library_pipeline; import std; @@ -158,6 +165,14 @@ export int build_and_pack_library(const std::string& targetName, auto be = mcpp::build::make_ninja_backend(); mcpp::build::BuildOptions bo; if (auto br = be->build(ctx->plan, bo); !br) { + // The compiler's own output, not just "build failed". `mcpp build` + // has always printed this; `mcpp pack` dropped it, so a failure + // inside the packer's build arrived as three words and CI logs had + // nothing to go on. + if (!br.error().diagnosticOutput.empty()) { + std::fputs(br.error().diagnosticOutput.c_str(), stderr); + if (br.error().diagnosticOutput.back() != '\n') std::fputs("\n", stderr); + } mcpp::ui::error(br.error().message); return 1; } diff --git a/src/pack/pipeline.cppm b/src/pack/pipeline.cppm index 2f8ae739..ab21b1da 100644 --- a/src/pack/pipeline.cppm +++ b/src/pack/pipeline.cppm @@ -86,6 +86,12 @@ export int build_and_pack(Options opts, bool modeFromUser, mcpp::build::BuildOptions bo; auto br = be->build(ctx->plan, bo); if (!br) { + // The compiler's own output, not just "build failed" — same reason as + // in the library pipeline. + if (!br.error().diagnosticOutput.empty()) { + std::fputs(br.error().diagnosticOutput.c_str(), stderr); + if (br.error().diagnosticOutput.back() != '\n') std::fputs("\n", stderr); + } mcpp::ui::error(br.error().message); return 1; } From 9bdb2842049eaf02a53792d63606588cedac3b9f Mon Sep 17 00:00:00 2001 From: speak-agent <248744407+speak-agent@users.noreply.github.com> Date: Tue, 18 Aug 2026 07:46:27 +0800 Subject: [PATCH 8/9] fix(flags): -fPIC is a property of the target format, not of the dialect MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Windows CI, once the packer stopped swallowing the compiler's output: error: unsupported option '-fPIC' for target 'x86_64-pc-windows-msvc' every MSVC-ABI shared build died in clang-scan-deps before compiling anything. The comment beside the condition already said the right thing — "PE code is position independent by design" — and the condition tested the DIALECT: `!isMsvcDialect`. Windows' default toolchain is clang, which speaks the GNU dialect while targeting the MSVC ABI, so the flag went out anyway. Same shape as the shared-library guard this PR replaced: asking which COMPILER when the question is which TARGET. It keys on the target format now. Unreachable until this PR, because `kind = "shared"` was refused on that ABI — the refusal was hiding an untested path, which is what refusals do. 257 pins the absence on every Linux CI pass through mingw-cross rather than only on Windows: the flag is equally meaningless for a PE target whichever compiler emits it, and GCC merely ignores it where clang refuses. --- src/build/flags.cppm | 21 +++++++++++++++++++-- tests/e2e/257_shared_library_pe.sh | 15 +++++++++++++++ 2 files changed, 34 insertions(+), 2 deletions(-) diff --git a/src/build/flags.cppm b/src/build/flags.cppm index 2058e12d..8c35f2be 100644 --- a/src/build/flags.cppm +++ b/src/build/flags.cppm @@ -21,6 +21,7 @@ import mcpp.platform.runtime_search; import mcpp.toolchain.clang; import mcpp.toolchain.detect; import mcpp.toolchain.dialect; +import mcpp.toolchain.triple; import mcpp.toolchain.hostflags; import mcpp.toolchain.linkmodel; import mcpp.toolchain.model; @@ -411,7 +412,23 @@ CompileFlags compute_flags(const BuildPlan& plan) { const bool isMsvcDialect = (d.id == "msvc"); - // PIC? (GNU-only concept; PE code is position independent by design.) + // PIC is a GNU concept and a property of the TARGET FORMAT: PE code is + // position independent by design (base relocations), and clang rejects the + // flag outright — `unsupported option '-fPIC' for target + // 'x86_64-pc-windows-msvc'`. + // + // ⚠️ The condition used to be `!isMsvcDialect`, i.e. the DIALECT. Windows' + // default toolchain is clang, which speaks the GNU dialect while targeting + // the MSVC ABI, so `-fPIC` was emitted and every MSVC-ABI shared build died + // in clang-scan-deps before compiling anything. It was unreachable while + // `kind = "shared"` was refused on that ABI; allowing it is what surfaced + // this. Same shape as the shared-library guard itself: asking which + // COMPILER when the question is which TARGET. + const bool peTarget = [&] { + if (auto t = mcpp::toolchain::triple::parse(plan.toolchain.targetTriple)) + return t->is_pe(); + return bool(mcpp::platform::is_windows); + }(); bool need_pic = false; for (auto& lu : plan.linkUnits) { if (lu.kind == LinkUnit::SharedLibrary) { @@ -419,7 +436,7 @@ CompileFlags compute_flags(const BuildPlan& plan) { break; } } - std::string pic_flag = (need_pic && !isMsvcDialect) ? " -fPIC" : ""; + std::string pic_flag = (need_pic && !isMsvcDialect && !peTarget) ? " -fPIC" : ""; // Include dirs — this is the TYPED PATH channel (bare paths from the // manifest; the dialect prefix is applied here at emission), not the diff --git a/tests/e2e/257_shared_library_pe.sh b/tests/e2e/257_shared_library_pe.sh index 83bafe62..925ef6a1 100755 --- a/tests/e2e/257_shared_library_pe.sh +++ b/tests/e2e/257_shared_library_pe.sh @@ -66,6 +66,21 @@ imp="$(find target -name 'libmathkit.dll.a' | head -1)" echo "FAIL: no import library. The .dll alone is a library only mingw's ld" echo " will link, so the package would be unusable everywhere else." exit 1; } +# ⚠️ And no `-fPIC`. PE code is position independent by design, and clang +# targeting the MSVC ABI REJECTS the flag — `unsupported option '-fPIC' for +# target 'x86_64-pc-windows-msvc'` — killing the build in clang-scan-deps before +# anything compiles. The condition used to be the DIALECT rather than the target, +# and Windows' default toolchain is clang, which speaks the GNU dialect while +# targeting MSVC. Asserted here rather than only on Windows because this runs on +# every Linux CI pass through mingw-cross, and the flag is equally meaningless +# for a PE target whichever compiler emits it. +nj_pic="$(find target -name build.ninja | head -1)" +grep -q '\-fPIC' "$nj_pic" && { + grep -n 'fPIC' "$nj_pic" | head -3 + echo "FAIL: -fPIC on a PE target. It means nothing here, and clang targeting" + echo " the MSVC ABI refuses it outright." + exit 1; } + # It is a declared output of the link edge, not a side effect ninja knows nothing # about — otherwise the consumer that links it has no producer and ninja stops # with 'no known rule to make it'. From 9e0ddf751e27b8de93809a9cd76e1559eb025d78 Mon Sep 17 00:00:00 2001 From: speak-agent <248744407+speak-agent@users.noreply.github.com> Date: Tue, 18 Aug 2026 08:09:35 +0800 Subject: [PATCH 9/9] fix(build): the import-library and .def spellings follow the target ABI MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Windows CI again, one layer further in: lld-link: warning: ignoring unknown argument '--out-implib' lld-link: error: could not open 'bin/mathkit-shared.lib': no such file Clang targeting the MSVC ABI speaks the GNU DIALECT while driving lld-link, so a dialect-keyed spelling handed the MSVC linker a MinGW flag. The `.def` had the same problem from the other side: `/DEF:` was added only to the MSVC-dialect rule, so a clang-driven MSVC-ABI link generated the file and never passed it. Three flags in this PR made the identical mistake — `-fPIC`, the import library, and `/DEF:` — and it is always the same one: asking which COMPILER when the question is which TARGET. So the spelling leaves the dialect table entirely and becomes `pe_link_flag`, which reads the target triple and wraps in `-Wl,` unless the linker is invoked directly. The dialect table now says, where the entry used to be, why it cannot answer this. Verified in both directions locally: a MinGW target still gets `-Wl,--out-implib,` and no def edge; ELF and Mach-O are untouched. --- src/build/ninja_backend.cppm | 54 ++++++++++++++++++++++++++---------- src/toolchain/dialect.cppm | 23 +++++---------- 2 files changed, 47 insertions(+), 30 deletions(-) diff --git a/src/build/ninja_backend.cppm b/src/build/ninja_backend.cppm index 335592fb..93f6adef 100644 --- a/src/build/ninja_backend.cppm +++ b/src/build/ninja_backend.cppm @@ -207,6 +207,31 @@ std::string join_flags(const std::vector& flags) { // default install name is the path it was LINKED at, so a package built in // /tmp/build-xyz records /tmp/build-xyz and cannot be relocated — which is // every distributed dylib. `@rpath/` is the only default that travels. +// A PE link flag, spelled for the TARGET ABI and wrapped for the driver. +// +// ⚠️ NOT a dialect-table entry, and Windows CI is why. Clang targeting the MSVC +// ABI speaks the GNU DIALECT while driving lld-link, so a dialect-keyed spelling +// handed it `-Wl,--out-implib,` and lld-link answered `warning: ignoring unknown +// argument '--out-implib'` followed by `could not open '…lib': no such file`. +// Three flags in this PR made the same mistake — `-fPIC`, the import library, +// and `/DEF:` — and the mistake is always the same one: asking which COMPILER +// when the question is which TARGET. +// +// `sep` is `LinkStyle::SeparateLinker`, i.e. link.exe invoked directly, where +// the flag needs no `-Wl,` wrapper. Everything else goes through a compiler +// driver. +std::string pe_link_flag(const BuildPlan& plan, bool sep, + std::string_view msvcForm, std::string_view gnuForm, + std::string_view path) +{ + const auto t = mcpp::toolchain::triple::parse(plan.toolchain.targetTriple); + const bool msvcAbi = t ? t->is_msvc_env() + : mcpp::toolchain::is_msvc_target(plan.toolchain); + if (!msvcAbi) return std::string(gnuForm) + std::string(path); + auto flag = std::string(msvcForm) + std::string(path); + return sep ? flag : "-Wl," + flag; +} + std::string shared_soname_flag(const LinkUnit& lu, const BuildPlan& plan) { if (lu.kind != LinkUnit::SharedLibrary) return ""; const auto t = mcpp::toolchain::triple::parse(plan.toolchain.targetTriple); @@ -1058,7 +1083,7 @@ std::string emit_ninja_string(const BuildPlan& plan) { link_rule("cxx_archive", std::string(dial.archiveCmd), "AR"); link_rule("cxx_shared", "$cxx -shared $in -o $out $ldflags $soname_flag " - "$implib_flag $unit_ldflags", + "$implib_flag $def_flag $unit_ldflags", "SHARED"); // mcpp#426: a link unit with no C++ translation unit in it is // linked by the C driver. `g++` appends `-lstdc++` unconditionally, @@ -1071,7 +1096,7 @@ std::string emit_ninja_string(const BuildPlan& plan) { "$cc $in -o $out $c_ldflags $unit_ldflags", "LINK"); link_rule("c_shared", "$cc -shared $in -o $out $c_ldflags $soname_flag " - "$implib_flag $unit_ldflags", + "$implib_flag $def_flag $unit_ldflags", "SHARED"); } } @@ -1853,20 +1878,21 @@ std::string emit_ninja_string(const BuildPlan& plan) { // agrees with — the name belongs to plan.cppm's import_library_for, and // this is how it gets to the command. The SPELLING belongs to the // dialect table, same as `archiveRemoveArg`. + const bool sepLinker = + dial.linkStyle == mcpp::toolchain::CommandDialect::LinkStyle::SeparateLinker; if (!lu.importLibrary.empty()) { - std::string arg{ dial.sharedImportLibArg }; - // `{}` or nothing: a row without the placeholder cannot say WHERE to - // write, so emitting its bare text would hand the linker a flag with - // no argument. Skipping is the honest reading of an empty row, and - // the implicit output above then fails loudly as a missing file - // rather than quietly linking against a stale one. - if (auto at = arg.find("{}"); at != std::string::npos) { - arg.replace(at, 2, escape_ninja_path(lu.importLibrary)); - out_line += " implib_flag = " + arg + "\n"; - } + out_line += " implib_flag = " + pe_link_flag( + plan, sepLinker, "/IMPLIB:", "-Wl,--out-implib,", + escape_ninja_path(lu.importLibrary)) + "\n"; + } + if (!lu.defFile.empty()) { + // `/DEF:` on both sides: it is a link.exe/lld-link flag, and the + // only targets that reach here are MSVC-ABI ones (MinGW auto-exports + // and gets no def edge at all). + out_line += " def_flag = " + pe_link_flag( + plan, sepLinker, "/DEF:", "/DEF:", + escape_ninja_path(lu.defFile)) + "\n"; } - if (!lu.defFile.empty()) - out_line += " def_flag = /DEF:" + escape_ninja_path(lu.defFile) + "\n"; { // Per-unit C++ runtime link, by ROLE. The kind→role map is the // only place that knows a TestBinary runs on the build machine diff --git a/src/toolchain/dialect.cppm b/src/toolchain/dialect.cppm index 98d0b210..e7b5167b 100644 --- a/src/toolchain/dialect.cppm +++ b/src/toolchain/dialect.cppm @@ -107,15 +107,14 @@ struct CommandDialect { std::string_view archiveRemoveArg; // "d" needs no {} | "/REMOVE:{}" bool archiveRemoveTakesArchiveFirst = true; - // How the linker is told where to write a shared library's IMPORT LIBRARY — - // the archive of stubs a PE consumer links against, as opposed to the `.dll` - // the loader opens. `{}` is the path. + // ⚠️ THE IMPORT-LIBRARY AND `.def` SPELLINGS ARE NOT HERE, on purpose. // - // Emitted only when the TARGET has import libraries at all (PE); on ELF and - // Mach-O the shared library is its own link input and there is nothing to - // write. So this being non-empty in both rows is not a contradiction: the - // rows describe how to SAY it, and the target decides whether to. - std::string_view sharedImportLibArg; // "-Wl,--out-implib,{}" | "/IMPLIB:{}" + // They were, keyed on the dialect, and that is wrong in a way Windows CI + // demonstrated: clang targeting the MSVC ABI speaks the GNU DIALECT while + // driving LLD-LINK, so it was handed `-Wl,--out-implib,` and answered + // `lld-link: warning: ignoring unknown argument '--out-implib'`. The + // spelling follows the target ABI, which this table does not know — see + // ninja_backend's pe_link_flag. }; // Dialect lookup. GCC / Clang / MinGW → gnu; MSVC → msvc. @@ -219,10 +218,6 @@ constexpr CommandDialect kGnuDialect{ // `ar d ...` — one verb, then every member. .archiveRemoveArg = "d", .archiveRemoveTakesArchiveFirst = true, - // ld/lld: `--out-implib` is what makes a PE shared library linkable at all. - // Without it mingw writes only the .dll, consumers link the .dll directly, - // and that works — until the same package is consumed by any other linker. - .sharedImportLibArg = "-Wl,--out-implib,{}", }; // Native cl.exe. Unreachable in builds until the MSVC backend lands @@ -258,10 +253,6 @@ constexpr CommandDialect kMsvcDialect{ // reported with the command that produced it rather than swallowed. .archiveRemoveArg = "/REMOVE:{}", .archiveRemoveTakesArchiveFirst = false, - // link.exe writes one whether asked or not; naming it explicitly is how the - // path stays the one plan.cppm chose, instead of the linker's `$out`-derived - // guess (`foo.dll.lib`) that nothing else in mcpp agrees with. - .sharedImportLibArg = "/IMPLIB:{}", }; } // namespace