diff --git a/.agents/docs/2026-08-18-windows-shared-library-and-module-extensions.md b/.agents/docs/2026-08-18-windows-shared-library-and-module-extensions.md new file mode 100644 index 00000000..08634bfb --- /dev/null +++ b/.agents/docs/2026-08-18-windows-shared-library-and-module-extensions.md @@ -0,0 +1,374 @@ +# Windows 动态库分发、cl.exe 消费,与 `.ixx` 的默认支持 + +> 2026-08-18 · 针对 2026.8.18.1 发布后遗留的 P1 / P2 / `.ixx` 三问 +> 状态:**分析与设计,未实施** + +三个问题看起来独立,实际共用一条主线:**mcpp 在这三处都把「工具链没有替我们 +做的事」当成了「做不了的事」**,而工业界对每一件都有已经跑了十年以上的做法。 + +| # | 问题 | 一句话结论 | +|---|---|---| +| **Q1** | `kind = "shared"` 在 `*-windows-msvc` 上被拒 | 拒绝的理由(符号导出)成立,但**结论不成立** —— CMake 十年前就用自动生成 `.def` 解决了,且不依赖 dumpbin | +| **Q2** | 打包库不能被原生 `cl.exe` 消费 | 是 mcpp 自己的 manifest 用了 GNU 拼写;方言中立通道已经存在,只是不在条件通道里 | +| **Q3** | `.ixx` 默认是否支持 | **不支持,而且失败方式很糟**:实测编译出对象、然后从链接里消失,报 `undefined reference` | + +--- + +## 1. Q3 先说,因为它是实测出来的,而且最脏 + +### 1.1 实测 + +```toml +[build] +sources = ["src/*.ixx", "src/*.cpp"] # 没有 module_extensions +``` + +``` +$ mcpp run + Compiling ixxprobe v0.1.0 (.) +error: build failed +failed: bin/ixxprobe +ld: obj/main.o: in function `main': + undefined reference to `mk::answer@mathkit()' +``` + +加上 `module_extensions = [".ixx"]` 之后同一份代码 `ixx-ok=42`,通过。 + +### 1.2 根因:两个答题者对「这是不是模块接口」给了不同答案 + +看未声明时生成的 `build.ninja`: + +```ninja +build obj/mathkit.ixx.ddi : cxx_scan .../src/mathkit.ixx +build obj/mathkit.ixx.o | gcm.cache/mathkit.gcm : cxx_object .../src/mathkit.ixx | ... + bmi_out = gcm.cache/mathkit.gcm ← BMI 产出了 + +build bin/ixxprobe : cxx_link obj/main.o ← obj/mathkit.ixx.o 不在里面 +``` + +- **扫描器**读到 `export module mathkit;`,记下 `provides = mathkit`,于是规划器给这条边挂了 `bmi_out`; +- **分类器** `classify()` 对不在表里的扩展名返回 `SourceKind::Other`(`source_kind.cppm:322`); +- 而链接集合只看 kind:`links_unconditionally(Other) == false`,`is_implementation_source(Other) == false`; +- 于是**对象被构建出来,然后没有任何人链接它**。 + +这是本仓库反复出现的那个形状:**同一个决定有两处推导,而下游只读其中一处**。 + +### 1.3 `.ixx` 在三个编译器上本来就能用 + +mcpp **从不依赖驱动的扩展名启发式**:模块接口的语言由 +`bmi_traits(tc).moduleInterfaceLangFlag` 显式给出 —— MSVC `/interface /TP`、 +clang `-x c++-module`、gcc `-x c++`(`model.cppm:309/325/346`)。所以 +`.ixx` 一旦进了表,三个编译器都直接可用;这与「clang 的驱动不认识 `.ixx`」 +并不矛盾,因为 mcpp 根本没让驱动去猜。 + +> ⚠️ 我此前的笔记记成「Clang 不认 `.ixx`」并把它当成产品限制。那说的是**驱动默认**, +> 而 mcpp 覆盖了它 —— 判据要落在 mcpp 实际发出的命令行上,不是编译器的默认行为。 + +### 1.4 因此:**不是内置 `.ixx`,而是让「已声明」真的自动适配、「未声明」不再静默** + +⚠️ **这一节在 review 中被更正过一次。** 第一版结论是「把 `.ixx` 放进 +`builtin_extension_table()`」。那是错的方向:**扩展名集合是配置,不是 mcpp 要 +逐个追加的内置清单** —— 今天补 `.ixx`,明天补 `.ccm`、`.cxxm`,而每一次追加都是 +mcpp 替工程做了一个工程自己能说清楚的决定。 + +真正要成立的是两件事: + +1. **已声明的扩展名必须在三个工具链上都直接可用 —— 这一条今天已经成立。** + 模块接口的语言由 `BmiTraits::moduleInterfaceLangFlag` 显式给出 + (`/interface /TP` / `-x c++-module` / `-x c++`),mcpp 从不让驱动去猜扩展名。 + 所以 `module_extensions = [".ixx"]` 一写,gcc / clang / MSVC 全部可用 + ——**实测 `ixx-ok=42`**。 +2. **未声明的扩展名必须当场拒绝,而不是编译出一个没人链接的对象。** + 这才是缺陷本体,而且它与 `.ixx` 无关:任何 `classify()` 归为 `Other` 的扩展名 + 都会掉进同一个洞。 + +拒绝写在**扫描器**里(分类发生的那一处),消息点名文件、扩展名与该写的键: + +``` +error: scanner errors: + …/src/mathkit.ixx: 'mathkit.ixx' is listed in [build] sources, and mcpp has + no role for the extension '.ixx'. + Its object would be compiled and then linked by nothing, so this is refused + rather than built. If it is a module interface, declare the extension: + [build] + module_extensions = [".ixx"] + Otherwise remove it from `sources` — headers belong in `include_dirs`, and + Windows resource scripts in `[resources]`. +``` + +**为什么不猜。** 让 mcpp 看到未知扩展名里有 `export module` 就自动当模块接口, +是给「这是不是模块接口」加**第三个**答题者 —— 而本节的缺陷正是前两个答题者 +(扫描器的 `provides` 与分类器的 `kind`)不一致造成的。 + +--- + +## 2. Q1:MSVC 的 `kind = "shared"` + +### 2.1 拒绝的理由成立 + +MSVC 在没有 `__declspec(dllexport)`、也没有 `.def` 时,DLL 不导出任何东西 —— +导入库为空,消费者拿到一片 unresolved externals,而那些符号就在对象里。 +这一点在链接器层面也被证实:**lld-link 的 MSVC 形态不做 auto-export**, +而 MinGW 形态做;LLVM 侧的说法是 MSVC 路径要控制导出数量,以免撞上 +**PE 的 65535 导出上限**([lld-link auto-export 讨论][lld])。 + +所以 `*-windows-gnu` 能用而 `*-windows-msvc` 不能,差别确实只在「链接器替不替你导出」。 + +### 2.2 但「做不了」是错的 —— 工业界有两条成熟路径 + +| 路径 | 代表 | 机制 | 正确性 | 需要改源码 | +|---|---|---|---|---| +| **A. 导出宏** | MSVC 官方指导、Qt、Boost、CMake `generate_export_header` | 作者在声明上标 `__declspec(dllexport/dllimport)` | **完全正确**,含数据符号与 vtable | 是 | +| **B. 自动 `.def`** | **CMake `WINDOWS_EXPORT_ALL_SYMBOLS`**(2015 年,CMake 3.4) | 扫描 `.obj` 的 COFF 符号表,生成 `.def` 交给链接器 | 函数正确;**数据符号与 vtable 有明确限制** | 否 | +| C. MinGW auto-export | GNU ld / lld-mingw | 无显式导出时导出全部 | 同 B,且不适用于 MSVC ABI | 否 | +| D. 干脆用静态库 | 相当一部分项目在 modules 时代的选择 | —— | —— | 否 | + +**B 是关键发现,因为它推翻了「需要 dumpbin」这个隐含前提。** CMake 的实现 +`bindexplib` **自己解析 COFF**,不调用任何外部工具([bindexplib.cxx][bx]): +读 image header 的 machine 字段(I386/AMD64/ARM/ARMNT/ARM64),遍历符号表,按下列 +规则筛选: + +- 只取 `IMAGE_SYM_CLASS_EXTERNAL` 且 `SectionNumber > 0`、Type 为 `0x20` 或 `0x0` 的符号; +- **DATA 判定**:Type == 0 且所在节带 `IMAGE_SCN_MEM_WRITE` ⇒ 写成 `name DATA`; +- 跳过 `??_G` / `??_E`(析构变体)、含 `.` 的托管符号、`__t2m` / `$$F` / `$$J`、 + ARM64EC 的 `$ientry_thunk` 等 thunk 变体; +- i386 与 `__cdecl` 情形去掉前导下划线。 + +CMake 官方文档同时**明确写出它的边界**([WINDOWS_EXPORT_ALL_SYMBOLS][cm]): + +> Global **data** symbols must be explicitly marked with `__declspec(dllimport)` +> in order to link to data in the `.dll`. +> +> In cases that the compiler generates references to the virtual function table, +> such as in a delegating constructor of a class with virtual functions, the +> whole class must be marked with `__declspec(dllimport)`. + +**这两条限制不是可以隐瞒的实现细节 —— 它们决定了 B 是默认值而不是终点。** + +### 2.3 C++20 modules 与 DLL 的官方姿势,恰好就是 mcpp 的模型 + +微软在两个 Q&A 里给出的答案一致([Q&A 1665106][ms1]、[Q&A 1695780][ms2]): + +1. 模块接口里的实体**必须**带 `__declspec(dllexport)`: + ```cpp + export __declspec(dllexport) void myFunction(); + export class __declspec(dllexport) Test { public: Test(); }; + ``` +2. 消费者要么拿到 `.ifc`(用 `/reference` 指过去),**要么把 `.ixx` 加进自己的工程, + 由自己编译出 `.ifc`**。 + +第二条正是 **mcpp 的分发模型**:发接口**源码**,消费者自己编。所以 mcpp +不需要发 `.ifc` —— 而 `.ifc` 与编译器构建逐位绑定,发它等同于发 BMI, +mcpp 已明确不做(docs/12「当前边界」)。 + +> 换句话说:**mcpp 在 modules × DLL 这件事上的模型是微软自己推荐的那一条, +> 缺的只有导出。** + +--- + +## 3. Q2:为什么原生 `cl.exe` 消费不了打包库 + +### 3.1 现状 + +生成的 manifest 用 GNU 拼写选每条腿: + +```toml +[target.'cfg(all(arch = "x86_64", os = "windows", env = "msvc"))'.build] +ldflags = ["-Llib/x86_64-windows-msvc", "-lmathkit"] +``` + +clang(Windows 默认工具链)吃这一套;**原生 `cl.exe` 不认 `-L`**。 + +### 3.2 两条看起来能走、实测走不通的路 + +**(a) 直接写文件路径。** 每个 driver 都接受裸路径作为链接输入,看起来是方言中立解。 +实测失败: + +``` +ld: cannot find lib/x86_64-windows-gnu/libmathkit.a +``` + +ninja 执行链接命令时 cwd 是**输出目录**,而只有 include 家族前缀(`-I`、`-L` …) +会被 `normalize_include_flags` 相对包根绝对化;无前缀 token 没有挂靠点。 +而 manifest 里写绝对路径就不再可重定位。 + +**(b) 把链接意图挪到方言中立通道。** `[runtime] link_library_dirs` + `libraries` +本来就按 flavor 渲染(`flags.cppm` 的 `LinkIntentFlavor::PeMsvc` → `/LIBPATH:` + +`.lib`)。问题是它**只在顶层读**,而包需要**按腿**给 —— 两条腿的库同名时, +顶层单一搜索路径会让链接器挑到错的那条。 + +而把它做成条件段 `[target.'cfg(…)'.runtime]` 有一个实测过的陷阱: +**旧版 mcpp 读到它不报错,而是静默忽略**。于是把 flag 从 `ldflags` 挪过去, +会让所有旧客户端**一个链接 flag 都拿不到**。 + +### 3.3 因此:两种拼写都带,新客户端优先中立形式 + +这是唯一同时满足「老客户端仍可用」与「cl.exe 可用」的形状: + +```toml +# 老客户端读这个(GNU 拼写,clang/gcc 可用)——保持现状,不动 +[target.'cfg(...)'.build] +ldflags = ["-Llib/", "-lmathkit"] + +# 新客户端读这个,并在读到时忽略上面那条 +[target.'cfg(...)'.runtime] +link_library_dirs = ["lib/"] +libraries = ["mathkit"] +``` + +**判据:重复不是问题,冲突才是。** 新客户端若两条都应用,`cl` 仍会因为 `-L` 失败; +所以规则必须是「条件 `runtime` 存在 ⇒ 同一条腿的 `ldflags` 不再应用」, +而不是简单相加。这条规则只对**分发包**(`provenance` 以 `mcpp-pack` 开头)生效, +普通工程的 `ldflags` 语义不变。 + +--- + +## 4. 方案:分四层,零新增 manifest 段 + +沿用库分发的既有判据 —— **新增段 0 个**;第 3 层复用的是已经存在的 `[runtime]` 键, +只是让它可以出现在条件通道里。 + +### L1 — 未知扩展名不再静默(独立、最小、立即可做) + +| 改动 | 位置 | +|---|---| +| `builtin_extension_table()` **保持 `{".cppm"}`** —— 扩展名集合是配置 | `source_kind.cppm` | +| `sources` 命中但 `classify()` 为 `Other` ⇒ **硬错误**,点名文件、扩展名与 `module_extensions` | `scanner.cppm::scan_file`,分类发生的那一处 | + +- **实测两侧**:未声明 `.ixx` ⇒ 上面那条消息;声明后 ⇒ `ixx-ok=42`。 +- ⚠️ 这是**行为变化**:`sources` 里混进 `.md` / `.txt` 的工程会开始报错。 + 需要一轮全量 e2e 确认没有工程靠「未知扩展名被静默忽略」活着。 +- docs/10 的「用了 `module_extensions` 的包要声明版本下限」**不变** —— + 它本来就是关于「旧客户端不认识这条键」的,与本层无关。 + +### L2 — MSVC 自动 `.def`(把拒绝变成默认可用) + +新增一个构建图节点,不是一个 flag: + +``` +obj/*.obj ──▶ [def-gen] ──▶ bin/.def ──▶ link /DEF:.def /DLL +``` + +- **实现:mcpp 自己解析 COFF**,规则照 §2.2 列的 bindexplib 语义。 + 判据:**不得依赖 `dumpbin`** —— 它只在 VS 开发者环境里,而 mcpp 的 Windows 默认 + 工具链是 clang,`mcpp build` 不进 VS 环境。`llvm-nm` 是可选的第二实现,不是首选: + 多一个外部依赖就多一处「这台机器上没有」。 +- 导出数量超过 **65535** 时必须**报错并说明**,不是截断。 +- 生成的 `.def` 是构建产物,进 `target/`,**不进分发包** —— 包里带的是 DLL 与导入库。 + +### L3 — 导出宏(正确性上限,与 L2 并存) + +L2 的两条限制(数据符号、vtable)**无法由工具消除**,只能由标注消除。所以: + +- mcpp 提供生成导出头的能力(等价 CMake 的 `generate_export_header`), + 或约定一个 `MCPP_EXPORT` 宏; +- 与 modules 的写法就是微软示例的那个:`export __declspec(dllexport) void f();` +- **默认走 L2,作者需要导出数据或跨 DLL 的多态类型时走 L3。** + docs 必须把这条边界写在「当前边界」里,而不是让人踩到。 + +### L4 — cl.exe 消费 + +- `ConditionalConfig` 增加承载 `[runtime]` 链接意图的能力(`link_library_dirs` / + `libraries`); +- `manifest_emit` 对每条腿**同时**写 `ldflags` 与条件 `runtime`; +- 消费侧:分发包的某条腿若有条件 `runtime`,**忽略同腿的 `ldflags`**。 + +--- + +## 5. 依赖与顺序 + +``` +L1(.ixx + 未知扩展名) ── 独立,可先合 +L2(自动 .def) ── 依赖:Windows CI 有 msvc job(已有) + └─ L3(导出宏) ── 依赖 L2 的边界文档 +L4(cl.exe 消费) ── 独立于 L2/L3,但只有 L2 落地后才有 msvc 动态库可消费 +``` + +## 6. 验证矩阵(每一格都要能指出「失败时报什么」) + +| 断言 | 载体 | 平台 | +|---|---|---| +| `.ixx` 无需声明即可用 | e2e | 三平台 | +| `sources` 里的未知扩展名被拒并点名 | e2e | 三平台 | +| MSVC DLL 导出非空、消费者链接通过 | e2e `# requires: msvc` | Windows | +| 导出的数据符号在消费端仍需 `dllimport`(**限制本身可复现**) | e2e | Windows | +| 导出数 > 65535 时报错而非截断 | 单测(合成符号表) | 三平台 | +| `cl.exe` 消费打包库 | e2e `# requires: msvc` | Windows | +| 老客户端仍能消费同一个包 | e2e(静态 + 真实) | 三平台 | + +## 7. 明确不做 + +- **不发 `.ifc` / BMI。** 与编译器构建逐位绑定;微软给的第二条路(发 `.ixx` 让消费者自编) + 正是 mcpp 已经在做的。 +- **不自动给源码插 `dllexport`。** 那是改用户代码。 +- **不在 MinGW 上生成 `.def`。** 链接器已经自动导出,再生成一份只会引入第二个真相。 + +--- + +## 参考 + +- [WINDOWS_EXPORT_ALL_SYMBOLS — CMake 文档][cm](限制原文) +- [Kitware/CMake `Source/bindexplib.cxx`][bx](COFF 筛选规则) +- [LLVM:lld-link 的 MSVC 形态不做 auto-export][lld] +- [Microsoft Q&A:C++20 modules 在共享库中的用法][ms1] +- [Microsoft Q&A:不发 `.ifc` 可行吗][ms2] + +[cm]: https://cmake.org/cmake/help/latest/prop_tgt/WINDOWS_EXPORT_ALL_SYMBOLS.html +[bx]: https://github.com/Kitware/CMake/blob/master/Source/bindexplib.cxx +[lld]: https://github.com/llvm/llvm-project/pull/71087 +[ms1]: https://learn.microsoft.com/en-gb/answers/questions/1665106/how-to-use-c-20-modules-in-shared-libraries +[ms2]: https://learn.microsoft.com/en-us/answers/questions/1695780/c-20-modules-in-shared-libraries + + +--- + +## 8. 实施记录(2026-08-18,2026.8.18.2) + +四层全部落地,单 PR。与设计的偏差各自有实测依据。 + +### 8.1 与设计不同的地方 + +| 设计说 | 实际做的 | 为什么 | +|---|---|---| +| L1「把 `.ixx` 放进 builtin」 | **不内置**,只做「未声明即拒绝」 | 扩展名集合是配置,不是 mcpp 逐个追加的清单(review 时更正) | +| L1 只涉及扫描器 | 还修了 **lib root 约定**与 **pack 的 manifest 输出** | `mcpp pack` 必须跟着 `module_extensions` 走,否则用户要配两次 | +| L3「提供导出头/宏」 | **检测 `.drectve`,标注优先** | 为「我标注过了」加一个键,就是给对象已经说过的事再加一个说法 | + +### 8.2 实测抓到的、设计里没有的缺陷 + +**`.ixx` 库打出来的包是静默错的。** lib root 约定把 `.cppm` 写死,于是闭包从一个 +不存在的文件开始: + +``` +$ mcpp pack mathkit + Interface (headers only) ← 模块接口整个没了 + Withheld (nothing) + Packed …-x86_64-linux-gnu ← C 表面的 tag +``` + +**两半都错,而第二半比第一半更糟**:空的发布集合正是打包器判定「C 表面」的依据, +所以丢掉接口的同时,这个包也不再约束 C++ ABI,兼容性闸门停止检查编译器与标准库。 +一个「说得比实际少」的包,正是整个分发设计要防的那种失败。 + +**给 `mcpp.manifest.types` 加一条模块边会让 GCC 16.1 ICE。** 探测型的 lib-root +解析需要扩展名表(`mcpp.source_kind`),而 `types` 是几乎所有东西都依赖的低层模块。 +加上那条 import 之后,GCC 在编译**与改动无关的 `src/main.cpp`** 时 ICE, +清掉 gcm.cache 也不行 —— 与本仓库此前遇到的模块毒化形状一致。 +**解法不是加边,而是把函数移到边已经存在的地方**(`mcpp.manifest.toml` 本就 import +了 `mcpp.source_kind`)。 + +### 8.3 验证到哪一步 + +| 断言 | 载体 | 平台 | +|---|---|---| +| 已声明的扩展名无需额外帮助;未声明的当场拒绝 | e2e 260 | 三平台 | +| pack 跟随扩展名;包自带声明;`.cppm` 包 manifest 不变 | e2e 261 | 三平台 | +| COFF 筛选规则(外部/已定义/DATA/跳过表/aux/i386 下划线/拒绝) | 单测 ×16(合成) | 三平台 | +| 真实 mingw 对象可读;真实标注对象被识别 | 单测 ×3(committed fixture) | 三平台 | +| MSVC 产出带非空导出的 DLL,消费者链通并运行 | e2e 258 | **Windows** | +| 标注过的库不被自动导出覆盖 | e2e 258 后半 | **Windows** | +| `.def` 是构建图节点,输入是链接同一批对象 | e2e 258 | **Windows** | +| MinGW 不生成 `.def`(链接器已自动导出) | 本机实测 `0 edges` | Linux | + +**尚未验证**:`cl.exe` 消费打包库的端到端(需要 msvc job 里再加一条消费用例); +数据符号在消费端仍需 `dllimport` 这条限制目前只写在文档里,没有做成可复现的测试。 diff --git a/CHANGELOG.md b/CHANGELOG.md index 14f64d80..85c44c12 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -3,6 +3,64 @@ > 本文件追踪 `mcpp-community/mcpp` 公开仓的版本演进。 > 格式参考 [Keep a Changelog](https://keepachangelog.com/zh-CN/1.1.0/)。 +## [2026.8.18.2] — 2026-08-18 + +### 新增 + +- **`kind = "shared"` 在 MSVC ABI 上可用了 —— mcpp 自己生成 `.def`。** + + MSVC 在没有 `__declspec(dllexport)`、也没有 `.def` 时,DLL 什么都不导出; + 导入库为空,消费者拿到一堆 unresolved externals,而符号明明就在对象里。 + **拒绝的理由成立,结论不成立** —— CMake 的 `WINDOWS_EXPORT_ALL_SYMBOLS` 自 3.4 + 起就是这么做的,而且它的 `bindexplib` **直接读 COFF、不依赖 dumpbin**。这一点是 + 决定性的:`dumpbin` 只在 Visual Studio 开发者环境里,而 mcpp 在 Windows 的默认 + 工具链是 clang,`mcpp build` 根本不在那个环境里。 + + `mcpp.build.coff_exports` 是那个读取器,写成**对字节的纯函数**,于是它能在任何 + 平台上被测试 —— 16 条单测逐字节构造对象(那是唯一能按需改变存储类的办法), + 外加一个**真实的 mingw-cross 对象**:只喂自己测试输出的读取器,只会与自己一致。 + 超过 **65535** 个可导出符号时**拒绝而不截断**:被截断的导出表能干净链完, + 然后在「恰好需要那个掉出去的符号」的消费者那里失败。 + + **标注优先。** 对象里已带 `/EXPORT:` 指令(即 `__declspec(dllexport)` 的产物)时, + mcpp 让开、不生成任何东西 —— 再加一份列表会把同名符号导出两次(`LNK4197`), + 更糟的是把其余所有符号也导出,用「全部」替换掉作者选定的公开面。 + **这件事靠检测而不是配置**:为「我标注过了」加一个 manifest 键,就是给对象已经 + 说过的事再加一个说法,而两者可以不一致。 + + 仍有两条限制是工具消不掉的(与 CMake 记录的同两条):导出的**数据**在消费端仍需 + `__declspec(dllimport)`;**vtable 被引用的类**要整类标注。两者都写进了 docs/12。 + +- **打包库可以被原生 `cl.exe` 消费。** 生成的 manifest 现在同时带方言中立的 + `[target..runtime]`(`link_library_dirs` / `libraries`),mcpp 会按 target + 渲染成 `/LIBPATH:` + `.lib` 或 `-L` + `-l`。**两种拼写都带**:旧版 mcpp + 只读 `ldflags` 并静默忽略新段,去掉它会让旧客户端一个 flag 都拿不到; + 而新版读到中立形式时**忽略**同腿的 `ldflags` 而不是叠加 —— 叠加会把 `-L` + 送回 `cl` 的命令行。 + +### 修复 + +- **`mcpp pack` 现在跟着工程的 `module_extensions` 走,不需要再配一次。** + + 实测:接口是 `.ixx` 的库打出来的包是**静默错的** —— + + + + 两半都错且都不出声:没有接口,消费者 `import` 不了;而**空的发布集合正是打包器 + 判定「C 表面」的依据**,于是这个包同时不再约束 C++ ABI,兼容性闸门也不再检查 + 编译器与标准库。真因是 lib root 约定把 `.cppm` 写死了。现在按**已声明的每个扩展名** + 各给一个候选并取存在的那个;生成的 manifest 也会**自己声明** `module_extensions` + —— 从**发布的文件**算出来,因此不可能与 `sources` 不一致。 + +- **`sources` 命中却分类不出角色的文件,现在当场拒绝。** + + 未声明的 `.ixx` 过去会编译出一个**没人链接的对象**,报错是 + `undefined reference to mk::answer@mathkit()` —— 既不点名扩展名也不点名那条键。 + 根因是「这是不是模块接口」有**两个答题者**:扫描器读到 `export module` 记下 + `provides`(所以边上挂了 `bmi_out`),而分类器说 `Other`,链接集合只读后者。 + + ⚠️ 这是**行为变化**:`sources` 里混进 `.md` / `.txt` 的工程会开始报错。 + ## [2026.8.18.1] — 2026-08-18 ### 新增 diff --git a/docs/12-binary-distribution.md b/docs/12-binary-distribution.md index 3ad47500..6b349d60 100644 --- a/docs/12-binary-distribution.md +++ b/docs/12-binary-distribution.md @@ -264,32 +264,46 @@ package published to a mixed-version audience. | `kind = "shared"` on Linux/ELF | ✅ — the package carries both the link name and the SONAME | | `kind = "shared"` on PE / MinGW (`*-windows-gnu`) | ✅ — the package carries the `.dll` **and** its import library | | `kind = "shared"` on Mach-O (`*-macos`) | ✅ — install name is `@rpath/`, so the `.dylib` relocates | -| `kind = "shared"` on PE / MSVC (`*-windows-msvc`) | ❌ refused — see below | +| `kind = "shared"` on PE / MSVC (`*-windows-msvc`) | ✅ — mcpp generates the `.def`; see below | | `kind = "shared"` on `*-musl` | ❌ a musl target links statically | | shipping prebuilt BMIs | ❌ not attempted; BMIs are compiler-build-exact | | bundling dependencies into the package | ❌ declare them instead (above) | | consuming a package with **native `cl.exe`** | ❌ see below | -### Why MSVC refuses `kind = "shared"` +### Exports on the MSVC ABI -Not the linker — `link /DLL /IMPLIB:` works. **Symbol export.** MSVC exports -nothing from a DLL unless the source says `__declspec(dllexport)` or a `.def` -file lists the symbols, so the import library comes out empty and every consumer -fails with unresolved externals naming symbols that are plainly in the object -files. mcpp refuses rather than produce a diagnostic that points nowhere near its -cause: +MSVC exports nothing from a DLL unless the source says `__declspec(dllexport)` or +a `.def` file lists the symbols. Without either, the import library comes out +empty and every consumer fails with unresolved externals for symbols that are +plainly in the object files. MinGW's linker auto-exports and hides this +entirely; lld-link's MSVC flavour does not, deliberately, because PE caps +exports at 65535. -``` -target 'mathkit': kind = "shared" is not supported for the MSVC ABI (x86_64-windows-msvc). - MSVC exports nothing from a DLL unless the source says `__declspec(dllexport)` - ... - Use kind = "lib" for this target, or build it for *-windows-gnu (MinGW), - where the linker auto-exports. -``` +mcpp generates the `.def` from the objects, which is what CMake's +`WINDOWS_EXPORT_ALL_SYMBOLS` has done since 3.4. It is a build-graph node whose +inputs are the same objects the link consumes, so the exported surface cannot +drift from what was compiled, and it reads COFF directly rather than shelling out +to `dumpbin` — that tool lives in a Visual Studio developer environment, and +mcpp's default Windows toolchain is clang. + +**Two limits survive that no tool can remove**, and they are the same two CMake +documents for the same mechanism: + +| | | +|---|---| +| exported **data** | the consumer still needs `__declspec(dllimport)` on its declaration; without it the linker reads a call thunk instead of the value | +| a class whose **vtable** is referenced | the whole class must be marked, e.g. in a delegating constructor of a class with virtual functions | -MinGW's linker auto-exports, which is why `*-windows-gnu` is supported and -`*-windows-msvc` is not. Closing this needs a generated `.def` — a symbol scan -over the objects — which is a build-graph node, not a flag. +Both are answered by annotation, and **annotation wins**: an object that already +carries `/EXPORT:` directives — which is what `__declspec(dllexport)` emits — +makes mcpp stand down and generate nothing. Adding a list on top would export the +same names twice (`LNK4197`) and export everything else besides, replacing a +chosen public surface with all of it. Nothing is configured for this; the objects +say it. + +Past 65535 exportable symbols mcpp refuses rather than truncating. A truncated +export table links cleanly and then fails at whichever consumer needed the symbol +that fell off the end. ### A package's link flags are GNU-spelled @@ -301,9 +315,33 @@ ldflags = ["-Llib/x86_64-windows-msvc", "-lmathkit"] ``` Every driver mcpp uses accepts that — including clang on the MSVC ABI, which is -Windows' default here. **Native `cl.exe` does not**: it rejects `-L`. So a -consumer that pins `[toolchain] windows = "msvc@system"` cannot link a packaged -library today. +Windows' default here. **Native `cl.exe` does not**: it rejects `-L`. So the +package carries the same statement a second time, without a dialect: + +```toml +[target.'cfg(all(arch = "x86_64", os = "windows", env = "msvc"))'.runtime] +link_library_dirs = ["lib/x86_64-windows-msvc"] +libraries = ["mathkit"] +``` + +mcpp renders those as `/LIBPATH:` + `.lib` or `-L` + `-l` from the +target, so a `cl.exe` consumer links the package. They are not new vocabulary — +`[runtime]` has had both keys at the top level all along; this makes them +per-target. + +**Both spellings are emitted, and a newer mcpp drops the leg's library +references rather than adding to them.** An older mcpp reads only the `ldflags` +and silently ignores the `runtime` block, so dropping the `ldflags` would leave +every older client with no link line at all; adding both would put `-L` back on +the `cl` command line, which is the thing being avoided. + +One leg is deliberately left out of this: a **PE/MinGW shared** leg links with +`-L… -Wl,-Bdynamic -lmathkit`, and `-Wl,-Bdynamic` only works immediately before +the `-l` it enables — mcpp gives PE executables `-static`, which otherwise leaves +the linker in static-only mode where it refuses an import library. The neutral +form cannot say "switch link mode first", so that leg keeps the spelling that +works. It costs nothing: a PE/MinGW leg is not an MSVC-ABI leg, and `cl.exe` +never reads it. Naming the file by path instead (`lib//mathkit.lib`) is the spelling every driver takes, and it does not work either: ninja runs link commands with diff --git a/docs/zh/12-binary-distribution.md b/docs/zh/12-binary-distribution.md index 33a454c6..a7692ffb 100644 --- a/docs/zh/12-binary-distribution.md +++ b/docs/zh/12-binary-distribution.md @@ -239,29 +239,38 @@ ldflags = ["-Llib/x86_64-linux-musl", "-lmathkit"] | `kind = "shared"` on Linux/ELF | ✅ —— 包里同时带链接名与 SONAME | | `kind = "shared"` on PE / MinGW(`*-windows-gnu`) | ✅ —— 包里同时带 `.dll` **和它的导入库** | | `kind = "shared"` on Mach-O(`*-macos`) | ✅ —— install name 是 `@rpath/`,`.dylib` 可重定位 | -| `kind = "shared"` on PE / MSVC(`*-windows-msvc`) | ❌ 拒绝 —— 见下 | +| `kind = "shared"` on PE / MSVC(`*-windows-msvc`) | ✅ —— mcpp 生成 `.def`;见下 | | `kind = "shared"` on `*-musl` | ❌ musl target 是静态链接的 | | 发布预编译 BMI | ❌ 未尝试;BMI 与编译器构建逐位绑定 | | 把依赖打包进去 | ❌ 改为声明依赖(见上) | | 用**原生 `cl.exe`** 消费这种包 | ❌ 见下 | -### MSVC 拒绝 `kind = "shared"` 的原因 +### MSVC ABI 上的符号导出 -**不是链接器的问题** —— `link /DLL /IMPLIB:` 本来就能用。是**符号导出**: -MSVC 在没有 `__declspec(dllexport)`、也没有 `.def` 列出符号时,DLL **什么都不导出**, -于是导入库是空的,每个消费者都会拿到一堆 unresolved externals,而那些符号 -明明就在对象文件里 —— 报错点离病因很远。mcpp 选择拒绝,而不是产出这种诊断: +MSVC 在源码没有 `__declspec(dllexport)`、也没有 `.def` 列出符号时,DLL 什么都不导出。 +两者皆无时导入库为空,每个消费者都会拿到一堆 unresolved externals,而那些符号 +明明就在对象文件里。MinGW 的链接器会自动导出,把这个问题整个遮住;lld-link 的 +MSVC 形态刻意不这么做,因为 PE 的导出上限是 65535。 -``` -target 'mathkit': kind = "shared" is not supported for the MSVC ABI (x86_64-windows-msvc). - ... - Use kind = "lib" for this target, or build it for *-windows-gnu (MinGW), - where the linker auto-exports. -``` +mcpp 从对象生成 `.def` —— 这正是 CMake 的 `WINDOWS_EXPORT_ALL_SYMBOLS` 自 3.4 起 +在做的事。它是一个构建图节点,输入就是链接所消费的那批对象,因此导出面不会与 +「实际编译了什么」发生漂移;而且它**直接读 COFF**,不调 `dumpbin` —— 那个工具在 +Visual Studio 开发者环境里才有,而 mcpp 在 Windows 上的默认工具链是 clang。 + +**有两条限制是任何工具都消不掉的**,与 CMake 为同一机制记录的是同两条: + +| | | +|---|---| +| 导出的**数据** | 消费者的声明仍需 `__declspec(dllimport)`;否则链接器读到的是调用桩而不是值 | +| **vtable** 被引用的类 | 整个类都要标注,例如带虚函数的类的委托构造函数 | + +两者都靠标注解决,而且**标注优先**:对象里若已带 `/EXPORT:` 指令(那正是 +`__declspec(dllexport)` 产生的),mcpp 就让开,不生成任何东西。在其之上再加一份 +列表会把同名符号导出两次(`LNK4197`),更糟的是把其余所有符号也一并导出 —— +用「全部」替换掉作者选定的公开面。这件事没有任何开关:对象自己说了算。 -MinGW 的链接器会自动导出,这就是 `*-windows-gnu` 支持而 `*-windows-msvc` 不支持的 -全部原因。要补齐它需要生成 `.def`(对对象做一次符号扫描)—— 那是一个构建图节点, -不是一个 flag。 +可导出符号超过 65535 时,mcpp 拒绝而不是截断。被截断的导出表能干净地链接完成, +随后在「恰好需要那个掉出去的符号」的消费者那里失败。 ### 包里的链接 flag 是 GNU 拼写 @@ -273,8 +282,29 @@ ldflags = ["-Llib/x86_64-windows-msvc", "-lmathkit"] ``` mcpp 用到的每个 driver 都吃这一套 —— 包括 Windows 上默认的、面向 MSVC ABI 的 -clang。**原生 `cl.exe` 不吃**:它不认 `-L`。所以固定了 -`[toolchain] windows = "msvc@system"` 的消费者目前链不上打包库。 +clang。**原生 `cl.exe` 不吃**:它不认 `-L`。所以包里会把同一句话再写一遍, +这一遍不带方言: + +```toml +[target.'cfg(all(arch = "x86_64", os = "windows", env = "msvc"))'.runtime] +link_library_dirs = ["lib/x86_64-windows-msvc"] +libraries = ["mathkit"] +``` + +mcpp 会按 target 把它们渲染成 `/LIBPATH:` + `.lib` 或 `-L` + `-l`, +于是 `cl.exe` 的消费者也能链上。这两个键不是新词表 —— `[runtime]` 顶层一直就有, +这里只是让它们可以按 target 给。 + +**两种拼写都会写出来,而新版 mcpp 读到中立形式时会丢掉同一条腿的库引用, +而不是叠加。** 旧版 mcpp 只读 `ldflags` 并静默忽略 `runtime` 段,所以去掉 +`ldflags` 会让所有旧客户端一个链接 flag 都拿不到;而两者都应用又会把 `-L` 送回 +`cl` 的命令行 —— 那正是要避免的事。 + +有一条腿被刻意排除在外:**PE/MinGW 的动态库腿**链接行是 +`-L… -Wl,-Bdynamic -lmathkit`,而 `-Wl,-Bdynamic` 只有**紧邻它所启用的那个 `-l`** +时才有效 —— mcpp 给 PE 可执行文件加 `-static`,否则链接器停在纯静态模式并拒绝 +导入库。中立形式没法表达「先切换链接模式」,所以那条腿保留能用的拼写。 +这不付出任何代价:PE/MinGW 的腿不是 MSVC ABI 的腿,`cl.exe` 永远读不到它。 **改成直接写文件路径也不行**(`lib//mathkit.lib` 才是每个 driver 都吃的 拼写):ninja 执行链接命令时 cwd 是**输出目录**,而只有 include 家族前缀 diff --git a/mcpp.toml b/mcpp.toml index 4ee2c54c..24fb84b8 100644 --- a/mcpp.toml +++ b/mcpp.toml @@ -1,6 +1,6 @@ [package] name = "mcpp" -version = "2026.8.18.1" +version = "2026.8.18.2" description = "Modern C++ build & package management tool" license = "Apache-2.0" authors = ["mcpp-community"] diff --git a/src/build/coff_exports.cppm b/src/build/coff_exports.cppm new file mode 100644 index 00000000..43d1aea8 --- /dev/null +++ b/src/build/coff_exports.cppm @@ -0,0 +1,329 @@ +// mcpp.build.coff_exports — which symbols of a COFF object a DLL should export. +// +// WHY THIS EXISTS +// +// MSVC exports nothing from a DLL unless the source says `__declspec(dllexport)` +// or a `.def` file lists the symbols. Without either, the import library comes +// out empty and every consumer fails with unresolved externals naming symbols +// that are plainly in the object files — a diagnostic pointing nowhere near its +// cause. MinGW's linker auto-exports and hides the whole problem; lld-link's +// MSVC flavour does not, deliberately, because PE caps exports at 65535. +// +// So mcpp generates the `.def`. This is not novel: CMake has shipped +// `WINDOWS_EXPORT_ALL_SYMBOLS` since 3.4 and its `bindexplib` does exactly this. +// The filter below follows that one's semantics, which are the de-facto standard +// for "what does auto-export mean on Windows". +// +// WHY IT PARSES COFF ITSELF +// +// `dumpbin` lives in a Visual Studio developer environment, and mcpp's default +// Windows toolchain is clang — a plain `mcpp build` is not inside that +// environment. `llvm-nm` would be a second external dependency with the same +// class of failure ("not on this machine"). COFF's symbol table is a fixed-size +// record array with a string table after it; reading it needs no library. +// +// WHY IT IS A PURE FUNCTION OVER BYTES +// +// So it can be tested on any host. Windows CI can only tell whether the linker +// accepted the result; the filtering rules are where mistakes hide, and those +// are decidable from a byte buffer. Linux CI produces real COFF objects through +// mingw-cross, so the parser is exercised against genuine input off-Windows too. +// +// Design: .agents/docs/2026-08-18-windows-shared-library-and-module-extensions.md §2. + +export module mcpp.build.coff_exports; + +import std; + +export namespace mcpp::build::coff { + +// One exportable symbol. +struct Export { + std::string name; + // `.def` needs `name DATA` for anything that is not code: the linker + // otherwise generates a call thunk for a variable, and the consumer reads + // the thunk instead of the value. + bool data = false; + + bool operator==(const Export&) const = default; + auto operator<=>(const Export&) const = default; +}; + +// PE's export directory addresses exports by 16-bit ordinal, so a DLL cannot +// have more than this many. Reaching it is refused rather than truncated: a +// truncated export table links and then fails at the consumer, naming whichever +// symbol happened to fall off the end. +inline constexpr std::size_t kMaxExports = 65535; + +// Machine types this reader accepts. An unknown one is not an error to guess +// at: the symbol-table layout is shared, but a machine it has never been tried +// against is exactly where an untested assumption would hide. +bool is_supported_machine(std::uint16_t machine); + +// Does this object ALREADY declare exports? +// +// `__declspec(dllexport)` makes the compiler write `/EXPORT:name` directives +// into the object's `.drectve` section, which the linker reads. When an author +// has annotated their surface, adding a generated `.def` on top would export +// the same names twice — `LNK4197: export specified multiple times` — and, +// worse, would export everything else as well, quietly replacing a chosen +// public surface with all of it. +// +// So the annotation wins and mcpp stays out of the way. Detected rather than +// configured: a manifest key for "I annotated my exports" would be a second +// place to say something the objects already say, and the two could disagree. +bool declares_exports(std::span bytes); + +// Read one object file's exportable symbols. `bytes` is a whole `.obj`. +// +// Returns an error string for input this reader cannot honestly interpret — +// truncated, an unsupported machine, or a symbol table that runs off the end. +// It never returns a partial list: half a symbol table is indistinguishable +// from a small one, and the caller would ship a DLL missing exports. +std::expected, std::string> +read_exports(std::span bytes); + +// The `.def` text for a set of exports, sorted and de-duplicated. +// +// `libraryName` goes in the `LIBRARY` statement — link.exe accepts a `.def` +// without one, but naming it makes the file self-describing when a human opens +// it while working out why a symbol is missing. +std::string write_def(std::string_view libraryName, std::vector exports); + +} // namespace mcpp::build::coff + +namespace mcpp::build::coff { + +namespace { + +// ── COFF, only the parts needed ────────────────────────────────────────── +// +// Offsets rather than structs: a packed-struct cast would depend on the host's +// alignment rules to read a file format, and this code runs on hosts that never +// produce COFF. + +constexpr std::size_t kFileHeaderSize = 20; +constexpr std::size_t kSymbolRecordSize = 18; +constexpr std::size_t kSectionHeaderSize = 40; + +// The subset of IMAGE_FILE_MACHINE_* that mcpp targets or can be handed. +constexpr std::uint16_t kMachineI386 = 0x014c; +constexpr std::uint16_t kMachineAmd64 = 0x8664; +constexpr std::uint16_t kMachineArm = 0x01c0; +constexpr std::uint16_t kMachineArmNT = 0x01c4; +constexpr std::uint16_t kMachineArm64 = 0xaa64; + +constexpr std::uint8_t kSymClassExternal = 2; // IMAGE_SYM_CLASS_EXTERNAL +constexpr std::uint16_t kSymTypeFunction = 0x20; // DTYPE_FUNCTION << 4 +constexpr std::uint32_t kScnMemExecute = 0x20000000u; // IMAGE_SCN_MEM_EXECUTE + +std::uint16_t rd16(std::span b, std::size_t off) { + return static_cast(std::to_integer(b[off]) + | (std::to_integer(b[off + 1]) << 8)); +} +std::uint32_t rd32(std::span b, std::size_t off) { + return static_cast(std::to_integer(b[off]) + | (std::to_integer(b[off + 1]) << 8) + | (std::to_integer(b[off + 2]) << 16) + | (std::to_integer(b[off + 3]) << 24)); +} + +// Symbols bindexplib skips, and why each one would be wrong to export. +bool is_skipped_name(std::string_view n) { + // Scalar-deleting and vector-deleting destructor thunks. They are emitted + // per-TU and exporting them makes the DLL's surface depend on which + // translation unit happened to instantiate one. + if (n.starts_with("??_G") || n.starts_with("??_E")) return true; + // Managed (C++/CLI) artefacts: `.` cannot appear in a C++ mangled name, and + // these forms name IL constructs that mean nothing to a native consumer. + if (n.find('.') != std::string_view::npos) return true; + if (n.find("__t2m") != std::string_view::npos) return true; + if (n.find("$$F") != std::string_view::npos) return true; + if (n.find("$$J") != std::string_view::npos) return true; + // ARM64EC thunks: linker-generated bridges between the ARM64 and x64 views + // of the same function. Exporting a bridge exports neither side. + if (n.find("$ientry_thunk") != std::string_view::npos) return true; + if (n.find("$entry_thunk") != std::string_view::npos) return true; + if (n.find("$iexit_thunk") != std::string_view::npos) return true; + if (n.find("$exit_thunk") != std::string_view::npos) return true; + return false; +} + +} // namespace + +bool declares_exports(std::span bytes) { + if (bytes.size() < kFileHeaderSize) return false; + const auto numSections = rd16(bytes, 2); + const std::size_t sectionsOff = kFileHeaderSize + rd16(bytes, 16); + + for (std::uint16_t i = 0; i < numSections; ++i) { + const auto hdr = sectionsOff + std::size_t(i) * kSectionHeaderSize; + if (hdr + kSectionHeaderSize > bytes.size()) return false; + + std::string name; + for (std::size_t c = 0; c < 8; ++c) { + auto ch = std::to_integer(bytes[hdr + c]); + if (ch == '\0') break; + name.push_back(ch); + } + if (name != ".drectve") continue; + + const auto size = rd32(bytes, hdr + 16); + const auto off = rd32(bytes, hdr + 20); + if (off == 0 || std::size_t(off) + size > bytes.size()) return false; + + // The section is plain text: a run of linker directives. Both spellings + // occur — cl emits `/EXPORT:`, clang-cl `-export:` — and a linker + // accepts either, so looking for one of them only would be a coin flip + // on which compiler produced the object. + std::string text; + text.reserve(size); + for (std::uint32_t k = 0; k < size; ++k) + text.push_back(std::to_integer(bytes[std::size_t(off) + k])); + for (auto& c : text) c = static_cast(std::tolower(static_cast(c))); + if (text.find("/export:") != std::string::npos) return true; + if (text.find("-export:") != std::string::npos) return true; + } + return false; +} + +bool is_supported_machine(std::uint16_t machine) { + return machine == kMachineI386 || machine == kMachineAmd64 + || machine == kMachineArm || machine == kMachineArmNT + || machine == kMachineArm64; +} + +std::expected, std::string> +read_exports(std::span bytes) +{ + if (bytes.size() < kFileHeaderSize) + return std::unexpected("not a COFF object: shorter than a file header"); + + const auto machine = rd16(bytes, 0); + + // `/bigobj` objects are a DIFFERENT container: machine 0 and a `0xFFFF` + // where the section count would be, followed by a class GUID and a much + // larger header. Named here rather than left to fall out as "unsupported + // machine 0x0000", which is true and useless — the reader would be blamed + // for a flag the project passed. + if (machine == 0 && bytes.size() >= 4 && rd16(bytes, 2) == 0xFFFF) { + return std::unexpected( + "this is a /bigobj object, whose header layout differs from an " + "ordinary COFF one.\n" + " mcpp's export reader does not parse it. Build the shared library " + "without /bigobj,\n" + " or mark its public surface with __declspec(dllexport) — an " + "annotated library needs\n" + " no generated .def at all."); + } + + if (!is_supported_machine(machine)) { + return std::unexpected(std::format( + "unsupported COFF machine 0x{:04x}. mcpp reads i386, amd64, arm, " + "armnt and arm64; anything else would be a guess at a layout this " + "has never been run against.", machine)); + } + + const auto numSections = rd16(bytes, 2); + const auto symTableOff = rd32(bytes, 8); + const auto numSymbols = rd32(bytes, 12); + if (symTableOff == 0 || numSymbols == 0) return std::vector{}; + + // Section characteristics, indexed 1-based the way symbols address them. + const std::size_t sectionsOff = kFileHeaderSize + rd16(bytes, 16); // + optional header + std::vector sectionFlags(numSections + 1, 0); + for (std::uint16_t i = 0; i < numSections; ++i) { + const auto off = sectionsOff + std::size_t(i) * kSectionHeaderSize; + if (off + kSectionHeaderSize > bytes.size()) + return std::unexpected("COFF section headers run past the end of the file"); + sectionFlags[i + 1] = rd32(bytes, off + 36); + } + + const std::size_t symbolsEnd = std::size_t(symTableOff) + + std::size_t(numSymbols) * kSymbolRecordSize; + if (symbolsEnd > bytes.size()) + return std::unexpected("COFF symbol table runs past the end of the file"); + + // The string table follows the symbol table; its first 4 bytes are its own + // size. Long names are `/` into it. + const std::size_t stringsOff = symbolsEnd; + const bool hasStrings = stringsOff + 4 <= bytes.size(); + + auto name_at = [&](std::size_t rec) -> std::expected { + // A name is 8 bytes: either inline (NUL-padded) or a zero word followed + // by an offset into the string table. + if (rd32(bytes, rec) != 0) { + std::string s; + for (std::size_t i = 0; i < 8; ++i) { + auto c = std::to_integer(bytes[rec + i]); + if (c == '\0') break; + s.push_back(c); + } + return s; + } + const auto off = rd32(bytes, rec + 4); + if (!hasStrings || stringsOff + off >= bytes.size()) + return std::unexpected("COFF long symbol name points outside the string table"); + std::string s; + for (std::size_t i = stringsOff + off; i < bytes.size(); ++i) { + auto c = std::to_integer(bytes[i]); + if (c == '\0') break; + s.push_back(c); + } + return s; + }; + + std::vector out; + for (std::uint32_t i = 0; i < numSymbols; ) { + const std::size_t rec = std::size_t(symTableOff) + + std::size_t(i) * kSymbolRecordSize; + const auto sectionNum = static_cast(rd16(bytes, rec + 12)); + const auto type = rd16(bytes, rec + 14); + const auto storage = std::to_integer(bytes[rec + 16]); + const auto numAux = std::to_integer(bytes[rec + 17]); + + // Advance past auxiliary records regardless of whether this symbol is + // taken — an aux record is not a symbol and reading it as one produces + // names out of raw section data. + i += 1u + numAux; + + if (storage != kSymClassExternal) continue; // not visible outside the TU + if (sectionNum <= 0) continue; // undefined, absolute or debug + if (type != kSymTypeFunction && type != 0) continue; + + auto nm = name_at(rec); + if (!nm) return std::unexpected(nm.error()); + if (nm->empty() || is_skipped_name(*nm)) continue; + + // i386 (and `__cdecl` on it) carries a leading underscore that is part + // of the calling convention rather than of the name. + std::string name = *nm; + if (machine == kMachineI386 && name.starts_with('_') && name.find('@') == std::string::npos) + name.erase(0, 1); + + const auto flags = std::size_t(sectionNum) < sectionFlags.size() + ? sectionFlags[sectionNum] : 0u; + // DATA when it is not code. Both halves matter: a function symbol in a + // writable section is still a function, and a variable in a read-only + // section (a `const`) is still data. + const bool isData = type != kSymTypeFunction + && (flags & kScnMemExecute) == 0; + out.push_back(Export{ std::move(name), isData }); + } + + return out; +} + +std::string write_def(std::string_view libraryName, std::vector exports) { + std::ranges::sort(exports); + exports.erase(std::ranges::unique(exports).begin(), exports.end()); + + std::string out; + if (!libraryName.empty()) out += std::format("LIBRARY {}\n", libraryName); + out += "EXPORTS\n"; + for (auto const& e : exports) + out += std::format(" {}{}\n", e.name, e.data ? " DATA" : ""); + return out; +} + +} // namespace mcpp::build::coff diff --git a/src/build/flags.cppm b/src/build/flags.cppm index 2058e12d..8c35f2be 100644 --- a/src/build/flags.cppm +++ b/src/build/flags.cppm @@ -21,6 +21,7 @@ import mcpp.platform.runtime_search; import mcpp.toolchain.clang; import mcpp.toolchain.detect; import mcpp.toolchain.dialect; +import mcpp.toolchain.triple; import mcpp.toolchain.hostflags; import mcpp.toolchain.linkmodel; import mcpp.toolchain.model; @@ -411,7 +412,23 @@ CompileFlags compute_flags(const BuildPlan& plan) { const bool isMsvcDialect = (d.id == "msvc"); - // PIC? (GNU-only concept; PE code is position independent by design.) + // PIC is a GNU concept and a property of the TARGET FORMAT: PE code is + // position independent by design (base relocations), and clang rejects the + // flag outright — `unsupported option '-fPIC' for target + // 'x86_64-pc-windows-msvc'`. + // + // ⚠️ The condition used to be `!isMsvcDialect`, i.e. the DIALECT. Windows' + // default toolchain is clang, which speaks the GNU dialect while targeting + // the MSVC ABI, so `-fPIC` was emitted and every MSVC-ABI shared build died + // in clang-scan-deps before compiling anything. It was unreachable while + // `kind = "shared"` was refused on that ABI; allowing it is what surfaced + // this. Same shape as the shared-library guard itself: asking which + // COMPILER when the question is which TARGET. + const bool peTarget = [&] { + if (auto t = mcpp::toolchain::triple::parse(plan.toolchain.targetTriple)) + return t->is_pe(); + return bool(mcpp::platform::is_windows); + }(); bool need_pic = false; for (auto& lu : plan.linkUnits) { if (lu.kind == LinkUnit::SharedLibrary) { @@ -419,7 +436,7 @@ CompileFlags compute_flags(const BuildPlan& plan) { break; } } - std::string pic_flag = (need_pic && !isMsvcDialect) ? " -fPIC" : ""; + std::string pic_flag = (need_pic && !isMsvcDialect && !peTarget) ? " -fPIC" : ""; // Include dirs — this is the TYPED PATH channel (bare paths from the // manifest; the dialect prefix is applied here at emission), not the diff --git a/src/build/ninja_backend.cppm b/src/build/ninja_backend.cppm index 930f0f33..93f6adef 100644 --- a/src/build/ninja_backend.cppm +++ b/src/build/ninja_backend.cppm @@ -207,6 +207,31 @@ std::string join_flags(const std::vector& flags) { // default install name is the path it was LINKED at, so a package built in // /tmp/build-xyz records /tmp/build-xyz and cannot be relocated — which is // every distributed dylib. `@rpath/` is the only default that travels. +// A PE link flag, spelled for the TARGET ABI and wrapped for the driver. +// +// ⚠️ NOT a dialect-table entry, and Windows CI is why. Clang targeting the MSVC +// ABI speaks the GNU DIALECT while driving lld-link, so a dialect-keyed spelling +// handed it `-Wl,--out-implib,` and lld-link answered `warning: ignoring unknown +// argument '--out-implib'` followed by `could not open '…lib': no such file`. +// Three flags in this PR made the same mistake — `-fPIC`, the import library, +// and `/DEF:` — and the mistake is always the same one: asking which COMPILER +// when the question is which TARGET. +// +// `sep` is `LinkStyle::SeparateLinker`, i.e. link.exe invoked directly, where +// the flag needs no `-Wl,` wrapper. Everything else goes through a compiler +// driver. +std::string pe_link_flag(const BuildPlan& plan, bool sep, + std::string_view msvcForm, std::string_view gnuForm, + std::string_view path) +{ + const auto t = mcpp::toolchain::triple::parse(plan.toolchain.targetTriple); + const bool msvcAbi = t ? t->is_msvc_env() + : mcpp::toolchain::is_msvc_target(plan.toolchain); + if (!msvcAbi) return std::string(gnuForm) + std::string(path); + auto flag = std::string(msvcForm) + std::string(path); + return sep ? flag : "-Wl," + flag; +} + std::string shared_soname_flag(const LinkUnit& lu, const BuildPlan& plan) { if (lu.kind != LinkUnit::SharedLibrary) return ""; const auto t = mcpp::toolchain::triple::parse(plan.toolchain.targetTriple); @@ -1049,7 +1074,7 @@ std::string emit_ninja_string(const BuildPlan& plan) { "LINK"); link_rule("cxx_archive", std::string(dial.archiveCmd), "AR"); link_rule("cxx_shared", - "$ld /nologo /DLL /OUT:$out $implib_flag " + "$ld /nologo /DLL /OUT:$out $implib_flag $def_flag " "$in $ldflags $unit_ldflags", "SHARED"); } else { @@ -1058,7 +1083,7 @@ std::string emit_ninja_string(const BuildPlan& plan) { link_rule("cxx_archive", std::string(dial.archiveCmd), "AR"); link_rule("cxx_shared", "$cxx -shared $in -o $out $ldflags $soname_flag " - "$implib_flag $unit_ldflags", + "$implib_flag $def_flag $unit_ldflags", "SHARED"); // mcpp#426: a link unit with no C++ translation unit in it is // linked by the C driver. `g++` appends `-lstdc++` unconditionally, @@ -1071,7 +1096,7 @@ std::string emit_ninja_string(const BuildPlan& plan) { "$cc $in -o $out $c_ldflags $unit_ldflags", "LINK"); link_rule("c_shared", "$cc -shared $in -o $out $c_ldflags $soname_flag " - "$implib_flag $unit_ldflags", + "$implib_flag $def_flag $unit_ldflags", "SHARED"); } } @@ -1086,6 +1111,15 @@ std::string emit_ninja_string(const BuildPlan& plan) { append(" description = CC $out\n\n"); } + // The auto-export edge. `mcpp coff-def` reads the same objects the link + // consumes, so the DLL's surface cannot drift from what was compiled — and + // it is an mcpp subcommand rather than a shell fragment because a generated + // POSIX-shell command is skipped entirely on Windows, the only platform this + // edge exists for. + append("rule coff_def\n"); + append(" command = $mcpp coff-def --output $out --name $def_name $in\n"); + append(" description = DEF $out\n\n"); + append("rule runtime_alias\n"); if constexpr (mcpp::platform::is_windows) { // PE has no soname symlink, so the alias is a copy — and a copy of a @@ -1816,6 +1850,24 @@ std::string emit_ninja_string(const BuildPlan& plan) { if (!lu.importLibrary.empty()) implicitOut = " | " + escape_ninja_path(lu.importLibrary); + // The `.def` edge, emitted BEFORE the link that consumes it. + // + // Its inputs are this unit's objects — the same list the link gets — so + // the exported surface is a function of what was compiled and cannot + // drift from it. It is an ordinary explicit input to the link rather + // than an implicit one: the linker reads it, so ninja should rebuild the + // DLL when it changes. + if (!lu.defFile.empty()) { + std::string defIns; + for (auto const& o : lu.objects) defIns += " " + escape_ninja_path(o); + append(std::format("build {} : coff_def{}\n", + escape_ninja_path(lu.defFile), defIns)); + append(std::format(" def_name = {}\n\n", + lu.output.filename().string())); + } + + if (!lu.defFile.empty()) implicit += " " + escape_ninja_path(lu.defFile); + std::string out_line = std::format("build {}{} : {}{}{}\n", escape_ninja_path(lu.output), implicitOut, rule, ins, implicit.empty() ? std::string{} : " |" + implicit); @@ -1826,17 +1878,20 @@ std::string emit_ninja_string(const BuildPlan& plan) { // agrees with — the name belongs to plan.cppm's import_library_for, and // this is how it gets to the command. The SPELLING belongs to the // dialect table, same as `archiveRemoveArg`. + const bool sepLinker = + dial.linkStyle == mcpp::toolchain::CommandDialect::LinkStyle::SeparateLinker; if (!lu.importLibrary.empty()) { - std::string arg{ dial.sharedImportLibArg }; - // `{}` or nothing: a row without the placeholder cannot say WHERE to - // write, so emitting its bare text would hand the linker a flag with - // no argument. Skipping is the honest reading of an empty row, and - // the implicit output above then fails loudly as a missing file - // rather than quietly linking against a stale one. - if (auto at = arg.find("{}"); at != std::string::npos) { - arg.replace(at, 2, escape_ninja_path(lu.importLibrary)); - out_line += " implib_flag = " + arg + "\n"; - } + out_line += " implib_flag = " + pe_link_flag( + plan, sepLinker, "/IMPLIB:", "-Wl,--out-implib,", + escape_ninja_path(lu.importLibrary)) + "\n"; + } + if (!lu.defFile.empty()) { + // `/DEF:` on both sides: it is a link.exe/lld-link flag, and the + // only targets that reach here are MSVC-ABI ones (MinGW auto-exports + // and gets no def edge at all). + out_line += " def_flag = " + pe_link_flag( + plan, sepLinker, "/DEF:", "/DEF:", + escape_ninja_path(lu.defFile)) + "\n"; } { // Per-unit C++ runtime link, by ROLE. The kind→role map is the diff --git a/src/build/plan.cppm b/src/build/plan.cppm index e050da48..e21a8779 100644 --- a/src/build/plan.cppm +++ b/src/build/plan.cppm @@ -100,6 +100,13 @@ struct LinkUnit { // producer; without that, ninja reports "no known rule to make it" naming a // file the link command does in fact write. std::filesystem::path importLibrary; // relative to plan.outputDir + // The generated module-definition file, on the MSVC ABI only. + // + // MSVC exports nothing from a DLL without `__declspec(dllexport)` or a + // `.def`, so without this the import library is empty and every consumer + // fails with unresolved externals for symbols that are visibly in the + // objects. MinGW's linker auto-exports and needs none of this. + std::filesystem::path defFile; // relative to plan.outputDir std::string soname; // ABI name for shared libraries std::vector runtimeAliases; // relative aliases, e.g. bin/libfoo.so.1 std::optional entryMain; // src path of main.cpp for bin @@ -1002,33 +1009,27 @@ make_plan(const mcpp::manifest::Manifest& manifest, return flag ? std::string(*flag) : std::string{}; }; - // WHAT `kind = "shared"` SUPPORTS, AND THE ONE THING IT STILL DOES NOT. + // `kind = "shared"` is supported on every format mcpp targets. // - // ELF, Mach-O and PE/MinGW are modelled: the import library PE consumers - // link is `import_library_for` above, and Mach-O's install name is set to - // `@rpath/` unconditionally (see shared_soname_flag) rather than - // defaulting to this machine's build path. + // ELF gets its SONAME, Mach-O an `@rpath/` install name, PE an import + // library — and on the MSVC ABI, a GENERATED `.def`. // - // PE/MSVC is refused, and the reason is not the linker — `link /DLL - // /IMPLIB:` has been in the rule table all along. It is symbol export: MSVC - // exports nothing from a DLL without `__declspec(dllexport)` or a `.def`, - // so the import library comes out EMPTY and the consumer fails with - // unresolved externals naming symbols that are plainly in the object files. - // Producing that is worse than refusing, because the diagnostic points - // nowhere near the cause. + // The MSVC case is the one that used to be refused, and the reason was never + // the linker (`link /DLL /IMPLIB:` has been in the rule table all along). It + // is symbol export: MSVC exports nothing from a DLL without + // `__declspec(dllexport)` or a `.def`, so the import library came out EMPTY + // and consumers failed with unresolved externals naming symbols that are + // plainly in the objects. MinGW's linker auto-exports and hides the whole + // problem; lld-link's MSVC flavour does not, deliberately, because PE caps + // exports at 65535. // - // ⚠️ THE HOST FALLBACK BELOW IS BELT AND BRACES, NOT A FIX. An earlier - // version of this comment claimed the previous guard — - // `!targetTriple.empty() && os != "linux"` — was inert on native builds - // because targetTriple would be empty there. It is not: `tc.targetTriple` - // is filled from the compiler's own `-dumpmachine` (detect.cppm), a native - // Linux build records `x86_64-linux-gnu` in resolution.json, and - // `parse("x86_64-pc-windows-msvc")` skips the vendor segment and succeeds. - // So the old guard did refuse native macOS and native Windows, and this - // change ADDS support rather than closing a hole. The fallback stays - // because a target that cannot be parsed must not be silently read as - // "not windows". - { + // So mcpp writes the `.def` (`msvc_needs_def` below → a def-gen edge in the + // backend → `/DEF:`). Two limits survive that no tool can remove, and they + // are documented rather than discovered: a consumer still needs + // `__declspec(dllimport)` to read exported DATA, and a class whose vtable is + // referenced must be marked whole. Both are CMake's documented limits for + // the same mechanism, and the answer to both is annotation. + const bool msvcTarget = [&] { const std::string targetOs = targetTriple.empty() ? (mcpp::platform::is_macos ? "macos" : mcpp::platform::is_windows ? "windows" : "linux") @@ -1036,32 +1037,12 @@ make_plan(const mcpp::manifest::Manifest& manifest, // The ABI, from the toolchain rather than from `naming`: on a native // Windows build the host naming constants are the MSVC ones whatever the // toolchain is (lib_prefix is "" and static_lib_ext is ".lib" for mingw - // too), so asking `naming` would refuse MinGW as well. `is_msvc_target` + // too), so asking `naming` would treat MinGW as MSVC. `is_msvc_target` // reads the compiler's own -dumpmachine answer, which distinguishes them // and also covers clang driving the MSVC ABI — clang auto-exports no more // than link.exe does, so "which compiler binary" is the wrong question. - if (targetOs == "windows" && mcpp::toolchain::is_msvc_target(tc)) { - for (auto const& t : manifest.targets) { - if (t.kind != mcpp::manifest::Target::SharedLibrary) continue; - return std::unexpected(std::format( - "target '{}': kind = \"shared\" is not supported for the MSVC " - "ABI ({}).\n" - " MSVC exports nothing from a DLL unless the source says " - "`__declspec(dllexport)`\n" - " or a `.def` file lists the symbols, so the import library " - "would be empty and\n" - " every consumer would fail with unresolved externals naming " - "symbols that are\n" - " visibly present in the objects. mcpp refuses rather than " - "produce that.\n" - " Use kind = \"lib\" for this target, or build it for " - "*-windows-gnu (MinGW),\n" - " where the linker auto-exports.", - t.name, - targetTriple.empty() ? "native" : targetTriple.str())); - } - } - } + return targetOs == "windows" && mcpp::toolchain::is_msvc_target(tc); + }(); bool experimentalStd = false; if (auto stdCfg = mcpp::manifest::normalize_cpp_standard(manifest.package.standard)) { @@ -1629,6 +1610,8 @@ make_plan(const mcpp::manifest::Manifest& manifest, lu.kind = LinkUnit::SharedLibrary; lu.output = dep.output; lu.importLibrary = import_library_for(dep.target, naming); + if (msvcTarget && !lu.importLibrary.empty()) + lu.defFile = std::filesystem::path("bin") / (dep.target.name + ".def"); lu.soname = dep.target.soname; lu.runtimeAliases = runtime_aliases_for_target(dep.target, naming); lu.loaderTagFlag = loader_tag_flag(lu.kind); @@ -1657,6 +1640,10 @@ make_plan(const mcpp::manifest::Manifest& manifest, lu.kind = LinkUnit::SharedLibrary; lu.output = target_output(t, naming); lu.importLibrary = import_library_for(t, naming); + // MSVC only: MinGW's linker auto-exports, and generating a second + // source of truth for what a DLL exports is how the two disagree. + if (msvcTarget && !lu.importLibrary.empty()) + lu.defFile = std::filesystem::path("bin") / (t.name + ".def"); lu.soname = t.soname; lu.runtimeAliases = runtime_aliases_for_target(t, naming); } else if (t.kind == mcpp::manifest::Target::TestBinary) { diff --git a/src/build/prepare.cppm b/src/build/prepare.cppm index 6d16502d..08d2939b 100644 --- a/src/build/prepare.cppm +++ b/src/build/prepare.cppm @@ -210,13 +210,53 @@ materialize_generated_files(const std::filesystem::path& root, void merge_conditional_config(mcpp::manifest::Manifest& m, const cfgpred::Ctx& ctx) { + // A DISTRIBUTION package may carry a leg's link line twice: as `ldflags` + // (GNU spelling, which is all an older mcpp reads) and as the neutral + // `[target..runtime]` pair, which mcpp renders per dialect. Applying + // both would put `-L` on a native `cl.exe` command line, which is exactly + // what the neutral form exists to avoid — so where the neutral form is + // present it REPLACES the ldflags rather than adding to them. + // + // Scoped to distribution packages on purpose: a hand-written manifest that + // states both may well mean both (`ldflags` also carries things like + // `-Wl,--as-needed`), and silently dropping half of it would be its own + // silent failure. + const bool generatedPackage = mcpp::pack::is_distribution_package(m); + for (auto const& cc : m.conditionalConfigs) { if (!cfgpred::matches(cc.predicate, ctx)) continue; + const bool neutralWins = generatedPackage + && (!cc.linkLibraryDirs.empty() || !cc.libraries.empty()); // One append() for every field the axis may carry (#258). Matching // sections land AFTER the base entries, so a conditional rule beats // a broader unconditional one under GNU last-wins — which is what // makes an off-OS REMOVAL expressible (`-U` after the base `-D`). - mcpp::manifest::append(m.buildConfig, cc.inputs); + if (neutralWins) { + // ⚠️ Drop the LIBRARY REFERENCES, not the whole ldflags list. + // + // Clearing it outright was a measured regression: a PE/MinGW shared + // leg's ldflags also carry `-Wl,-Bdynamic`, without which `-static` + // leaves ld in static-only mode and it refuses the import library + // with `have you installed the static version of the mathkit + // library?`. e2e 257 caught it. + // + // The neutral form replaces exactly what it can express — a library + // and where to find it. Anything else in that block says something + // it cannot say, and must survive. + auto inputs = cc.inputs; + std::erase_if(inputs.ldflags, [](std::string_view f) { + return f.starts_with("-L") || f.starts_with("-l") + || f.starts_with("/LIBPATH:"); + }); + mcpp::manifest::append(m.buildConfig, inputs); + } else { + mcpp::manifest::append(m.buildConfig, cc.inputs); + } + // The neutral half goes where `render_link_intent_flags` will find it. + for (auto const& d : cc.linkLibraryDirs) + m.runtimeConfig.linkIntent.linkLibraryDirs.push_back(d); + for (auto const& l : cc.libraries) + m.runtimeConfig.linkIntent.libraries.push_back(l); // `modules.sources` is the scanner's own view and is not part of // BuildInputs, so conditional sources are mirrored into it here. for (auto const& s : cc.inputs.sources) diff --git a/src/cli.cppm b/src/cli.cppm index 9f98060e..8d4fd079 100644 --- a/src/cli.cppm +++ b/src/cli.cppm @@ -652,6 +652,11 @@ int run(int argc, char** argv) { .option(cl::Option("verify").takes_value().value_name("MODE") .help("Already-staged check: size (default) | content")) .action(wrap_rc(cmd_stage))) + .subcommand(cl::App("coff-def") + .description("(internal: invoked by ninja) Write a .def of every exportable symbol in the given COFF objects") + .option(cl::Option("output").takes_value().value_name("PATH").help("the .def to write")) + .option(cl::Option("name").takes_value().value_name("DLL").help("LIBRARY name recorded in the .def")) + .action(wrap_rc(cmd_coff_def))) .subcommand(cl::App("bmi-equal") .description("(internal: invoked by ninja) Compare two BMIs ignoring the compiler's embedded timestamp") .action(wrap_rc(cmd_bmi_equal))) @@ -752,7 +757,7 @@ int run(int argc, char** argv) { "new", "build", "run", "test", "clean", "add", "remove", "update", "search", "publish", "pack", "emit", "xpkg", "toolchain", "cache", "index", "self", "explain", - "version", "dyndep", "why", "resolve", "stage", "bmi-equal", + "version", "dyndep", "why", "resolve", "stage", "bmi-equal", "coff-def", "bmi-compile", "bmi-supervise", "bmi-await", }); bool ok = false; diff --git a/src/cli/cmd_build.cppm b/src/cli/cmd_build.cppm index 09be73c8..380a8db4 100644 --- a/src/cli/cmd_build.cppm +++ b/src/cli/cmd_build.cppm @@ -13,6 +13,7 @@ import mcpplibs.cmdline; import mcpp.build.prepare; import mcpp.build.execute; import mcpp.build.configure; +import mcpp.build.coff_exports; import mcpp.build.stage; import mcpp.build.schedule.detach_codegen; import mcpp.build.test_targets; @@ -483,6 +484,80 @@ export int cmd_bmi_equal(const mcpplibs::cmdline::ParsedArgs& parsed) { return same ? 0 : 1; } +// `mcpp coff-def --output --name ...` — the auto-export edge. +// +// A subcommand rather than a shell fragment, for the reason bmi-equal is one: +// a generated ninja command written in POSIX shell is skipped entirely on +// Windows, which is the only platform this edge exists for. It also keeps the +// COFF reader testable as a pure function over bytes (tests/unit/test_coff_exports) +// instead of as whatever a command line happened to produce. +export int cmd_coff_def(const mcpplibs::cmdline::ParsedArgs& parsed) { + std::filesystem::path out; + if (auto v = parsed.value("output")) out = *v; + if (out.empty()) { + std::println(stderr, "error: coff-def requires --output"); + return 2; + } + std::string libName; + if (auto v = parsed.value("name")) libName = *v; + + std::vector all; + // When the AUTHOR has annotated the surface, the annotation wins and this + // edge writes an empty EXPORTS section — the linker then takes its export + // set from the objects' own `/EXPORT:` directives, exactly as if mcpp were + // not here. Adding a generated list on top would export the same names twice + // (LNK4197) and, worse, would export everything else as well, replacing a + // chosen public surface with all of it. + bool annotated = false; + for (std::size_t i = 0; i < parsed.positional_count(); ++i) { + const std::filesystem::path obj{ parsed.positional(i) }; + std::ifstream in(obj, std::ios::binary); + if (!in) { + std::println(stderr, "error: cannot read object '{}'", obj.string()); + return 1; + } + std::vector bytes; + for (char c; in.get(c); ) bytes.push_back(static_cast(c)); + if (mcpp::build::coff::declares_exports(bytes)) annotated = true; + auto syms = mcpp::build::coff::read_exports(bytes); + if (!syms) { + // Named with the object, because "which one" is the whole question + // when one file out of two hundred is the problem. + std::println(stderr, "error: {}: {}", obj.string(), syms.error()); + return 1; + } + all.insert(all.end(), syms->begin(), syms->end()); + } + + // Refused, not truncated. A `.def` cut at the ceiling links cleanly and + // then fails at whichever consumer happens to need a symbol that fell off + // the end — a diagnostic with no path back to this decision. + if (annotated) all.clear(); + std::ranges::sort(all); + all.erase(std::ranges::unique(all).begin(), all.end()); + if (all.size() > mcpp::build::coff::kMaxExports) { + std::println(stderr, + "error: {} exportable symbols, and PE addresses exports by 16-bit " + "ordinal (max {}).\n" + " Auto-export cannot express this library. Mark its public surface " + "with __declspec(dllexport)\n" + " and the export set becomes what you declared instead of " + "everything.", + all.size(), mcpp::build::coff::kMaxExports); + return 1; + } + + std::error_code ec; + if (out.has_parent_path()) std::filesystem::create_directories(out.parent_path(), ec); + std::ofstream o(out, std::ios::binary | std::ios::trunc); + if (!o) { + std::println(stderr, "error: cannot write '{}'", out.string()); + return 1; + } + o << mcpp::build::coff::write_def(libName, std::move(all)); + return o.good() ? 0 : 1; +} + // The three edges of the DetachCodegen shape. They are `mcpp` subcommands // rather than shell fragments for two reasons: the previous BMI-equivalence // logic lived in the generated ninja command as POSIX shell and was therefore diff --git a/src/manifest/toml.cppm b/src/manifest/toml.cppm index b73deaab..9198877c 100644 --- a/src/manifest/toml.cppm +++ b/src/manifest/toml.cppm @@ -35,6 +35,18 @@ std::expected upsert_dependency_text(std::string_view source, const DependencyTextEdit& edit); +// Every path the lib-root convention would accept, in extension-table order +// (`.cppm` first, then whatever `[build] module_extensions` declares). An +// explicit `[lib] path` collapses this to that one entry. +std::vector lib_root_candidates(const Manifest& manifest); + +// The lib root that EXISTS under `projectRoot`. `mcpp.manifest.types` has the +// non-probing form, which answers with the conventional NAME and is the right +// one for a diagnostic; this is the right one for "which file is actually +// there", and a project whose interfaces are `.ixx` needs it. +std::filesystem::path resolve_lib_root_path(const Manifest& manifest, + const std::filesystem::path& projectRoot); + } // namespace mcpp::manifest namespace mcpp::manifest { @@ -1444,6 +1456,18 @@ std::expected parse_string(std::string_view content, // prepare_build. ConditionalConfig cc; cc.predicate = triple; + // `[target..runtime]` — the dialect-neutral link intent. Two + // keys only, and the same two `[runtime]` already has at the top + // level: this makes them per-target, it does not invent a vocabulary. + if (auto rit = body.find("runtime"); rit != body.end() && rit->second.is_table()) { + auto& rt = rit->second.as_table(); + if (auto f = rt.find("link_library_dirs"); f != rt.end() && f->second.is_array()) + for (auto& v : f->second.as_array()) + if (v.is_string()) cc.linkLibraryDirs.emplace_back(v.as_string()); + if (auto f = rt.find("libraries"); f != rt.end() && f->second.is_array()) + for (auto& v : f->second.as_array()) + if (v.is_string()) cc.libraries.push_back(v.as_string()); + } if (auto bit = body.find("build"); bit != body.end() && bit->second.is_table()) { auto& bt = bit->second.as_table(); auto read_list = [&](const char* key, std::vector& out) { @@ -2026,4 +2050,64 @@ upsert_dependency_text(std::string_view source, return text; } + +// ── the lib root that actually exists ──────────────────────────────────── +// +// ⚠️ LIVES HERE, NOT IN mcpp.manifest.types, AND THE REASON IS MEASURED. +// Probing needs the extension table (`mcpp.source_kind`), which this module +// already imports and `types` does not. Adding that import to `types` — a +// module nearly everything depends on — made GCC 16.1 ICE while compiling +// `src/main.cpp`, a file unrelated to the change, and clearing gcm.cache did +// not help. That is the module-poisoning shape this project has hit before: a +// NEW edge into a low-level module whose interface carries std types. So the +// edge is not added; the function moves to where the edge already is. + +std::vector lib_root_candidates(const Manifest& manifest) { + if (!manifest.lib.path.empty()) return { manifest.lib.path }; + + // Convention: `src/.` — ONE + // candidate per DECLARED extension, not just `.cppm`. + // + // A project whose interfaces are `.ixx` says so in + // `[build] module_extensions`, and the convention used to look only for + // `src/.cppm`, a file that does not exist there. Measured on such a + // package: + // + // $ mcpp pack mathkit + // Interface (headers only) ← the module interface, gone + // Withheld (nothing) + // Packed …-x86_64-linux-gnu ← the C-SURFACE tag + // + // Both halves silently wrong: nothing publishes the interface, so no + // consumer can `import mathkit`; and an empty published set is exactly how + // the packer recognises a C surface, so the package also stops constraining + // the C++ ABI and the compatibility gate stops checking compiler and + // stdlib. `mcpp pack` has to follow the project's extension choice on its + // own — that is what makes `module_extensions` a knob rather than a knob + // plus a second thing to remember. + // + // `extension_table_for` keeps `.cppm` first, so a project that has both + // keeps today's answer. + std::string tail = manifest.package.name; + if (auto p = tail.rfind('.'); p != std::string::npos) tail = tail.substr(p + 1); + + const auto table = mcpp::extension_table_for(manifest.buildConfig.moduleExtensions); + std::vector out; + out.reserve(table.moduleInterface.size()); + for (auto const& ext : table.moduleInterface) + out.push_back(std::filesystem::path("src") / (tail + ext)); + return out; +} + +std::filesystem::path resolve_lib_root_path(const Manifest& manifest, + const std::filesystem::path& projectRoot) { + auto candidates = lib_root_candidates(manifest); + std::error_code ec; + for (auto const& rel : candidates) + if (std::filesystem::is_regular_file(projectRoot / rel, ec)) return rel; + // None on disk: hand back the conventional first candidate so the caller's + // diagnostic names the file it expected rather than nothing at all. + return candidates.front(); +} + } // namespace mcpp::manifest diff --git a/src/manifest/types.cppm b/src/manifest/types.cppm index 6240cfe7..bde125f6 100644 --- a/src/manifest/types.cppm +++ b/src/manifest/types.cppm @@ -702,6 +702,17 @@ struct ConditionalConfig { // `!`-exclusion globs work there too (the scanner handles positive+ // negative sets). BuildInputs inputs; + // `[target..runtime]` — the DIALECT-NEUTRAL half of a link line. + // + // `inputs.ldflags` above is spelled the GNU way and native `cl.exe` rejects + // `-L`. These two keys say the same thing without committing to a spelling, + // and `render_link_intent_flags` renders them as `/LIBPATH:` + `.lib` or + // `-L` + `-l` depending on the target. A generated package carries BOTH, + // because an older mcpp reads only the first — see the note where they are + // merged for why the newer client must then IGNORE the ldflags rather than + // add to them. + std::vector linkLibraryDirs; + std::vector libraries; // Conditional dependencies (Phase 1b): merged into the corresponding // manifest maps in prepare_build when the predicate matches the resolved // target — before dependency resolution, so they resolve like any dep. @@ -918,6 +929,12 @@ std::vector dialect_flags(const BuildConfig& bc); // libstdc++/libc++ headers declare, or participates in BMI dialect checks). bool is_dialect_flag(std::string_view flag); +// The lib root's CONVENTIONAL name: `src/.cppm`, or `[lib] path` +// when the manifest states one. It does not touch the filesystem, so it is the +// right answer for a diagnostic or a validator's expectation and the wrong one +// for "which file is actually there" — `mcpp.manifest.toml` owns the probing +// form, because probing needs the extension table and this module deliberately +// does not import it (see the note there). std::filesystem::path resolve_lib_root_path(const Manifest& manifest); // True if the manifest declares at least one `kind = "lib"` target. @@ -1094,14 +1111,9 @@ bool has_lib_target(const Manifest& manifest) { } std::filesystem::path resolve_lib_root_path(const Manifest& manifest) { - if (!manifest.lib.path.empty()) { - return manifest.lib.path; - } - // Convention: src/.cppm + if (!manifest.lib.path.empty()) return manifest.lib.path; std::string tail = manifest.package.name; - if (auto p = tail.rfind('.'); p != std::string::npos) { - tail = tail.substr(p + 1); - } + if (auto p = tail.rfind('.'); p != std::string::npos) tail = tail.substr(p + 1); return std::filesystem::path("src") / (tail + ".cppm"); } diff --git a/src/modgraph/scanner.cppm b/src/modgraph/scanner.cppm index 90d929fe..2b64fb7a 100644 --- a/src/modgraph/scanner.cppm +++ b/src/modgraph/scanner.cppm @@ -574,6 +574,45 @@ std::expected scan_file(const std::filesystem::path& file // `u.kind`; nothing re-derives it from the extension. u.kind = mcpp::classify(file, extTable); + // ⚠️ A file `sources` matched but the classifier cannot place. + // + // This used to be accepted, and what it produced was a compile edge whose + // object NOTHING LINKS. Measured with an `.ixx` that no + // `[build] module_extensions` declared: + // + // build obj/mathkit.ixx.o | gcm.cache/mathkit.gcm : cxx_object … + // bmi_out = gcm.cache/mathkit.gcm ← the BMI was produced + // build bin/app : cxx_link obj/main.o ← the object is not here + // + // ld: undefined reference to `mk::answer@mathkit()' + // + // Two answers to "is this a module interface" and only one of them read: + // the SCANNER sees `export module mathkit;` and records `provides`, which + // is why the edge got a `bmi_out`; the CLASSIFIER says `Other`, and the + // link set is built from the classifier. So the author is told about a + // symbol, when what happened is a missing line of configuration. + // + // Refused here rather than repaired downstream: the extension set is the + // project's to declare, and mcpp guessing which unknown extension "must + // have meant" a module interface would be a second answer to the same + // question — the very thing that produced this defect. + if (u.kind == mcpp::SourceKind::Other) { + return std::unexpected(ScanError{ file, 0, std::format( + "'{}' is listed in [build] sources, and mcpp has no role for the " + "extension '{}'.\n" + " Its object would be compiled and then linked by nothing, so this " + "is refused rather\n" + " than built. If it is a module interface, declare the extension:\n" + " [build]\n" + " module_extensions = [\"{}\"]\n" + " Otherwise remove it from `sources` — headers belong in " + "`include_dirs`, and\n" + " Windows resource scripts in `[resources]`.", + file.filename().string(), + file.extension().string().empty() ? "(none)" : file.extension().string(), + file.extension().string().empty() ? ".ixx" : file.extension().string()) }); + } + // C-like files are not C++ modules: they cannot legally contain `module` / `import` // declarations, and we route them to the C-language compile rule (no // P1689 scan, no BMI lookups). Skip the line-by-line module scan to diff --git a/src/pack/library.cppm b/src/pack/library.cppm index a4b85812..4260b3bd 100644 --- a/src/pack/library.cppm +++ b/src/pack/library.cppm @@ -38,6 +38,7 @@ export module mcpp.pack.library; import std; import mcpp.pack.digest; import mcpp.pack.manifest_emit; +import mcpp.source_kind; // builtin_extension_table — what needs no declaring import mcpp.pack.zip; import mcpp.platform; @@ -350,6 +351,20 @@ run_library_pack(const LibraryPackPlan& plan) doc.version = plan.packageVersion; doc.builtBy = plan.builtBy; doc.interfaceFiles = interfaceNames; + // Whatever the published set uses beyond the built-in `.cppm`. + // Computed from the FILES, so it cannot disagree with `sources` above. + { + const auto builtin = mcpp::builtin_extension_table(); + std::set extras; + for (auto const& src : plan.interfaceSources) { + auto ext = src.extension().string(); + if (ext.empty()) continue; + if (std::ranges::find(builtin.moduleInterface, ext) + != builtin.moduleInterface.end()) continue; + extras.insert(ext); + } + doc.moduleExtensions.assign(extras.begin(), extras.end()); + } doc.hasIncludeDir = std::filesystem::is_directory(plan.stagingRoot / "include", ec); doc.interfaceDigest = plan.interfaceSources.empty() && !doc.hasIncludeDir ? std::string{} diff --git a/src/pack/library_pipeline.cppm b/src/pack/library_pipeline.cppm index 14d89e83..844adfef 100644 --- a/src/pack/library_pipeline.cppm +++ b/src/pack/library_pipeline.cppm @@ -14,6 +14,13 @@ // // Design: .agents/docs/2026-08-17-library-distribution-design.md §2. +module; +// `stderr` / `fputs`: the packer prints the compiler's own diagnostics, and a +// build failure that arrives as three words is a build failure nobody can act +// on. This global module fragment was removed once as an unused ``; +// it has a use now. +#include + export module mcpp.pack.library_pipeline; import std; @@ -158,6 +165,14 @@ export int build_and_pack_library(const std::string& targetName, auto be = mcpp::build::make_ninja_backend(); mcpp::build::BuildOptions bo; if (auto br = be->build(ctx->plan, bo); !br) { + // The compiler's own output, not just "build failed". `mcpp build` + // has always printed this; `mcpp pack` dropped it, so a failure + // inside the packer's build arrived as three words and CI logs had + // nothing to go on. + if (!br.error().diagnosticOutput.empty()) { + std::fputs(br.error().diagnosticOutput.c_str(), stderr); + if (br.error().diagnosticOutput.back() != '\n') std::fputs("\n", stderr); + } mcpp::ui::error(br.error().message); return 1; } @@ -180,7 +195,13 @@ export int build_and_pack_library(const std::string& targetName, } // ── the interface closure ───────────────────────────────────── - auto libRoot = ctx->projectRoot / mcpp::manifest::resolve_lib_root_path(ctx->manifest); + // PROBING form: the convention is `src/.`, and + // which extension that is belongs to the project. A package whose + // interfaces are `.ixx` used to resolve to a `src/.cppm` that does + // not exist, and the packer then published nothing and tagged the + // result as a C surface — both silently. + auto libRoot = ctx->projectRoot + / mcpp::manifest::resolve_lib_root_path(ctx->manifest, ctx->projectRoot); std::error_code ec; InterfaceClosure here; std::string qualifiedPackage; diff --git a/src/pack/manifest_emit.cppm b/src/pack/manifest_emit.cppm index e2c858a6..622462c3 100644 --- a/src/pack/manifest_emit.cppm +++ b/src/pack/manifest_emit.cppm @@ -68,6 +68,16 @@ struct PackageDoc { // `sources = []` says exactly that (an omitted key would be filled with // the default glob and would sweep up whatever sits under src/). std::vector interfaceFiles; + // Module-interface extensions the PUBLISHED SET uses beyond the built-in + // `.cppm` — emitted as `[build] module_extensions`. + // + // A function of what is in the package, not a copy of what the producer + // declared: the packer publishes a computed subset, so the extensions it + // needs are computed from that subset too. Without this the consumer is + // handed `sources = ["interface/mathkit.ixx"]` and no way to know what an + // `.ixx` is, which is the producer remembering to configure something the + // package could state for itself. + std::vector moduleExtensions; bool hasIncludeDir = false; std::string interfaceDigest; // over the ordered interface set @@ -168,6 +178,12 @@ std::string emit_package_manifest(const PackageDoc& doc) { // ── the interface ────────────────────────────────────────────────── o += "[build]\n"; o += std::format("sources = [{}]\n", join_quoted(doc.interfaceFiles)); + // Emitted only when the published set actually uses one, so a `.cppm` + // package's manifest is byte-identical to before. A package whose interface + // is `.ixx` states that itself rather than requiring the consumer to have + // guessed the producer's convention. + if (!doc.moduleExtensions.empty()) + o += std::format("module_extensions = [{}]\n", join_quoted(doc.moduleExtensions)); if (doc.hasIncludeDir) o += "include_dirs = [\"include\"]\n"; if (!doc.cxxRuntime.empty()) o += std::format("cxx_runtime = {}\n", quote(doc.cxxRuntime)); @@ -226,6 +242,33 @@ std::string emit_package_manifest(const PackageDoc& doc) { leg.triple, peGnuShared ? "\"-Wl,-Bdynamic\", " : "", leg.linkName); + + // …and, where it says the SAME thing, the dialect-neutral form. mcpp + // renders these as `/LIBPATH:` + `.lib` or `-L` + `-l` from the + // target, which is what lets a consumer driven by native `cl.exe` link + // this package at all — cl rejects `-L`. + // + // BOTH are emitted, deliberately. An older mcpp reads only the ldflags + // above and silently ignores this block (measured), so dropping the + // ldflags would leave every older client with no link line at all. A + // newer mcpp seeing this block drops that leg's library references and + // uses these instead — see merge_conditional_config. + // + // ⚠️ NOT FOR A PE/MinGW SHARED LEG, and this was measured rather than + // reasoned. That leg's line is `-L… -Wl,-Bdynamic -lmathkit`, and + // `-Wl,-Bdynamic` only works IMMEDIATELY BEFORE the `-l` it enables: + // mcpp gives PE executables `-static`, which leaves ld in static-only + // mode where it refuses an import library. The neutral form has no way + // to say "and switch link mode first", and rendering the two halves + // through different slots separates the flag from its argument — e2e + // 257 fails with `have you installed the static version of the mathkit + // library?`. So that one leg keeps the spelling that works, and cl.exe + // never sees it: a PE/GNU leg is not an MSVC-ABI leg. + if (!peGnuShared) { + o += std::format("[target.'{}'.runtime]\n", cfg_predicate_for(leg.triple)); + o += std::format("link_library_dirs = [\"lib/{}\"]\n", leg.triple); + o += std::format("libraries = [\"{}\"]\n\n", leg.linkName); + } } // A shared library has to be FOUND at run time as well as linked, and the // two are different search paths — `link_library_dirs` is not rpath. diff --git a/src/pack/pipeline.cppm b/src/pack/pipeline.cppm index 2f8ae739..ab21b1da 100644 --- a/src/pack/pipeline.cppm +++ b/src/pack/pipeline.cppm @@ -86,6 +86,12 @@ export int build_and_pack(Options opts, bool modeFromUser, mcpp::build::BuildOptions bo; auto br = be->build(ctx->plan, bo); if (!br) { + // The compiler's own output, not just "build failed" — same reason as + // in the library pipeline. + if (!br.error().diagnosticOutput.empty()) { + std::fputs(br.error().diagnosticOutput.c_str(), stderr); + if (br.error().diagnosticOutput.back() != '\n') std::fputs("\n", stderr); + } mcpp::ui::error(br.error().message); return 1; } diff --git a/src/source_kind.cppm b/src/source_kind.cppm index 28992404..e59674ad 100644 --- a/src/source_kind.cppm +++ b/src/source_kind.cppm @@ -251,6 +251,22 @@ std::string normalize_extension(std::string_view raw) { } ExtensionTable builtin_extension_table() { + // `.cppm` alone, on purpose. `.ixx`, `.ccm`, `.cxxm` and anything else are + // the PROJECT's to declare via `[build] module_extensions` — the extension + // set is configuration, not a built-in list that mcpp grows one entry at a + // time as extensions come into fashion. + // + // What has to be true for that to be a real knob is that a DECLARED + // extension works everywhere without further help, and it does: a module + // interface's language is stated explicitly per toolchain + // (`BmiTraits::moduleInterfaceLangFlag` — `/interface /TP`, + // `-x c++-module`, `-x c++`) rather than inferred by the driver from the + // extension. Clang's driver not recognising `.ixx` is therefore beside the + // point: mcpp never lets it guess. + // + // What was NOT true, and is fixed in the scanner rather than here: an + // UNdeclared extension used to be accepted silently. See the classification + // check there for what that cost. return ExtensionTable{ .moduleInterface = { ".cppm" } }; } diff --git a/src/toolchain/dialect.cppm b/src/toolchain/dialect.cppm index 98d0b210..e7b5167b 100644 --- a/src/toolchain/dialect.cppm +++ b/src/toolchain/dialect.cppm @@ -107,15 +107,14 @@ struct CommandDialect { std::string_view archiveRemoveArg; // "d" needs no {} | "/REMOVE:{}" bool archiveRemoveTakesArchiveFirst = true; - // How the linker is told where to write a shared library's IMPORT LIBRARY — - // the archive of stubs a PE consumer links against, as opposed to the `.dll` - // the loader opens. `{}` is the path. + // ⚠️ THE IMPORT-LIBRARY AND `.def` SPELLINGS ARE NOT HERE, on purpose. // - // Emitted only when the TARGET has import libraries at all (PE); on ELF and - // Mach-O the shared library is its own link input and there is nothing to - // write. So this being non-empty in both rows is not a contradiction: the - // rows describe how to SAY it, and the target decides whether to. - std::string_view sharedImportLibArg; // "-Wl,--out-implib,{}" | "/IMPLIB:{}" + // They were, keyed on the dialect, and that is wrong in a way Windows CI + // demonstrated: clang targeting the MSVC ABI speaks the GNU DIALECT while + // driving LLD-LINK, so it was handed `-Wl,--out-implib,` and answered + // `lld-link: warning: ignoring unknown argument '--out-implib'`. The + // spelling follows the target ABI, which this table does not know — see + // ninja_backend's pe_link_flag. }; // Dialect lookup. GCC / Clang / MinGW → gnu; MSVC → msvc. @@ -219,10 +218,6 @@ constexpr CommandDialect kGnuDialect{ // `ar d ...` — one verb, then every member. .archiveRemoveArg = "d", .archiveRemoveTakesArchiveFirst = true, - // ld/lld: `--out-implib` is what makes a PE shared library linkable at all. - // Without it mingw writes only the .dll, consumers link the .dll directly, - // and that works — until the same package is consumed by any other linker. - .sharedImportLibArg = "-Wl,--out-implib,{}", }; // Native cl.exe. Unreachable in builds until the MSVC backend lands @@ -258,10 +253,6 @@ constexpr CommandDialect kMsvcDialect{ // reported with the command that produced it rather than swallowed. .archiveRemoveArg = "/REMOVE:{}", .archiveRemoveTakesArchiveFirst = false, - // link.exe writes one whether asked or not; naming it explicitly is how the - // path stays the one plan.cppm chose, instead of the linker's `$out`-derived - // guess (`foo.dll.lib`) that nothing else in mcpp agrees with. - .sharedImportLibArg = "/IMPLIB:{}", }; } // namespace diff --git a/src/version.cppm b/src/version.cppm index f7d3c7c4..884646df 100644 --- a/src/version.cppm +++ b/src/version.cppm @@ -31,6 +31,6 @@ import std; export namespace mcpp { -inline constexpr std::string_view MCPP_VERSION = "2026.8.18.1"; +inline constexpr std::string_view MCPP_VERSION = "2026.8.18.2"; } // namespace mcpp diff --git a/tests/e2e/251_pack_library_shared.sh b/tests/e2e/251_pack_library_shared.sh index 6eb0587d..26ca0288 100755 --- a/tests/e2e/251_pack_library_shared.sh +++ b/tests/e2e/251_pack_library_shared.sh @@ -7,17 +7,16 @@ # handled" from "there is no gate". Which half applies is now a property of the # TARGET, not of "is it Linux": # -# ELF, Mach-O, PE/MinGW produced — the package carries every name the -# platform needs to link it and to find it later -# PE/MSVC refused — MSVC exports nothing from a DLL without -# `__declspec(dllexport)`, so the import library would -# be empty and consumers would fail with unresolved -# externals for symbols visibly present in the objects +# ELF, Mach-O, PE/MinGW, PE/MSVC produced — the package carries every name +# the platform needs to link it and to find +# it later # -# Windows' default toolchain here is clang on the MSVC ABI, so this file sees the -# refusal there. 258 pins that refusal in detail, 259 pins Mach-O relocatability, -# 257 pins the PE/MinGW path; what this one adds is that the two outcomes are -# reachable from the same fixture and the same command. +# Every format mcpp targets now produces one. MSVC was the last holdout, and +# what it was missing was never the linker: it exports nothing from a DLL +# without `__declspec(dllexport)` or a `.def`, so mcpp generates the `.def` from +# the objects (258 pins that path in detail, 259 pins Mach-O relocatability, 257 +# the PE/MinGW one). What this file adds is that one fixture and one command +# produce a usable package on whichever platform it runs. # # A shared library is LINKED by `lib.so` and FOUND at run time by its # SONAME, and those are two different filenames. The first version of this @@ -58,37 +57,17 @@ EOF cd mathkit -# ── Windows: the MSVC ABI is refused, and says what to do instead ────── -if [[ "$(uname -s)" != "Linux" && "$(uname -s)" != "Darwin" ]]; then - if "$MCPP" pack mathkit-shared > refuse.log 2>&1; then - cat refuse.log - echo "FAIL: a shared library was packed for the MSVC ABI. Its import" - echo " library has no exports, so consumers fail with unresolved" - echo " externals naming symbols that are in the objects." - exit 1 - fi - grep -qi 'dllexport' refuse.log || { - cat refuse.log - echo "FAIL: it refused, but not for the export reason — 'shared libraries" - echo " are not supported here' does not tell the reader what to change." - exit 1; } - grep -qi 'windows-gnu' refuse.log || { - cat refuse.log - echo "FAIL: the refusal names no way forward. MinGW auto-exports." - exit 1; } - echo "PASS: a shared library package is refused on the MSVC ABI, with the reason" - exit 0 -fi - -# ── macOS: it is produced, and the deep claims live in 259 ────────────── -if [[ "$(uname -s)" == "Darwin" ]]; then +# ── Windows and macOS: it is produced; the deep claims live in 258 / 259 ── +if [[ "$(uname -s)" != "Linux" ]]; then "$MCPP" pack mathkit-shared > pack.log 2>&1 \ - || { cat pack.log; echo "FAIL: shared pack failed on Mach-O"; exit 1; } - macpkg="$(find target/dist -maxdepth 1 -type d -name 'mathkit-0.1.0-*' | head -1)" - [[ -n "$(find "$macpkg" -name 'libmathkit-shared.dylib' | head -1)" ]] || { - find "$macpkg" \( -type f -o -type l \) - echo "FAIL: no .dylib in the package"; exit 1; } - echo "PASS: a shared library package is produced on Mach-O" + || { cat pack.log; echo "FAIL: shared pack failed off ELF"; exit 1; } + nonelf="$(find target/dist -maxdepth 1 -type d -name 'mathkit-0.1.0-*' | head -1)" + # `.dylib` on macOS, `.dll` on Windows — asked for by shape rather than by + # `uname`, so the assertion is about the artifact and not about the host. + [[ -n "$(find "$nonelf" \( -name '*.dylib' -o -name '*.dll' \) | head -1)" ]] || { + find "$nonelf" \( -type f -o -type l \) + echo "FAIL: no shared library in the package"; exit 1; } + echo "PASS: a shared library package is produced off ELF too" exit 0 fi diff --git a/tests/e2e/257_shared_library_pe.sh b/tests/e2e/257_shared_library_pe.sh index 85e13937..925ef6a1 100755 --- a/tests/e2e/257_shared_library_pe.sh +++ b/tests/e2e/257_shared_library_pe.sh @@ -66,6 +66,21 @@ imp="$(find target -name 'libmathkit.dll.a' | head -1)" echo "FAIL: no import library. The .dll alone is a library only mingw's ld" echo " will link, so the package would be unusable everywhere else." exit 1; } +# ⚠️ And no `-fPIC`. PE code is position independent by design, and clang +# targeting the MSVC ABI REJECTS the flag — `unsupported option '-fPIC' for +# target 'x86_64-pc-windows-msvc'` — killing the build in clang-scan-deps before +# anything compiles. The condition used to be the DIALECT rather than the target, +# and Windows' default toolchain is clang, which speaks the GNU dialect while +# targeting MSVC. Asserted here rather than only on Windows because this runs on +# every Linux CI pass through mingw-cross, and the flag is equally meaningless +# for a PE target whichever compiler emits it. +nj_pic="$(find target -name build.ninja | head -1)" +grep -q '\-fPIC' "$nj_pic" && { + grep -n 'fPIC' "$nj_pic" | head -3 + echo "FAIL: -fPIC on a PE target. It means nothing here, and clang targeting" + echo " the MSVC ABI refuses it outright." + exit 1; } + # It is a declared output of the link edge, not a side effect ninja knows nothing # about — otherwise the consumer that links it has no producer and ninja stops # with 'no known rule to make it'. @@ -129,11 +144,11 @@ grep -q 'ok=42' <<<"$out" || { echo "$out"; echo "FAIL: wrong answer from the PE # ── 4. windows-msvc is refused here, but by the TARGET gate ───────────── # -# The MSVC shared-library refusal cannot be observed from a Linux host: this -# machine cannot serve `x86_64-windows-msvc` at all, so the target gate answers -# first and make_plan is never reached. Asserting `dllexport` here would be -# asserting a message this host cannot produce — the MSVC-ABI half lives in 258, -# under `# requires: msvc`. +# The MSVC-ABI shared library cannot be observed from a Linux host at all: this +# machine cannot serve `x86_64-windows-msvc`, so the target gate answers first +# and make_plan is never reached. That half lives in 258, under +# `# requires: msvc`, where a real link.exe can say whether the generated `.def` +# was accepted. # # What IS worth pinning here is that the refusal happens at all, and names the # host rather than silently building an ELF: that is precisely what this used to diff --git a/tests/e2e/258_shared_library_msvc_auto_def.sh b/tests/e2e/258_shared_library_msvc_auto_def.sh new file mode 100755 index 00000000..14203f0c --- /dev/null +++ b/tests/e2e/258_shared_library_msvc_auto_def.sh @@ -0,0 +1,160 @@ +#!/usr/bin/env bash +# requires: msvc +# 258_shared_library_msvc_auto_def.sh — a DLL on the MSVC ABI, exported without +# a single `__declspec(dllexport)` in the source. +# +# MSVC exports nothing from a DLL unless the source says `__declspec(dllexport)` +# or a `.def` lists the symbols. Without either, the import library comes out +# empty and every consumer fails with unresolved externals for symbols that are +# plainly in the object files — a diagnostic pointing nowhere near its cause. +# MinGW's linker auto-exports and hides the whole problem; lld-link's MSVC +# flavour does not, deliberately, because PE caps exports at 65535. +# +# So mcpp writes the `.def` from the objects, which is what CMake's +# `WINDOWS_EXPORT_ALL_SYMBOLS` has done since 3.4. The filtering rules are unit +# tested against synthetic and real COFF (tests/unit/test_coff_exports.cpp); what +# only a real link can answer is whether link.exe accepts the result and whether +# a consumer can then resolve a symbol through it. That is this test. +# +# ⚠️ THE TWO LIMITS ARE ASSERTED, NOT ASSUMED. Auto-export cannot make a data +# symbol readable without `__declspec(dllimport)` on the consumer's declaration, +# and CMake documents the same limit for the same mechanism. The test exercises a +# FUNCTION across the boundary, which is what auto-export does cover, and says so +# — a test that quietly used only functions would read as if the limit did not +# exist. +set -e +source "$(dirname "$0")/_host_path.sh" + +TMP=$(mktemp -d) +trap "rm -rf $TMP" EXIT +cd "$TMP" + +mkdir -p mathkit/src +cat > mathkit/src/mathkit.cppm <<'EOF' +export module mathkit; +export namespace mk { int answer(); } +EOF +# No dllexport anywhere. That is the point. +cat > mathkit/src/impl.cpp <<'EOF' +module mathkit; +namespace mk { int answer() { return 42; } } +EOF +cat > mathkit/mcpp.toml <<'EOF' +[package] +name = "mathkit" +version = "0.1.0" +[build] +sources = ["src/*.cppm", "src/*.cpp"] +[targets.mathkit] +kind = "shared" +[toolchain] +windows = "msvc@system" +EOF + +cd mathkit +"$MCPP" build > build.log 2>&1 || { + cat build.log + echo "FAIL: a kind=\"shared\" target did not build on the MSVC ABI." + echo " If the message mentions dllexport, the old refusal is still in place." + exit 1; } + +# ── the .def was generated, and it is a build-graph node ──────────────── +nj="$(find target -name build.ninja | head -1)" +grep -qE "^build .*mathkit\.def : coff_def " "$nj" || { + grep -n 'def' "$nj" | head + echo "FAIL: no coff_def edge. The .def has to be produced from the same" + echo " objects the link consumes, or the exported surface can drift" + echo " from what was compiled." + exit 1; } +def="$(find target -name 'mathkit.def' | head -1)" +[[ -n "$def" ]] || { echo "FAIL: no .def was written"; exit 1; } +grep -q '^EXPORTS' "$def" || { cat "$def"; echo "FAIL: malformed .def"; exit 1; } +# Non-empty is the criterion: an empty EXPORTS section links fine and produces +# exactly the import library this whole mechanism exists to avoid. +[[ "$(grep -cvE '^(LIBRARY|EXPORTS|$)' "$def")" -gt 0 ]] || { + cat "$def" + echo "FAIL: the .def exports nothing, so the import library is empty and" + echo " every consumer will fail with unresolved externals." + exit 1; } + +# ── the DLL and its import library both exist ─────────────────────────── +[[ -n "$(find target -name 'mathkit.dll' | head -1)" ]] || { echo "FAIL: no DLL"; exit 1; } +[[ -n "$(find target -name 'mathkit.lib' | head -1)" ]] || { + find target -name '*.lib' + echo "FAIL: no import library beside the DLL"; exit 1; } + +# ── and a consumer resolves a function through it ─────────────────────── +# +# A function, deliberately: auto-export covers code. Exported DATA additionally +# needs `__declspec(dllimport)` on the consumer's declaration — a limit of the +# mechanism, documented in docs/12, not a defect of this test. +"$MCPP" pack mathkit > pack.log 2>&1 || { cat pack.log; echo "FAIL: pack"; exit 1; } +pkg="$TMP/mathkit/$(find target/dist -maxdepth 1 -type d -name 'mathkit-0.1.0-*' | head -1)" +PKG_HOST="$(host_path "$pkg")" + +cd "$TMP" +mkdir -p app/src +cat > app/src/main.cpp <<'EOF' +#include +import mathkit; +int main(){ std::printf("msvc-dll=%d\n", mk::answer()); return 0; } +EOF +cat > app/mcpp.toml < run.log 2>&1 ) || { + cat app/run.log + echo "FAIL: the consumer could not link or start against the MSVC DLL." + echo " 'unresolved external symbol' here means the .def did not reach" + echo " the link, or exported the wrong names." + exit 1; } +grep -q 'msvc-dll=42' app/run.log || { cat app/run.log; echo "FAIL: wrong answer"; exit 1; } + +# ── and an ANNOTATED library keeps its own surface ────────────────────── +# +# `__declspec(dllexport)` writes `/EXPORT:` directives into the object, and a +# generated list on top would export the same names twice (LNK4197) AND export +# everything else besides — replacing a chosen public surface with all of it. +# So the annotation wins and the generated `.def` stays empty. +cd "$TMP" +mkdir -p annotated/src +cat > annotated/src/a.cppm <<'EOF' +export module annkit; +export namespace ak { __declspec(dllexport) int chosen(); } +EOF +cat > annotated/src/a.cpp <<'EOF' +module annkit; +namespace ak { __declspec(dllexport) int chosen() { return 7; } } +EOF +cat > annotated/mcpp.toml <<'EOF' +[package] +name = "annkit" +version = "0.1.0" +[build] +sources = ["src/*.cppm", "src/*.cpp"] +[targets.annkit] +kind = "shared" +[toolchain] +windows = "msvc@system" +EOF +( cd annotated && "$MCPP" build > build.log 2>&1 ) || { + cat annotated/build.log; echo "FAIL: the annotated library did not build"; exit 1; } +anndef="$(find annotated/target -name 'annkit.def' | head -1)" +[[ -n "$anndef" ]] || { echo "FAIL: no .def edge for the annotated library"; exit 1; } +[[ "$(grep -cvE '^(LIBRARY|EXPORTS|$)' "$anndef")" -eq 0 ]] || { + cat "$anndef" + echo "FAIL: mcpp generated an export list for a library that declares its own." + echo " That exports every other symbol too, which replaces the author's" + echo " chosen surface with all of it." + exit 1; } + +echo "PASS: MSVC exports without dllexport, and defers to dllexport when present" diff --git a/tests/e2e/258_shared_library_msvc_refused.sh b/tests/e2e/258_shared_library_msvc_refused.sh deleted file mode 100755 index 81f8bf73..00000000 --- a/tests/e2e/258_shared_library_msvc_refused.sh +++ /dev/null @@ -1,89 +0,0 @@ -#!/usr/bin/env bash -# requires: msvc -# 258_shared_library_msvc_refused.sh — `kind = "shared"` on the MSVC ABI is -# refused, and the message says why. -# -# The refusal is NOT about the linker. `link /DLL /IMPLIB:` has been in mcpp's -# rule table all along. It is about symbol export: MSVC exports nothing from a -# DLL unless the source says `__declspec(dllexport)` or a `.def` file lists the -# symbols. Without that the import library comes out EMPTY and every consumer -# fails with unresolved externals naming symbols that are plainly in the object -# files — a diagnostic pointing nowhere near its cause. Producing that is worse -# than refusing. -# -# ⚠️ WHY THIS TEST IS SEPARATE FROM 257. On a Linux host this cannot be observed: -# the machine cannot serve `x86_64-windows-msvc`, so the target-vocabulary gate -# answers first and make_plan is never reached. A `# requires:`-less version of -# this test would assert a message its host cannot produce. -# -# ⚠️ AND WHY IT PINS BOTH SIDES. `kind = "lib"` must still build in the same -# project with the same toolchain. Asserting only the refusal cannot distinguish -# "shared is refused" from "this project does not build at all". -set -e - -TMP=$(mktemp -d) -trap "rm -rf $TMP" EXIT -cd "$TMP" - -mkdir -p mathkit/src -cat > mathkit/src/mathkit.cppm <<'EOF' -export module mathkit; -export namespace mk { int answer(); } -EOF -cat > mathkit/src/impl.cpp <<'EOF' -module mathkit; -namespace mk { int answer() { return 42; } } -EOF - -manifest() { # $1 = kind - cat > "$TMP/mathkit/mcpp.toml" < shared.log 2>&1; then - echo "FAIL: a kind=\"shared\" target built for the MSVC ABI." - find target -name '*.dll' -o -name '*.lib' | head - echo " If the import library is empty, consumers fail with unresolved" - echo " externals for symbols that are visibly present in the objects." - exit 1 -fi -grep -qi 'dllexport' shared.log || { - cat shared.log - echo "FAIL: refused, but not by the shared-library gate — the message must" - echo " name symbol export, or the reader cannot tell what to do about it." - exit 1; } -# And it must point somewhere: a refusal with no way forward is a dead end. -grep -q 'windows-gnu' shared.log || { - cat shared.log - echo "FAIL: the refusal names no alternative. MinGW auto-exports, and that is" - echo " the answer for anyone who actually needs a DLL here." - exit 1; } - -# ── and the static form still builds, same project, same toolchain ────── -manifest lib -rm -rf target -"$MCPP" build > lib.log 2>&1 || { - cat lib.log - echo "FAIL: kind=\"lib\" does not build either, so the refusal above proves" - echo " nothing about shared libraries specifically." - exit 1; } -[[ -n "$(find target -name 'mathkit.lib' | head -1)" ]] || { - find target -type f | head - echo "FAIL: no mathkit.lib — the MSVC static path did not produce its artifact" - exit 1; } - -echo "PASS: MSVC refuses kind=\"shared\" for the export reason, and still builds kind=\"lib\"" diff --git a/tests/e2e/260_module_extension_is_configuration.sh b/tests/e2e/260_module_extension_is_configuration.sh new file mode 100755 index 00000000..9fa97053 --- /dev/null +++ b/tests/e2e/260_module_extension_is_configuration.sh @@ -0,0 +1,92 @@ +#!/usr/bin/env bash +# requires: +# (no capability: mcpp states a module interface's LANGUAGE explicitly per +# toolchain, so a declared extension behaves the same on all three.) +# +# 260_module_extension_is_configuration.sh — `[build] module_extensions` is the +# knob, and everything downstream follows it without a second declaration. +# +# `.ixx` is NOT built in, on purpose: the extension set is configuration, not a +# list mcpp grows one entry at a time as extensions come into fashion. What that +# demands in return is two things, and this test is both of them: +# +# 1. a DECLARED extension works with no further help; +# 2. an UNDECLARED one is refused where the mistake is, not four steps later. +# +# ⚠️ WHY (2) IS THE INTERESTING HALF. Before this, an undeclared `.ixx` matched +# by `sources` was accepted and produced a compile edge whose object NOTHING +# LINKS — measured: +# +# build obj/mathkit.ixx.o | gcm.cache/mathkit.gcm : cxx_object … +# bmi_out = gcm.cache/mathkit.gcm ← the BMI was produced +# build bin/app : cxx_link obj/main.o ← the object is not here +# +# ld: undefined reference to `mk::answer@mathkit()' +# +# Two answers to "is this a module interface" and only one of them read: the +# scanner sees `export module` and records `provides`, which is why the edge got +# a `bmi_out`; the classifier says `Other`, and the link set is built from the +# classifier. So the author was told about a symbol when what happened was a +# missing line of configuration. +set -e + +TMP=$(mktemp -d) +trap "rm -rf $TMP" EXIT +cd "$TMP" + +mkdir -p proj/src +cat > proj/src/mathkit.ixx <<'EOF' +export module mathkit; +export namespace mk { int answer() { return 42; } } +EOF +cat > proj/src/main.cpp <<'EOF' +#include +import mathkit; +int main(){ std::printf("ixx-ok=%d\n", mk::answer()); return 0; } +EOF + +manifest() { # $1 = extra [build] body + cat > "$TMP/proj/mcpp.toml" < undeclared.log 2>&1; then + echo "FAIL: an undeclared '.ixx' was accepted." + nj="$(find target -name build.ninja | head -1)" + grep -n 'mathkit' "$nj" | head + echo " Check whether its object is in the link edge — if it is not," + echo " this 'success' produces an undefined reference at link time." + exit 1 +fi +grep -q "no role for the extension '.ixx'" undeclared.log || { + cat undeclared.log + echo "FAIL: refused, but not for the classification reason. A message that" + echo " does not name the extension leaves the author with a symbol." + exit 1; } +grep -q "module_extensions" undeclared.log || { + cat undeclared.log + echo "FAIL: the message does not name the key that fixes it, so it is a" + echo " diagnosis without a remedy." + exit 1; } + +# ── 2. declared: works, on this toolchain, with nothing else ──────────── +manifest 'module_extensions = [".ixx"]' +rm -rf target +"$MCPP" run > declared.log 2>&1 || { cat declared.log; echo "FAIL: declared .ixx did not build"; exit 1; } +grep -q 'ixx-ok=42' declared.log || { cat declared.log; echo "FAIL: wrong answer"; exit 1; } + +echo "PASS: a declared module extension needs no further help; an undeclared one is refused" diff --git a/tests/e2e/261_pack_follows_the_module_extension.sh b/tests/e2e/261_pack_follows_the_module_extension.sh new file mode 100755 index 00000000..f8267eb9 --- /dev/null +++ b/tests/e2e/261_pack_follows_the_module_extension.sh @@ -0,0 +1,150 @@ +#!/usr/bin/env bash +# requires: +# (no capability: nothing here is toolchain-specific.) +# +# 261_pack_follows_the_module_extension.sh — `mcpp pack` follows the project's +# module extension on its own, and the package it writes states that extension +# for its consumers. +# +# The principle: a project says `module_extensions = [".ixx"]` ONCE. Packing it +# and consuming the package must both work with nothing further declared on +# either side. Anything else makes the knob a knob plus two things to remember. +# +# ⚠️ TWO DEFECTS SAT BEHIND THIS, and the first was silent in both directions. +# +# 1. The lib-root convention hard-coded `.cppm`: `src/.cppm`. For an +# `.ixx` project that file does not exist, so the closure started nowhere: +# +# $ mcpp pack mathkit +# Interface (headers only) ← the module interface, gone +# Withheld (nothing) +# Packed …-x86_64-linux-gnu ← the C-SURFACE tag +# +# Both halves wrong without a word. No consumer could `import mathkit`; and +# an empty published set is precisely how the packer recognises a C surface, +# so the package ALSO stopped constraining the C++ ABI and the compatibility +# gate stopped checking compiler and stdlib. A package that claims less than +# it should is the failure this whole design exists to prevent. +# +# 2. The generated manifest listed `sources = ["interface/mathkit.ixx"]` and did +# not say what an `.ixx` is, so the consumer was refused by the classifier. +# Now the package declares it — computed from the published FILES, so it +# cannot disagree with `sources`. +set -e +source "$(dirname "$0")/_host_path.sh" + +TMP=$(mktemp -d) +trap "rm -rf $TMP" EXIT +cd "$TMP" + +mkdir -p mathkit/src +cat > mathkit/src/mathkit.ixx <<'EOF' +export module mathkit; +export import :api; +EOF +cat > mathkit/src/api.ixx <<'EOF' +export module mathkit:api; +export namespace mk { int answer(); } +EOF +cat > mathkit/src/impl.cpp <<'EOF' +module mathkit; +namespace mk { int answer() { return 42; } } +EOF +cat > mathkit/mcpp.toml <<'EOF' +[package] +name = "mathkit" +version = "0.1.0" +[build] +sources = ["src/*.ixx", "src/*.cpp"] +module_extensions = [".ixx"] +[targets.mathkit] +kind = "lib" +EOF + +cd mathkit +"$MCPP" pack mathkit --format dir > pack.log 2>&1 || { cat pack.log; echo "pack failed"; exit 1; } +pkg="$TMP/mathkit/$(find target/dist -maxdepth 1 -type d -name 'mathkit-0.1.0-*' | head -1)" + +# ── the closure actually started ──────────────────────────────────────── +[[ -f "$pkg/interface/mathkit.ixx" && -f "$pkg/interface/api.ixx" ]] || { + find "$pkg" -type f + cat pack.log + echo "FAIL: the module interface was not published. 'Interface (headers only)'" + echo " in the log above means the lib root resolved to a file that does" + echo " not exist — the convention looked for .cppm." + exit 1; } +[[ ! -e "$pkg/interface/impl.cpp" ]] || { echo "FAIL: the implementation unit was published"; exit 1; } + +# ── and the package is tagged as the C++ surface it is ────────────────── +# +# Asserted separately because it fails INDEPENDENTLY: an empty published set is +# how a C surface is recognised, so losing the interface also silently downgrades +# the compatibility tag, and a package that constrains nothing is accepted by +# every toolchain. +grep -qE '^abi +=.*-c\+\+[0-9]+"' "$pkg/mcpp.toml" || { + grep -n 'abi' "$pkg/mcpp.toml" + echo "FAIL: the leg carries a C-surface tag. A package that publishes a C++" + echo " module interface constrains the C++ ABI, and this one now says" + echo " it does not — the gate will accept any compiler." + exit 1; } + +# ── the package tells consumers what an .ixx is ───────────────────────── +grep -q 'module_extensions' "$pkg/mcpp.toml" || { + cat "$pkg/mcpp.toml" + echo "FAIL: the generated manifest lists .ixx sources without declaring the" + echo " extension, so every consumer is refused by the classifier." + exit 1; } + +# ── a consumer that declares NOTHING builds and runs ──────────────────── +PKG_HOST="$(host_path "$pkg")" +cd "$TMP" +mkdir -p app/src +cat > app/src/main.cpp <<'EOF' +#include +import mathkit; +int main(){ std::printf("pack-ixx=%d\n", mk::answer()); return 0; } +EOF +cat > app/mcpp.toml < run.log 2>&1 ) || { + cat app/run.log + echo "FAIL: the consumer had to be told about .ixx. The package should carry it." + exit 1; } +grep -q 'pack-ixx=42' app/run.log || { cat app/run.log; echo "FAIL: wrong answer"; exit 1; } + +# ── a .cppm package's manifest is unchanged ───────────────────────────── +# +# The new key must appear only when the published set needs it, or every +# existing package's manifest changes shape for no reason. +cd "$TMP" +mkdir -p plain/src +cat > plain/src/plain.cppm <<'EOF' +export module plain; +export namespace pl { int v() { return 1; } } +EOF +cat > plain/mcpp.toml <<'EOF' +[package] +name = "plain" +version = "0.1.0" +[build] +sources = ["src/*.cppm"] +[targets.plain] +kind = "lib" +EOF +( cd plain && "$MCPP" pack plain --format dir > pack.log 2>&1 ) || { + cat plain/pack.log; echo "plain pack failed"; exit 1; } +plainpkg="$(find plain/target/dist -maxdepth 1 -type d -name 'plain-0.1.0-*' | head -1)" +grep -q 'module_extensions' "$plainpkg/mcpp.toml" && { + cat "$plainpkg/mcpp.toml" + echo "FAIL: a .cppm-only package gained a module_extensions key it does not need" + exit 1; } + +echo "PASS: pack follows the project's module extension, and the package carries it" diff --git a/tests/fixtures/coff/README.md b/tests/fixtures/coff/README.md new file mode 100644 index 00000000..4a2794f9 --- /dev/null +++ b/tests/fixtures/coff/README.md @@ -0,0 +1,33 @@ +# COFF fixtures + +`probe-amd64.obj` is a real amd64 COFF object, produced by mingw-cross GCC 16.1 +from `probe-amd64.cpp.in`: + +``` +x86_64-w64-mingw32-g++ -c probe-amd64.cpp.in -o probe-amd64.obj -O1 +``` + +It is committed rather than generated because the parser it exercises +(`mcpp.build.coff_exports`) must be correct on hosts that cannot produce COFF at +all — macOS CI has no mingw, and that is exactly where a byte-level reader is +most likely to be wrong and least likely to be noticed. + +The `.cpp.in` extension is not decoration: `mcpp test` discovers `tests/**/*.cpp` +and would compile a `.cpp` here into a test binary with no `main`. + +Its external defined symbols, per `x86_64-w64-mingw32-nm --defined-only +--extern-only`: + +``` +0000000000000000 R exported_const +0000000000000000 D exported_data +0000000000000000 T exported_fn +0000000000000004 T _Z3usev +000000000000000a T _ZN2ns10mangled_fnEi +``` + +`annotated-amd64.obj` is the same shape with one symbol marked +`__declspec(dllexport)`, from `annotated-amd64.cpp.in`. It exists so +`declares_exports` is tested against a real `.drectve` section rather than one +this repository wrote itself — the point of that function is to recognise what a +COMPILER emits. diff --git a/tests/fixtures/coff/annotated-amd64.cpp.in b/tests/fixtures/coff/annotated-amd64.cpp.in new file mode 100644 index 00000000..a361ae75 --- /dev/null +++ b/tests/fixtures/coff/annotated-amd64.cpp.in @@ -0,0 +1,2 @@ +extern "C" __declspec(dllexport) int chosen(int x) { return x + 1; } +extern "C" int not_chosen(int x) { return x - 1; } diff --git a/tests/fixtures/coff/annotated-amd64.obj b/tests/fixtures/coff/annotated-amd64.obj new file mode 100644 index 00000000..30de43b6 Binary files /dev/null and b/tests/fixtures/coff/annotated-amd64.obj differ diff --git a/tests/fixtures/coff/probe-amd64.cpp.in b/tests/fixtures/coff/probe-amd64.cpp.in new file mode 100644 index 00000000..28201598 --- /dev/null +++ b/tests/fixtures/coff/probe-amd64.cpp.in @@ -0,0 +1,7 @@ +extern "C" int exported_fn(int x) { return x + 1; } +extern "C" int exported_data = 7; +extern "C" const int exported_const = 9; +static int hidden_fn(int x) { return x - 1; } +static int hidden_data = 3; +int use() { return hidden_fn(hidden_data); } +namespace ns { int mangled_fn(int) { return 0; } } diff --git a/tests/fixtures/coff/probe-amd64.obj b/tests/fixtures/coff/probe-amd64.obj new file mode 100644 index 00000000..adb13f81 Binary files /dev/null and b/tests/fixtures/coff/probe-amd64.obj differ diff --git a/tests/unit/test_coff_exports.cpp b/tests/unit/test_coff_exports.cpp new file mode 100644 index 00000000..7694b69a --- /dev/null +++ b/tests/unit/test_coff_exports.cpp @@ -0,0 +1,334 @@ +#include + +import std; +import mcpp.build.coff_exports; + +using mcpp::build::coff::Export; +using mcpp::build::coff::read_exports; +using mcpp::build::coff::write_def; +using mcpp::build::coff::kMaxExports; +using mcpp::build::coff::declares_exports; + +// The `.def` generator decides what a DLL's public surface is. Getting the +// filter wrong does not fail loudly: too few exports and the consumer sees +// unresolved externals for symbols that are visibly in the objects; too many +// and the DLL's surface depends on which translation unit instantiated what. +// +// So both halves are tested here, on every host: +// +// * SYNTHETIC objects, built byte by byte, for the filter rules — storage +// class, section number, type, section characteristics, the skip list. +// Nothing else can vary a symbol's storage class on demand. +// * A REAL object (tests/fixtures/coff/probe-amd64.obj, from mingw-cross +// GCC 16.1), because a synthetic file only proves the reader agrees with +// the test's idea of COFF. macOS and Windows CI cannot produce one, which +// is precisely why it is committed rather than generated. + +namespace { + +// ── a minimal COFF writer, so a test can state its input exactly ────────── +struct Sym { + std::string name; // ≤ 8 chars: written inline + std::int16_t section = 1; + std::uint16_t type = 0x20; // function + std::uint8_t storage = 2; // IMAGE_SYM_CLASS_EXTERNAL + std::uint8_t aux = 0; +}; + +void put16(std::vector& b, std::size_t off, std::uint16_t v) { + b[off] = std::byte(v & 0xff); + b[off + 1] = std::byte((v >> 8) & 0xff); +} +void put32(std::vector& b, std::size_t off, std::uint32_t v) { + for (int i = 0; i < 4; ++i) b[off + i] = std::byte((v >> (8 * i)) & 0xff); +} + +// `sectionFlags` is 1-based to match how symbols address sections. +std::vector make_obj(std::vector syms, + std::vector sectionFlags = { 0x60000020u }, + std::uint16_t machine = 0x8664) +{ + constexpr std::size_t kHdr = 20, kSec = 40, kSym = 18; + const std::size_t nsec = sectionFlags.size(); + const std::size_t secOff = kHdr; + const std::size_t symOff = secOff + nsec * kSec; + + std::vector b(symOff + syms.size() * kSym + 4, std::byte{0}); + put16(b, 0, machine); + put16(b, 2, static_cast(nsec)); + put32(b, 8, static_cast(symOff)); + put32(b, 12, static_cast(syms.size())); + put16(b, 16, 0); // no optional header + + for (std::size_t i = 0; i < nsec; ++i) + put32(b, secOff + i * kSec + 36, sectionFlags[i]); + + for (std::size_t i = 0; i < syms.size(); ++i) { + const auto rec = symOff + i * kSym; + for (std::size_t c = 0; c < syms[i].name.size() && c < 8; ++c) + b[rec + c] = std::byte(syms[i].name[c]); + put16(b, rec + 12, static_cast(syms[i].section)); + put16(b, rec + 14, syms[i].type); + b[rec + 16] = std::byte(syms[i].storage); + b[rec + 17] = std::byte(syms[i].aux); + } + put32(b, b.size() - 4, 4); // empty string table + return b; +} + +constexpr std::uint32_t kText = 0x60000020u; // CODE | MEM_EXECUTE | MEM_READ +constexpr std::uint32_t kData = 0xC0000040u; // INITIALIZED | MEM_READ | MEM_WRITE +constexpr std::uint32_t kRdata = 0x40000040u; // INITIALIZED | MEM_READ + +std::vector names(const std::vector& v) { + std::vector out; + for (auto const& e : v) out.push_back(e.name); + std::ranges::sort(out); + return out; +} + +} // namespace + +// ─── what is exported ────────────────────────────────────────────────────── + +TEST(CoffExports, TakesExternalDefinedSymbols) { + auto obj = make_obj({ Sym{ .name = "f" } }); + auto r = read_exports(obj); + ASSERT_TRUE(r) << r.error(); + EXPECT_EQ(names(*r), (std::vector{ "f" })); +} + +TEST(CoffExports, SkipsStaticSymbols) { + // IMAGE_SYM_CLASS_STATIC (3). Exporting a TU-local symbol would put a name + // in the DLL's surface that the author never made public. + auto obj = make_obj({ Sym{ .name = "s", .storage = 3 } }); + auto r = read_exports(obj); + ASSERT_TRUE(r); + EXPECT_TRUE(r->empty()); +} + +TEST(CoffExports, SkipsUndefinedSymbols) { + // SectionNumber 0 is UNDEFINED — a symbol this object REFERENCES. Exporting + // it would have the DLL claim to provide what it is asking someone else for. + auto obj = make_obj({ Sym{ .name = "u", .section = 0 } }); + auto r = read_exports(obj); + ASSERT_TRUE(r); + EXPECT_TRUE(r->empty()); +} + +TEST(CoffExports, SkipsAbsoluteAndDebugSections) { + // -1 IMAGE_SYM_ABSOLUTE, -2 IMAGE_SYM_DEBUG: neither has an address to + // export. + auto obj = make_obj({ Sym{ .name = "a", .section = -1 }, + Sym{ .name = "d", .section = -2 } }); + auto r = read_exports(obj); + ASSERT_TRUE(r); + EXPECT_TRUE(r->empty()); +} + +// ─── DATA, which is the half that fails silently ─────────────────────────── + +TEST(CoffExports, AVariableIsMarkedDATA) { + // Without `DATA` the linker generates a call thunk for a variable and the + // consumer reads the thunk instead of the value. + auto obj = make_obj({ Sym{ .name = "v", .section = 2, .type = 0 } }, + { kText, kData }); + auto r = read_exports(obj); + ASSERT_TRUE(r); + ASSERT_EQ(r->size(), 1u); + EXPECT_TRUE((*r)[0].data); +} + +TEST(CoffExports, AConstInAReadOnlySectionIsStillDATA) { + // `.rdata` is not writable, so a rule keyed on MEM_WRITE alone calls this a + // function. It is a variable, and consumers read it as one. + auto obj = make_obj({ Sym{ .name = "c", .section = 2, .type = 0 } }, + { kText, kRdata }); + auto r = read_exports(obj); + ASSERT_TRUE(r); + ASSERT_EQ(r->size(), 1u); + EXPECT_TRUE((*r)[0].data); +} + +TEST(CoffExports, AFunctionIsNotDATA) { + auto obj = make_obj({ Sym{ .name = "f", .section = 1, .type = 0x20 } }, { kText }); + auto r = read_exports(obj); + ASSERT_TRUE(r); + ASSERT_EQ(r->size(), 1u); + EXPECT_FALSE((*r)[0].data); +} + +// ─── the skip list, each entry for its own reason ────────────────────────── + +TEST(CoffExports, SkipsDestructorThunksAndManagedArtefacts) { + auto obj = make_obj({ + Sym{ .name = "??_Gx" }, // scalar-deleting destructor + Sym{ .name = "??_Ex" }, // vector-deleting destructor + Sym{ .name = "a.b" }, // managed: '.' cannot appear in a C++ mangling + Sym{ .name = "keep" }, + }); + auto r = read_exports(obj); + ASSERT_TRUE(r); + EXPECT_EQ(names(*r), (std::vector{ "keep" })); +} + +TEST(CoffExports, SkipsAuxiliaryRecords) { + // An aux record follows its symbol and is NOT a symbol. Reading one as a + // symbol yields a name made of raw section data — which is how a `.def` + // ends up with entries no linker can resolve. + auto obj = make_obj({ Sym{ .name = "fn", .aux = 1 }, + Sym{ .name = "\x01\x02\x03" }, // the aux record + Sym{ .name = "after" } }); + auto r = read_exports(obj); + ASSERT_TRUE(r); + EXPECT_EQ(names(*r), (std::vector{ "after", "fn" })); +} + +TEST(CoffExports, StripsTheLeadingUnderscoreOnI386Only) { + auto i386 = make_obj({ Sym{ .name = "_f" } }, { kText }, 0x014c); + auto r32 = read_exports(i386); + ASSERT_TRUE(r32); + EXPECT_EQ(names(*r32), (std::vector{ "f" })); + + // On amd64 a leading underscore is part of the name. + auto amd64 = make_obj({ Sym{ .name = "_f" } }, { kText }, 0x8664); + auto r64 = read_exports(amd64); + ASSERT_TRUE(r64); + EXPECT_EQ(names(*r64), (std::vector{ "_f" })); +} + +// ─── refusals, because a partial answer here ships a broken DLL ──────────── + +TEST(CoffExports, RefusesAnUnsupportedMachine) { + auto obj = make_obj({ Sym{ .name = "f" } }, { kText }, 0x5032 /* RISC-V */); + auto r = read_exports(obj); + ASSERT_FALSE(r); + EXPECT_NE(r.error().find("machine"), std::string::npos); +} + +TEST(CoffExports, RefusesATruncatedSymbolTable) { + auto obj = make_obj({ Sym{ .name = "f" }, Sym{ .name = "g" } }); + obj.resize(obj.size() - 20); // cut into the symbol table + auto r = read_exports(obj); + ASSERT_FALSE(r) << "a truncated table must not read as a short one"; + EXPECT_NE(r.error().find("past the end"), std::string::npos); +} + +TEST(CoffExports, NamesBigobjRatherThanCallingItAnUnknownMachine) { + // A `/bigobj` object is a different container: machine 0 and 0xFFFF where + // the section count would be. Falling through to "unsupported machine + // 0x0000" is true and useless — it blames the reader for a flag the project + // passed. + auto obj = make_obj({ Sym{ .name = "f" } }); + obj[0] = std::byte{0}; obj[1] = std::byte{0}; + obj[2] = std::byte{0xFF}; obj[3] = std::byte{0xFF}; + auto r = read_exports(obj); + ASSERT_FALSE(r); + EXPECT_NE(r.error().find("bigobj"), std::string::npos); + // And it says what to do, or it is a dead end. + EXPECT_NE(r.error().find("dllexport"), std::string::npos); +} + +TEST(CoffExports, RefusesSomethingThatIsNotAnObject) { + std::vector tiny(4, std::byte{0}); + EXPECT_FALSE(read_exports(tiny)); +} + +// ─── the .def text ───────────────────────────────────────────────────────── + +TEST(CoffExports, DefIsSortedDeduplicatedAndMarksData) { + auto def = write_def("mathkit.dll", { + { "zeta", false }, { "alpha", true }, { "zeta", false } }); + EXPECT_EQ(def, + "LIBRARY mathkit.dll\n" + "EXPORTS\n" + " alpha DATA\n" + " zeta\n"); +} + +TEST(CoffExports, TheExportCeilingIsPesNotOurs) { + // Stated as a constant a reader can check against the format, not as a + // number chosen here: PE addresses exports by 16-bit ordinal. + EXPECT_EQ(kMaxExports, 65535u); +} + +// ─── a real object, from a real compiler ─────────────────────────────────── + +TEST(CoffExports, ReadsARealMingwObject) { + // Committed fixture: macOS and Windows CI cannot produce a COFF object, and + // a byte-level reader that is only ever fed its own test's output is exactly + // the kind that agrees with itself and with nothing else. + for (auto const& base : { "tests/fixtures/coff/probe-amd64.obj", + "../tests/fixtures/coff/probe-amd64.obj", + "../../tests/fixtures/coff/probe-amd64.obj" }) { + std::ifstream in(base, std::ios::binary); + if (!in) continue; + std::vector bytes; + for (char c; in.get(c); ) bytes.push_back(static_cast(c)); + + auto r = read_exports(bytes); + ASSERT_TRUE(r) << r.error(); + auto got = names(*r); + + // From `x86_64-w64-mingw32-nm --defined-only --extern-only` on the same + // object; see tests/fixtures/coff/README.md. + EXPECT_NE(std::ranges::find(got, "exported_fn"), got.end()); + EXPECT_NE(std::ranges::find(got, "exported_data"), got.end()); + EXPECT_NE(std::ranges::find(got, "exported_const"), got.end()); + EXPECT_NE(std::ranges::find(got, "_ZN2ns10mangled_fnEi"), got.end()); + // `static` at namespace scope is internal linkage: not in the surface. + EXPECT_EQ(std::ranges::find(got, "_ZL9hidden_fni"), got.end()); + + for (auto const& e : *r) { + if (e.name == "exported_data" || e.name == "exported_const") + EXPECT_TRUE(e.data) << e.name << " is a variable"; + if (e.name == "exported_fn") EXPECT_FALSE(e.data); + } + return; + } + GTEST_SKIP() << "fixture not reachable from this working directory"; +} + +// ─── annotation wins ─────────────────────────────────────────────────────── + +TEST(CoffExports, RecognisesAnObjectThatAlreadyDeclaresExports) { + // `__declspec(dllexport)` makes the compiler write `/EXPORT:` directives + // into `.drectve`. Generating a `.def` on top of that would export the same + // names twice (LNK4197) AND export everything else besides, replacing a + // chosen public surface with all of it. + // + // Tested against a real object because the whole point is recognising what + // a COMPILER emits; a hand-built `.drectve` would only prove this agrees + // with the test. + for (auto const& base : { "tests/fixtures/coff/annotated-amd64.obj", + "../tests/fixtures/coff/annotated-amd64.obj", + "../../tests/fixtures/coff/annotated-amd64.obj" }) { + std::ifstream in(base, std::ios::binary); + if (!in) continue; + std::vector bytes; + for (char c; in.get(c); ) bytes.push_back(static_cast(c)); + EXPECT_TRUE(declares_exports(bytes)); + return; + } + GTEST_SKIP() << "fixture not reachable from this working directory"; +} + +TEST(CoffExports, AnObjectWithNoDirectivesDeclaresNothing) { + // The negative half: without it, a `declares_exports` that always said true + // would pass the test above and silently disable auto-export everywhere. + for (auto const& base : { "tests/fixtures/coff/probe-amd64.obj", + "../tests/fixtures/coff/probe-amd64.obj", + "../../tests/fixtures/coff/probe-amd64.obj" }) { + std::ifstream in(base, std::ios::binary); + if (!in) continue; + std::vector bytes; + for (char c; in.get(c); ) bytes.push_back(static_cast(c)); + EXPECT_FALSE(declares_exports(bytes)); + return; + } + GTEST_SKIP() << "fixture not reachable from this working directory"; +} + +TEST(CoffExports, SyntheticObjectsHaveNoDirectiveSection) { + EXPECT_FALSE(declares_exports(make_obj({ Sym{ .name = "f" } }))); +}