X++: add extractor and QL library generated from the compiler's own AST

[Giulia Stocco (gfs)]

Why

CodeQL has no support for X++, the language behind Dynamics 365 Finance & Operations. The only existing option is running the C# extractor over compiled X++, which captures CIL semantics but loses everything X++-specific: select statements, ttsbegin blocks, tables, forms, chain-of-command. Those are exactly where the interesting security questions live.

This adds a first-class X++ extractor. The guiding decision is that the dbscheme and QL classes are generated 1:1 from the X++ compiler's own AST hierarchy, so there is no hand-maintained mapping to keep in sync and no second-guessing the first-party model.

About the file count

Most of this diff is generated and marked linguist-generated, so it collapses in review. For calibration, xpp/ql/lib is about 1000 files; rust/ql/lib is 1019 and swift/ql/lib is 1578. Per schema class we are slightly leaner than rust. The stubs cannot be dropped without patching misc/codegen, which is shared with rust and swift, because generated classes import XImpl::Impl and constructX.

The genuinely hand-written surface is about 20 files, all under xpp/extractor/** (excluding Generated/), plus xpp/schema/prelude.py, xpp/codegen.conf, xpp/codeql-extractor.yml and xpp/tools/.

One cleanup is still outstanding: xpp/ql/test/extractor-tests/ holds 209 empty MISSING_SOURCE.txt placeholders, one per node type, generated before any fixtures existed. codegen.conf no longer produces them, but the committed files still need deleting:

git rm -r xpp/ql/test/extractor-tests

Approach

Xlnt.XppCore.dll --reflect--> xpp/schema/ast.py --misc/codegen--> dbscheme + QL classes + stubs
                                     |
                                     +--> generated C# TRAP writer

AxClass/*.xml --> Pass1 --> Ast --> TRAP --> CodeQL database

Xpp.SchemaGenerator reflects over the 243 types deriving from Microsoft.Dynamics.AX.Metadata.XppCompiler.Ast, using MetadataLoadContext so no code from the proprietary compiler package is executed. The resulting schema/ast.py feeds misc/codegen, the same suite Swift and Rust use.

Extraction calls Pass1, the compiler's own parser, in process. There is no reimplemented grammar anywhere in this change.

Reconciling the CLR hierarchy with the dbscheme

Most of the interesting work was in the mismatches between what the compiler's class model expresses and what a dbscheme can:

• Tuples. The compiler uses CLR tuples for grouped values (a catch with its handler, a switch case with its body, a call argument with its by-ref flag). The schema has no tuple type, so the generator synthesizes a named class per tuple-valued property.
• Dictionaries of AST values are represented by their values, since the key is always the value's own name.
• Re-declared inherited properties are collapsed onto the ancestor, which would otherwise emit conflicting non-overriding QL predicates.
• HasX booleans that duplicate the hasX() predicate generated for an optional X are dropped.
• Instantiable base classes. A class that is both instantiable and extended gets only a union in the dbscheme, leaving instances of exactly that class with nowhere to bind. Seven X++ classes are in that position, so the generator emits an ...Internal leaf for each, following the Rust schema.
• Open generics cannot be named in generated code, so they are treated as abstract with members bound through dynamic.

Five properties remain unmapped, all reasonably so: two open generic parameters, two object-typed accessors that have typed equivalents on subclasses, and one System.Type.

The TRAP writer is checked against the dbscheme

The TRAP writer is generated from the reflected model rather than from schema/ast.py, because the schema keeps only snake_case names and loses the CLR binding needed to read a property back off a node. Relation names, though, come from inflection.tableize inside misc/codegen, so the generator reimplements those rules and then validates every relation it emits against the committed dbscheme. A divergence fails generation rather than silently producing a database with missing tuples.

That check earned its keep immediately, catching two naming bugs: plural rules applied in the wrong order, and predicate tables using underscore rather than tableize.

Verification

• codeql database create --language=xpp over four AxClass objects: 65 source blocks, 2224 nodes, no unsupported nodes, no errors. TRAP imported, source archive written, and queries return the expected counts.
• A QL library test covers the class and its methods, control flow, and the database statements. Writing it immediately found a bug: the tree walk stepped over CLR tuples, so a ttsabort inside a catch extracted as zero statements. Traversal now looks through tuple slots and the test pins counts for both tuple shapes.
• Three AxClass objects from the MIT-licensed TrudAX/XppTools parse to 2181 nodes with no unsupported nodes and no errors.
• The generated QL library compiles, and regeneration is byte-for-byte deterministic.

Worth noting: Pass1 runs standalone on .NET 8 including macOS, so the extractor is not tied to Windows. Method bodies parse with no metadata provider; only class and interface headers need one to resolve extends, and an empty one supplied through DispatchProxy is sufficient.

Notes for reviewers

• The compiler package is proprietary and LCS-gated. It is not committed, is never copied into build output, and is resolved at run time from XPP_COMPILER_PACKAGE. The generated schema, dbscheme and QL are committed so the QL layer can be worked on without it.
• No CI workflow yet, deliberately. Building or testing the extractor needs the LCS-gated package, which a public runner cannot fetch. That needs either a private feed plus a secret, or splitting the job so QL tests run against committed TRAP fixtures with no proprietary assembly present. Adding a workflow that cannot pass seemed worse than adding none.
• misc/codegen requires Python 3.13 or earlier. On 3.14, PEP 649 lazy annotations cause cls.__dict__["__annotations__"] to come back empty and every property is silently dropped from the schema.
• ql/lib/codeql/xpp/elements/internal/ElementImpl.qll is hand-edited to supply the default toStringImpl. Its "generated" marker comment has been removed deliberately; leave it off, and do not delete the generated tree wholesale when regenerating.

Not included

Still to come: the base QL layer over the shared/ packs (CFG, SSA, dataflow), Concepts and MaD models, CI and packaging wiring, and the security queries themselves.

[Copilot]

Copilot wasn't able to review this pull request because it exceeds the maximum number of files (300). Try reducing the number of changed files and requesting a review from Copilot again.