Compile inflector regexes once instead of per call - #54
Conversation
d278f48 to
a6ab8fd
Compare
dduugg
left a comment
There was a problem hiding this comment.
Approving. I checked this specifically for the ways a "just hoist the regexes" refactor can quietly change output, and it's clean on all of them.
Verified
- Pattern strings and evaluation order are identical.
SINGULARIZEis built withCLASS_CASE_TO_SINGULAR.map(...), which preserves the const array's order (Censuse, Leafe, Lefe, Daum), and.iter().for_eachiterates that same order — so cascading replacements behave as before. - The guard rewrite is equivalent:
class_name.contains(re.as_str())vs the oldcontains(plural).Regex::as_str()returns the original pattern text, and all four table patterns are plain literals with no regex metacharacters, sore.as_str() == pluralexactly. The outercontains("Statu")guard is untouched. - No escaping hazard — every pattern here is a static literal, and construction is still
Regex::new, just hoisted. - The discarded
"Statuss"result is still discarded, sostatussremains untouched and the("statuss", false, "Statuss")case still pins it. Agreed that fixing it belongs in its own PR — I'd rather see that tradeoff stated explicitly, as you did, than silently "improved" inside a perf change. - The
unwrap()timing shift (per-call -> first-use) is inert: every pattern is a static, known-valid literal, so there's no input that could make one fail to compile, in a rayon worker or anywhere else. LazyLockis the right primitive — stable since 1.80, and the repo has noonce_cell/lazy_staticdependency or existing convention to match.- Your call on
file_utils.rs:108andrails_utils.rs:19holds up:rails_utils.rs:19(get_acronyms_from_disk) is called once per run atzeitwerk/mod.rs:120, andfile_utils.rs:108runs once per.erbfile — neither is per-file-hot the wayto_class_case/camelizeare. Leaving them is right.
Worth knowing: this doesn't actually depend on #53. It touches only inflector_shim.rs; #53 touches Cargo.toml, dev/*, main.rs, packs.rs, checker.rs. Zero file overlap and no symbol dependency — cherry-picked straight onto main, cargo build --lib succeeds (LazyLock needs 1.80, main already pins 1.92.0). So if #53 stalls in review, this can be retargeted to #52 or main directly rather than waiting. #53's LTO change is what makes your numbers reproducible, but that's a measurement convenience, not a code dependency.
| LazyLock::new(|| Regex::new("Statu$").unwrap()); | ||
| static STATUSS: LazyLock<Regex> = | ||
| LazyLock::new(|| Regex::new("Statuss").unwrap()); | ||
| static SINGULARIZE: LazyLock<[(Regex, &str); CLASS_CASE_TO_SINGULAR.len()]> = |
There was a problem hiding this comment.
Non-blocking test-coverage note, since this table is the part where order matters most.
test_to_class_case covers "lefe" but not "leafe", and has no case where a single input triggers two replacements in sequence (e.g. something matching both a Statu* pattern and a singular-table entry). Those cascading paths are exactly what would break if the iteration order of CLASS_CASE_TO_SINGULAR ever changed — and this diff is what makes that order an explicit property of a static rather than an incidental property of a loop.
Pre-existing gap and not this PR's job to close: the refactor doesn't touch transformation logic, only where the regexes get compiled. Just a good candidate for a follow-up case or two while the reasoning is fresh.
a6ab8fd to
88e790f
Compare
88e790f to
47b2383
Compare
`camelize` and `to_class_case` run once per file while the Zeitwerk constant map is built -- roughly 50k calls per invocation on a large codebase -- and each call was compiling between two and seven regexes from scratch. Building a regex constructs a DFA, so this was not a marginal cost. Hoisted every pattern into a `LazyLock<Regex>` compiled once per process. MEASURED on a 51,513-file app: wall clock 5.127s -> 4.656s (-9.2%) user cpu time 13.561s -> 7.370s (-45.7%) "infer constants from filename" phase 0.689s -> <0.12s Wall clock improves less than CPU time because this phase is parallelized across cores, so the wasted work was partly hidden. It was still burning nearly half the process's total CPU. Behavior is deliberately unchanged, including one oddity: the "Statuss" replacement has always had its result discarded, so "statuss" is left alone. That is preserved verbatim with a comment, and the existing test that pins it still passes. It looks like a bug, but fixing it belongs in its own change. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
47b2383 to
3bef8d6
Compare
Best return per line of any change I've measured on
pks check: ~40 lines, and it cuts total CPU time by 45%.Note
Stacked on #53 (which is itself stacked on #52). The diff above shows only this change — one file. GitHub retargets the base automatically as each parent merges.
The problem
camelizeandto_class_caserun once per file while the Zeitwerk constant map is built — roughly 50k calls per invocation on a large codebase. Each call was compiling between two and seven regexes from scratch:Regex::newconstructs a DFA. Doing that per filename, for every file in the app, on every run, is not a marginal cost.The change
Every pattern becomes a
LazyLock<Regex>, compiled once per process. That's the whole change — one file, no API or behavior change.Measured
On a 51,513-file application:
Two honest notes on those numbers:
One thing to know about the diff
This preserves an existing oddity rather than fixing it:
That result has always been thrown away, so
statussis left untouched — andtest_to_class_casepins that behavior with a("statuss", false, "Statuss")case. I kept it verbatim with a comment explaining why. It looks like a bug, but fixing it changes inflection output and belongs in its own PR, not one whose only job is to stop recompiling regexes.Two other files have the same compile-per-call pattern (
file_utils.rs:108,rails_utils.rs:19). I left them alone: once this landed, the phase was down to 0.047 s and there was nothing left to win. Not worth the churn.Verification
cargo test— 257 passingcargo clippy --all-targets --all-features— cleancargo fmt --all -- --check— cleanHeads-up: a pre-existing flaky test
While verifying, I hit
gitignore_test::test_respect_gitignore_can_be_disabledfailing on roughly 1 run in 3. It is not caused by this change — it reproduces at the same rate on the base branch, and passes 6/6 when run in isolation.Cause:
common::teardown()globstests/fixtures/*/tmp/cache/packwerkand deletes the cache for every fixture, not just the one the calling test used.gitignore_test.rscalls it from 7 different tests, which Rust runs in parallel threads within the same binary — so one test deletes another's cache mid-run.Worth fixing separately (scope
teardownto the fixture the test actually touched). Flagging it here because it will make CI intermittently red regardless of this PR.🤖 Generated with Claude Code