Add modulepath-ignore command to filter out modulepath content - #682
Merged
Conversation
Design notes for the modulepath-ignore modulefile command introduced to solve issue envmodules#562. Assisted-by: Claude:claude-fable-5 Signed-off-by: Xavier Delaruelle <xavier.delaruelle@cea.fr>
Modulepath directories may contain files that are not modulefiles, like helper scripts sourced by modulefiles or software installation files when modulefiles are mixed with the software trees they describe. These files are read to check if they are valid modulefiles and recorded in cache files, which makes cachebuild take a huge amount of time and memory when they are numerous. Add the modulepath-ignore modulefile command to define from rc files gitignore-style patterns describing modulepath content to ignore when searching for modulefiles. Ignored files and directories are handled as if they do not exist: they are not checked, not walked down, not reported and not recorded in cache files. Rc file evaluation entirely moves from getModules consolidation phase to the modulepath walk procedure, so that the patterns rc files define apply to the elements walked next. Walk becomes depth-first with directory content processed in lexical order to preserve the rc file evaluation order that module resolution and symbol definition results depend on. Rc files recorded in cache are evaluated at walk start, as their patterns may apply to the walked down limited access elements. Also add the modulepath_ignore configuration option, enabled by default, and the --no-modulepath-ignore command line switch to control whether pattern definitions are applied. Fixes envmodules#562 Assisted-by: Claude:claude-fable-5 Signed-off-by: Xavier Delaruelle <xavier.delaruelle@cea.fr>
Both modulepath walk procedures applied the ignored_dirs configuration option check side by side with the isPathIgnored call. Move this check into isPathIgnored as an early test made prior applying patterns, so the behavior of this option is preserved: negated patterns cannot re-include a matching directory and directories keep being ignored when modulepath_ignore option is disabled. Assisted-by: Claude:claude-fable-5 Signed-off-by: Xavier Delaruelle <xavier.delaruelle@cea.fr>
Both modulepath walk procedures skipped backup or version control files (*~, *,v and #*#) with a specific switch pattern set side by side with the isPathIgnored call. Move this check into isPathIgnored as an early test made prior applying patterns, like the ignored_dirs option check: negated patterns cannot re-include these files and they keep being ignored when modulepath_ignore option is disabled. Assisted-by: Claude:claude-fable-5 Signed-off-by: Xavier Delaruelle <xavier.delaruelle@cea.fr>
Rc files of a directory were evaluated when reached in the walked element list, right after the directory content fetch that returns them first. Evaluate and record them at fetch time instead, prior the element list trimming made for the stop_mod optimization: the hiding or ignore rules they may define are then in place for this trimming, which does not need to source rc files on its own anymore. Fetched directory content is turned into an array: rc files are directly looked up by name in it then withdrawn once evaluated and recorded, which lifts the need to get them first in the list returned by getFilesInDirectory (Tcl implementation of this procedure is set back to its original form). Remaining element paths, sorted in lexical order, are transmitted to trimElemListForStopmod, which now returns the trimmed list, and are then inserted in the walked list right after current position: depth-first insertion is what matters to evaluate the rc files of a sub-directory prior those of directories lying next or deeper. Only the rc files that are not part of a fetched directory content flow through the walked list and get handled by the walk loop (modulepath root rc file from the initial listing, limited access rc files from a cache file listing). Assisted-by: Claude:claude-fable-5 Signed-off-by: Xavier Delaruelle <xavier.delaruelle@cea.fr>
Assisted-by: Claude:claude-fable-5 Signed-off-by: Xavier Delaruelle <xavier.delaruelle@cea.fr>
Assisted-by: Claude:claude-fable-5 Signed-off-by: Xavier Delaruelle <xavier.delaruelle@cea.fr>
Assisted-by: Claude:claude-fable-5 Signed-off-by: Xavier Delaruelle <xavier.delaruelle@cea.fr>
Assisted-by: Claude:claude-fable-5 Signed-off-by: Xavier Delaruelle <xavier.delaruelle@cea.fr>
Assisted-by: Claude:claude-fable-5 Signed-off-by: Xavier Delaruelle <xavier.delaruelle@cea.fr>
Assisted-by: Claude:claude-fable-5 Signed-off-by: Xavier Delaruelle <xavier.delaruelle@cea.fr>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Add the
modulepath-ignoremodulefile command to define, from rc files, gitignore-style patterns describing modulepath content to ignore when searching for modulefiles. Ignored files and directories are handled as if they do not exist: they are not checked, not walked down, not reported and not recorded in modulepath cache file. This solves the cachebuild time and memory issue observed when a modulepath contains a very large amount of non-modulefiles.Patterns follow the same syntax and behavior as git with
.gitignorefiles (negation, anchoring,**, character ranges, directory-only trailing slash). They are anchored to the directory of the rc file defining them; global and user rc files define global patterns applying to every modulepath.To have patterns in place when modulepath content is walked through, rc file evaluation moves from the
getModulesconsolidation phase to the walk procedure itself, which becomes depth-first: rc files of a directory are evaluated and recorded as soon as its content is fetched. Theignored_dirsoption check and the historical backup file skip are integrated into the path ignore matcher.The
modulepath_ignoreconfiguration option (enabled by default) and the--no-modulepath-ignorecommand line switch control whether pattern definitions are applied.Design notes are provided in
doc/source/design/modulepath-ignore.rst.Fixes #562