Fix "Too many open files" in QNN DeepLabV3 example - #21874
Conversation
### Summary
`deeplab_v3.py --download` dies partway through calibration with
`OSError: [Errno 24] Too many open files` on any host whose soft
descriptor limit is the usual 1024 -- a stock Docker container, a systemd
service and an ordinary login shell all get that.
`get_dataset` materializes all 1449 VOC 2012 val samples with `list()` so
that `random.shuffle` can pick 100 of them. torchvision's
`VOCSegmentation.__getitem__` calls `.convert("RGB")` on the image, which
forces `load()` and lets Pillow close the file it opened, but it leaves
the mask as a lazy `Image.open`. The example sets `transform` and not
`target_transform`, so nothing ever reads the mask pixels and every
retained sample keeps a `SegmentationClass/*.png` descriptor open -- one
per sample, until the limit is reached and the run dies. The file named
in the error is a JPEG only because that is whichever `open()` happened
to tip it over.
Sampling the index range instead never holds more than the sample being
converted, and peak descriptors across `get_dataset` drop from 1454 to 6.
Setting `target_transform` would also stop the leak, but it treats the
symptom and keeps the waste, since `list()` still decodes, resizes,
normalizes and retains 1449 images in order to use 100. `random.sample`
over the index range and shuffle-then-take-100 draw from the same
distribution, and nothing under `examples/qualcomm/` seeds `random`, so
which 100 you get is exactly as reproducible as it was before. This also
brings the script in line with `get_imagenet_dataset` in
`examples/qualcomm/utils.py` and with the oss_scripts examples, none of
which materialize the dataset.
Fixes pytorch#21870
### Test plan
No device and no source build are needed. The 1.4.0 wheel ships
`examples/qualcomm/`, its copy of this script is byte-identical to the
file before this patch, and the backend downloads the pinned QNN SDK
itself on first import. Starting from nothing:
python3 -m venv dlv3-venv
. dlv3-venv/bin/activate
pip install \
--index-url https://download.pytorch.org/whl/cpu \
--extra-index-url https://pypi.org/simple \
executorch==1.4.0 torchvision==0.28.0 py-cpuinfo transformers pydot
ulimit -n 1024 # the stock soft limit; some shells raise it already
export QNN_SDK_ROOT=$HOME/.cache/executorch/qnn/sdk-2.37.0.250724
export LD_LIBRARY_PATH=$QNN_SDK_ROOT/lib/x86_64-linux-clang:$LD_LIBRARY_PATH
# before: the wheel's own copy
python -m executorch.examples.qualcomm.scripts.deeplab_v3 \
--build_folder build-x86 --soc_model SM8550 \
--artifact ./dlv3 --compile_only --download
# after: this checkout's copy, against the same installed wheel
python <checkout>/examples/qualcomm/scripts/deeplab_v3.py \
--build_folder build-x86 --soc_model SM8550 \
--artifact ./dlv3 --compile_only --download
`py-cpuinfo`, `transformers` and `pydot` are imported on the way to the
example but are not dependencies of the wheel. `--compile_only` exits
after writing the .pte, so no phone is involved, and `--build_folder` is
asserted by `QnnConfig` but never used on that path. Running the second
command by path rather than with `-m` puts the script's own directory on
`sys.path` and not the checkout root, so the two runs differ only in this
file.
The first died in `get_dataset` with `Exception: [Errno 24] Too many open
files` -- the top-level handler re-raises bare, which hides the errno
class from anything reading only the tail of a log -- after 34 min,
nearly all of it the 2 GB VOCSegmentation download. The second reused
that download and completed in 2 min 12 s, exit 0, writing a
61,732,608-byte `dlv3_qnn.pte`. No two runs produce the same bytes,
patched or not, because the calibration draw is unseeded.
Descriptors were counted from /proc/self/fd around `get_dataset` against
a generated 1449-sample stand-in for VOC 2012 val: 8x8 images in the same
layout, since nothing in the mechanism depends on their contents. Peak
held was 1454 before (1449 masks plus the process's own five) and 6
after. At `ulimit -n 1024` the unpatched code stops at 1022 descriptors
and the patched code completes. Where both complete they agree, on 100
inputs of (1, 3, 224, 224) and 100 targets of (224, 224) uint8, and both
return all 50 when asked for 100 from a 50-sample set.
Measured with Python 3.14.4, executorch 1.4.0+cpu, torch 2.13.0+cpu,
torchvision 0.28.0+cpu and Pillow 12.3.0.
Authored with Claude Code (Claude Opus 5).
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01RyGcV8dicAd91uZPFUCLA9
🔗 Helpful Links🧪 See artifacts and rendered test results at hud.pytorch.org/pr/pytorch/executorch/21874
Note: Links to docs will display an error until the docs builds have been completed.
|
|
Hi @mcollinswisc! Thank you for your pull request and welcome to our community. Action RequiredIn order to merge any pull request (code, docs, etc.), we require contributors to sign our Contributor License Agreement, and we don't seem to have one on file for you. ProcessIn order for us to review and merge your suggested changes, please sign at https://code.facebook.com/cla. If you are contributing on behalf of someone else (eg your employer), the individual CLA may not be sufficient and your employer may need to sign the corporate CLA. Once the CLA is signed, our tooling will perform checks and validations. Afterwards, the pull request will be tagged with If you have received this in error or have any questions, please contact us at cla@meta.com. Thanks! |
|
One or more co-authors of this pull request were not found. You must specify co-authors in commit message trailer via: Supported
Alternatively, if the co-author should not be included, remove the Please update your commit message(s) by doing |
This PR needs a
|
|
/easycla |
|
Thank you for signing our Contributor License Agreement. We can now accept your code for this (and any) Meta Open Source project. Thanks! |
meta-cla
Bot
added
the
CLA Signed
1 participant
Summary
Fixes #21870
Avoids exhausting the process limit of file descriptors by sampling indices, and loading only the sample. This avoids materializing the full VOC val dataset with
list(...).Test plan
Since this is just editing an example Python script, we tested it against a built executorch installed from pip too. With
CWD in a working copy pointed to this branch:
Run at the
mainbranch (or withpython -m executorch.examples.qualcomm.scripts.deeplab_v3 ...this fails with:after this change it succeeds & writes the
.ptefile: