task: refactor ASV benchmarks

.gitignore

@@ -4,6 +4,9 @@ build_cython
 cython_debug
 dpnp.egg-info
 
+# Airspeed Velocity (asv) benchmark environments, results and html
+benchmarks/.asv/
+
 # Byte-compiled / optimized / DLL files
 __pycache__/
 

benchmarks/README.md

@@ -0,0 +1,144 @@
+# dpnp benchmarks
+
+Benchmarking dpnp using Airspeed Velocity.
+Read more about ASV [here](https://asv.readthedocs.io/en/stable/index.html).
+
+## Usage
+
+Unlike a pure-Python project, dpnp is a SYCL/DPC++ extension that requires the
+Intel oneAPI compiler and a lengthy build, so ASV does not build dpnp itself:
+`build_command` in `asv.conf.json` is empty and the benchmarks are run against
+an **existing environment** that already has dpnp installed.
+
+Create an environment
+[following these instructions](https://intelpython.github.io/dpnp/quick_start_guide.html)
+and install the benchmarking tooling into it.
+
+Install the tooling directly, which leaves the already-built dpnp untouched:
+
+```bash
+conda install -c conda-forge asv scipy
+```
+
+Do **not** use `pip install ".[benchmark]"` for an environment that already has
+dpnp: dpnp is a scikit-build project, so pip reinstalls the `dpnp` package
+itself and triggers a full oneAPI/DPC++ rebuild of the backend just to pull in
+two pure-Python dependencies. The `benchmark` extra exists for the case where
+dpnp is being built from source anyway, e.g.:
+
+```bash
+pip install --no-build-isolation --no-deps -e ".[benchmark]"
+```
+
+Then activate the environment and run the benchmarks against it. The simplest
+way is to point ASV at the currently active environment with `--python=same`:
+
+```bash
+conda activate dpnp_env
+asv run --python=same --launch-method spawn --quick HEAD^!
+```
+
+Alternatively, point ASV explicitly at an environment's python binary:
+
+```bash
+asv run --environment existing:/full/conda/path/envs/dpnp_env/bin/python \
+    --launch-method spawn
+```
+
+Compare two commits or check for regressions:
+
+```bash
+asv continuous --python=same --launch-method spawn HEAD~1 HEAD
+```
+
+**Always pass `--launch-method spawn`.** ASV defaults to a forkserver, which
+`fork()`s a process that has already initialized a SYCL runtime; the SYCL
+runtime is multi-threaded and not fork-safe, so benchmarks may hang until
+`default_benchmark_timeout` expires (reported as `failed`) or fail with
+`USM Allocation` errors on `level_zero` devices. `spawn` starts a fresh
+interpreter per benchmark and avoids this entirely.
+
+By default, dpnp selects a default SYCL device. Use the `ONEAPI_DEVICE_SELECTOR`
+environment variable to target a specific device, e.g.:
+
+```bash
+ONEAPI_DEVICE_SELECTOR=level_zero:gpu asv run \
+    --launch-method spawn \
+    --python=same
+```
+
+## Benchmarks
+
+### `bench_dpbench.py` -- dpBench workloads
+
+`bench_dpbench.py` runs a set of dpnp workloads vendored from
+[dpBench](https://github.com/IntelPython/dpbench). The kernels, their data
+initialization, and the data-size presets are copied from dpBench and live in
+`benchmarks/dpbench/workloads`. Each workload is exposed as its own benchmark
+class (e.g. `BlackScholes.time_black_scholes`) and is parametrized by the
+dpBench data-size preset (`S`, `M16Gb`, `M`, `L`) and by floating-point
+precision (`single`, `double`).
+
+Currently vendored workloads:
+
+| Workload            | Domain             |
+| ------------------- | ------------------ |
+| `black_scholes`     | Finance            |
+| `l2_norm`           | Distance Compute   |
+| `pairwise_distance` | Distance Compute   |
+| `rambo`             | Particle Physics   |
+| `gpairs`            | Astrophysics       |
+
+Host input data is generated and copied to the device exactly the way dpBench
+does, and each kernel ends with `dpnp.synchronize_array_data`, so a single call
+blocks until the device work has finished. The `time_*` methods invoke the
+workload once and let ASV wall-clock-time it (handling repeats, samples and
+statistics natively) -- the same end-to-end quantity dpBench itself measures,
+and the same plain `time_*` style used by the mkl_fft ASV benchmarks.
+
+**Precision.** Both `single` and `double` are benchmarked. Devices without fp64
+support (common on iGPUs) skip the `double` parametrization via
+`SkipNotImplemented` rather than failing the run, so such a device still
+produces `single`-precision results. dpBench's own configs request `double`
+throughout; that value is kept in each workload's `PRECISION` for reference.
+
+**Preset selection.** Presets are chosen per device instead of being hard-coded:
+`_dpbench_runner.select_presets` keeps every preset whose estimated peak device
+footprint (each workload's `peak_elements`) fits within a fraction of the
+device's `global_mem_size`. A large discrete GPU therefore exercises the bigger
+problem sizes automatically, while a small iGPU stays on `S`. Note that dpBench's
+preset names are not ordered by size -- `M16Gb` is *smaller* than `M`.
+
+Prefer the largest preset your device fits when looking for regressions. The
+smallest sizes are dominated by per-call dispatch overhead and are noticeably
+noisier: on a CPU device the run-to-run spread of the median at `S` was measured
+at 4-14%, against the 20% `regressions_thresholds` in `asv.conf.json`, whereas
+the larger presets settled to a few percent. Timings at `S` are still useful for
+a quick smoke test, and ASV's repeat/sample handling absorbs part of the noise.
+
+**Validation.** Each workload also ships the NumPy `reference` implementation
+from dpBench, and every benchmark's `setup` compares the dpnp results for all
+`OUTPUT_ARGS` against it (mirroring dpBench's
+`infrastructure/benchmark_validation.py`, same `1e-05` relative-error
+tolerance). A numerically wrong kernel therefore fails the benchmark instead of
+being silently timed. Validation runs outside the timed region and does not
+affect the reported numbers.
+
+### Other benchmark modules
+
+The remaining `bench_*.py` modules (`bench_linalg.py`, `bench_elementwise.py`,
+`bench_random.py`) are plain ASV benchmarks comparing dpnp against NumPy.
+
+## Writing new benchmarks
+
+Read ASV's guidelines for writing benchmarks
+[here](https://asv.readthedocs.io/en/stable/writing_benchmarks.html).
+
+To add another dpBench workload, copy its `<name>_dpnp.py` kernel,
+`<name>_numpy.py` reference (as `reference`) and `<name>_initialize.py`
+initializer into a new module under `benchmarks/dpbench/workloads`, translate its
+`bench_info` TOML presets into the module's `PRESETS` and argument-metadata
+constants, add a `peak_elements` estimate (see the existing workloads for the
+exact shape), and add the module to `WORKLOADS` in
+`benchmarks/dpbench/workloads/__init__.py`. `bench_dpbench.py` will generate a
+benchmark class for it automatically.

benchmarks/asv.conf.json

@@ -1,89 +1,26 @@
 {
-    // The version of the config file format.  Do not change, unless
-    // you know what you are doing.
     "version": 1,
-
-    // The name of the project being benchmarked
     "project": "dpnp",
-
-    // The project's homepage
-    "project_url": "",
-
-    // The URL or local path of the source code repository for the
-    // project being benchmarked
+    "project_url": "https://github.com/IntelPython/dpnp",
     "repo": "..",
-
-    // List of branches to benchmark. If not provided, defaults to "master"
-    // (for git) or "tip" (for mercurial).
+    "show_commit_url": "https://github.com/IntelPython/dpnp/commit/",
+    "build_command": [],
     "branches": [
         "HEAD"
     ],
-
-    // The DVCS being used.  If not set, it will be automatically
-    // determined from "repo" by looking at the protocol in the URL
-    // (if remote), or by looking for special directories, such as
-    // ".git" (if local).
     "dvcs": "git",
-
-    // The tool to use to create environments.  May be "conda",
-    // "virtualenv" or other value depending on the plugins in use.
-    // If missing or the empty string, the tool will be automatically
-    // determined by looking for tools on the PATH environment
-    // variable.
-    "environment_type": "virtualenv",
-
-    // the base URL to show a commit for the project.
-    "show_commit_url": "",
-
-    // The Pythons you'd like to test against.  If not provided, defaults
-    // to the current version of Python used to run `asv`.
-    "pythons": [
-        "3.7"
+    "environment_type": "conda",
+    "conda_channels": [
+        "https://software.repos.intel.com/python/conda/",
+        "conda-forge"
     ],
-
-    // The matrix of dependencies to test.  Each key is the name of a
-    // package (in PyPI) and the values are version numbers.  An empty
-    // list indicates to just test against the default (latest)
-    // version.
-    "matrix": {
-        "Cython": [],
-    },
-
-    // The directory (relative to the current directory) that benchmarks are
-    // stored in.  If not provided, defaults to "benchmarks"
     "benchmark_dir": "benchmarks",
-
-    // The directory (relative to the current directory) to cache the Python
-    // environments in.  If not provided, defaults to "env"
-    "env_dir": "env",
-
-    // The directory (relative to the current directory) that raw benchmark
-    // results are stored in.  If not provided, defaults to "results".
-    "results_dir": "results",
-
-    // The directory (relative to the current directory) that the html tree
-    // should be written to.  If not provided, defaults to "html".
-    "html_dir": "html",
-
-    // The number of characters to retain in the commit hashes.
-    // "hash_length": 8,
-
-    // `asv` will cache wheels of the recent builds in each
-    // environment, making them faster to install next time.  This is
-    // number of builds to keep, per environment.
-    "build_cache_size": 8,
-
-    // The commits after which the regression search in `asv publish`
-    // should start looking for regressions. Dictionary whose keys are
-    // regexps matching to benchmark names, and values corresponding to
-    // the commit (exclusive) after which to start looking for
-    // regressions.  The default is to start from the first commit
-    // with results. If the commit is `null`, regression detection is
-    // skipped for the matching benchmark.
-    //
-    // "regressions_first_commits": {
-    //    "some_benchmark": "352cdf",  // Consider regressions only after this
-    //    commit
-    //    "another_benchmark": null,   // Skip regression detection altogether
-    // }
+    "env_dir": ".asv/env",
+    "results_dir": ".asv/results",
+    "html_dir": ".asv/html",
+    "build_cache_size": 2,
+    "default_benchmark_timeout": 500,
+    "regressions_thresholds": {
+        ".*": 0.2
+    }
 }

benchmarks/benchmarks/bench_dpbench.py

@@ -0,0 +1,128 @@
+# *****************************************************************************
+# Copyright (c) 2026, Intel Corporation
+# All rights reserved.
+#
+# Redistribution and use in source and binary forms, with or without
+# modification, are permitted provided that the following conditions are met:
+# - Redistributions of source code must retain the above copyright notice,
+#   this list of conditions and the following disclaimer.
+# - Redistributions in binary form must reproduce the above copyright notice,
+#   this list of conditions and the following disclaimer in the documentation
+#   and/or other materials provided with the distribution.
+# - Neither the name of the copyright holder nor the names of its contributors
+#   may be used to endorse or promote products derived from this software
+#   without specific prior written permission.
+#
+# THIS SOFTWARE IS PROVIDED BY THE COPYRIGHT HOLDERS AND CONTRIBUTORS "AS IS"
+# AND ANY EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT LIMITED TO, THE
+# IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS FOR A PARTICULAR PURPOSE
+# ARE DISCLAIMED. IN NO EVENT SHALL THE COPYRIGHT HOLDER OR CONTRIBUTORS BE
+# LIABLE FOR ANY DIRECT, INDIRECT, INCIDENTAL, SPECIAL, EXEMPLARY, OR
+# CONSEQUENTIAL DAMAGES (INCLUDING, BUT NOT LIMITED TO, PROCUREMENT OF
+# SUBSTITUTE GOODS OR SERVICES; LOSS OF USE, DATA, OR PROFITS; OR BUSINESS
+# INTERRUPTION) HOWEVER CAUSED AND ON ANY THEORY OF LIABILITY, WHETHER IN
+# CONTRACT, STRICT LIABILITY, OR TORT (INCLUDING NEGLIGENCE OR OTHERWISE)
+# ARISING IN ANY WAY OUT OF THE USE OF THIS SOFTWARE, EVEN IF ADVISED OF
+# THE POSSIBILITY OF SUCH DAMAGE.
+# *****************************************************************************
+
+"""ASV benchmarks for dpnp workloads vendored from dpBench.
+
+The workloads (kernels + data initialization) and their data-size presets are
+copied from dpBench (https://github.com/IntelPython/dpbench); see
+``benchmarks/benchmarks/dpbench``.
+
+Each vendored kernel ends with ``dpnp.synchronize_array_data`` on its output,
+so a single call blocks until the device work has finished. The ``time_*``
+methods below simply invoke the workload once and let ASV wall-clock-time it
+(handling repeats, samples and statistics natively) -- the same end-to-end
+quantity dpBench itself measures, and the same plain ``time_*`` style used by
+the mkl_fft ASV benchmarks.
+
+A separate benchmark class is generated for each workload -- e.g.
+``BlackScholes.time_black_scholes`` -- parametrized by the data-size preset and
+the floating-point precision. The presets are chosen per device so that only
+problem sizes fitting into device memory are benchmarked, and a precision the
+device does not support (typically fp64 on an iGPU) is skipped rather than
+failing the run.
+
+``setup`` also validates the dpnp results against the workload's NumPy
+reference, so a numerically wrong kernel fails the benchmark instead of being
+timed. Validation happens outside the timed region and therefore does not
+affect the reported numbers, but it is limited to the cheapest preset: the
+reference runs on the host, and at the larger presets it costs far more than
+the benchmark it guards (measured at ~70 s for ``pairwise_distance`` at
+``M16Gb``) while checking numerics that do not depend on the problem size.
+"""
+
+import dpctl
+
+from . import benchmark_utils as bench_utils
+from .dpbench import _dpbench_runner as runner
+from .dpbench.workloads import WORKLOADS
+
+# Default-device queue, used to query device capabilities (fp64 support, memory
+# size) so the parameter matrix can be tailored to the device. This is the
+# device dpnp allocates on by default.
+DEVICE_QUEUE = dpctl.SyclQueue()
+DEVICE = DEVICE_QUEUE.sycl_device
+
+
+def _camel_case(name):
+    """``black_scholes`` -> ``BlackScholes``, ``l2_norm`` -> ``L2Norm``."""
+    return "".join(part.capitalize() for part in name.split("_"))
+
+
+def _make_benchmark_class(workload):
+    """Build an ASV benchmark class for a single dpBench workload."""
+
+    class WorkloadBenchmark:
+        # The per-benchmark timeout is governed by ``default_benchmark_timeout``
+        # in ``asv.conf.json``; larger presets on a busy device can take a
+        # while.
+
+        params = [
+            runner.select_presets(workload, DEVICE),
+            list(runner.PRECISIONS),
+        ]
+        param_names = ["preset", "precision"]
+
+        # Preset the results are validated against; see the module docstring.
+        _validated_preset = runner.presets_by_size(workload)[0]
+
+        def setup(self, preset, precision):
+            # Skip precisions the device does not support (e.g. fp64 on many
+            # iGPUs), mirroring the dpctl ASV benchmarks.
+            bench_utils.skip_unsupported_dtype(
+                DEVICE_QUEUE, runner.float_dtype(precision)
+            )
+
+            self._runner = runner.WorkloadRunner(workload, preset, precision)
+            self._runner.setup()
+            if preset == self._validated_preset:
+                self._runner.validate()
+
+        def time_workload(self, preset, precision):
+            self._runner.run()
+
+    # Name things so ASV displays e.g. ``BlackScholes.time_black_scholes``.
+    WorkloadBenchmark.__name__ = _camel_case(workload.NAME)
+    WorkloadBenchmark.__qualname__ = WorkloadBenchmark.__name__
+
+    time_method = WorkloadBenchmark.time_workload
+    time_method.__name__ = f"time_{workload.NAME}"
+    setattr(WorkloadBenchmark, time_method.__name__, time_method)
+    del WorkloadBenchmark.time_workload
+
+    return WorkloadBenchmark
+
+
+def _generate_benchmark_classes():
+    """Create and register a benchmark class for every vendored workload."""
+    for workload in WORKLOADS:
+        cls = _make_benchmark_class(workload)
+        # Register the class at module scope so ASV can discover it.
+        globals()[cls.__name__] = cls
+
+
+_generate_benchmark_classes()