Skip to content

Add combined results and plot generation #219

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Open
avolkov-intel wants to merge 8 commits into
base: main
Choose a base branch
from
+589 −0
Open

Add combined results and plot generation #219

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
8 commits
Select commit Hold shift + click to select a range
49bd845
Add an option --combined-results to create list with all use cases
avolkov-intel Feb 19, 2026
356dc0a
Add options to draw comparison plots
avolkov-intel Feb 19, 2026
24ab534
Handle n_jobs param correctly and add Summary 2 and All cases 2 pages
avolkov-intel Jun 2, 2026
e842a63
Add footnote for plot and add metrics
avolkov-intel Aug 12, 2026
98fa891
Update footnote
avolkov-intel Aug 17, 2026
1886459
Update docs
avolkov-intel Aug 17, 2026
59276a2
Code format
avolkov-intel Aug 17, 2026
2f671bd
Minor docs update
avolkov-intel Aug 18, 2026
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
4 changes: 4 additions & 0 deletions sklbench/report/README.md
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -18,6 +18,10 @@ Raw results are converted into a pandas dataframe and the final report is made b
| `--report-file` | str | report.xlsx | | Report file path. |
| `--report-type` | str | separate-tables | ('separate-tables',) | Report type ("separate-tables" is the only supported now). |
| `--compatibility-mode` | | False | | [EXPERIMENTAL] Compatibility mode drops and modifies results to make them comparable (for example, sklearn and cuML parameters). |
| `--performance-stability-metrics`</br>`-psm` | | False | | Adds performance stability metrics (`1st run time[ms]`, `1st-mean run ratio`, `median time[ms]`, `time CV`) to the report. |
| `--combined-results` | | False | | [EXPERIMENTAL] Creates `All cases` and `Summary (for plots)` sheets combining time and speedup of all algorithms with per-dtype/total GEOMEAN rows. This option can only be used for comparing scikit-learn-intelex with stock scikit-learn (use with `--compatibility-mode`). |
| `--draw-plots` | | False | | [EXPERIMENTAL] Draws Training/Inference speedup bar charts from the combined results (requires `--combined-results`). |

@david-cortes-intel david-cortes-intel Aug 18, 2026

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This option without --plot-output doesn't appear to do anything. It doesn't show anything interactive for me, or at least not when used in an SSH session.

| `--plot-output` | str | None | | [EXPERIMENTAL] Output file path for plots (e.g. `plots.png`); if unset, plots are shown interactively. |

@david-cortes-intel david-cortes-intel Aug 18, 2026

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Does this require passing --draw-plots?

@avolkov-intel avolkov-intel Aug 18, 2026 •
edited
Loading

Copy link
Copy Markdown
Collaborator Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Well technically no, you can provide it without draw plots it would be just ignored

@david-cortes-intel david-cortes-intel Aug 18, 2026

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Please add that kind of detail in these docs.

| `--drop-columns`</br>`--drop-cols` | str | [] | | Columns to drop from report. |
| `--diff-columns`</br>`--diff-cols` | str | ['environment_name', 'library', 'format', 'device'] | | Columns to show difference between. |
| `--split-columns` | str | ['estimator', 'method', 'function'] | | Splitting columns for subreports/sheets. |
Expand Down
22 changes: 22 additions & 0 deletions sklbench/report/arguments.py
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -93,6 +93,28 @@ def add_report_generator_arguments(
help="Selects which part of one-vs-one difference to show "
"(all matrix or one of triangles).",
)
parser.add_argument(
"--combined-results",
default=False,
action="store_true",
help="[EXPERIMENTAL] Creates 'All cases' and 'Summary (for plots)' sheets "
"combining time[ms] and speedup of all algorithms into single pages with "
"per-dtype and total GEOMEAN rows. Use together with --compatibility-mode.",
)
parser.add_argument(
"--draw-plots",
default=False,
action="store_true",
help="[EXPERIMENTAL] Draw Training/Inference speedup bar charts from combined "
"results (requires --combined-results).",
)
parser.add_argument(
"--plot-output",
type=str,
default=None,
help="[EXPERIMENTAL] Output file path for plots (e.g., plots.png). "
"If not specified, plots are displayed.",
)
# color scale settings
parser.add_argument(
"--perf-color-scale",
Expand Down
6 changes: 6 additions & 0 deletions sklbench/report/compatibility.py
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -34,6 +34,12 @@ def transform_results_to_compatible(results: pd.DataFrame):
"min_bin_size",
],
)
if (
"n_jobs" in results.columns
and results["n_jobs"].isna().any()
and results["n_jobs"].notna().any()

@ethanglaser ethanglaser Aug 17, 2026

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Would it make more sense to drop and results["n_jobs"].notna().any() and just drop n_jobs col if any are na? Or what is the reason for this change

@avolkov-intel avolkov-intel Aug 18, 2026

Copy link
Copy Markdown
Collaborator Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

If we compare sklearn algorithm with sklearnex counterpart, n_jobs parameter might only be present in sklearnex but not in sklearn. In this case the measurements will not be compared and report & plots would not be generated. So the idea of this change if both not nans and nans are present we can drop this column so that reports are properly generated. If all values are na there's no need to drop the column

@avolkov-intel avolkov-intel Aug 18, 2026

Copy link
Copy Markdown
Collaborator Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This feature should only be used for comparison of sklearn vs sklearnex btw

):
results.drop(inplace=True, columns=["n_jobs"])
if results["environment_name"].unique().size > 1:
# DBSCAN `eps` parameter drop for different CPUs
results.drop(
Expand Down
Loading
Loading