Support user-defined benchmark suites. - #109

Merged

ericsnowcurrently merged 127 commits into

python:mainfrom

ericsnowcurrently:benchmark-management

Dec 8, 2021

Commits on Oct 4, 2021

benchmarks -> _benchmarks
ericsnowcurrently
committed
509d93c
Use a new API for pyperformance.benchmarks.
ericsnowcurrently
committed
e16141d
Deal with benchmark objects instead of names.
ericsnowcurrently
committed
d2101b2
Refactor select_benchmarks().
ericsnowcurrently
committed
20b2c23
Add and use the default manifest file.
ericsnowcurrently
committed
20e8492
Move run_perf_script() back.
ericsnowcurrently
committed
4f74cb8
Clean up benchmark/__init__.py.
ericsnowcurrently
committed
3917d22
Make the utils a package.
ericsnowcurrently
committed
32e15ec
Move each of the default benchmarks into its own directory.
ericsnowcurrently
committed
b041e2e
Make BenchmarkSpec.metafile as "secondary" attribute.
ericsnowcurrently
committed
d79eba5
Fix benchmark selection.
ericsnowcurrently
committed
f30421d
Fix the default benchmarks selection.
ericsnowcurrently
committed
043836b
Fix the run script filename.
ericsnowcurrently
committed
6ba9603
Run benchmarks from the metadata instead of hard-coded.
ericsnowcurrently
committed
0f15e8e
Fix the requirements.
ericsnowcurrently
committed
91c27f9
Pass "name" through to parse_pyproject_toml().
ericsnowcurrently
committed
a4f97ad
Leave a note about classifiers.
ericsnowcurrently
committed
6d385bf
Drop an unused file.
ericsnowcurrently
committed
5308b29
Load manifest and select benchmarks before running the command.
ericsnowcurrently
committed
4c8a18f
Fix Benchmark.__repr__().
ericsnowcurrently
committed
0f63a54
Fix a default arg in load_metadata().
ericsnowcurrently
committed
c7a92f4
Fix the packaging data.
ericsnowcurrently
committed
c45ece7
Support per-benchmark venvs in VirtualEnvironment.
ericsnowcurrently
committed
e940e24
Ignore pyproject.toml name only if provided.
ericsnowcurrently
committed
0420e96
Add requirements lock files to the benchmarks.
ericsnowcurrently
committed
6bce386
Make a venv for each benchmark instead of sharing one.
ericsnowcurrently
committed
29cf228
Support "libsdir" in metadata.
ericsnowcurrently
committed
6cadadc
Fix an error message.
ericsnowcurrently
committed
3fb5187
Use the default resolve() if the default manifest is explicit.
ericsnowcurrently
committed
abf56e9
Merge in the version properly.
ericsnowcurrently
committed
4bf223a
Preserve PYTHONPATH (with libsdir) when invoking pyperf via benchmark scripts.
ericsnowcurrently
committed
9acbbe4
Add a note about using an upstream lib for parsing pyproject.toml.
ericsnowcurrently
committed
d06aa7b
Use the full benchmark version rather than the canonicalized form.
ericsnowcurrently
committed
e48d29b
Add iter_clean_lines() to _utils.
ericsnowcurrently
committed
31f6021
Use a run ID when running benchmarks.
ericsnowcurrently
committed
4b13645
Finish implementing "pre" and "post" script support.
ericsnowcurrently
committed
b045e9a
Drop "pre" and "post" script support. (It isn't necessary.)
ericsnowcurrently
committed
54b8ba0
Use the correct venv name for each benchmark.
ericsnowcurrently
committed
362bb6e
Stop supporting "libsdir" in the benchmark metadata.
ericsnowcurrently
committed
e4fc5d4
Clean up run_perf_script() and related code.
ericsnowcurrently
committed
9816103
Add a missing import.
ericsnowcurrently
committed
b7f90f3
Drop accidental files.
ericsnowcurrently
committed
7d42a40
Install all dependencies when running tests.
ericsnowcurrently
committed
bb599e9
Temporarily get tests passing on Windows.
ericsnowcurrently
committed
df30711
Ignore stdlib_dir mismatch for now.
ericsnowcurrently
committed
8d563f9

Commits on Nov 2, 2021

Move requirements.txt into the data dir.
ericsnowcurrently
committed
d05b07c
Move the benchmarks to the data dir.
ericsnowcurrently
committed
701c910
Move _pythoninfo out of the _utils dir.
ericsnowcurrently
committed
afafda6
Move _pyproject_toml out of the _utils dir.
ericsnowcurrently
committed
cf4679a
Make _utils a single module.
ericsnowcurrently
committed
d3bdaad
Move benchmarks.* up to the top level.
ericsnowcurrently
committed
8f0b3e3
Move benchmark.* up to the top level.
ericsnowcurrently
committed
3ce65a6
Clean up the metadata files.
ericsnowcurrently
committed
3479e27
Do not import _manifest in itself.
ericsnowcurrently
committed
35e0c7a
Drop version and origin from the manifest.
ericsnowcurrently
committed
0fe268a
Add a project-level symlink to the default benchmarks.
ericsnowcurrently
committed
698dec9
Names starting with a digit.
ericsnowcurrently
committed
f3c6c4b
Drop the base metadata file.
ericsnowcurrently
committed
0987842
All extra project fields even if there is a metabase.
ericsnowcurrently
committed
2c99d0f
Fix a typo.
ericsnowcurrently
committed
121a281
Allow specifying the supported groups in the manifest.
ericsnowcurrently
committed
0f9d202
Add tags to all the benchmarks.
ericsnowcurrently
committed
3d19edf

Commits on Nov 3, 2021

Use the tags to get the groups in the default manifest.
ericsnowcurrently
committed
7270c76
Show the default group before the others.
ericsnowcurrently
committed
efeb77e
Drop an outdated comment.
ericsnowcurrently
committed
3b3de1d
Allow excluding benchmarks in a group.
ericsnowcurrently
committed
8ceb990
Finish _init_metadata().
ericsnowcurrently
committed
0f10859
metabase -> inherits
ericsnowcurrently
committed
9e3124d
Document the manifest and benchmark formats.
ericsnowcurrently
committed
62f7c23

Commits on Nov 6, 2021

Commits on Nov 8, 2021

Only re-install reqs for benchmark venvs.
ericsnowcurrently
committed
6fb7403

Commits on Nov 9, 2021

Support an "includes" section in the manifest.
ericsnowcurrently
committed
4979d5b
doc fixes
ericsnowcurrently
committed
7f7b571
"all" and "default" are always valid groups.
ericsnowcurrently
committed
0addec2
Do not import pyperformance._manifest unless already installed.
ericsnowcurrently
committed
57e7070
Ensure we run in a venv when needed.
ericsnowcurrently
committed
c55d690
Do not re-install the shared venv.
ericsnowcurrently
committed
01697d5
Only list the "all" and "default" groups once.
ericsnowcurrently
committed
94ac28a
Add the manifest to the "compile" config.
ericsnowcurrently
committed
f4b09bb
Adjust the stdlib_dir check.
ericsnowcurrently
committed
b56b25a
Be sure to set base_executable.
ericsnowcurrently
committed
f8338ae

Commits on Nov 15, 2021

Use the --venv opt to the "compile" command.
ericsnowcurrently
committed
dd4597b
Separate the logic for create vs. recreate.
ericsnowcurrently
committed
e22eeb8
Do not re-create the venv if already running in it.
ericsnowcurrently
committed
6bb3a9e
Ensure all requirements are always isntalled.
ericsnowcurrently
committed
9c664ff
Do not buffer stdout during tests.
ericsnowcurrently
committed
91c09d3
Fix a check.
ericsnowcurrently
committed
7a33956
Do not buffer stdout during tests.
ericsnowcurrently
committed
62e2014
Always switch to a venv if running out of the repo.
ericsnowcurrently
committed
d7ea256
Distinguish message from runtests.py.
ericsnowcurrently
committed
8ed6fd5
Pass values into VirtualEnvironment instead of the options object.
ericsnowcurrently
committed
ce6d09e
Do not buffer stdout during tests.
ericsnowcurrently
committed
4048199
Distinguish message from runtests.py.
ericsnowcurrently
committed
df8cccf
Print out the --venv option.
ericsnowcurrently
committed
8719636
Print out the --venv option.
ericsnowcurrently
committed
3c5d7da
Print out the --venv option.
ericsnowcurrently
committed
14761b2

Commits on Nov 16, 2021

Do not add args directly to the "venv" command.
ericsnowcurrently
committed
b975c51
Be explicit about "create".
ericsnowcurrently
committed
062745b
Drop debug messages.
ericsnowcurrently
committed
d918429
Resolve the manifest file in the compile config.
ericsnowcurrently
committed
5842d48
Add a "dryrun" mode for testing "compile".
ericsnowcurrently
committed
f1f9db1
Add BenchmarkManifest.show().
ericsnowcurrently
committed
35166b7
Use --manifest and --benchmarks when creating venv for "compile".
ericsnowcurrently
committed
561d271
Add the resolve_file() util.
ericsnowcurrently
committed
6831508
Resolve the manifest file in includes.
ericsnowcurrently
committed
bb88341
Default BenchmarkRevision._dryrun to False.
ericsnowcurrently
committed
1e51f8e
Set the default for --benchmarks manually.
ericsnowcurrently
committed
ef231fe
Allow the "venv" command to not install benchmark requirements.
ericsnowcurrently
committed
648bd18
Use <NONE> as a marker for "no benchmarks".
ericsnowcurrently
committed
7bb8d94
Require --benchmarks (or default) for some commands.
ericsnowcurrently
committed
15fd559
Do not always install the first benchmark venv.
ericsnowcurrently
committed
f041503
Separate creating venv from installing requirements.
ericsnowcurrently
committed
452b541
Only install per-benchmark requirements when running them.
ericsnowcurrently
committed
b049a4a
Print the benchmark number.
ericsnowcurrently
committed
a083cd5

Commits on Nov 17, 2021

Factor out Python.resolve_program().
ericsnowcurrently
committed
7b18753
Do not pass --benchmarks when creating venv for "compile".
ericsnowcurrently
committed
c8a7789
Skip a benchmark if its requirements could not be installed.
ericsnowcurrently
committed
6f6df4d
Set Python.program to None if the resolved path does not exist.
ericsnowcurrently
committed
b379536
Factor out resolve_python().
ericsnowcurrently
committed
686a96d
Add a blank line.
ericsnowcurrently
committed
664a909
Do not print a traceback for skipped benchmarks.
ericsnowcurrently
committed
9899813
Fix a typo.
ericsnowcurrently
committed
da1b6c3

Commits on Dec 7, 2021

Merge main.
ericsnowcurrently
committed
09ffb6b