mirror of
https://github.com/RGBCube/uutils-coreutils
synced 2025-07-29 12:07:46 +00:00
factor::benchmarking(doc): Add guidance on writing µbenches
This commit is contained in:
parent
e9f8194266
commit
1d75f09743
1 changed files with 31 additions and 0 deletions
|
@ -32,3 +32,34 @@ as possible:
|
||||||
[lemire]: https://lemire.me/blog/2018/01/16/microbenchmarking-calls-for-idealized-conditions/
|
[lemire]: https://lemire.me/blog/2018/01/16/microbenchmarking-calls-for-idealized-conditions/
|
||||||
[isolate a **physical** core]: https://pyperf.readthedocs.io/en/latest/system.html#isolate-cpus-on-linux
|
[isolate a **physical** core]: https://pyperf.readthedocs.io/en/latest/system.html#isolate-cpus-on-linux
|
||||||
[frequency stays constant]: XXXTODO
|
[frequency stays constant]: XXXTODO
|
||||||
|
|
||||||
|
|
||||||
|
### Guidance for designing µbenchmarks
|
||||||
|
|
||||||
|
*Note:* this guidance is specific to `factor` and takes its application domain
|
||||||
|
into account; do not expect it to generalise to other projects. It is based
|
||||||
|
on Daniel Lemire's [*Microbenchmarking calls for idealized conditions*][lemire],
|
||||||
|
which I recommend reading if you want to add benchmarks to `factor`.
|
||||||
|
|
||||||
|
1. Select a small, self-contained, deterministic component
|
||||||
|
`gcd` and `table::factor` are good example of such:
|
||||||
|
- no I/O or access to external data structures ;
|
||||||
|
- no call into other components ;
|
||||||
|
- behaviour is deterministic: no RNG, no concurrency, ... ;
|
||||||
|
- the test's body is *fast* (~100ns for `gcd`, ~10µs for `factor::table`),
|
||||||
|
so each sample takes a very short time, minimizing variability and
|
||||||
|
maximizing the numbers of samples we can take in a given time.
|
||||||
|
|
||||||
|
2. Benchmarks are immutable (once merged in `uutils`)
|
||||||
|
Modifying a benchmark means previously-collected values cannot meaningfully
|
||||||
|
be compared, silently giving nonsensical results. If you must modify an
|
||||||
|
existing benchmark, rename it.
|
||||||
|
|
||||||
|
3. Test common cases
|
||||||
|
We are interested in overall performance, rather than specific edge-cases;
|
||||||
|
use **reproducibly-randomised inputs**, sampling from either all possible
|
||||||
|
input values or some subset of interest.
|
||||||
|
|
||||||
|
4. Use [`criterion`], `criterion::black_box`, ...
|
||||||
|
`criterion` isn't perfect, but it is also much better than ad-hoc
|
||||||
|
solutions in each benchmark.
|
||||||
|
|
Loading…
Add table
Add a link
Reference in a new issue