The default baseline is now the oldest run on the same machine that covered most of the benchmarks the latest run did, not a narrow run of a few. The docs say how noisy short benchmarks are on a busy machine.
- commit
- 176dc9b152f519abdc988cf1e4e6397f60792a9b
- parent
- 3573ef954fd1a4026c396ac31e86453486593194
- author
- Dami <dami.etoile@gmail.com>
| M | docs/ | 10 | +++++++--- |
| M | scripts/ | 7 | +++++-- |
@@ -114,9 +114,13 @@ FROM read_json_auto('events.jsonl') WHERE msg = 'jev';
114114 `MB-file` or `us/answer`. `make stress-report` loads them with115115 `scripts/stress/load.sql` and runs `scripts/stress/report.sql`: the116116 latest run against the previous one on the same CPU and against a117-baseline (`BASELINE=run_id`, default the oldest), flagging changes past118-`THRESHOLD` (0.10) and `MIN_DELTA_NS` (2000), and the headline119-benchmarks over the last eight runs. `FAIL_ON_REGRESSION=1` makes it117+baseline (`BASELINE=run_id`, default the oldest run that covered most118+of the same benchmarks), flagging changes past `THRESHOLD` (0.10) and119+`MIN_DELTA_NS` (2000), and the headline benchmarks over the last eight120+runs. With the default 500 ms per benchmark, runs on a busy machine (say121+with the Docker stack ingesting) differ by 10 to 30% on benchmarks of a122+few tens of milliseconds; use `BENCHTIME=2s` on a quiet machine before123+trusting a single regression. `FAIL_ON_REGRESSION=1` makes it120124 exit 1 on a regression, for CI. The same records load into ClickHouse121125 (`stress.results`) with the same columns.122126 @@ -2,7 +2,8 @@
22 -- on the same CPU, regressions flagged, and trends of the headline33 -- benchmarks. Run by scripts/stress/report.sh after load.sql, with44 -- variables threshold (fraction, e.g. 0.10), min_delta_ns (smaller5--- changes are noise) and baseline (a run_id, or '' for the oldest run).5+-- changes are noise) and baseline (a run_id, or '' for the oldest run that6+-- covered most of the same benchmarks).67 78 .mode box89 .maxrows 400@@ -20,7 +21,9 @@ same AS (SELECT r.* FROM runs r, latest l WHERE r.cpu = l.cpu AND r.goos = l.goo
2021 SELECT2122 (SELECT run_id FROM latest) AS latest,2223 (SELECT run_id FROM same WHERE age > 1 ORDER BY age LIMIT 1) AS previous,23- coalesce(nullif(getvariable('baseline'), ''), (SELECT run_id FROM same ORDER BY age DESC LIMIT 1)) AS baseline;24+ -- By default the oldest run that covered most of what the latest did.25+ coalesce(nullif(getvariable('baseline'), ''),26+ (SELECT run_id FROM same WHERE benchmarks >= 0.9 * (SELECT benchmarks FROM latest) ORDER BY age DESC LIMIT 1)) AS baseline;2427 2528 CREATE OR REPLACE TEMP TABLE compared AS2629 WITH l AS (SELECT r.* FROM results r, picked p WHERE r.run_id = p.latest)