The latest run is compared with the recorded runs of the newest tag reachable from HEAD; a benchmark regresses when its median is slower by more than THRESHOLD (10%) plus its noise, the larger spread of its samples on either side, and the report exits 1. COUNT=n records n samples per benchmark, and the per-run comparisons use their median.
- commit
- 2205e51f199081aba631629fdc57f2c9f9169551
- parent
- 77fde551eae0f41488a3653066111378499b87cc
- author
- Dami <dami.etoile@gmail.com>
| M | Makefile | 4 | +- |
| M | docs/ | 2 | +- |
| M | docs/ | 47 | ++++++++++++++++--- |
| M | docs/ | 2 | +- |
| M | scripts/ | 12 | +++++- |
| A | scripts/ | 71 | ++++++++++++++++++++++++++++++ |
| M | scripts/ | 59 | +++++++++++++++++++++---- |
| M | scripts/ | 8 | ++-- |
| M | scripts/ | 4 | +- |
@@ -65,7 +65,9 @@ stress:
6565 stress-data:6666 scripts/stress-data.sh6767 68-# Compare the latest run with the previous one and a baseline (DuckDB).68+# Compare the latest run with the previous one, a baseline and the last69+# tagged release's run (DuckDB); exits 1 on a regression against the70+# release past THRESHOLD (0.10) plus the benchmark's noise.6971 stress-report:7072 scripts/stress/report.sh7173 @@ -513,7 +513,7 @@ In order of value for effort. The first two are what raising
513513 ```sh514514 make stress # fetch datasets, run everything, record the run (5 minutes)515515 BENCH='Edit|Frame' make stress # a subset; BENCHTIME=2s for steadier numbers516-make stress-report # latest against previous and baseline, with trends516+make stress-report # latest against previous, baseline and last release, with trends517517 make stress-e2e # key press to screen through libghostty518518 ```519519 @@ -278,18 +278,49 @@ FROM read_json_auto('events.jsonl') WHERE msg = 'jev';
278278 `git_sha`, `git_dirty`, `go_version`, `goos`, `goarch`, `cpu`, `host`,279279 `pkg`, `name`, `procs`, `n`, `ns_op`, `b_op`, `allocs_op`, `mb_s`, and280280 `metrics`, a map of custom units such as `B/cell`, `cells/s`,281-`MB-file` or `us/answer`. `make stress-report` loads them with282-`scripts/stress/load.sql` and runs `scripts/stress/report.sql`: the283-latest run against the previous one on the same CPU and against a284-baseline (`BASELINE=run_id`, default the oldest run that covered most285-of the same benchmarks), flagging changes past `THRESHOLD` (0.10) and281+`MB-file` or `us/answer`. `COUNT=n` runs each benchmark n times, one282+record per sample. `make stress-report` loads them with283+`scripts/stress/load.sql` (a benchmark's time in a run is the median of284+its samples) and runs `scripts/stress/report.sql`: the latest run285+against the previous one on the same CPU and against a baseline286+(`BASELINE=run_id`, default the oldest run that covered most of the287+same benchmarks), flagging changes past `THRESHOLD` (0.10) and286288 `MIN_DELTA_NS` (2000), and the headline benchmarks over the last eight287289 runs. With the default 500 ms per benchmark, runs on a busy machine (say288290 with the Docker stack ingesting) differ by 10 to 30% on benchmarks of a289291 few tens of milliseconds; use `BENCHTIME=2s` on a quiet machine before290-trusting a single regression. `FAIL_ON_REGRESSION=1` makes it291-exit 1 on a regression, for CI. The same records load into ClickHouse292-(`stress.results`) with the same columns.292+trusting a single regression. `FAIL_ON_REGRESSION=1` also makes it exit293+1 on a regression against the previous run. The same records load into294+ClickHouse (`stress.results`) with the same columns; there, samples of295+one benchmark in one run collapse into one row.296+297+### Regressions against the last release298+299+`make stress-report` then compares the latest run with the last300+release's (`scripts/stress/release.sql`) and exits 1 when anything301+regressed, so a release checklist ([Releasing](releasing.md)) can stop302+on it:303+304+- **The release** is the newest `v*` tag reachable from HEAD whose305+ commit has a clean recorded run on the latest run's CPU and OS (other306+ than the latest run, and not on the latest run's commit). All such307+ runs of that commit are pooled. `RELEASE=v1.2.3` picks the tag,308+ `RELEASE_RUN=run_id` one run instead. With none recorded, the report309+ says so and exits 0: record one with `make stress` on the tagged310+ commit.311+- **Noise** is each benchmark's spread, `(max - min) / median` of its312+ samples, the larger of the release's and the latest run's. It is 0313+ for a benchmark with one sample on each side, so `COUNT=3` or more on314+ both runs gives the threshold room where a benchmark is noisy.315+- **A regression** is a median slower than the release's by more than316+ `THRESHOLD` plus the noise (10% plus, by default) and by more than317+ `MIN_DELTA_NS`. Faster past the same bar is listed as `faster`.318+319+```sh320+COUNT=3 BENCHTIME=2s make stress # on the tagged commit, then on the change321+make stress-report # exit 1 on a regression against the tag322+THRESHOLD=0.2 RELEASE=v0.2.0 make stress-report323+```293324 294325 ## The stack295326 @@ -18,7 +18,7 @@ publishes there, and `make dist` uploads nothing.
1818 2. **Stress** (optional for small releases): `make stress` on a quiet1919 machine, then `make stress-report`, which compares the run with the2020 last release's and exits non-zero on a regression past the threshold21- (see [Observability](observability.md#stress-runs)).21+ (see [Observability](observability.md#regressions-against-the-last-release)).2222 Record the run on the release commit so the next release compares2323 against it.2424 3. **XLSX in spreadsheet apps.** Export a workbook that uses what the@@ -30,7 +30,17 @@ CREATE OR REPLACE TEMP MACRO fmt_ns(ns) AS
3030 CREATE OR REPLACE TEMP TABLE runs AS3131 SELECT run_id, min(ts) AS ts, any_value(git_sha) AS git_sha, bool_or(git_dirty) AS dirty,3232 any_value(cpu) AS cpu, any_value(goos) AS goos, any_value(go_version) AS go_version,33- count(*) AS benchmarks,33+ count(DISTINCT (pkg, name)) AS benchmarks,3434 row_number() OVER (ORDER BY min(ts) DESC) AS age3535 FROM results3636 GROUP BY run_id;37+38+-- One row per benchmark per run: the median of its samples (a run made39+-- with COUNT=n has n) and their spread, (max - min) / median.40+CREATE OR REPLACE TEMP TABLE bench AS41+SELECT run_id, pkg, name, count(*) AS samples,42+ median(ns_op) AS ns_op, min(ns_op) AS min_ns, max(ns_op) AS max_ns,43+ CASE WHEN median(ns_op) > 0 THEN (max(ns_op) - min(ns_op)) / median(ns_op) ELSE 0 END AS spread,44+ median(allocs_op) AS allocs_op45+FROM results46+GROUP BY run_id, pkg, name;@@ -0,0 +1,71 @@
1+-- The latest run against the last release's recorded runs, with a2+-- threshold that allows for noise. Run by scripts/stress/report.sh after3+-- load.sql, with variables threshold, min_delta_ns and release_tags, a4+-- list of 'tag commit' strings, newest release first (from git), or a5+-- single run id in release_run.6+--7+-- The release's runs are every clean run of the newest tagged commit that8+-- has one on the latest run's CPU and OS, other than the latest run9+-- itself; its samples are pooled. A benchmark regresses when its median10+-- is slower than the release's by more than threshold + noise and by more11+-- than min_delta_ns, where noise is the larger spread,12+-- (max - min) / median, of its samples in either side.13+14+CREATE OR REPLACE TEMP TABLE release_tags AS15+SELECT split_part(t, ' ', 1) AS tag, split_part(t, ' ', 2) AS commit, i AS ord16+FROM (SELECT unnest(getvariable('release_tags')) AS t, generate_subscripts(getvariable('release_tags'), 1) AS i)17+WHERE t <> '';18+19+CREATE OR REPLACE TEMP TABLE release_pick AS20+WITH latest AS (SELECT * FROM runs WHERE age = 1),21+cand AS (22+ SELECT t.tag, t.ord, r.run_id23+ FROM release_tags t24+ JOIN runs r ON starts_with(t.commit, r.git_sha) AND length(r.git_sha) >= 725+ JOIN latest l ON r.cpu = l.cpu AND r.goos = l.goos26+ WHERE NOT r.dirty AND r.run_id <> l.run_id AND r.git_sha <> l.git_sha27+),28+tag AS (29+ SELECT tag FROM cand ORDER BY ord LIMIT 130+)31+SELECT run_id, tag FROM cand WHERE getvariable('release_run') = '' AND tag = (SELECT tag FROM tag)32+UNION ALL33+SELECT run_id, run_id AS tag FROM runs WHERE run_id = getvariable('release_run');34+35+CREATE OR REPLACE TEMP TABLE release_compared AS36+WITH l AS (SELECT r.* FROM bench r JOIN runs u USING (run_id) WHERE u.age = 1),37+rel AS (38+ SELECT r.pkg, r.name, count(*) AS samples, median(r.ns_op) AS ns_op,39+ CASE WHEN median(r.ns_op) > 0 THEN (max(r.ns_op) - min(r.ns_op)) / median(r.ns_op) ELSE 0 END AS spread40+ FROM results r JOIN release_pick p USING (run_id)41+ GROUP BY r.pkg, r.name42+)43+SELECT l.pkg, l.name, rel.ns_op AS before_ns, l.ns_op AS after_ns,44+ (l.ns_op - rel.ns_op) / rel.ns_op AS change,45+ greatest(l.spread, rel.spread) AS noise,46+ rel.samples AS release_samples, l.samples AS latest_samples,47+ CASE48+ WHEN (l.ns_op - rel.ns_op) / rel.ns_op > getvariable('threshold') + greatest(l.spread, rel.spread)49+ AND l.ns_op - rel.ns_op > getvariable('min_delta_ns') THEN 'REGRESSION'50+ WHEN (rel.ns_op - l.ns_op) / rel.ns_op > getvariable('threshold') + greatest(l.spread, rel.spread)51+ AND rel.ns_op - l.ns_op > getvariable('min_delta_ns') THEN 'faster'52+ ELSE ''53+ END AS flag54+FROM l JOIN rel USING (pkg, name);55+56+.print57+.print # Latest against the last release (threshold plus each benchmark's noise)58+SELECT (SELECT any_value(tag) FROM release_pick) AS release,59+ (SELECT count(*) FROM release_pick) AS release_runs,60+ count(*) AS compared,61+ count(*) FILTER (WHERE flag = 'REGRESSION') AS regressions,62+ count(*) FILTER (WHERE flag = 'faster') AS faster,63+ printf('%.0f%%', getvariable('threshold') * 100) AS threshold64+FROM release_compared;65+66+SELECT pkg, name, fmt_ns(before_ns) AS release, fmt_ns(after_ns) AS latest,67+ printf('%+.1f%%', change * 100) AS change,68+ printf('%.1f%%', noise * 100) AS noise,69+ release_samples || '/' || latest_samples AS samples, flag70+FROM release_compared WHERE flag <> ''71+ORDER BY flag DESC, change DESC;@@ -1,11 +1,17 @@
11 2-# report.sh - compare the latest stress run with the previous one and a3-# baseline, using DuckDB over the JSONL run records. No server needed.2+# report.sh - compare the latest stress run with the previous one, a3+# baseline and the last tagged release, using DuckDB over the JSONL run4+# records. No server needed. Exits 1 when the latest run regressed5+# against the last release's recorded run.46 #5-# THRESHOLD=0.10 flag changes over 10%7+# THRESHOLD=0.10 flag changes over 10% (plus the benchmark's noise8+# against the release)69 # MIN_DELTA_NS=2000 ignore changes under 2 us (noise)710 # BASELINE=run_id compare with this run (default: the oldest run)8-# FAIL_ON_REGRESSION=1 exit 1 when the latest run regressed against the previous11+# RELEASE=v1.2.3 compare with this tag's runs (default: the newest12+# tag reachable from HEAD that has a recorded run)13+# RELEASE_RUN=run_id compare with this run instead of a release's14+# FAIL_ON_REGRESSION=1 also exit 1 when the latest run regressed against the previous915 set -euo pipefail1016 1117 root=$(cd "$(dirname "$0")/../.." && pwd)@@ -19,20 +25,53 @@ if ! command -v duckdb >/dev/null 2>&1; then
1925 exit 12026 fi2127 28+# Release tags, newest version first, as 'tag commit' pairs.29+if [ -n "${RELEASE:-}" ]; then30+ tags=$(git -C "$root" rev-parse -q --verify "refs/tags/$RELEASE^{commit}" | sed "s/^/$RELEASE /") || {31+ echo "stress-report: no tag $RELEASE" >&232+ exit 133+ }34+else35+ tags=$(git -C "$root" tag -l 'v*' --merged HEAD --sort=-v:refname \36+ --format='%(refname:short) %(if)%(*objectname)%(then)%(*objectname)%(else)%(objectname)%(end)' 2>/dev/null || true)37+fi38+list=$(printf '%s\n' "$tags" | awk 'NF == 2 { printf "%s'\''%s'\''", (n++ ? ", " : ""), $0 }')39+2240 vars="SET VARIABLE results = '$results';2341 SET VARIABLE threshold = ${THRESHOLD:-0.10};2442 SET VARIABLE min_delta_ns = ${MIN_DELTA_NS:-2000};25-SET VARIABLE baseline = '${BASELINE:-}';"43+SET VARIABLE baseline = '${BASELINE:-}';44+SET VARIABLE release_run = '${RELEASE_RUN:-}';45+SET VARIABLE release_tags = [$list]::VARCHAR[];"2646 27-cat <(echo "$vars") "$root/scripts/stress/load.sql" "$root/scripts/stress/report.sql" | duckdb47+sql() {48+ cat <(echo "$vars") "$root/scripts/stress/load.sql" "$@"49+}2850 29-if [ "${FAIL_ON_REGRESSION:-}" = 1 ]; then30- n=$(cat <(echo "$vars") "$root/scripts/stress/load.sql" "$root/scripts/stress/report.sql" <(echo "51+sql "$root/scripts/stress/report.sql" "$root/scripts/stress/release.sql" | duckdb52+53+count() {54+ sql "$root/scripts/stress/report.sql" "$root/scripts/stress/release.sql" <(echo "3155 .mode list3256 .header off33-SELECT count(*) FROM compared WHERE kind = 'previous' AND flag = 'REGRESSION';") | duckdb | tail -1)57+$1") | duckdb | tail -158+}59+60+status=061+if [ "$(count 'SELECT count(*) FROM release_pick;')" = 0 ]; then62+ echo "stress-report: no recorded run of a release to compare with; record one with make stress on a tagged commit" >&263+else64+ n=$(count "SELECT count(*) FROM release_compared WHERE flag = 'REGRESSION';")65+ if [ "$n" != 0 ]; then66+ echo "stress-report: $n regressions against the last release" >&267+ status=168+ fi69+fi70+if [ "${FAIL_ON_REGRESSION:-}" = 1 ]; then71+ n=$(count "SELECT count(*) FROM compared WHERE kind = 'previous' AND flag = 'REGRESSION';")3472 if [ "$n" != 0 ]; then3573 echo "stress-report: $n regressions against the previous run" >&236- exit 174+ status=13775 fi3876 fi77+exit $status@@ -26,7 +26,7 @@ SELECT
2626 (SELECT run_id FROM same WHERE benchmarks >= 0.9 * (SELECT benchmarks FROM latest) ORDER BY age DESC LIMIT 1)) AS baseline;2727 2828 CREATE OR REPLACE TEMP TABLE compared AS29-WITH l AS (SELECT r.* FROM results r, picked p WHERE r.run_id = p.latest)29+WITH l AS (SELECT r.* FROM bench r, picked p WHERE r.run_id = p.latest)3030 SELECT l.pkg, l.name, kind, o.ns_op AS before_ns, l.ns_op AS after_ns,3131 (l.ns_op - o.ns_op) / o.ns_op AS change,3232 l.allocs_op - o.allocs_op AS allocs_change,@@ -39,9 +39,9 @@ SELECT l.pkg, l.name, kind, o.ns_op AS before_ns, l.ns_op AS after_ns,
3939 END AS flag4040 FROM l4141 JOIN (42- SELECT r.*, 'previous' AS kind FROM results r, picked p WHERE r.run_id = p.previous42+ SELECT r.*, 'previous' AS kind FROM bench r, picked p WHERE r.run_id = p.previous4343 UNION ALL44- SELECT r.*, 'baseline' AS kind FROM results r, picked p WHERE r.run_id = p.baseline AND p.baseline <> p.latest44+ SELECT r.*, 'baseline' AS kind FROM bench r, picked p WHERE r.run_id = p.baseline AND p.baseline <> p.latest4545 ) o USING (pkg, name);4646 4747 .print@@ -62,7 +62,7 @@ ORDER BY flag DESC, change DESC;
6262 .print # Headline benchmarks over the last 8 runs (time per op)6363 CREATE OR REPLACE TEMP TABLE headline AS6464 SELECT r.name, r.run_id, u.age, fmt_ns(r.ns_op) AS t65-FROM results r JOIN runs u USING (run_id)65+FROM bench r JOIN runs u USING (run_id)6666 WHERE u.age <= 8 AND regexp_matches(r.name,6767 '^(Edit/(chain|fanin|running|volatile)|RecalcAll/(dense|fanin)|Frame/(dense-8192x256|longtext-8192x500)/200x60|Keystroke/(arrow/dense-8192x256|extend-data/dense-8192x256)/80x24|Import/(owid|airport)|JEV/1000|Open/dense-8192x256|Save/dense-8192x256)');6868 PIVOT headline ON age USING first(t) GROUP BY name ORDER BY name;@@ -7,6 +7,8 @@
77 #88 # BENCH=Edit|Frame only these benchmarks (a go test -bench pattern)99 # BENCHTIME=500ms per benchmark (slow ones still run once)10+# COUNT=1 samples per benchmark (make stress-report reads11+# their spread as noise)1012 # PKGS="./internal/sheet"1113 set -euo pipefail1214 @@ -21,7 +23,7 @@ pkgs=${PKGS:-"./internal/sheet ./internal/ui ./internal/fileio ./internal/chart
2123 2224 # shellcheck disable=SC2086 # pkgs is a list2325 STRESS_DIR="$root/.deps/stress" go test -tags stress -run '^$' \24- -bench "${BENCH:-.}" -benchmem -benchtime "${BENCHTIME:-500ms}" -timeout 60m $pkgs 2>&1 |26+ -bench "${BENCH:-.}" -benchmem -benchtime "${BENCHTIME:-500ms}" -count "${COUNT:-1}" -timeout 60m $pkgs 2>&1 |2527 tee "$raw" | awk '/^Benchmark/ { print " " $1; fflush() } /^(FAIL|panic|--- FAIL)/ { print; fflush() }'2628 2729 sha=$(git -C "$root" rev-parse --short HEAD 2>/dev/null || echo unknown)