TestSpeed measures ten cases on mid-sized stress sheets: allocations per operation (deterministic, 1.25x plus 16) and time relative to a calibration workload timed beside it (best of seven, 2x, same OS and arch as the baseline only), retrying once before failing. It holds /tmp/012-bench.lock while timing, as a flock'd file or a mkdir lock. make speed-update rewrites internal/ui/testdata/speed.json.
- commit
- dbce1a12a75029cc3810f6bde3f86e08ce1ba6cb
- parent
- 49b7f2640d0e2bfabd76bb81fb8b249988e71499
- author
- Dami <dami.etoile@gmail.com>
- kind
- feat risk score 1.1 of 3, Jev's estimate
| M | Makefile | 17 | +++- |
| M | docs/ | 45 | +++++++++ |
| M | docs/ | 3 | +- |
| A | internal/ | 133 | ++++++++++++++++++++++++++++ |
| A | internal/ | 9 | ++ |
| A | internal/ | 71 | +++++++++++++++ |
| A | internal/ | 144 | ++++++++++++++++++++++++++++++ |
| A | internal/ | 56 | ++++++++++++ |
@@ -7,7 +7,7 @@ GHOSTTY_SRC := $(DEPS)/ghostty-src
77 GHOSTTY_OUT := $(DEPS)/ghostty88 GHOSTTY_STAMP := $(GHOSTTY_OUT)/.built-$(GHOSTTY_COMMIT)99 10-10+1111 1212 build:1313 CGO_ENABLED=0 go build -o bin/012 ./cmd/012@@ -25,13 +25,14 @@ test:
2525 STRESS_DIR=$(DEPS)/stress go test ./...2626 2727 # Everything that must pass before a push: formatting, vet (also with the28-# stress benchmarks), shape limits, unit tests, the excelize oracle and the29-# end-to-end tests in libghostty.28+# stress benchmarks), shape limits, unit tests, the speed gate, the29+# excelize oracle and the end-to-end tests in libghostty.3030 check:3131 @test -z "$$(gofmt -l cmd internal demos e2e oracle)" || { gofmt -l cmd internal demos e2e oracle; echo "gofmt: files above need formatting"; exit 1; }3232 go vet -tags stress ./...3333 $(MAKE) lint3434 STRESS_DIR=$(DEPS)/stress go test ./...35+ $(MAKE) speed3536 $(MAKE) oracle3637 $(MAKE) e2e3738 @@ -40,6 +41,16 @@ lint:
4041 go vet ./...4142 scripts/lint.sh4243 44+# The speed gate: frames and recalculation on mid-sized sheets against45+# internal/ui/testdata/speed.json, in about two seconds, alone so that46+# no other package's tests share the CPU. speed-update rewrites the47+# baseline. See docs/contributing/limits.md#the-speed-gate.48+speed:49+ go test ./internal/ui -run '^TestSpeed$$' -count=1 -speed50+51+speed-update:52+ go test ./internal/ui -run '^TestSpeed$$' -count=1 -speed-update -v53+4354 fuzz:4455 go test ./internal/sheet -run '^$$' -fuzz FuzzParse -fuzztime 60s4556 go test ./internal/sheet -run '^$$' -fuzz FuzzRead -fuzztime 60s@@ -624,3 +624,48 @@ overhead), `internal/fileio/stress_test.go` (imports and exports),
624624 `internal/stress`, up to `dense-1Mx10`, ten million numbers, the625625 default `max-cells`. Results accumulate in `.deps/stress/results` and, with626626 the observability stack up, in ClickHouse.627+628+### The speed gate629+630+`make speed`, part of `make check`, catches a clear slowdown before a631+push rather than at the next stress run. `TestSpeed`632+(`internal/ui/speed_test.go`) builds mid-sized sheets with633+`internal/stress` and draws frames through the stress benchmarks' fake634+terminal, so it measures what `BenchmarkKeystroke` and `BenchmarkEdit`635+do, on fewer cases and briefly: an arrow key through to its frame at636+200 x 60 (dense, laid out, color scale and chart sheets of 8192 rows),637+single edits (fan-in, a chain, criteria functions, lookups) and full638+recalculations (running totals, arrays). It takes about two seconds.639+640+Each case is held to `internal/ui/testdata/speed.json` two ways:641+642+- **Allocations** an operation, counted with `testing.AllocsPerRun`,643+ so the same code gives the same count on any machine and under any644+ load. A case fails past 1.25 times its baseline plus 16.645+- **Time** relative to a calibration workload (formatting numbers,646+ string map lookups, sorting) timed beside it in the same process,647+ the fastest of seven timings of each, so a slower or busier machine648+ slows both. A case fails past twice its baseline's ratio. Ratios649+ still differ between CPUs, so they are compared only on the650+ baseline's OS and architecture (the file records them); elsewhere651+ only allocations are.652+653+A case over its margin is measured once more before it fails. Runs on654+one machine vary by up to about a third, under load from other tests655+too, well inside the margin; the margins are for regressions of a656+multiple, and finer ones are what `make stress-report` is for.657+658+The gate runs alone (`go test ./...` skips `TestSpeed` without659+`-speed`), and while it times it holds `/tmp/012-bench.lock`, the lock660+other timed runs on the machine take: with flock(2) when the path is a661+file (`flock /tmp/012-bench.lock make stress` where the `flock` command662+exists), and otherwise as a directory it makes and removes, which needs663+no `flock` command. It waits up to ten minutes for a directory someone664+else holds.665+666+When it fails, run `make speed` again on a quiet machine, then look at667+the case with the stress benchmarks (`BENCH='Keystroke|Edit' make668+stress`). A slowdown that is intended, or a new case, is taken into the669+baseline with `make speed-update`, which measures every case and670+rewrites `speed.json` for the machine it runs on; commit it with the671+change that explains it.@@ -6,10 +6,11 @@ sidebar_position: 5
66 # Testing77 88 ```sh9-make check # all of the below that must pass before a push: gofmt, vet, lint, test, oracle, e2e9+make check # all of the below that must pass before a push: gofmt, vet, lint, test, speed, oracle, e2e1010 make lint # go vet, staticcheck, cognitive complexity at most 25, Go files at most 500 lines, doclint and doccheck (see Docs below)1111 make test # engine, file formats and UI unit tests1212 make fuzz # fuzz the formula parser and the .012, CSV, .wk1, XLSX and NUON readers13+make speed # frames and recalculation against a checked-in baseline (see limits.md)1314 make oracle # compare formulas and number formats with excelize1415 make e2e # run the real binary in a terminal emulator (needs Zig and pkg-config)1516 make screens # rewrite the golden screens and build the review gallery@@ -0,0 +1,133 @@
1+package ui2+3+import (4+ "flag"5+ "fmt"6+ "runtime"7+ "testing"8+9+ tea "charm.land/bubbletea/v2"10+11+ "github.com/FelineStateMachine/012/internal/sheet"12+ "github.com/FelineStateMachine/012/internal/stress"13+)14+15+// The speed gate (make speed, part of make check): frames and16+// recalculation on mid-sized sheets from internal/stress, held to17+// testdata/speed.json. See docs/contributing/limits.md#the-speed-gate.18+var (19+ speedFlag = flag.Bool("speed", false, "run the speed gate against testdata/speed.json")20+ speedUpdateFlag = flag.Bool("speed-update", false, "measure the speed gate and rewrite testdata/speed.json")21+)22+23+// speedCase is one measured operation: setup builds its state and24+// returns the operation, which must leave the state as it found it25+// after an even number of calls.26+type speedCase struct {27+ name string28+ setup func() func()29+}30+31+// frameCase is an arrow key through to its frame at 200 x 60, down and32+// up in turn, on the sheet build makes.33+func frameCase(name string, build func() *sheet.Sheet) speedCase {34+ return speedCase{"frame/" + name, func() func() {35+ m := sized(build(), 200, 60)36+ t := newFakeTerm(200, 60)37+ keys := [2]tea.KeyPressMsg{{Code: tea.KeyDown}, {Code: tea.KeyUp}}38+ i := 039+ return func() {40+ m.Update(keys[i%2])41+ t.frame(m)42+ i++43+ }44+ }}45+}46+47+// editCase is typing into one cell of a stress shape and the48+// incremental recalculation that follows, two entries in turn.49+func editCase(sh stress.Shape) speedCase {50+ return speedCase{"edit/" + sh.Name, func() func() {51+ s := sh.Build()52+ inputs := [2]string{sh.Input, sh.Input + "1"}53+ i := 054+ return func() {55+ if err := s.Set(sh.Edit, inputs[i%2]); err != nil {56+ panic(err)57+ }58+ i++59+ }60+ }}61+}62+63+// recalcCase is a full recalculation of the sheet build makes.64+func recalcCase(name string, build func() *sheet.Sheet) speedCase {65+ return speedCase{"recalc/" + name, func() func() {66+ s := build()67+ return s.RecalcAll68+ }}69+}70+71+func speedCases() []speedCase {72+ dense := func() *sheet.Sheet { return stress.Dense(stress.Rows, 26) }73+ laidout := func() *sheet.Sheet { return stress.Laidout(stress.Rows, 26) }74+ charts := func() *sheet.Sheet {75+ s := stress.Table(200, 3)76+ stress.Charts(s, 8)77+ return s78+ }79+ shape := func(name string, build func() *sheet.Sheet, a string, input string) stress.Shape {80+ at, _ := sheet.ParseAddr(a)81+ return stress.Shape{Name: name, Build: build, Edit: at, Input: input}82+ }83+ fanin := shape("fanin-1000xSUM8192", func() *sheet.Sheet { return stress.FanIn(stress.Rows, 1000) }, "A4001", "7")84+ return []speedCase{85+ frameCase("dense-8192x26", dense),86+ frameCase("laidout-8192x26", laidout),87+ frameCase("scale-8192x26", scaled),88+ frameCase("charts-8", charts),89+ editCase(fanin),90+ editCase(shape("chain-8192", func() *sheet.Sheet { return stress.Chain(stress.Rows) }, "A1", "2")),91+ editCase(shape("criteria-10xSUMIF8192", func() *sheet.Sheet { return stress.Criteria(stress.Rows, 10) }, "A4001", "7")),92+ editCase(shape("lookup-100xVLOOKUP8192", func() *sheet.Sheet { return stress.Lookup(stress.Rows, 100) }, "A4001", "4000")),93+ recalcCase("running-8192", func() *sheet.Sheet { return stress.RunningTotals(stress.Rows) }),94+ recalcCase("arrays-50xFILTER8192", func() *sheet.Sheet { return stress.Arrays(stress.Rows, 50) }),95+ }96+}97+98+// TestSpeed measures every case and compares it with the baseline:99+// allocations per operation, and time per operation relative to a100+// calibration workload timed beside it. A case over its margin is101+// measured once more before it fails, so one noisy run doesn't.102+func TestSpeed(t *testing.T) {103+ if !*speedFlag && !*speedUpdateFlag {104+ t.Skip("the speed gate runs with -speed (make speed)")105+ }106+ unlock := benchLock(t)107+ defer unlock()108+ base, err := readSpeedBaseline()109+ if err != nil && !*speedUpdateFlag {110+ t.Fatal(err)111+ }112+ next := speedBaseline{GOOS: runtime.GOOS, GOARCH: runtime.GOARCH, Cases: map[string]speedResult{}}113+ for _, c := range speedCases() {114+ op := c.setup()115+ got := measureSpeed(op)116+ if !*speedUpdateFlag {117+ if verdict := base.judge(c.name, got); verdict != "" {118+ got = measureSpeed(op) // a second chance, against noise119+ if verdict = base.judge(c.name, got); verdict != "" {120+ t.Errorf("%s: %s", c.name, verdict)121+ }122+ }123+ }124+ t.Logf("%-36s %8.3f ms %6.2fx calibration %8.0f allocs", c.name, got.NS/1e6, got.Ratio, got.Allocs)125+ next.Cases[c.name] = got126+ }127+ if *speedUpdateFlag {128+ if err := next.write(); err != nil {129+ t.Fatal(err)130+ }131+ fmt.Println("wrote", speedBaselinePath)132+ }133+}@@ -0,0 +1,71 @@
1+//go:build unix2+3+package ui4+5+import (6+ "errors"7+ "io/fs"8+ "os"9+ "syscall"10+ "testing"11+ "time"12+)13+14+// benchLockPath is the lock timed runs on this machine share, so their15+// timings don't overlap. Two conventions meet there: a file locked with16+// flock(2), as `flock /tmp/012-bench.lock <command>` takes it where the17+// flock command exists, and a directory made with mkdir and removed18+// after, which needs no flock (macOS has none). The speed gate takes19+// whichever the path holds, a directory when it holds nothing.20+const benchLockPath = "/tmp/012-bench.lock"21+22+// benchLockWait is how long the gate waits for a directory lock before23+// timing without it (the lock may be stale, left by a run that was24+// killed).25+const benchLockWait = 10 * time.Minute26+27+// benchLock waits for the lock and returns its release.28+func benchLock(t *testing.T) func() {29+ t.Helper()30+ deadline := time.Now().Add(benchLockWait)31+ logged := false32+ for {33+ err := os.Mkdir(benchLockPath, 0o755)34+ if err == nil {35+ return func() { os.Remove(benchLockPath) }36+ }37+ info, serr := os.Stat(benchLockPath)38+ switch {39+ case !errors.Is(err, fs.ErrExist) || serr != nil:40+ t.Logf("timing without the lock: %v", err)41+ return func() {}42+ case info.Mode().IsRegular():43+ return flockFile(t)44+ case time.Now().After(deadline):45+ t.Logf("timing without the lock: %s held for %v", benchLockPath, benchLockWait)46+ return func() {}47+ }48+ if !logged {49+ t.Logf("waiting for %s", benchLockPath)50+ logged = true51+ }52+ time.Sleep(200 * time.Millisecond)53+ }54+}55+56+// flockFile takes the lock as a file, with flock(2).57+func flockFile(t *testing.T) func() {58+ f, err := os.OpenFile(benchLockPath, os.O_RDWR, 0)59+ if err != nil {60+ t.Logf("timing without the lock: %v", err)61+ return func() {}62+ }63+ if err := syscall.Flock(int(f.Fd()), syscall.LOCK_EX); err != nil {64+ f.Close()65+ t.Fatalf("locking %s: %v", benchLockPath, err)66+ }67+ return func() {68+ syscall.Flock(int(f.Fd()), syscall.LOCK_UN)69+ f.Close()70+ }71+}@@ -0,0 +1,144 @@
1+package ui2+3+import (4+ "encoding/json"5+ "fmt"6+ "math"7+ "os"8+ "runtime"9+ "slices"10+ "strconv"11+ "testing"12+ "time"13+)14+15+// speedBaselinePath holds the speed gate's baseline, rewritten by16+// make speed-update.17+const speedBaselinePath = "testdata/speed.json"18+19+// The margins a case may exceed its baseline by before it fails: time20+// relative to the calibration may double (the ratio still moves with21+// the CPU, the Go version and the machine's load, and a clear22+// regression is a multiple, not a percentage), and allocations, which23+// are counted, not timed, may grow by a quarter plus a few.24+const (25+ ratioMargin = 2.026+ allocsMargin = 1.2527+ allocsSlack = 1628+)29+30+// speedRounds is how many times each case is timed, beside the31+// calibration, keeping the fastest of each; speedBatch is about how long32+// one timing of a case runs.33+const (34+ speedRounds = 735+ speedBatch = 2 * time.Millisecond36+)37+38+// speedResult is one case measured: nanoseconds per operation (for39+// reading), the same relative to the calibration workload, and40+// allocations per operation.41+type speedResult struct {42+ NS float64 `json:"ns"`43+ Ratio float64 `json:"ratio"`44+ Allocs float64 `json:"allocs"`45+}46+47+// speedBaseline is testdata/speed.json: the machine it was measured on48+// and each case's result. Ratios are compared only on the same OS and49+// architecture; allocations everywhere.50+type speedBaseline struct {51+ GOOS string `json:"goos"`52+ GOARCH string `json:"goarch"`53+ Cases map[string]speedResult `json:"cases"`54+}55+56+func readSpeedBaseline() (speedBaseline, error) {57+ var b speedBaseline58+ data, err := os.ReadFile(speedBaselinePath)59+ if err != nil {60+ return b, fmt.Errorf("%w (make speed-update writes it)", err)61+ }62+ return b, json.Unmarshal(data, &b)63+}64+65+func (b speedBaseline) write() error {66+ for name, r := range b.Cases {67+ r.NS = math.Round(r.NS)68+ r.Ratio = math.Round(r.Ratio*1000) / 100069+ r.Allocs = math.Round(r.Allocs)70+ b.Cases[name] = r71+ }72+ data, err := json.MarshalIndent(b, "", " ")73+ if err != nil {74+ return err75+ }76+ return os.WriteFile(speedBaselinePath, append(data, '\n'), 0o644)77+}78+79+// judge says how got is over the baseline's margins, or "" when it80+// isn't.81+func (b speedBaseline) judge(name string, got speedResult) string {82+ want, ok := b.Cases[name]83+ if !ok {84+ return "not in " + speedBaselinePath + " (make speed-update)"85+ }86+ if limit := want.Allocs*allocsMargin + allocsSlack; got.Allocs > limit {87+ return fmt.Sprintf("%.0f allocations an operation, over %.0f (baseline %.0f)", got.Allocs, limit, want.Allocs)88+ }89+ sameMachine := b.GOOS == runtime.GOOS && b.GOARCH == runtime.GOARCH90+ if limit := want.Ratio * ratioMargin; sameMachine && got.Ratio > limit {91+ return fmt.Sprintf("%.2fx the calibration, over %.2fx (baseline %.2fx, %.3f ms now)", got.Ratio, limit, want.Ratio, got.NS/1e6)92+ }93+ return ""94+}95+96+// measureSpeed times op against the calibration and counts its allocations.97+// Calls come in even numbers, so an operation alternating two states98+// ends where it started.99+func measureSpeed(op func()) speedResult {100+ op()101+ op()102+ n := 2103+ for start := time.Now(); time.Since(start) < speedBatch/2; n += 2 {104+ op()105+ op()106+ }107+ allocs := testing.AllocsPerRun(n, op)108+ op() // AllocsPerRun calls it once more than n109+ best, calib := time.Duration(math.MaxInt64), time.Duration(math.MaxInt64)110+ for range speedRounds {111+ calib = min(calib, calibrate())112+ start := time.Now()113+ for range n {114+ op()115+ }116+ best = min(best, time.Since(start))117+ }118+ per := float64(best) / float64(n)119+ return speedResult{NS: per, Ratio: per / float64(calib), Allocs: allocs}120+}121+122+// calibrationSink keeps the calibration's result alive.123+var calibrationSink float64124+125+// calibrate times a fixed workload of the kinds of work the cases do126+// (formatting numbers, map lookups by string, sorting, allocating) and127+// returns how long it took: the unit the cases' times are measured in.128+func calibrate() time.Duration {129+ start := time.Now()130+ for range 4 {131+ m := make(map[string]float64)132+ xs := make([]float64, 0, 2048)133+ for i := range 2048 {134+ k := strconv.Itoa(i * 7919 % 4099)135+ m[k] += float64(i)136+ xs = append(xs, float64(i*7919%4099))137+ }138+ slices.Sort(xs)139+ for i := range 2048 {140+ calibrationSink += m[strconv.Itoa(i)] + xs[i]141+ }142+ }143+ return time.Since(start)144+}@@ -0,0 +1,56 @@
1+{2+ "goos": "darwin",3+ "goarch": "arm64",4+ "cases": {5+ "edit/chain-8192": {6+ "ns": 1637375,7+ "ratio": 3.051,8+ "allocs": 82989+ },10+ "edit/criteria-10xSUMIF8192": {11+ "ns": 4130156,12+ "ratio": 7.574,13+ "allocs": 37514+ },15+ "edit/fanin-1000xSUM8192": {16+ "ns": 481802,17+ "ratio": 0.911,18+ "allocs": 109819+ },20+ "edit/lookup-100xVLOOKUP8192": {21+ "ns": 2673615,22+ "ratio": 4.88,23+ "allocs": 90624+ },25+ "frame/charts-8": {26+ "ns": 1746531,27+ "ratio": 3.205,28+ "allocs": 729529+ },30+ "frame/dense-8192x26": {31+ "ns": 774323,32+ "ratio": 1.442,33+ "allocs": 486734+ },35+ "frame/laidout-8192x26": {36+ "ns": 1166000,37+ "ratio": 2.162,38+ "allocs": 522339+ },40+ "frame/scale-8192x26": {41+ "ns": 1137615,42+ "ratio": 2.096,43+ "allocs": 1014244+ },45+ "recalc/arrays-50xFILTER8192": {46+ "ns": 5452500,47+ "ratio": 10,48+ "allocs": 295449+ },50+ "recalc/running-8192": {51+ "ns": 1641448,52+ "ratio": 2.97,53+ "allocs": 826954+ }55+ }56+}