/pstack:benchmark-checklist
AutoIn one lineChecks a performance number against 7 questions. The verdict is only faster, slower, no difference, or not sure.
- When to use
- You measured performance, or someone's numbers look suspiciously good.
/pstack:benchmark-checklist Check this 40% speedup before it goes into the PRJust asking vs. using it
The same request, typed plainly or with /pstack:benchmark-checklist. Here's what changes.
With /pstack:benchmark-checklistWhat the document says
4 sections, about 850 words, 4 min to read. Click a section to read it.
OpeningUse it whenever you report a performance number. A ballpark you asked for needs one run, but still checks questions 4 and 7.
- 1Before you run anythingWrites down the claim to test, reads the measurement script, and checks machine load with uptime and nproc.
- 2The questionsAnswers 7 questions: what limits it, was it tuned, did it break limits, did it error, reproduce, matter, happen.
- 3ReportLeads with the verdict and the number, run count, range and limiter. No named limiter means inconclusive.
- 4Other perf materialVets the Perf issue playbook's baseline and the Hillclimb harness with this checklist before they are used.
Wording worth borrowing
A few sentences from the original: what each one makes the AI do, why it works, and how you can borrow it.
Write the claim before testing
Write down the claim you expect to make, in the words you would ship
- Makes the AI
- Before measuring, it writes the sentence it plans to report, and every question then tests that sentence.
- Why it works
- One concrete claim gives the checks a target, so it can't measure first and invent the story after.
- Borrow it
- Before research, add: "First write the one-line conclusion you expect to give me, like 'the home page loads twice as fast', then check if it holds."
Use arithmetic as a guardrail
Removing a piece that takes 10% of the run can make the run at most about 11% faster.
- Makes the AI
- When a speedup looks too good, it checks whether it's even possible, and admits a bad measurement if not.
- Why it works
- A worked example beats "don't exaggerate", because the AI can redo the math itself.
- Borrow it
- Give a limit it can compute: "This section is a tenth of the text, so cutting it saves at most a tenth of the reading time."
Limit the verdict to a few options
Lead with the verdict: faster, slower, no measurable difference, or inconclusive.
- Makes the AI
- The report opens with one of four verdicts, and "inconclusive" is a real answer.
- Why it works
- A short fixed list that includes "inconclusive" stops the AI from forcing a flattering result.
- Borrow it
- When you want a verdict, list the options: "Start with one of: ready to ship, not ready, can't tell yet."
Turn checks into short questions
Why not double?
- Makes the AI
- Every number has to answer seven questions, like "why not double?", which makes it find what limits the speed.
- Why it works
- A short question is easy to remember and demands an answer, so it's harder to brush past than a rule.
- Borrow it
- Before a draft is done, have the AI answer: "Who reads this? What do they do next? Which line can go?"
Full text
The SKILL.md the AI reads, as written.
descriptionVet a perf measurement (limiter, tuning, limits, errors, repeatability, relevance, and whether the work happened) before you report or act on it. Use when you run a benchmark or report a speedup or regression you measured.
Benchmark checklist
Use this when you produce a performance number: a PR's before and after, a regression claim, a hillclimb harness, or a library or config choice. Answer each question with evidence from a run, not a guess about the code.
For a quick ballpark the user asked for, one run is enough. Still check questions 4 and 7, and say that it is one run. Skip the rest unless that run looks wrong. A choice between options is never a ballpark.
Before you run anything
- Write down the claim you expect to make, in the words you would ship ("export is 30% faster at p50 on the 60k-row dataset"). The questions test that sentence.
- Read the measurement script. Note what it times, counts, and ignores.
- Check the load average with
uptimeand the core count withnproc. If the machine is busy, find out what is running. If you cannot stop it, interleave the sides so both see the same noise, and say so in the report.
The questions
- Why not double? Name the limiter. Profile in a run you do not report, because profilers and tracers slow the work down. Use CPU per process (
top,pidstat), a runtime profiler (node --cpu-prof,py-spy,perf), I/O wait, and syscall counts (strace -con Linux), then map the hot spot to source. Watch the load generator too: if it saturates first, you measured the load generator. If a change did not move the number, the limiter explains why, so find it before you call the change useless. - Was it tuned? Run every side the way production runs it: release builds, production flags and env, batching and transaction settings, connection pools, caches as warm or cold as production sees them, and the same versions and data. If one side runs on defaults, you compared configurations, not implementations. A limiter that is a setting (a commit per row, a debug build, a missing index) means that side is untuned. Tune it and measure again before you pick a winner. If you cannot tune it, do not pick a winner from that run. When the user is choosing what to adopt, narrowing the claim to the code as it ships today does not fix this, because they adopt the option, not today's settings.
- Did it break limits? Do the arithmetic. Compare bytes per second with disk and network bandwidth, operations per second times cost per operation with your cores, and the time saved with the time the changed piece took. Removing a piece that takes 10% of the run can make the run at most about 11% faster. A result past a limit measured something other than the work, such as a cache, a no-op, or a bug.
- Did it error? Count failures and non-success responses, and check that outputs are correct, not just present. Rejections are often fast, and timeouts and retries are slow. If the script does not count errors, add the count.
- Does it reproduce? Run each side at least 5 times, alternating (A, B, A, B, and so on) so warmup, lazy initialization, caches, and drift favor neither. Report the median and range. A gap smaller than the run-to-run variation is no measurable difference. When the call is close, use a rank-sum test or the harness's own statistics.
- Does it matter? Next to any micro result, measure the end-to-end path a user waits on, with realistic data sizes and concurrency, and report the micro result as a share of the whole. A helper that takes 1% of a request can make the request at most 1% faster, however fast the helper gets.
- Did it even happen? Confirm the work ran inside the timed region: the request reached the server, the rows were written, the bytes were read, and the code used the result. Lazy code (generators nobody iterates, promises nobody awaits, results the JIT can discard) and timeouts produce numbers for work that never happened.
Report
- Lead with the verdict: faster, slower, no measurable difference, or inconclusive.
- Give the number with its unit, the run count, the range, and the limiter. For example, "p50 41 ms → 33 ms, median of 7 runs per side, range 32 to 35 ms after, bound by JSON parsing on one core."
- Call the verdict inconclusive when you claim a difference but cannot name the limiter, when a side ran untuned, or when you could not check questions 4 and 7. Name the gap.
- Keep a PR body to one primary number, per the Opening a PR playbook. Put the runs, range, and limiter evidence in a linked artifact or notes file.
Other perf material
- Vet the Perf issue playbook's baseline before the playbook plans from it, and every number after that.
- Vet the Hillclimb playbook's harness before it is frozen. The frozen harness then prints error and work counts, so each keep-or-revert checks questions 4 and 7 for free.
Original by Lauren Tan (poteto), MIT license. Checked against the pstack mirror at 97de40d. View on GitHub