/pstack:create-formal-verification-skill
按需点名一句话测试碰不到的情况(比如并发、重试),检查某条规则永远成立。
- 什么时候用
- 有些地方出错代价很高,测试又很难触发。
- 要一起装
- maintain-formal-verification-skill、statechart
- 详细教程
- 测试碰不到的 bug:形式化验证
/pstack:create-formal-verification-skill 任务租约和重试逻辑。同一个任务绝不能同时跑两次。用它和不用它
同一件事,直接跟 AI 说,和点名用 /pstack:create-formal-verification-skill,AI 的做法不一样:
用 /pstack:create-formal-verification-skill文档写了什么
原文分 6 节,约 900 个英文单词,中文译文约 1500 字,读完约 4 分钟。点一节直接跳到全文。
开头开头对比测试和形式化验证,说明这个技能要挑一处验证一条规则,再留下一个下一个 AI 能拿来复查改动的技能。
- 1从代码里选目标从代码里找一条出错会伤到用户、测试又很少碰到的承诺,你点名了就从那里开始。
- 2让问题来选工具按问题挑工具:步骤顺序用 TLA+ 建小模型,每个输入都要对就用直接跑代码的验证器。
- 3怎样才算好七条标准:对得上代码、能失败、发现照实报、说清范围、不懂工具也能读、重跑便宜、跟代码同步。
- 4生成验证技能写带 frontmatter 的项目技能,记下覆盖的代码和同步的提交,TLA+ 结果交给
/pstack:statechart讲。 - 5交出去之前先证明它管用在真代码上做一个弄坏规则的临时改动,检查必须抓到,撤掉后回到原来的结果。
- 6交接告诉你验证了什么、为什么选这个工具、上限、发现了什么、怎么跑、哪些没覆盖。
写法亮点
挑了几句原文:每句让 AI 做什么、为什么这样写管用、你写提示词时怎么借来用。
先让检查失败一次
A check that you have never seen fail proves nothing.
一个你从没见它失败过的检查,什么也证明不了。
- 让 AI
- 它会先故意弄坏代码,确认检查真会报错,再告诉你检查通过了。
- 为什么管用
- 一句短而绝对的话,说穿了「通过」凭什么算数,AI 就没法拿一个从不报错的检查来交差。
- 借来用
- 设好一条提醒规则后加一句:「先造一条该触发的数据,证明它真的会提醒。」
点破省事的那条路
Choose the strongest such target in the repo, not the one that is easiest to verify.
选仓库里最有分量的那个目标,而不是最容易验证的那个。
- 让 AI
- 它去查最要紧的那部分代码,而不是挑最好做的来交差。
- 为什么管用
- AI 天然偏向省事的选项,直接点出这个诱惑,它就不会顺手走过去。
- 借来用
- 让 AI 分析时写:「挑最可能让客户流失的问题,不要挑最好分析的。」
不许为了过关改标准
Do not weaken the property or the model until it passes
不要削弱性质或模型直到它通过
- 让 AI
- 代码真有问题时,它把问题和反例报给你,而不是悄悄放宽检查让它通过。
- 为什么管用
- 点名最常见的一种作弊办法,比泛泛地说「要诚实」管用得多。
- 借来用
- 让 AI 修 bug 时加一句:「测试不过就告诉我哪里错了,不要改测试让它通过。」
说清结论管到哪
A pass covers the model or the bound, not the whole system.
通过只覆盖模型或那个上限,不覆盖整个系统。
- 让 AI
- 报告会写明验证了多大范围,你不会误以为整个系统都没问题。
- 为什么管用
- 给结论划好边界,读的人就不会把局部的通过当成全局的保证。
- 借来用
- 让 AI 报测试结果时写:「告诉我这次测了哪些浏览器,没测的也列出来。」
全文
AI 读到的就是这份 SKILL.md。可以看中文、看原文,或者一段一段对照着看。
descriptionGenerate a project-local skill that formally verifies the code where a bug costs the most and tests reach the least, in any language, with whichever verification tool fits. Use for /pstack:create-formal-verification-skill, "model-check this race", "prove this invariant", or when concurrency, retries, state machines, parsers, or unsafe code need more than tests.
description生成一个放在项目里的技能:找出 bug 代价最大、测试又最难碰到的那部分代码,对它做形式化验证。任何语言都行,用最合适的验证工具。用于 /pstack:create-formal-verification-skill、「model-check this race」、「prove this invariant」,或者并发、重试、状态机、解析器、unsafe 代码光靠测试不够的时候。
Create a formal verification skill
Create a formal verification skill(创建一个形式化验证技能)
A test checks the cases that someone thought to write. Formal verification checks every case in a stated scope: every order of steps in a model, every input up to a bound, or every input. This skill finds the part of the repo where that matters most, verifies one property of it, and leaves a project-local skill that lets the next agent check a change against that property. Write that skill for an agent who reads it cold, mid-change, and has never seen your work. The user must be able to trust the result and act on it without learning the verification language.
测试只检查有人想到要写的情况。形式化验证检查一个说明白的范围里的每一种情况:模型里步骤的每一种先后顺序、某个上限以内的每一个输入,或者全部输入。这个技能会找出仓库里最需要这样做的部分,验证它的一条性质,并留下一个放在项目里的技能,让下一个 AI 能拿改动去对照这条性质检查。写这个技能时,要写给这样一个 AI:它改代码改到一半,第一次打开这个技能,从没见过你做的事。用户必须能信任结果、照着结果行动,而不用去学验证语言。
Choose the target from the code
从代码里选目标
Answer from the codebase, and ask the user only what you cannot observe. If the user named an area or a bug, start there.
从代码库里找答案,只问用户你观察不到的东西。如果用户点名了某个区域或某个 bug,就从那里开始。
A good target is a promise the code makes, where a break harms users and tests rarely reach the case that breaks it. Choose the strongest such target in the repo, not the one that is easiest to verify.
好的目标是代码做出的一个承诺:一旦被打破就会伤到用户,而测试又很少碰到会打破它的情况。选仓库里最有分量的那个目标,而不是最容易验证的那个。
Let the question choose the tool
让问题来选工具
Pick the tool that answers the question at the lowest cost: a model checker, a proof assistant, an SMT-backed verifier, or another tool. Whether events can happen in a bad order usually suits a model checker over a small model of the design, and TLA+ is the default for that in any language. Whether the code itself behaves for every input, as with memory safety, overflow, or a data-structure invariant, usually suits a verifier that runs on the code. If no verifier runs on this language, say so, and test the real code with generated inputs, not a hand model of the function. A property that must hold at every size, where a bound is not enough, can need a proof. Prefer a tool that the repo's developers can install and run, and tell the user why you chose it.
挑能以最低成本回答这个问题的工具:模型检查器、证明助手、基于 SMT 的验证器,或者别的工具。「事件会不会按坏的顺序发生」这类问题,通常适合用模型检查器去查一个按设计做的小模型;这种情况不管什么语言,默认都用 TLA+。「代码本身是不是对每个输入都表现正常」,比如内存安全、溢出或某个数据结构的不变式,通常适合用直接在代码上运行的验证器。如果这门语言没有能用的验证器,就说出来,然后用自动生成的输入去测真实的代码,而不是测一个手写的函数模型。一条在任何规模下都必须成立、有上限还不够的性质,可能需要证明。优先选仓库的开发者自己能装、能跑的工具,并告诉用户你为什么选它。
What makes it good
怎样才算好
- True to the code. Every part of the model or proof points to code that a reviewer can open. What the verification treats as one indivisible step is one indivisible step in the code. Names come from this repo.
- Able to fail. A check that you have never seen fail proves nothing. Show each property failing on a broken variant of the code it guards, such as a removed check or a reordered step.
- Findings stay findings. When the current code breaks the property, report it with the counterexample in the code's terms. Do not weaken the property or the model until it passes, and do not change product code unless the user asks.
- Honest about scope. Say what is verified, under which bounds and assumptions, and what is left out. A pass covers the model or the bound, not the whole system.
- Readable without the tool. State the property and the harm in one plain sentence each. Tell a counterexample as a short sequence of steps in the code's words.
- Cheap to rerun. One command, a reproducible tool version, and a run short enough that the next agent will use it on every relevant change.
- Kept in step with the code. Record which files and functions each check covers, and the commit at which you last confirmed that the verification describes them. When the covered code changes after that commit, the next agent can see that the verification needs another look.
- 忠于代码。 模型或证明的每一部分,都指向审阅的人能打开的代码。验证里当作一个不可分割的步骤的,在代码里也必须是一个不可分割的步骤。名字都取自这个仓库。
- 能失败。 一个你从没见它失败过的检查,什么也证明不了。给它守护的代码做一个弄坏的版本,比如删掉一个检查、调换一个步骤的顺序,让每条性质都在上面失败一次给人看。
- 发现的问题就是问题。 当前代码打破了性质时,就报告出来,并用代码里的说法写出反例。不要削弱性质或模型直到它通过;除非用户要求,也不要改产品代码。
- 如实说明范围。 说清楚验证了什么、在哪些上限和假设下验证的、哪些没包括。通过只覆盖模型或那个上限,不覆盖整个系统。
- 不懂工具也能看懂。 性质和危害各用一句大白话说清。反例讲成一小串步骤,用代码里的说法。
- 重跑成本低。 一条命令,能复现的工具版本,跑得够快,让下一个 AI 每次有相关改动都会去跑。
- 跟着代码同步。 记下每个检查覆盖了哪些文件和函数,以及你最后一次确认验证与它们相符时所在的提交。那个提交之后,被覆盖的代码一改,下一个 AI 就能看出验证需要再看一遍。
Generate the verify skill
生成验证技能
Write a project-local skill with frontmatter that names the area and says when to use it. Without frontmatter, the skill never registers. From the skill alone, the next agent must be able to tell whether a change touches the verified code, run the check, tell a real bug from a modeling error, and update the verification when a change is intended. Put the coverage record and the sync commit in the skill, and tell the next agent to move the sync commit forward only after it confirms that the verification describes the code at the new commit. Use this repo's real commands, paths, and nouns, and leave no placeholders.
写一个放在项目里的技能,带上 frontmatter,写明它管哪个区域、什么时候用。没有 frontmatter,这个技能永远不会被注册。下一个 AI 必须光看这个技能就能做到:判断一个改动有没有碰到验证过的代码,运行检查,分清是真 bug 还是建模出了错,以及在改动是有意为之时更新验证。把覆盖记录和同步提交写进技能里,并告诉下一个 AI:只有在确认验证与新提交上的代码相符之后,才能把同步提交往前移。用这个仓库真实的命令、路径和名词,不留任何占位符。
If the check is a TLA+ model, tell the next agent to read every result with the pstack /pstack:statechart skill. It shows each invariant violation the checker finds as its shortest path in plain words. When the check passes, /pstack:statechart draws a chart of how the system moves between states. Refer to /pstack:statechart by name. Its commands, options, and file paths change between releases and machines, so copy none of them into the generated skill or its scripts.
如果检查是一个 TLA+ 模型,告诉下一个 AI 用 pstack 的 /pstack:statechart 技能去读每一个结果。它会把检查器找到的每一次不变式违反,用大白话展示成一条最短的路径。检查通过时,/pstack:statechart 会画一张图,说明系统怎么在各个状态之间移动。提到 /pstack:statechart 时只用名字。它的命令、选项和文件路径会随版本和机器变化,所以一样都不要抄进生成的技能或它的脚本里。
Prove it before you hand off
交出去之前先证明它管用
Follow the generated skill once, as the next agent would, on a scratch change that breaks the property in the real code. It must catch the break. Then drop the scratch change and confirm that the check is back where it started: passing, or failing only on the findings you will report. A verify skill that was never run is a draft. If the skill misses the break or a step does not run, fix the skill or the verification, not the scratch change.
像下一个 AI 那样,照着生成的技能走一遍:用一个临时改动,在真实代码里打破那条性质。它必须抓到这次破坏。然后撤掉这个临时改动,确认检查回到了开始时的样子:通过,或者只在你要报告的那些问题上失败。一个从没跑过的验证技能只是草稿。如果技能没抓到破坏,或者某一步跑不起来,就去修技能或验证,不要去改那个临时改动。
Hand off
交接
Tell the user what you verified, which tool you used and why, the bounds, what the check found in plain words, how to run it, and what it does not cover. Suggest a second target only if you found a close one. Point the user at /pstack:maintain-formal-verification-skill for bringing the verification back in line after changes that did not recheck it.
告诉用户:你验证了什么,用了什么工具、为什么用它,上限是多少,检查发现了什么(用大白话说),怎么运行,以及它不覆盖什么。只有找到了一个相近的目标时,才建议第二个目标。如果之后的改动没有重新检查,告诉用户可以用 /pstack:maintain-formal-verification-skill 让验证重新和代码对上。
原文作者 Lauren Tan(poteto),MIT 许可。译文由本站翻译,核对到 pstack 镜像 97de40d。 在 GitHub 上看