跳到正文
  1. 技能速查
  2. 验证
  3. /pstack:maintain-formal-verification-skill

/pstack:maintain-formal-verification-skill

按需点名

一句话验证过的代码改了,它更新并重跑检查,告诉你是检查过时还是真有 bug。

什么时候用
合并或重构碰到了验证过的代码。
这样说
/pstack:maintain-formal-verification-skill

用它和不用它

同一件事,直接跟 AI 说,和点名用 /pstack:maintain-formal-verification-skill,AI 的做法不一样:

直接说用 /pstack:maintain-formal-verification-skill
检查通过了,就当代码没问题通过后故意弄坏新代码,确认检查还能抓到
分不清是验证过时还是代码有 bug对着真代码一步步回放反例,每步都能发生才算 bug
检查失败,就改模型改到通过先照代码写好更新再重跑,不按结果去凑

文档写了什么

原文分 4 节,约 1150 个英文单词,中文译文约 1750 字,读完约 5 分钟。点一节直接跳到全文。

开头开头点出最危险的情况是代码改了检查照样通过,并说明这个技能处理合并、手改这类绕过验证技能的改动。

  1. 1结果每个检查给四种结论之一:in sync、updated、bug、blocked,updated 和 bug 可以同时成立。
  2. 2能改什么只改模型、harness、规则文件和验证技能,产品代码不碰,规则失败就照实报告。
  3. 3一轮检查六步:找检查,先原样跑,读每个改动并分四类,照代码更新,重跑判断,更新同步提交。
  4. 4报告每个检查报结论、读过的提交、改动原因、用来证明能失败的破坏,并分开说哪些是过时哪些是 bug。

看写法亮点读全文

写法亮点

挑了几句原文:每句让 AI 做什么、为什么这样写管用、你写提示词时怎么借来用。

  1. 点破最让人放心的假象

    That pass is the dangerous case, because it looks like a proof.

    这种通过最危险,因为它看起来像一个证明。

    让 AI
    检查通过时,它不会就此放心,而是先怀疑这个通过还算不算数。
    为什么管用
    「通过」平时是让人放心的信号,直接说它最危险,AI 就会主动去查,而不是顺手打勾。
    借来用
    想让 AI 别轻信好消息,就说:「没有客户投诉不代表大家满意,先当它是危险信号查一遍。」
  2. 规定先写后看结果

    Write the update from the code before you rerun, and do not shape it by what makes the check pass.

    重跑之前,先根据代码写好更新,不要按「怎样能让检查通过」去塑造它。

    让 AI
    它照着代码改验证,不会为了让检查变成通过去凑答案。
    为什么管用
    把「看结果之前就动手」写成先后顺序,AI 就没有机会照着结果倒推。
    借来用
    让 AI 做分析时写:「先写下你的预测,再去看数据,看完数据不要回头改预测。」
  3. 要求亲手弄坏一次

    Break the new code the way the property guards against

    按性质要防的那种方式把新代码弄坏

    让 AI
    检查通过后,它会故意把代码弄坏一次,确认检查真的会报错。
    为什么管用
    一次证明「检查会失败」的破坏,比「我觉得检查有效」有说服力得多。
    借来用
    让 AI 写完表单校验后补一句:「故意填一个错的手机号,确认它真的会拦下来。」
  4. 只许几个固定的结论

    Give each check one outcome, and say which:

    每个检查给一个结果,并说出是哪一个:

    让 AI
    每个检查最后都落在四种结果之一,你一眼就知道该担心哪个。
    为什么管用
    事先列出所有可能的结论并逐个定义,AI 就写不出「基本没问题」这种含糊话。
    借来用
    让 AI 审稿时规定:「每篇的结论只能是『能发』『改完能发』『不能发』之一,并说是哪个。」
  5. 缺信息时先备好退路

    If the skill does not record a sync commit, use the last commit that changed the verification files, and say that you did.

    如果技能没有记录同步提交,就用最后一次修改验证文件的那个提交,并说明你这么做了。

    让 AI
    找不到记录时,它有现成的替代做法,并且会告诉你它用了替代做法。
    为什么管用
    提前写好「缺了怎么办」,再加一句「说出来」,AI 既不会卡住,也不会悄悄地猜。
    借来用
    整理报销时写:「找不到发票日期就用下单日期,并在表里标上『按下单日期』。」

全文

AI 读到的就是这份 SKILL.md。可以看中文、看原文,或者一段一段对照着看。

descriptionBring a project's formal verification back in line with the code after the code changes: find the commits that touched verified code, update the model, harness, or properties to describe the code as it is now, rerun, and say whether each failure is drift in the verification or a bug in the code. Use for /pstack:maintain-formal-verification-skill, "is the model still true to the code", or after merges and refactors that touched verified code without rechecking it.

description代码改了以后,让项目的形式化验证重新和代码对上:找出碰过已验证代码的提交,更新模型、harness(驱动检查的代码)或性质,让它们描述代码现在的样子,然后重跑,并说出每个失败是验证偏离了代码,还是代码里有 bug。用于 /pstack:maintain-formal-verification-skill、「模型和代码还对得上吗」,或者合并、重构碰了已验证的代码却没有重新检查之后。

Maintain a formal verification skill

Maintain a formal verification skill(维护形式化验证技能)

A formal check proves a property of what it describes. When the code changes and the verification does not, the check can keep passing while it describes code that no longer exists. That pass is the dangerous case, because it looks like a proof. This skill is the upkeep loop for a verify skill that /pstack:create-formal-verification-skill generated, or for any project-local skill that runs a model checker, prover, or code-level verifier against named code. The project-local skill handles a change that an agent makes while it uses that skill. This skill handles the changes that skipped it: merges, edits made without the skill, and drift that builds up across many commits.

形式化检查证明的,是它所描述的东西的某个性质。代码改了、验证没改,检查就可能一直通过,而它描述的代码早已不存在。这种通过最危险,因为它看起来像一个证明。这个技能是一套维护流程。维护的对象是 /pstack:create-formal-verification-skill 生成的验证技能,或者任何对指定代码运行模型检查器、证明器或代码级验证器的项目本地技能。项目本地技能负责的是:AI 在用那个技能时做出的改动。这个技能负责绕过了它的那些改动:合并、没用那个技能时做的修改,以及在很多次提交里一点点积累起来的偏离。

Outcomes

结果

Give each check one outcome, and say which:

每个检查给一个结果,并说出是哪一个:

  • in sync: no change since the last sync alters what the check describes, and the check passes.
  • updated: the code changed, the verification now describes the new code, and the check passes.
  • bug: the verification describes the current code, and the code breaks the property. This is a finding about the code. It is not drift.
  • blocked: the check could not be brought back in line or could not run. Say exactly why.
  • in sync(已同步):自上次同步以来,没有任何改动影响这个检查所描述的东西,而且检查通过。
  • updated(已更新):代码改了,验证现在描述的是新代码,而且检查通过。
  • bug:验证描述的是当前代码,而代码违反了这个性质。这是关于代码的发现。它不是偏离。
  • blocked(卡住):检查没能重新和代码对上,或者没能运行。说清楚具体原因。

A check can be both updated and bug: the update makes the verification describe the code again, and then the property fails on it.

一个检查可以同时是 updated 和 bug:更新让验证重新描述了代码,然后性质在新代码上失败了。

Edit scope

能改什么

Edit only the verification: models, harnesses, property files, and the verify skill itself. Never edit product code during a run, unless the user asks you to. When a property fails on the current code, do not weaken the property or the model to make it pass. Report the failure.

只改验证:模型、harness、性质文件,以及验证技能本身。运行期间绝不改产品代码,除非用户要你改。某个性质在当前代码上失败时,不要为了让它通过而削弱性质或模型。报告这个失败。

Pass

一轮检查

  1. Find the checks. A verify skill has no fixed name, so find it by what it does: a project skill that runs a model checker, a prover, or another verifier against named code. Read each one you find, or the one the user named. For each check, collect the property, the code it covers, the bounds and assumptions, the command that runs it, and the commit where it was last confirmed to describe the code. If no verify skill exists, stop and point at /pstack:create-formal-verification-skill. If the skill does not record a sync commit, use the last commit that changed the verification files, and say that you did.
  1. 找到检查。 验证技能没有固定的名字,所以按它做的事来找:一个对指定代码运行模型检查器、证明器或其他验证器的项目技能。读你找到的每一个,或者用户点名的那一个。对每个检查,收集这几样:性质、它覆盖的代码、边界和假设、运行它的命令,以及最后一次确认它描述了代码的那个提交。如果没有验证技能,就停下,指向 /pstack:create-formal-verification-skill。如果技能没有记录同步提交,就用最后一次修改验证文件的那个提交,并说明你这么做了。
  1. Run it as it is. Run each check once before you change anything, and record the result. A pass here does not tell you that the code is correct. It tells you that the old description still holds. A failure here can be drift or a bug. Do not decide which until step 5.
  1. 原样跑一遍。 改任何东西之前,每个检查先跑一次,记下结果。这里通过,不代表代码是对的。它只说明旧的描述还成立。这里失败,可能是偏离,也可能是 bug。到第 5 步之前,不要下判断。
  1. Read every change to covered code. Diff the covered files from the sync commit to the current commit, and read each commit message. Also look for code that moved: a covered function that is renamed, split, or moved to another file is still covered code. Also look for new code that touches the state that a check guards, such as a new function that writes the same fields. For each change, decide which of these it is:
    • No effect. The change does not alter anything the verification describes, such as formatting, comments, or code paths that the verification does not cover. Check this against the model or harness, not against the commit message.
    • The verification no longer describes the code. A step that was one indivisible action is now two, a branch was added or removed, a check moved, a new path reaches the guarded state, or a value now has a different range. The model or harness must change.
    • The promise changed on purpose. The change alters what the code promises, and evidence in the change says that this is intended: the doc comment, the tests, the changelog, or the commit message. The property must change, and you must cite that evidence.
    • The promise may be broken. The code changed, and no evidence says that the promise changed. Treat a failure that follows as a bug candidate.
  1. 读每一处碰到被覆盖代码的改动。 对被覆盖的文件,从同步提交到当前提交做 diff,并读每一条提交信息。还要找被挪走的代码:一个被覆盖的函数被改名、拆分或挪到别的文件,它仍然是被覆盖的代码。还要找碰到检查所守护的状态的新代码,比如一个写同样字段的新函数。对每个改动,判断它属于下面哪一种:
    • 没有影响。 改动没有改变验证所描述的任何东西,比如格式、注释,或者验证没有覆盖的代码路径。拿模型或 harness 来核对这一点,不要拿提交信息来核对。
    • 验证不再描述代码。 原来不可分割的一步现在变成了两步,加了或删了一个分支,某个检查挪了位置,有一条新路径能到达被守护的状态,或者某个值的取值范围变了。模型或 harness 必须改。
    • 承诺是有意改的。 改动改变了代码的承诺,而且改动里有证据说明这是有意的:文档注释、测试、更新日志或提交信息。性质必须改,而且你必须引用那份证据。
    • 承诺可能被破坏了。 代码改了,而且没有证据说明承诺变了。之后出现的失败,当作 bug 候选。
  1. Update from the code, not from the result. Change the model or harness so that each changed part again points to code that a reviewer can open, and keep one indivisible step in the code as one step in the verification. Write the update from the code before you rerun, and do not shape it by what makes the check pass. Change a property only for a promise that changed on purpose. Write down the old wording, the new wording, and the evidence. Keep the bounds unless the change makes them wrong. If you must reduce a bound, say so in the report.
  1. 根据代码更新,不根据结果更新。 修改模型或 harness,让每个改过的部分重新对应到审阅者能打开查看的代码,并且代码里不可分割的一步,在验证里也保持为一步。重跑之前,先根据代码写好更新,不要按「怎样能让检查通过」去塑造它。只有对有意改变的承诺,才修改性质。写下旧的措辞、新的措辞和证据。保持边界不变,除非改动让它们变得不对。如果你必须缩小某个边界,在报告里说出来。
  1. Rerun and judge each result.
    • Pass: the check can now pass on a description that has lost the ability to fail. Break the new code the way the property guards against, such as removing a new check or reordering a new step, and confirm that the check catches it. Then drop the break.
    • Fail: replay the counterexample against the real code, step by step. If a step cannot happen in the code, the verification is wrong. Fix the verification, and say what was wrong. If every step can happen, it is a bug. Name the commit that introduced it, tell the counterexample as a sequence of steps in the code's own words, and state the harm in one sentence. If a cheap test or reproduction against the real code shows the bug, add it to the report as evidence. Do not commit it to the product code.
  1. 重跑,并判断每个结果。
    • 通过: 检查现在有可能是在一个已经不会失败的描述上通过的。按性质要防的那种方式把新代码弄坏,比如去掉一个新加的检查,或者调换一个新步骤的顺序,确认检查能抓到它。然后撤掉这个破坏。
    • 失败: 拿反例对着真实代码,一步一步重放。如果有一步在代码里不可能发生,那就是验证错了。修好验证,并说出错在哪。如果每一步都可能发生,那就是 bug。指出引入它的提交,用代码自己的说法把反例讲成一串步骤,并用一句话说出危害。如果一个简单的测试,或者在真实代码上让 bug 再出现一次,就能展示这个 bug,就把它加进报告当证据。不要把它提交进产品代码。
  1. Update the verify skill. Update the code that each check covers, the commands, the bounds, the assumptions, and the sync commit. Set the sync commit to the commit that you checked. Do not move the sync commit for a blocked check, and say so in the report. Then follow the verify skill once as the next agent would, so that you know its steps still run.
  1. 更新验证技能。 更新每个检查覆盖的代码、命令、边界、假设和同步提交。把同步提交设成你检查过的那个提交。卡住的检查,不要移动它的同步提交,并在报告里说明。然后像下一个 AI 那样,照着验证技能走一遍,这样你才知道它的步骤还能跑通。

Report

报告

Tell the user, for each check:

对每个检查,告诉用户:

  • the outcome, and the commits that you read;
  • what changed in the verification and why, with a short reason for each change: the code changed shape, or the promise changed on purpose (with the evidence);
  • the rerun result, and the break that you used to show the updated check can still fail;
  • each bug, in plain words: the counterexample as steps in the code's words, the harm, and the commit that introduced it;
  • what the verification still does not cover, including new code that it should cover and does not.
  • 结果,以及你读过的提交;
  • 验证改了什么、为什么改,每处改动附一个简短理由:代码的形状变了,或者承诺是有意改的(附证据);
  • 重跑的结果,以及你用来证明更新后的检查仍然会失败的那个破坏;
  • 每个 bug,用大白话说:用代码的说法讲成步骤的反例、危害,以及引入它的提交;
  • 验证仍然没有覆盖的东西,包括它本该覆盖却没有覆盖的新代码。

Say outright which failures were drift in the verification and which were bugs in the code. For a check that passed before you updated it and fails after, say that the old pass was on a stale description.

明确说出哪些失败是验证偏离了代码,哪些是代码里的 bug。如果一个检查在你更新前通过、更新后失败,要说明旧的通过是建立在过时的描述上的。

Put the verification updates in a change of their own, separate from any product fix.

验证的更新单独放进一个改动,和任何产品修复分开。

原文作者 Lauren Tan(poteto),MIT 许可。译文由本站翻译,核对到 pstack 镜像 97de40d。 在 GitHub 上看

本站是 pstack 的非官方入门教程,不是官方文档;pstack 由 poteto 开发,命令和行为以官方仓库为准。教程原始源码见 GitHub。

由 airali部署,基于小小东的 MIT 授权教程改编。