skill-eval: lean the review and run paths toward trimming
A clean suite is the only safe license to cut, so a green full run now comes back as a trim invitation and the report names bundled files no case ever opened. Assisted-by: Claude Opus 5 via pi
diff --git a/ai/pi/extensions/skill-eval/README.md b/ai/pi/extensions/skill-eval/README.md
index 531c91e3..975773ab 100644
--- a/ai/pi/extensions/skill-eval/README.md
+++ b/ai/pi/extensions/skill-eval/README.md
@@ -1,7 +1,8 @@
# Skill Eval
Three ways to judge a skill or `AGENTS.md` file, from cheapest to most
-expensive:
+expensive. All three push toward a smaller file: efficacy is the bar every
+change has to clear, and length is what gets optimized once it's cleared.
| Command | What it does | Cost |
|---------|--------------|------|
@@ -94,9 +95,19 @@ problem in the file, so one run of the command surfaces all of them.
The command confirms the run count first. Trigger cases cost one model run
each; adherence cases cost two, because grading is its own run. Use `--case`
while authoring, so iterating on one case doesn't re-bill the ones that already
-pass. A report lands in `$XDG_STATE_HOME/pi/skill-eval/reports/` and, when
-anything failed, the failures arrive as a follow-up message so the conversation
-can fix the text.
+pass. A report lands in `$XDG_STATE_HOME/pi/skill-eval/reports/`.
+
+Failures arrive as a follow-up message so the conversation can fix the text,
+with the ask being to sharpen or cut what's there before adding anything. A
+clean run over the whole suite arrives as a trim invitation instead: a green
+suite is the safety net that makes cutting cheap, since you can cut, re-run,
+and read a still-green score as proof the text wasn't load-bearing. Runs
+narrowed by `--case` or `--only` skip the invitation, because they only prove
+part of the suite.
+
+The report also lists bundled reference files no run opened. That's the
+cheapest trim signal the run produces, and it cuts both ways: either the file
+isn't earning its place, or no case reaches the behavior it covers.
A report that opens with a provider-retry line is not a verdict on the skill.
Overload and rate limiting inside a nested run can empty a reply, which lands as
@@ -122,7 +133,7 @@ behavior contradicts a git-shaped instinct.
| `index.ts` | Command and tool registration, triage, review, target resolution |
| `evals.ts` | `evals.yml` loading and validation |
| `subagent.ts` | Spawning nested `pi` processes and parsing their JSON event stream |
-| `run.ts` | Case orchestration, judging, scoring, report formatting |
+| `run.ts` | Case orchestration, judging, scoring, report and follow-up formatting |
Tests run without spending API credit; a stub entrypoint injected through
`PI_BIN` stands in for `pi`.
diff --git a/ai/pi/extensions/skill-eval/index.ts b/ai/pi/extensions/skill-eval/index.ts
index a42ee3e7..8844e392 100644
--- a/ai/pi/extensions/skill-eval/index.ts
+++ b/ai/pi/extensions/skill-eval/index.ts
@@ -20,7 +20,15 @@ import type {
ExtensionCommandContext,
} from "@mariozechner/pi-coding-agent";
import { EVALS_FILENAME, loadEvalSuite } from "./evals.ts";
-import { filterSuite, formatEvalReport, runEvals, summarizeFailures, type EvalRunConfig } from "./run.ts";
+import {
+ buildFailureFollowUp,
+ buildTrimFollowUp,
+ filterSuite,
+ formatEvalReport,
+ runEvals,
+ summarizeFailures,
+ type EvalRunConfig,
+} from "./run.ts";
// ---------- Types ----------
@@ -355,18 +363,29 @@ export function resolveTarget(
// ---------- Review prompt ----------
-const SKILL_RUBRIC = `Review this skill for trim opportunities:
-- Redundant or repeated instructions
-- Sections that just restate the description
-- Unclear or unnecessary instructions
-- Structural drift from the skill-notes authoring conventions
-Propose specific cuts, not just observations.`;
+const SKILL_RUBRIC = `Cut this skill down. Load the \`tighten-docs\` skill and work its audit, then check the
+result against the \`skill-notes\` authoring conventions.
-const AGENTS_RUBRIC = `Review this AGENTS.md file for trim opportunities:
+What this file is likely paying for and not using:
+- Instructions the model already follows without being told
+- Repetition across sections, or a section that restates the description
+- Examples past the one that carries the point
+- Reference files nothing in SKILL.md routes to
+
+Efficacy outranks size. Text that changes behavior stays even when it reads badly, and the
+\`description\` frontmatter keeps every trigger context it has — a smaller skill that stops
+loading is a worse skill. Propose specific cuts, name what the skill would lose if each one
+landed, and give the line count you'd end at.`;
+
+const AGENTS_RUBRIC = `Cut this AGENTS.md down. Load the \`tighten-docs\` skill and work its audit.
+
+What this file is likely paying for and not using:
- Stale guidance that no longer reflects current practice
-- Guidance duplicated elsewhere (in this file or in a referenced skill)
-- Instructions that don't actually change behavior
-Propose specific cuts, not just observations.`;
+- Guidance duplicated here and in a skill this file already routes to
+- Instructions that don't change behavior
+
+Every line here loads into every session, so length is the whole cost — but text that changes
+behavior stays regardless. Propose specific cuts and give the line count you'd end at.`;
export function buildReviewPrompt(
kind: "skill" | "agents",
@@ -382,7 +401,17 @@ export function buildReviewPrompt(
}.`
: "No usage signal available for this file type.";
- return [`# Trim review: ${name}`, "", usageLine, "", rubric, "", "```", content, "```"].join("\n");
+ return [
+ `# Trim review: ${name}`,
+ "",
+ `${countLines(content)} line${countLines(content) === 1 ? "" : "s"}. ${usageLine}`,
+ "",
+ rubric,
+ "",
+ "```",
+ content,
+ "```",
+ ].join("\n");
}
// ---------- Command arguments ----------
@@ -682,20 +711,14 @@ async function handleRun(
const failures = summarizeFailures(report);
announce(ctx, `${markdown}\n\nReport saved to ${reportPath}`, failures.length > 0 ? "warning" : "info");
- if (failures.length === 0) return;
- pi.sendUserMessage(
- [
- `# Eval failures: ${report.skillName}`,
- "",
- `Behavioral evals for ${target.meta.path} came back with failures. Full report: ${reportPath}`,
- "",
- ...failures.map((note) => `- ${note}`),
- "",
- "Read the skill and propose specific edits to its text that would fix these, or say why a case is",
- "wrong about the skill rather than the skill being wrong.",
- ].join("\n"),
- { deliverAs: "followUp" },
- );
+ if (failures.length > 0) {
+ pi.sendUserMessage(buildFailureFollowUp(report, failures, reportPath), { deliverAs: "followUp" });
+ return;
+ }
+
+ // A green run only licenses trimming if it covered the whole suite.
+ if (parsed.cases.length > 0 || parsed.only !== undefined) return;
+ pi.sendUserMessage(buildTrimFollowUp(report, target.meta.lineCount, reportPath), { deliverAs: "followUp" });
}
export default async function (pi: ExtensionAPI) {
diff --git a/ai/pi/extensions/skill-eval/run.test.ts b/ai/pi/extensions/skill-eval/run.test.ts
index ea3ac7b5..f45dfdfc 100644
--- a/ai/pi/extensions/skill-eval/run.test.ts
+++ b/ai/pi/extensions/skill-eval/run.test.ts
@@ -1,11 +1,15 @@
import { test } from "node:test";
import assert from "node:assert/strict";
-import { mkdtemp, rm, writeFile } from "node:fs/promises";
+import { mkdir, mkdtemp, rm, writeFile } from "node:fs/promises";
import { tmpdir } from "node:os";
import { join } from "node:path";
import {
+ buildFailureFollowUp,
buildJudgePrompt,
+ buildTrimFollowUp,
+ bundledReads,
formatEvalReport,
+ listBundledFiles,
loadedSkill,
mapWithConcurrency,
filterSuite,
@@ -14,6 +18,7 @@ import {
runEvals,
skillLoadPath,
summarizeFailures,
+ unusedBundledFiles,
type EvalReport,
} from "./run.ts";
@@ -38,6 +43,36 @@ test("loadedSkill counts only the file itself for a bare .md skill", () => {
assert.equal(loadedSkill([{ name: "read", args: { path: "/skills/bash.md" } }], skillPath), false);
});
+test("bundledReads lists skill-relative reads and ignores everything else", () => {
+ const skillPath = "/skills/jj/SKILL.md";
+ const calls = [
+ { name: "read", args: { path: "/skills/jj/SKILL.md" } },
+ { name: "read", args: { path: "/skills/jj/references/rebase.md" } },
+ { name: "read", args: { path: "/skills/jj/references/rebase.md" } },
+ { name: "read", args: { path: "/skills/other/SKILL.md" } },
+ { name: "bash", args: { command: "cat /skills/jj/pitfalls.md" } },
+ ];
+ assert.deepEqual(bundledReads(calls, skillPath), ["SKILL.md", "references/rebase.md"]);
+ assert.deepEqual(bundledReads(calls, "/skills/jj.md"), []);
+});
+
+test("listBundledFiles finds references and skips SKILL.md, evals.yml, and dotfiles", async () => {
+ const dir = await mkdtemp(join(tmpdir(), "skill-eval-bundle-"));
+ try {
+ await writeFile(join(dir, "SKILL.md"), "# skill");
+ await writeFile(join(dir, "evals.yml"), "trigger: {}");
+ await writeFile(join(dir, "pitfalls.md"), "# pitfalls");
+ await writeFile(join(dir, ".hidden"), "x");
+ await mkdir(join(dir, "references"));
+ await writeFile(join(dir, "references", "rebase.md"), "# rebase");
+
+ assert.deepEqual(await listBundledFiles(join(dir, "SKILL.md")), ["pitfalls.md", "references/rebase.md"]);
+ assert.deepEqual(await listBundledFiles(join(dir, "bare.md")), []);
+ } finally {
+ await rm(dir, { recursive: true, force: true });
+ }
+});
+
test("skillLoadPath passes the bundle directory for SKILL.md and the file otherwise", () => {
assert.equal(skillLoadPath("/skills/jj/SKILL.md"), "/skills/jj");
assert.equal(skillLoadPath("/skills/jj.md"), "/skills/jj.md");
@@ -104,10 +139,11 @@ const report: EvalReport = {
model: "anthropic/claude-opus-5",
thinking: "high",
repeat: 1,
+ bundledFiles: ["calibration.md", "torque.md"],
trigger: [
- { kind: "trigger", expected: "load", prompt: "calibrate the bench widget", attempts: [{ loaded: true, retries: [] }], passes: 1, costUsd: 0.01 },
- { kind: "trigger", expected: "load", prompt: "check widget torque", attempts: [{ loaded: false, retries: [] }], passes: 0, costUsd: 0.01 },
- { kind: "trigger", expected: "skip", prompt: "what is 2 + 2", attempts: [{ loaded: true, retries: [] }], passes: 0, costUsd: 0.01 },
+ { kind: "trigger", expected: "load", prompt: "calibrate the bench widget", attempts: [{ loaded: true, reads: ["SKILL.md"], retries: [] }], passes: 1, costUsd: 0.01 },
+ { kind: "trigger", expected: "load", prompt: "check widget torque", attempts: [{ loaded: false, reads: [], retries: [] }], passes: 0, costUsd: 0.01 },
+ { kind: "trigger", expected: "skip", prompt: "what is 2 + 2", attempts: [{ loaded: true, reads: ["SKILL.md"], retries: [] }], passes: 0, costUsd: 0.01 },
],
adherence: [
{
@@ -116,6 +152,7 @@ const report: EvalReport = {
attempts: [
{
transcript: "[assistant] wrangle --torque 42",
+ reads: ["torque.md"],
retries: [],
expectations: [
{ expectation: "names the torque value 42", verdict: "pass", reason: "said 42" },
@@ -153,17 +190,72 @@ test("summarizeFailures reports each miss and each unmet expectation", () => {
assert.match(notes[2], /"reports newton-metres" came back fail: no units given/);
});
+const clean: EvalReport = {
+ ...report,
+ trigger: [
+ {
+ kind: "trigger",
+ expected: "load",
+ prompt: "calibrate the bench widget",
+ attempts: [{ loaded: true, reads: ["SKILL.md", "calibration.md"], retries: [] }],
+ passes: 1,
+ costUsd: 0,
+ },
+ ],
+ adherence: [],
+};
+
test("summarizeFailures says nothing when every case passed", () => {
- const clean: EvalReport = {
- ...report,
- trigger: [
- { kind: "trigger", expected: "load", prompt: "p", attempts: [{ loaded: true, retries: [] }], passes: 1, costUsd: 0 },
- ],
- adherence: [],
- };
assert.deepEqual(summarizeFailures(clean), []);
});
+test("unusedBundledFiles names the references no attempt opened", () => {
+ assert.deepEqual(unusedBundledFiles(report), ["calibration.md"]);
+ assert.deepEqual(unusedBundledFiles(clean), ["torque.md"]);
+ assert.deepEqual(unusedBundledFiles({ ...report, bundledFiles: [] }), []);
+});
+
+test("formatEvalReport flags unopened bundled files as trim candidates", () => {
+ const markdown = formatEvalReport(report);
+ assert.match(markdown, /## Trim signals\n\nNever opened, 1 of 2 bundled files: calibration\.md\./);
+ assert.match(markdown, /Either no case reaches it or the skill doesn't need it\./);
+});
+
+test("formatEvalReport says so when every bundled file earned a read", () => {
+ const covered: EvalReport = { ...clean, bundledFiles: ["calibration.md"] };
+ assert.match(formatEvalReport(covered), /Some run opened the one bundled file\./);
+ const both: EvalReport = { ...clean, bundledFiles: ["SKILL.md", "calibration.md"] };
+ assert.match(formatEvalReport(both), /Some run opened all 2 bundled files\./);
+});
+
+test("formatEvalReport skips trim signals for a skill with no bundled files", () => {
+ assert.doesNotMatch(formatEvalReport({ ...report, bundledFiles: [] }), /Trim signals/);
+});
+
+test("buildFailureFollowUp asks for cuts before additions", () => {
+ const message = buildFailureFollowUp(report, summarizeFailures(report), "/reports/eval.md");
+ assert.match(message, /# Eval failures: widget-wrangler/);
+ assert.match(message, /\/reports\/eval\.md/);
+ assert.match(message, /- The description did not fire/);
+ assert.match(message, /sharpening or cutting what's already there before adding anything/);
+ assert.match(message, /name something else in the file to delete/);
+});
+
+test("buildTrimFollowUp turns a green suite into a trim invitation", () => {
+ const message = buildTrimFollowUp(clean, 212, "/reports/eval.md");
+ assert.match(message, /# Trim pass: widget-wrangler/);
+ assert.match(message, /Every case passed/);
+ assert.match(message, /The skill file is 212 lines/);
+ assert.match(message, /No run opened torque\.md, so start there\./);
+ assert.match(message, /tighten-docs/);
+ assert.match(message, /trigger vocabulary alone/);
+});
+
+test("buildTrimFollowUp stays quiet about unopened files when there are none", () => {
+ const message = buildTrimFollowUp({ ...clean, bundledFiles: [] }, 40, "/reports/eval.md");
+ assert.doesNotMatch(message, /No run opened/);
+});
+
test("formatEvalReport warns when the provider retried, so scores aren't misread", () => {
const retried: EvalReport = {
...report,
@@ -172,7 +264,7 @@ test("formatEvalReport warns when the provider retried, so scores aren't misread
kind: "trigger",
expected: "load",
prompt: "p",
- attempts: [{ loaded: false, retries: [{ attempt: 1, maxAttempts: 3, errorMessage: "overloaded_error" }] }],
+ attempts: [{ loaded: false, reads: [], retries: [{ attempt: 1, maxAttempts: 3, errorMessage: "overloaded_error" }] }],
passes: 0,
costUsd: 0,
},
diff --git a/ai/pi/extensions/skill-eval/run.ts b/ai/pi/extensions/skill-eval/run.ts
index ae05b589..caa09782 100644
--- a/ai/pi/extensions/skill-eval/run.ts
+++ b/ai/pi/extensions/skill-eval/run.ts
@@ -12,12 +12,15 @@
*
* Runs are hermetic apart from globally discovered skills: no context files, no
* extensions, no session, and a fresh sandbox directory per case.
+ *
+ * Scores are the point, but the run also feeds trimming: it records which
+ * bundled files anything opened, and a clean suite is the license to cut.
*/
-import { mkdtemp, mkdir, rm } from "node:fs/promises";
+import { mkdtemp, mkdir, readdir, rm } from "node:fs/promises";
import { tmpdir } from "node:os";
import { basename, dirname, join, relative, resolve, sep } from "node:path";
-import type { AdherenceCase, EvalSuite, TriggerCase } from "./evals.ts";
+import { EVALS_FILENAME, type AdherenceCase, type EvalSuite, type TriggerCase } from "./evals.ts";
import { formatTranscript, runSetupScript, runSubagent, type RetryNote, type SubagentResult } from "./subagent.ts";
export interface EvalRunConfig {
@@ -43,6 +46,8 @@ export interface ExpectationResult {
export interface TriggerAttempt {
loaded: boolean;
+ /** Bundled files this attempt opened, relative to the skill directory. */
+ reads: string[];
retries: RetryNote[];
error?: string;
}
@@ -60,6 +65,8 @@ export interface TriggerResult {
export interface AdherenceAttempt {
expectations: ExpectationResult[];
transcript: string;
+ /** Bundled files this attempt opened, relative to the skill directory. */
+ reads: string[];
retries: RetryNote[];
error?: string;
}
@@ -79,6 +86,8 @@ export interface EvalReport {
model: string;
thinking?: string;
repeat: number;
+ /** Files bundled beside SKILL.md, relative to its directory. Empty for a bare `.md` skill. */
+ bundledFiles: string[];
trigger: TriggerResult[];
adherence: AdherenceResult[];
costUsd: number;
@@ -97,23 +106,65 @@ export function skillLoadPath(skillPath: string): string {
}
/**
- * True when the run read the skill itself, or one of its bundled references.
- * A bare `.md` skill shares its directory with unrelated skills, so only the
- * file itself counts there.
+ * Which bundled references a run opened, relative to the skill directory. A
+ * bare `.md` skill shares its directory with unrelated skills, so it has no
+ * bundle to attribute reads to.
*/
+export function bundledReads(
+ toolCalls: { name: string; args: Record<string, unknown> }[],
+ skillPath: string,
+): string[] {
+ if (basename(skillPath) !== "SKILL.md") return [];
+ const bundleDir = resolve(dirname(skillPath));
+ const reads = new Set<string>();
+ for (const call of toolCalls) {
+ if (call.name !== "read") continue;
+ const raw = call.args?.path;
+ if (typeof raw !== "string") continue;
+ const rel = relative(bundleDir, resolve(raw));
+ if (rel === "" || rel.startsWith("..")) continue;
+ reads.add(rel.split(sep).join("/"));
+ }
+ return [...reads].sort();
+}
+
+/** True when the run read the skill itself, or one of its bundled references. */
export function loadedSkill(toolCalls: { name: string; args: Record<string, unknown> }[], skillPath: string): boolean {
const target = resolve(skillPath);
- const bundleDir = basename(skillPath) === "SKILL.md" ? resolve(dirname(skillPath)) : undefined;
- return toolCalls.some((call) => {
- if (call.name !== "read") return false;
- const raw = call.args?.path;
- if (typeof raw !== "string") return false;
- const read = resolve(raw);
- if (read === target) return true;
- if (!bundleDir) return false;
- const rel = relative(bundleDir, read);
- return rel !== "" && !rel.startsWith("..") && !rel.startsWith(`..${sep}`);
- });
+ const readTarget = toolCalls.some(
+ (call) => call.name === "read" && typeof call.args?.path === "string" && resolve(call.args.path) === target,
+ );
+ return readTarget || bundledReads(toolCalls, skillPath).length > 0;
+}
+
+/**
+ * Files bundled beside SKILL.md, as trim candidates: a reference no case ever
+ * opens is either dead weight or a gap in the suite. `evals.yml` is the suite
+ * itself, so it never counts.
+ */
+export async function listBundledFiles(skillPath: string): Promise<string[]> {
+ if (basename(skillPath) !== "SKILL.md") return [];
+ const root = resolve(dirname(skillPath));
+
+ const walk = async (dir: string): Promise<string[]> => {
+ let entries;
+ try {
+ entries = await readdir(dir, { withFileTypes: true });
+ } catch {
+ return [];
+ }
+ const found: string[] = [];
+ for (const entry of entries) {
+ if (entry.name.startsWith(".")) continue;
+ const full = join(dir, entry.name);
+ if (entry.isDirectory()) found.push(...(await walk(full)));
+ else if (entry.isFile()) found.push(relative(root, full).split(sep).join("/"));
+ }
+ return found;
+ };
+
+ const files = await walk(root);
+ return files.filter((file) => file !== "SKILL.md" && file !== EVALS_FILENAME).sort();
}
/** Case matches when any filter appears in its prompt or note, compared case-insensitively. */
@@ -228,6 +279,7 @@ async function runTriggerAttempt(
return {
attempt: {
loaded: loadedSkill(result.toolCalls, skillPath),
+ reads: bundledReads(result.toolCalls, skillPath),
retries: result.retries,
...(result.error ? { error: result.error } : {}),
},
@@ -300,6 +352,7 @@ async function runAdherenceAttempt(
attempt: {
expectations: unclearAll(testCase.expect, `setup script failed: ${setup.output || "no output"}`),
transcript: "",
+ reads: [],
retries: [],
error: `setup script failed: ${setup.output || "no output"}`,
},
@@ -325,7 +378,7 @@ async function runAdherenceAttempt(
} catch (error) {
const message = (error as Error).message;
return {
- attempt: { expectations: unclearAll(testCase.expect, message), transcript: "", retries: [], error: message },
+ attempt: { expectations: unclearAll(testCase.expect, message), transcript: "", reads: [], retries: [], error: message },
costUsd: 0,
};
}
@@ -336,6 +389,7 @@ async function runAdherenceAttempt(
attempt: {
expectations: unclearAll(testCase.expect, `run failed: ${result.error}`),
transcript,
+ reads: bundledReads(result.toolCalls, skillPath),
retries: result.retries,
error: result.error,
},
@@ -348,6 +402,7 @@ async function runAdherenceAttempt(
attempt: {
expectations: graded.expectations,
transcript,
+ reads: bundledReads(result.toolCalls, skillPath),
retries: [...result.retries, ...graded.retries],
},
costUsd: result.costUsd + graded.costUsd,
@@ -457,6 +512,7 @@ export async function runEvals({ skillName, skillPath, suite, config, only }: Ru
model: `${config.provider}/${config.model}`,
thinking: config.thinking,
repeat: config.repeat,
+ bundledFiles: await listBundledFiles(skillPath),
trigger,
adherence,
costUsd: results.reduce((sum, r) => sum + r.costUsd, 0),
@@ -473,6 +529,17 @@ function firstLine(text: string, max = 80): string {
return line.length > max ? `${line.slice(0, max - 1)}…` : line;
}
+/** Bundled references no attempt opened — the cheapest trim candidates in the skill. */
+export function unusedBundledFiles(report: EvalReport): string[] {
+ const opened = new Set<string>();
+ for (const row of [...report.trigger, ...report.adherence]) {
+ for (const attempt of row.attempts) {
+ for (const file of attempt.reads) opened.add(file);
+ }
+ }
+ return report.bundledFiles.filter((file) => !opened.has(file));
+}
+
export function formatEvalReport(report: EvalReport): string {
const lines: string[] = [];
const triggerPasses = report.trigger.reduce((sum, r) => sum + r.passes, 0);
@@ -544,6 +611,21 @@ export function formatEvalReport(report: EvalReport): string {
}
}
+ const unused = unusedBundledFiles(report);
+ if (report.bundledFiles.length > 0) {
+ const total = report.bundledFiles.length;
+ const them = unused.length === 1 ? "it" : "them";
+ lines.push("## Trim signals");
+ lines.push("");
+ lines.push(
+ unused.length === 0
+ ? `Some run opened ${total === 1 ? "the one bundled file" : `all ${total} bundled files`}.`
+ : `Never opened, ${unused.length} of ${total} bundled ${total === 1 ? "file" : "files"}: ` +
+ `${unused.join(", ")}. Either no case reaches ${them} or the skill doesn't need ${them}.`,
+ );
+ lines.push("");
+ }
+
return lines.join("\n");
}
@@ -571,3 +653,46 @@ export function summarizeFailures(report: EvalReport): string[] {
}
return notes;
}
+
+/**
+ * Follow-up for a run with failures. Efficacy comes first, but a fix that only
+ * adds text is how a skill bloats, so the ask is to pay for new lines with old
+ * ones.
+ */
+export function buildFailureFollowUp(report: EvalReport, failures: string[], reportPath: string): string {
+ return [
+ `# Eval failures: ${report.skillName}`,
+ "",
+ `Behavioral evals for ${report.skillPath} came back with failures. Full report: ${reportPath}`,
+ "",
+ ...failures.map((note) => `- ${note}`),
+ "",
+ "Read the skill and propose specific edits to its text that would fix these, or say why a case is",
+ "wrong about the skill rather than the skill being wrong.",
+ "",
+ "Reach the fix by sharpening or cutting what's already there before adding anything. If a failure",
+ "genuinely needs new instruction, name something else in the file to delete alongside it.",
+ ].join("\n");
+}
+
+/**
+ * Follow-up for a clean run. A green suite is the safety net that makes cutting
+ * cheap: cut, re-run, and the score says whether the text was load-bearing.
+ */
+export function buildTrimFollowUp(report: EvalReport, lineCount: number, reportPath: string): string {
+ const unused = unusedBundledFiles(report);
+ return [
+ `# Trim pass: ${report.skillName}`,
+ "",
+ `Every case passed for ${report.skillPath} (${report.runs} run(s)). Full report: ${reportPath}`,
+ "",
+ `The suite is now a safety net, so cuts are cheap to verify: cut, re-run \`/skill-eval run ${report.skillName}\`,`,
+ `and a still-green score says the text wasn't load-bearing. The skill file is ${lineCount} lines.`,
+ ...(unused.length > 0
+ ? ["", `No run opened ${unused.join(" or ")}, so start there.`]
+ : []),
+ "",
+ "Load the `tighten-docs` skill and propose specific cuts. Leave the `description` frontmatter's",
+ "trigger vocabulary alone — a smaller skill that stops loading is a worse skill.",
+ ].join("\n");
+}
diff --git a/ai/pi/extensions/skill-eval/skill-eval.test.ts b/ai/pi/extensions/skill-eval/skill-eval.test.ts
index e69ce33b..e1af0d68 100644
--- a/ai/pi/extensions/skill-eval/skill-eval.test.ts
+++ b/ai/pi/extensions/skill-eval/skill-eval.test.ts
@@ -308,15 +308,17 @@ test("resolveTarget matches AGENTS.md by basename", () => {
}
});
-test("buildReviewPrompt includes usage line and content for skills", () => {
- const prompt = buildReviewPrompt("skill", "jj", "# jj skill content", { count: 3, lastUsed: 1735689600000 });
- assert.match(prompt, /Usage in the last 90 days: 3/);
+test("buildReviewPrompt includes size, usage, and content for skills", () => {
+ const prompt = buildReviewPrompt("skill", "jj", "# jj skill content\nline two\n", { count: 3, lastUsed: 1735689600000 });
+ assert.match(prompt, /2 lines\. Usage in the last 90 days: 3/);
assert.match(prompt, /jj skill content/);
+ assert.match(prompt, /tighten-docs/);
+ assert.match(prompt, /Efficacy outranks size/);
});
test("buildReviewPrompt notes no usage signal for AGENTS.md", () => {
const prompt = buildReviewPrompt("agents", "/home/user/AGENTS.md", "# instructions", null);
- assert.match(prompt, /No usage signal available/);
+ assert.match(prompt, /1 line\. No usage signal available/);
});
import { parseCommandArgs, tokenizeArgs } from "./index.ts";