ai: evaluate skill text with nested pi runs
Reviewing a skill only checks whether its text reads well. Trigger and adherence cases in evals.yml check what the text does, using nested `pi -p --mode json` processes in place of subagents. Assisted-by: Claude Opus 5 via pi
diff --git a/AGENTS.md b/AGENTS.md
index 21cc3e0c..1e824b08 100644
--- a/AGENTS.md
+++ b/AGENTS.md
@@ -46,6 +46,8 @@ Skills live in `ai/skills/<name>/SKILL.md`. See the `skill-notes` skill for auth
Pi loads these via `ai/pi/settings.json`, which points to `~/.dotfiles/ai/skills`. Claude reaches them through `ai/claude/skills` (a symlink to `../skills`), so both agents read the same files — only ever edit `ai/skills/`.
+A skill can declare behavioral evals in `ai/skills/<name>/evals.yml` — prompts that should (and should not) load it, plus tasks with expectations to grade once it is loaded. Run them with `/skill-eval run <name>`; see `ai/pi/extensions/skill-eval/README.md`.
+
## Pi Packages
Pi manages plugins natively via `pi install` / `pi remove` / `pi update` / `pi list`. Global packages are declared in `~/.pi/agent/settings.json` under `packages`. Project-scoped packages use `.pi/settings.json` with the `-l` flag.
diff --git a/ai/pi/extensions/skill-eval/README.md b/ai/pi/extensions/skill-eval/README.md
new file mode 100644
index 00000000..fed98f91
--- /dev/null
+++ b/ai/pi/extensions/skill-eval/README.md
@@ -0,0 +1,116 @@
+# Skill Eval
+
+Three ways to judge a skill or `AGENTS.md` file, from cheapest to most
+expensive:
+
+| Command | What it does | Cost |
+|---------|--------------|------|
+| `/skill-eval` or `/skill-eval triage` | Ranks in-context skills and `AGENTS.md` files by line count and 90-day usage from session logs | Free |
+| `/skill-eval review <target>` | Queues the file's text into the conversation with a trim rubric | One turn |
+| `/skill-eval run <skill>` | Runs the skill's `evals.yml` through nested `pi` processes | API credit |
+
+Triage and review are also available to the model through the `skill_eval`
+tool. Evals are not: they spend real money, so only you start them.
+
+## Behavioral evals
+
+Reviewing a skill asks a model whether the text *looks* right. An eval checks
+what the text actually does, in two parts.
+
+Trigger fidelity answers whether the `description` frontmatter makes the agent
+reach for the skill. Each case runs a fresh `pi` with ordinary skill discovery
+on — competing descriptions are part of the test — and only the `read` tool
+available. A case passes when the run reads a file in the skill's directory
+(positive cases) or leaves it alone (negative cases).
+
+Instruction adherence answers whether the body changes behavior once loaded.
+Each case forces the skill in with `/skill:<name>`, runs the task in a fresh
+temporary directory, then hands the transcript to a second `pi` run that grades
+every declared expectation as pass, fail, or unclear. A case passes only when
+all of its expectations pass.
+
+Runs use the model and thinking level of the session you launch them from, so
+scores describe the setup you actually work in. Everything else is hermetic:
+no session file, no extensions, no `AGENTS.md`, no prompt templates, and a
+scratch working directory per case. Globally discovered skills are the one
+exception, and they are deliberate.
+
+## Writing evals.yml
+
+Put `evals.yml` next to `SKILL.md`:
+
+```yaml
+trigger:
+ positive:
+ # Tasks the skill should be loaded for. Phrase them the way you would.
+ - prompt: I need to squash this change into its parent
+ - prompt: |
+ The description of my last commit is wrong.
+ Fix it.
+ note: covers the describe-versus-commit distinction
+ negative:
+ # Nearby tasks the skill should stay out of.
+ - prompt: Add fzf-git as a submodule with a shallow clone
+
+adherence:
+ - prompt: Commit the change in this repo
+ expect:
+ - runs `jj commit` rather than `jj describe`
+ - the commit message subject is a plain sentence with no conventional-commit prefix
+ - the message carries an `Assisted-by` trailer
+ tools: [read, bash]
+ setup: |
+ jj git init
+ echo "hello" > file.txt
+```
+
+Fields:
+
+`trigger.positive` and `trigger.negative` take a list of cases. A case is
+either a bare string or a mapping with `prompt` and an optional `note`.
+
+`adherence` takes a list of mappings. `prompt` and `expect` are required;
+`expect` holds one checkable claim per entry, since the judge grades each
+separately. `tools` defaults to `[read]`, which keeps a case read-only and
+grades what the agent says it would do; add `bash`, `edit`, or `write` when the
+task only means something if the agent carries it out. `setup` is a bash script
+run in the sandbox before the agent starts, for scaffolding fixtures.
+
+A prompt cannot start with `-` or `@`, which `pi` reads as a flag or a file
+include. Validation rejects those with the case's path, along with every other
+problem in the file, so one run of the command surfaces all of them.
+
+## Running
+
+```
+/skill-eval run jj
+/skill-eval run jj --only trigger # skip the expensive half
+/skill-eval run jj --repeat 3 # attempts per case, to expose flakiness
+/skill-eval run jj --jobs 1 # cases in flight at once (default 3)
+```
+
+The command confirms the run count first. Trigger cases cost one model run
+each; adherence cases cost two, because grading is its own run. A report lands
+in `$XDG_STATE_HOME/pi/skill-eval/reports/` and, when anything failed, the
+failures arrive as a follow-up message so the conversation can fix the text.
+
+Trigger misses usually mean the `description` field needs the vocabulary of the
+prompt that missed. False positives on negative cases mean it claims too much
+ground. Adherence failures point at the body: an instruction stated once in
+passing, buried under prose, or contradicted elsewhere in the file.
+
+## Layout
+
+| File | Contents |
+|------|----------|
+| `index.ts` | Command and tool registration, triage, review, target resolution |
+| `evals.ts` | `evals.yml` loading and validation |
+| `subagent.ts` | Spawning nested `pi` processes and parsing their JSON event stream |
+| `run.ts` | Case orchestration, judging, scoring, report formatting |
+
+Tests run without spending API credit; a stub entrypoint injected through
+`PI_BIN` stands in for `pi`.
+
+```bash
+node --test "ai/pi/extensions/skill-eval/*.test.ts"
+```
diff --git a/ai/pi/extensions/skill-eval/evals.test.ts b/ai/pi/extensions/skill-eval/evals.test.ts
new file mode 100644
index 00000000..f9d89332
--- /dev/null
+++ b/ai/pi/extensions/skill-eval/evals.test.ts
@@ -0,0 +1,133 @@
+import { test } from "node:test";
+import assert from "node:assert/strict";
+import { mkdtemp, rm, writeFile } from "node:fs/promises";
+import { tmpdir } from "node:os";
+import { join } from "node:path";
+import { evalsPathFor, loadEvalSuite, parseEvalSuite } from "./evals.ts";
+
+test("parseEvalSuite accepts bare strings and mappings for trigger cases", () => {
+ const result = parseEvalSuite({
+ trigger: {
+ positive: ["squash this into its parent", { prompt: "describe my change", note: "phrasing check" }],
+ negative: [{ prompt: "add a git submodule" }],
+ },
+ });
+
+ assert.equal(result.ok, true);
+ if (!result.ok) return;
+ assert.deepEqual(result.suite.triggerPositive, [
+ { prompt: "squash this into its parent" },
+ { prompt: "describe my change", note: "phrasing check" },
+ ]);
+ assert.deepEqual(result.suite.triggerNegative, [{ prompt: "add a git submodule" }]);
+ assert.deepEqual(result.suite.adherence, []);
+});
+
+test("parseEvalSuite defaults adherence tools to read and keeps setup scripts", () => {
+ const result = parseEvalSuite({
+ adherence: [
+ { prompt: "commit my work", expect: ["uses jj commit"] },
+ { prompt: "rebase onto main", expect: ["uses jj rebase"], tools: ["read", "bash"], setup: "jj git init\n" },
+ ],
+ });
+
+ assert.equal(result.ok, true);
+ if (!result.ok) return;
+ assert.deepEqual(result.suite.adherence[0].tools, ["read"]);
+ assert.equal(result.suite.adherence[0].setup, undefined);
+ assert.deepEqual(result.suite.adherence[1].tools, ["read", "bash"]);
+ assert.equal(result.suite.adherence[1].setup, "jj git init\n");
+});
+
+test("parseEvalSuite rejects prompts pi would read as a flag or file include", () => {
+ const result = parseEvalSuite({ trigger: { positive: ["--torque is wrong, fix it"] } }, "evals.yml");
+ assert.equal(result.ok, false);
+ if (result.ok) return;
+ assert.equal(result.errors.length, 1);
+ assert.match(result.errors[0], /evals\.yml:trigger\.positive\[0\]/);
+ assert.match(result.errors[0], /cannot start with/);
+});
+
+test("parseEvalSuite requires at least one expectation per adherence case", () => {
+ const result = parseEvalSuite({ adherence: [{ prompt: "commit my work" }] });
+ assert.equal(result.ok, false);
+ if (result.ok) return;
+ assert.match(result.errors[0], /needs at least one expectation/);
+});
+
+test("parseEvalSuite collects every problem rather than stopping at the first", () => {
+ const result = parseEvalSuite({
+ trigger: { positive: [{ note: "no prompt here" }], sideways: [] },
+ adherence: [{ prompt: "do a thing", expect: ["fine"], tools: "bash" }],
+ extra: true,
+ });
+
+ assert.equal(result.ok, false);
+ if (result.ok) return;
+ assert.equal(result.errors.length, 4);
+ assert.ok(result.errors.some((e) => /unknown top-level key "extra"/.test(e)));
+ assert.ok(result.errors.some((e) => /trigger: unknown key "sideways"/.test(e)));
+ assert.ok(result.errors.some((e) => /missing a non-empty "prompt"/.test(e)));
+ assert.ok(result.errors.some((e) => /tools: expected a list of tool names/.test(e)));
+});
+
+test("parseEvalSuite reports an empty or non-mapping file", () => {
+ const empty = parseEvalSuite(null, "evals.yml");
+ assert.equal(empty.ok, false);
+ if (!empty.ok) assert.match(empty.errors[0], /file is empty/);
+
+ const list = parseEvalSuite([1, 2], "evals.yml");
+ assert.equal(list.ok, false);
+ if (!list.ok) assert.match(list.errors[0], /expected a mapping/);
+
+ const noCases = parseEvalSuite({ trigger: { positive: [] } }, "evals.yml");
+ assert.equal(noCases.ok, false);
+ if (!noCases.ok) assert.match(noCases.errors[0], /no cases found/);
+});
+
+test("evalsPathFor puts the eval file next to SKILL.md", () => {
+ assert.equal(evalsPathFor("/skills/jj/SKILL.md"), "/skills/jj/evals.yml");
+});
+
+test("loadEvalSuite reads the file next to the skill and validates the decoded data", async () => {
+ const dir = await mkdtemp(join(tmpdir(), "skill-eval-evals-"));
+ try {
+ await writeFile(join(dir, "evals.yml"), "trigger:\n positive:\n - squash it\n");
+ const seen: string[] = [];
+ const result = await loadEvalSuite(join(dir, "SKILL.md"), async (text) => {
+ seen.push(text);
+ return { trigger: { positive: ["squash it"] } };
+ });
+
+ assert.equal(result.ok, true);
+ if (!result.ok) return;
+ assert.deepEqual(result.suite.triggerPositive, [{ prompt: "squash it" }]);
+ assert.equal(seen[0], "trigger:\n positive:\n - squash it\n");
+ } finally {
+ await rm(dir, { recursive: true, force: true });
+ }
+});
+
+test("loadEvalSuite reports a missing file with a template to copy", async () => {
+ const result = await loadEvalSuite("/nonexistent/skill/SKILL.md");
+ assert.equal(result.ok, false);
+ if (result.ok) return;
+ assert.equal((result as { missing?: true }).missing, true);
+ assert.match(result.errors[0], /No eval file at \/nonexistent\/skill\/evals\.yml/);
+ assert.ok(result.errors.some((e) => /trigger:/.test(e)));
+});
+
+test("loadEvalSuite surfaces YAML decoding failures", async () => {
+ const dir = await mkdtemp(join(tmpdir(), "skill-eval-evals-"));
+ try {
+ await writeFile(join(dir, "evals.yml"), "trigger: [\n");
+ const result = await loadEvalSuite(join(dir, "SKILL.md"), async () => {
+ throw new Error("unexpected end of stream");
+ });
+ assert.equal(result.ok, false);
+ if (result.ok) return;
+ assert.match(result.errors[0], /could not parse YAML — unexpected end of stream/);
+ } finally {
+ await rm(dir, { recursive: true, force: true });
+ }
+});
diff --git a/ai/pi/extensions/skill-eval/evals.ts b/ai/pi/extensions/skill-eval/evals.ts
new file mode 100644
index 00000000..a2245ba6
--- /dev/null
+++ b/ai/pi/extensions/skill-eval/evals.ts
@@ -0,0 +1,266 @@
+/**
+ * evals.yml parsing and validation.
+ *
+ * An eval file sits next to SKILL.md and declares two kinds of cases:
+ * trigger cases (does the description make the agent reach for the skill?)
+ * and adherence cases (with the skill loaded, does the agent follow it?).
+ *
+ * `parseEvalSuite` is pure and works on already-parsed YAML data;
+ * `loadEvalSuite` handles reading and YAML decoding.
+ */
+
+import { readFile } from "node:fs/promises";
+import { dirname, join } from "node:path";
+
+export const EVALS_FILENAME = "evals.yml";
+
+export const DEFAULT_ADHERENCE_TOOLS = ["read"];
+
+export interface TriggerCase {
+ prompt: string;
+ note?: string;
+}
+
+export interface AdherenceCase {
+ prompt: string;
+ expect: string[];
+ tools: string[];
+ setup?: string;
+ note?: string;
+}
+
+export interface EvalSuite {
+ triggerPositive: TriggerCase[];
+ triggerNegative: TriggerCase[];
+ adherence: AdherenceCase[];
+}
+
+export type ParseResult =
+ | { ok: true; suite: EvalSuite }
+ | { ok: false; errors: string[] };
+
+export const EVALS_TEMPLATE = `trigger:
+ positive:
+ # Tasks the skill should be loaded for.
+ - prompt: a task phrased the way you'd actually phrase it
+ negative:
+ # Nearby tasks the skill should stay out of.
+ - prompt: a task that sounds similar but needs a different skill
+
+adherence:
+ - prompt: a task to carry out with the skill loaded
+ expect:
+ # One checkable claim per line, graded against the transcript.
+ - names the specific command the skill prescribes
+ # Optional. Defaults to ["read"]; add tools the task genuinely needs.
+ tools: [read, bash]
+ # Optional shell script run in the sandbox before the agent starts.
+ setup: |
+ echo "scaffold fixtures here"
+`;
+
+/** Path of the eval file belonging to a SKILL.md. */
+export function evalsPathFor(skillPath: string): string {
+ return join(dirname(skillPath), EVALS_FILENAME);
+}
+
+function isRecord(value: unknown): value is Record<string, unknown> {
+ return typeof value === "object" && value !== null && !Array.isArray(value);
+}
+
+/** A prompt is passed to pi as an argv message, where these prefixes are consumed as flags or file includes. */
+function checkPromptPrefix(prompt: string, path: string, errors: string[]): void {
+ if (prompt.startsWith("-") || prompt.startsWith("@")) {
+ errors.push(
+ `${path}: prompt cannot start with "-" or "@" — pi reads those as a flag or a file include. Reword the prompt.`,
+ );
+ }
+}
+
+function parseTriggerCases(raw: unknown, path: string, errors: string[]): TriggerCase[] {
+ if (raw === undefined || raw === null) return [];
+ if (!Array.isArray(raw)) {
+ errors.push(`${path}: expected a list of cases, got ${describe(raw)}.`);
+ return [];
+ }
+
+ const cases: TriggerCase[] = [];
+ raw.forEach((entry, index) => {
+ const entryPath = `${path}[${index}]`;
+ if (typeof entry === "string") {
+ const prompt = entry.trim();
+ if (!prompt) {
+ errors.push(`${entryPath}: prompt is empty.`);
+ return;
+ }
+ checkPromptPrefix(prompt, entryPath, errors);
+ cases.push({ prompt });
+ return;
+ }
+ if (!isRecord(entry)) {
+ errors.push(`${entryPath}: expected a string or a mapping with a "prompt" key, got ${describe(entry)}.`);
+ return;
+ }
+ const prompt = typeof entry.prompt === "string" ? entry.prompt.trim() : "";
+ if (!prompt) {
+ errors.push(`${entryPath}: missing a non-empty "prompt".`);
+ return;
+ }
+ checkPromptPrefix(prompt, entryPath, errors);
+ const note = typeof entry.note === "string" ? entry.note : undefined;
+ cases.push(note ? { prompt, note } : { prompt });
+ });
+ return cases;
+}
+
+function parseAdherenceCases(raw: unknown, path: string, errors: string[]): AdherenceCase[] {
+ if (raw === undefined || raw === null) return [];
+ if (!Array.isArray(raw)) {
+ errors.push(`${path}: expected a list of cases, got ${describe(raw)}.`);
+ return [];
+ }
+
+ const cases: AdherenceCase[] = [];
+ raw.forEach((entry, index) => {
+ const entryPath = `${path}[${index}]`;
+ if (!isRecord(entry)) {
+ errors.push(`${entryPath}: expected a mapping with "prompt" and "expect" keys, got ${describe(entry)}.`);
+ return;
+ }
+
+ const prompt = typeof entry.prompt === "string" ? entry.prompt.trim() : "";
+ if (!prompt) errors.push(`${entryPath}: missing a non-empty "prompt".`);
+ else checkPromptPrefix(prompt, entryPath, errors);
+
+ const expect: string[] = [];
+ if (!Array.isArray(entry.expect) || entry.expect.length === 0) {
+ errors.push(`${entryPath}.expect: needs at least one expectation to grade against.`);
+ } else {
+ entry.expect.forEach((item, i) => {
+ if (typeof item !== "string" || !item.trim()) {
+ errors.push(`${entryPath}.expect[${i}]: expected a non-empty string, got ${describe(item)}.`);
+ return;
+ }
+ expect.push(item.trim());
+ });
+ }
+
+ let tools = DEFAULT_ADHERENCE_TOOLS;
+ if (entry.tools !== undefined) {
+ if (!Array.isArray(entry.tools) || entry.tools.some((t) => typeof t !== "string" || !t.trim())) {
+ errors.push(`${entryPath}.tools: expected a list of tool names, got ${describe(entry.tools)}.`);
+ } else {
+ tools = (entry.tools as string[]).map((t) => t.trim());
+ }
+ }
+
+ let setup: string | undefined;
+ if (entry.setup !== undefined) {
+ if (typeof entry.setup !== "string") {
+ errors.push(`${entryPath}.setup: expected a shell script string, got ${describe(entry.setup)}.`);
+ } else if (entry.setup.trim()) {
+ setup = entry.setup;
+ }
+ }
+
+ if (!prompt || expect.length === 0) return;
+ const note = typeof entry.note === "string" ? entry.note : undefined;
+ cases.push({ prompt, expect, tools, ...(setup ? { setup } : {}), ...(note ? { note } : {}) });
+ });
+ return cases;
+}
+
+function describe(value: unknown): string {
+ if (value === null) return "null";
+ if (Array.isArray(value)) return "a list";
+ return typeof value;
+}
+
+/** Validate decoded evals.yml data. Collects every problem rather than failing on the first. */
+export function parseEvalSuite(data: unknown, source = EVALS_FILENAME): ParseResult {
+ const errors: string[] = [];
+
+ if (data === null || data === undefined) {
+ return { ok: false, errors: [`${source}: file is empty.`] };
+ }
+ if (!isRecord(data)) {
+ return { ok: false, errors: [`${source}: expected a mapping with "trigger" and/or "adherence" keys, got ${describe(data)}.`] };
+ }
+
+ const known = new Set(["trigger", "adherence"]);
+ for (const key of Object.keys(data)) {
+ if (!known.has(key)) errors.push(`${source}: unknown top-level key "${key}" (expected "trigger" or "adherence").`);
+ }
+
+ let triggerPositive: TriggerCase[] = [];
+ let triggerNegative: TriggerCase[] = [];
+ if (data.trigger !== undefined && data.trigger !== null) {
+ if (!isRecord(data.trigger)) {
+ errors.push(`${source}:trigger: expected a mapping with "positive" and/or "negative" keys, got ${describe(data.trigger)}.`);
+ } else {
+ for (const key of Object.keys(data.trigger)) {
+ if (key !== "positive" && key !== "negative") {
+ errors.push(`${source}:trigger: unknown key "${key}" (expected "positive" or "negative").`);
+ }
+ }
+ triggerPositive = parseTriggerCases(data.trigger.positive, `${source}:trigger.positive`, errors);
+ triggerNegative = parseTriggerCases(data.trigger.negative, `${source}:trigger.negative`, errors);
+ }
+ }
+
+ const adherence = parseAdherenceCases(data.adherence, `${source}:adherence`, errors);
+
+ if (triggerPositive.length + triggerNegative.length + adherence.length === 0 && errors.length === 0) {
+ errors.push(`${source}: no cases found. Add trigger cases, adherence cases, or both.`);
+ }
+
+ if (errors.length > 0) return { ok: false, errors };
+ return { ok: true, suite: { triggerPositive, triggerNegative, adherence } };
+}
+
+/**
+ * The `yaml` package ships inside pi but isn't resolvable from an extension
+ * directory, so decode through pi's own frontmatter parser, which wraps it.
+ */
+async function defaultYamlParser(text: string): Promise<unknown> {
+ // The wrapper below ends at the first unindented `---`, which would silently
+ // truncate the file, so reject that line outright.
+ if (text.split("\n").some((line) => line.trimEnd() === "---")) {
+ throw new Error('a line containing only "---" is not supported; indent it or quote it');
+ }
+ const mod: any = await import("@earendil-works/pi-coding-agent");
+ const wrapped = text.endsWith("\n") ? text : `${text}\n`;
+ return mod.parseFrontmatter(`---\n${wrapped}---\n`).frontmatter;
+}
+
+export type LoadResult = ParseResult | { ok: false; errors: string[]; missing: true };
+
+/** Read and validate the evals.yml next to a SKILL.md. */
+export async function loadEvalSuite(
+ skillPath: string,
+ parseYaml: (text: string) => Promise<unknown> = defaultYamlParser,
+): Promise<LoadResult> {
+ const path = evalsPathFor(skillPath);
+ let text: string;
+ try {
+ text = await readFile(path, "utf8");
+ } catch {
+ return {
+ ok: false,
+ missing: true,
+ errors: [
+ `No eval file at ${path}. Create one to describe what this skill should do:`,
+ "",
+ EVALS_TEMPLATE,
+ ],
+ };
+ }
+
+ let data: unknown;
+ try {
+ data = await parseYaml(text);
+ } catch (error) {
+ return { ok: false, errors: [`${path}: could not parse YAML — ${(error as Error).message}`] };
+ }
+ return parseEvalSuite(data, path);
+}
diff --git a/ai/pi/extensions/skill-eval/index.ts b/ai/pi/extensions/skill-eval/index.ts
index 90e1b650..d4ae4269 100644
--- a/ai/pi/extensions/skill-eval/index.ts
+++ b/ai/pi/extensions/skill-eval/index.ts
@@ -6,6 +6,8 @@
* trim/removal candidates, using Pi session logs as a usage signal.
* Deep-reviews a single target by handing its content and usage
* stats to the current conversation with a trim-focused rubric.
+ * Also runs behavioral evals declared in a skill's evals.yml through nested pi
+ * processes; see run.ts and README.md.
*/
import { existsSync, statSync } from "node:fs";
@@ -17,6 +19,8 @@ import type {
ExtensionContext,
ExtensionCommandContext,
} from "@mariozechner/pi-coding-agent";
+import { EVALS_FILENAME, loadEvalSuite } from "./evals.ts";
+import { formatEvalReport, runEvals, summarizeFailures, type EvalRunConfig } from "./run.ts";
// ---------- Types ----------
@@ -299,10 +303,11 @@ export function formatTriageMarkdown(
return lines.join("\n");
}
-export function resolveReportPath(now: Date, stateBaseDir?: string): string {
+export function resolveReportPath(now: Date, stateBaseDir?: string, label?: string): string {
const base = stateBaseDir ?? process.env.XDG_STATE_HOME ?? join(homedir(), ".local", "state");
const timestamp = now.toISOString().replace(/[:.]/g, "-");
- return join(base, "pi", "skill-eval", "reports", `${timestamp}.md`);
+ const suffix = label ? `-${label.replace(/[^a-zA-Z0-9._-]+/g, "_")}` : "";
+ return join(base, "pi", "skill-eval", "reports", `${timestamp}${suffix}.md`);
}
export async function writeReport(markdown: string, reportPath: string): Promise<void> {
@@ -380,9 +385,81 @@ export function buildReviewPrompt(
return [`# Trim review: ${name}`, "", usageLine, "", rubric, "", "```", content, "```"].join("\n");
}
+// ---------- Command arguments ----------
+
+export type ParsedCommand =
+ | { action: "triage" }
+ | { action: "review"; target: string }
+ | { action: "run"; target: string; repeat: number; jobs: number; only?: "trigger" | "adherence" }
+ | { action: "error"; message: string };
+
+const DEFAULT_REPEAT = 1;
+const DEFAULT_JOBS = 3;
+
+/**
+ * `/skill-eval` and `/skill-eval triage` triage; `/skill-eval run <skill>` runs
+ * behavioral evals; anything else is treated as a review target, which keeps the
+ * original bare-target form working.
+ */
+export function parseCommandArgs(args: string): ParsedCommand {
+ const tokens = args.trim().split(/\s+/).filter(Boolean);
+ if (tokens.length === 0) return { action: "triage" };
+
+ const [head, ...rest] = tokens;
+ if (head === "triage") {
+ if (rest.length > 0) return { action: "error", message: "triage takes no arguments." };
+ return { action: "triage" };
+ }
+
+ if (head === "review") {
+ if (rest.length !== 1) return { action: "error", message: "review takes exactly one target." };
+ return { action: "review", target: rest[0] };
+ }
+
+ if (head === "run") {
+ let target: string | undefined;
+ let repeat = DEFAULT_REPEAT;
+ let jobs = DEFAULT_JOBS;
+ let only: "trigger" | "adherence" | undefined;
+
+ for (let i = 0; i < rest.length; i++) {
+ const token = rest[i];
+ if (token === "--repeat" || token === "--jobs") {
+ const value = Number(rest[++i]);
+ if (!Number.isInteger(value) || value < 1) {
+ return { action: "error", message: `${token} needs a positive integer.` };
+ }
+ if (token === "--repeat") repeat = value;
+ else jobs = value;
+ continue;
+ }
+ if (token === "--only") {
+ const value = rest[++i];
+ if (value !== "trigger" && value !== "adherence") {
+ return { action: "error", message: "--only takes either trigger or adherence." };
+ }
+ only = value;
+ continue;
+ }
+ if (token.startsWith("-")) return { action: "error", message: `Unknown flag ${token}.` };
+ if (target !== undefined) return { action: "error", message: "run takes exactly one skill." };
+ target = token;
+ }
+
+ if (target === undefined) {
+ return { action: "error", message: "run needs a skill name: /skill-eval run <skill>." };
+ }
+ return { action: "run", target, repeat, jobs, ...(only ? { only } : {}) };
+ }
+
+ if (tokens.length > 1) return { action: "error", message: `Unknown subcommand "${head}".` };
+ return { action: "review", target: head };
+}
+
// ---------- Wiring ----------
const NINETY_DAYS_MS = 90 * 24 * 60 * 60 * 1000;
+const RUN_TIMEOUT_MS = 5 * 60 * 1000;
function sessionsDirFor(): string {
return join(homedir(), ".pi", "agent", "sessions");
@@ -441,30 +518,182 @@ async function runReview(
return { ok: true, prompt, name: resolved.meta.path };
}
+async function resolveEvalTarget(
+ ctx: ExtensionContext,
+ target: string,
+): Promise<{ ok: true; meta: SkillMeta } | { ok: false; message: string }> {
+ const contextSkills = parseAvailableSkillsBlock(ctx.getSystemPrompt());
+ const skills = await loadSkillsFromContext(contextSkills);
+ const agentsFiles = await discoverAgentsFiles(ctx.cwd);
+ const resolved = resolveTarget(target, skills, agentsFiles);
+
+ if (!resolved.ok) {
+ const label =
+ resolved.reason === "ambiguous" ? "Ambiguous target" : "No matching skill";
+ return {
+ ok: false,
+ message: `${label} for "${target}". Candidates: ${resolved.candidates.join(", ") || "none"}`,
+ };
+ }
+ if (resolved.kind !== "skill") {
+ return {
+ ok: false,
+ message: `"${target}" is an AGENTS.md file. Behavioral evals only work on skills, since they measure whether a skill loads and gets followed. Use /skill-eval review ${target} instead.`,
+ };
+ }
+ return { ok: true, meta: resolved.meta };
+}
+
+function evalConfig(ctx: ExtensionContext, repeat: number, jobs: number): EvalRunConfig | null {
+ const model = ctx.model;
+ if (!model) return null;
+ return {
+ provider: model.provider,
+ model: model.id,
+ thinking: ctx.thinkingLevel,
+ repeat,
+ jobs,
+ timeoutMs: RUN_TIMEOUT_MS,
+ };
+}
+
+/** UI methods are no-ops without a UI, so fall back to the console for headless runs. */
+function announce(ctx: ExtensionCommandContext, text: string, level: "info" | "warning" | "error"): void {
+ ctx.ui.notify(text, level);
+ if (!ctx.hasUI) {
+ if (level === "error") console.error(text);
+ else console.log(text);
+ }
+}
+
+async function handleRun(
+ pi: ExtensionAPI,
+ ctx: ExtensionCommandContext,
+ parsed: { target: string; repeat: number; jobs: number; only?: "trigger" | "adherence" },
+): Promise<void> {
+ const target = await resolveEvalTarget(ctx, parsed.target);
+ if (!target.ok) {
+ announce(ctx, target.message, "error");
+ return;
+ }
+
+ const loaded = await loadEvalSuite(target.meta.path);
+ if (!loaded.ok) {
+ announce(ctx, loaded.errors.join("\n"), "error");
+ return;
+ }
+ const suite = loaded.suite;
+
+ const config = evalConfig(ctx, parsed.repeat, parsed.jobs);
+ if (!config) {
+ announce(ctx, "No active model, so there's nothing to run evals with. Pick a model with /model first.", "error");
+ return;
+ }
+
+ const triggerCases = parsed.only === "adherence" ? 0 : suite.triggerPositive.length + suite.triggerNegative.length;
+ const adherenceCases = parsed.only === "trigger" ? 0 : suite.adherence.length;
+ if (triggerCases + adherenceCases === 0) {
+ const kind = parsed.only ? `${parsed.only} ` : "";
+ announce(ctx, `${EVALS_FILENAME} for ${target.meta.name} has no ${kind}cases to run.`, "error");
+ return;
+ }
+
+ // Each adherence case costs a judge run on top of the agent run.
+ const modelRuns = (triggerCases + adherenceCases * 2) * parsed.repeat;
+ if (ctx.hasUI) {
+ const proceed = await ctx.ui.confirm(
+ `Run evals for ${target.meta.name}?`,
+ `${modelRuns} nested ${config.provider}/${config.model} runs, ${parsed.jobs} at a time. This spends API credit.`,
+ );
+ if (!proceed) return;
+ }
+
+ ctx.ui.setStatus("skill-eval", `evals: ${target.meta.name} starting`);
+ let report;
+ try {
+ report = await runEvals({
+ skillName: target.meta.name,
+ skillPath: target.meta.path,
+ suite,
+ only: parsed.only,
+ config: {
+ ...config,
+ onProgress: (done, total, label) => {
+ ctx.ui.setStatus("skill-eval", `evals: ${target.meta.name} ${done}/${total} — ${label.slice(0, 40)}`);
+ },
+ },
+ });
+ } finally {
+ ctx.ui.setStatus("skill-eval", undefined);
+ }
+
+ const markdown = formatEvalReport(report);
+ const reportPath = resolveReportPath(report.finishedAt, undefined, `eval-${report.skillName}`);
+ await writeReport(markdown, reportPath);
+
+ const failures = summarizeFailures(report);
+ announce(ctx, `${markdown}\n\nReport saved to ${reportPath}`, failures.length > 0 ? "warning" : "info");
+
+ if (failures.length === 0) return;
+ pi.sendUserMessage(
+ [
+ `# Eval failures: ${report.skillName}`,
+ "",
+ `Behavioral evals for ${target.meta.path} came back with failures. Full report: ${reportPath}`,
+ "",
+ ...failures.map((note) => `- ${note}`),
+ "",
+ "Read the skill and propose specific edits to its text that would fix these, or say why a case is",
+ "wrong about the skill rather than the skill being wrong.",
+ ].join("\n"),
+ { deliverAs: "followUp" },
+ );
+}
+
export default async function (pi: ExtensionAPI) {
const { Type } = await import("typebox");
pi.registerCommand("skill-eval", {
- description: "Triage in-context skills/AGENTS.md for trim candidates, or deep-review one target",
+ description:
+ "triage | review <target> | run <skill> [--repeat N] [--jobs N] [--only trigger|adherence]",
+ getArgumentCompletions: (prefix: string) => {
+ const items = [
+ { value: "triage", label: "triage — rank skills and AGENTS.md files by trim potential" },
+ { value: "review ", label: "review <target> — critique one file's text" },
+ { value: "run ", label: "run <skill> — run its evals.yml through subagents" },
+ ].filter((item) => item.value.startsWith(prefix));
+ return items.length > 0 ? items : null;
+ },
handler: async (args: string, ctx: ExtensionCommandContext) => {
- const target = args.trim();
- if (!target) {
+ const parsed = parseCommandArgs(args);
+
+ if (parsed.action === "error") {
+ announce(ctx, `${parsed.message} Usage: /skill-eval [triage | review <target> | run <skill>]`, "error");
+ return;
+ }
+
+ if (parsed.action === "triage") {
const result = await runTriage(ctx);
if (!result.ok) {
- ctx.ui.notify(result.message, "error");
+ announce(ctx, result.message, "error");
return;
}
- ctx.ui.notify(`${result.markdown}\n\nReport saved to ${result.reportPath}`, "info");
+ announce(ctx, `${result.markdown}\n\nReport saved to ${result.reportPath}`, "info");
return;
}
- const result = await runReview(ctx, target);
- if (!result.ok) {
- ctx.ui.notify(result.message, "error");
+ if (parsed.action === "review") {
+ const result = await runReview(ctx, parsed.target);
+ if (!result.ok) {
+ announce(ctx, result.message, "error");
+ return;
+ }
+ pi.sendUserMessage(result.prompt, { deliverAs: "followUp" });
+ announce(ctx, `Queued review for ${result.name}`, "info");
return;
}
- pi.sendUserMessage(result.prompt, { deliverAs: "followUp" });
- ctx.ui.notify(`Queued review for ${result.name}`, "info");
+
+ await handleRun(pi, ctx, parsed);
},
});
@@ -476,6 +705,7 @@ export default async function (pi: ExtensionAPI) {
promptGuidelines: [
"Use skill_eval with action 'triage' when asked to find skills or AGENTS.md files worth trimming.",
"Use skill_eval with action 'review' and a target name when asked to critique a specific skill or AGENTS.md file.",
+ "Behavioral evals spend API credit, so they are user-driven: tell the user to run /skill-eval run <skill> rather than trying to run them yourself.",
],
parameters: Type.Object({
action: Type.Union([Type.Literal("triage"), Type.Literal("review")]),
diff --git a/ai/pi/extensions/skill-eval/run.test.ts b/ai/pi/extensions/skill-eval/run.test.ts
new file mode 100644
index 00000000..8aa0565f
--- /dev/null
+++ b/ai/pi/extensions/skill-eval/run.test.ts
@@ -0,0 +1,327 @@
+import { test } from "node:test";
+import assert from "node:assert/strict";
+import { mkdtemp, rm, writeFile } from "node:fs/promises";
+import { tmpdir } from "node:os";
+import { join } from "node:path";
+import {
+ buildJudgePrompt,
+ formatEvalReport,
+ loadedSkill,
+ mapWithConcurrency,
+ parseJudgeVerdicts,
+ runEvals,
+ skillLoadPath,
+ summarizeFailures,
+ type EvalReport,
+} from "./run.ts";
+
+test("loadedSkill matches reads anywhere inside the skill directory", () => {
+ const skillPath = "/skills/jj/SKILL.md";
+ assert.equal(loadedSkill([{ name: "read", args: { path: "/skills/jj/SKILL.md" } }], skillPath), true);
+ assert.equal(loadedSkill([{ name: "read", args: { path: "/skills/jj/references/rebase.md" } }], skillPath), true);
+ assert.equal(loadedSkill([{ name: "read", args: { path: "/skills/jj/../jj/SKILL.md" } }], skillPath), true);
+});
+
+test("loadedSkill ignores other skills, other tools, and malformed arguments", () => {
+ const skillPath = "/skills/jj/SKILL.md";
+ assert.equal(loadedSkill([{ name: "read", args: { path: "/skills/jj-workspaces/SKILL.md" } }], skillPath), false);
+ assert.equal(loadedSkill([{ name: "bash", args: { command: "cat /skills/jj/SKILL.md" } }], skillPath), false);
+ assert.equal(loadedSkill([{ name: "read", args: {} }], skillPath), false);
+ assert.equal(loadedSkill([], skillPath), false);
+});
+
+test("loadedSkill counts only the file itself for a bare .md skill", () => {
+ const skillPath = "/skills/jj.md";
+ assert.equal(loadedSkill([{ name: "read", args: { path: "/skills/jj.md" } }], skillPath), true);
+ assert.equal(loadedSkill([{ name: "read", args: { path: "/skills/bash.md" } }], skillPath), false);
+});
+
+test("skillLoadPath passes the bundle directory for SKILL.md and the file otherwise", () => {
+ assert.equal(skillLoadPath("/skills/jj/SKILL.md"), "/skills/jj");
+ assert.equal(skillLoadPath("/skills/jj.md"), "/skills/jj.md");
+});
+
+test("buildJudgePrompt numbers expectations and carries task and transcript", () => {
+ const prompt = buildJudgePrompt("jj", "commit my work", "[assistant] jj commit", ["uses jj commit", "adds a trailer"]);
+ assert.match(prompt, /"jj" skill/);
+ assert.match(prompt, /1\. uses jj commit/);
+ assert.match(prompt, /2\. adds a trailer/);
+ assert.match(prompt, /commit my work/);
+ assert.match(prompt, /\[assistant\] jj commit/);
+});
+
+test("buildJudgePrompt notes an empty transcript rather than sending nothing", () => {
+ assert.match(buildJudgePrompt("jj", "task", "", ["something"]), /the agent produced no output/);
+});
+
+test("parseJudgeVerdicts reads a fenced array and matches verdicts by index", () => {
+ const reply = [
+ "Here are my verdicts:",
+ "```json",
+ '[{"index": 2, "verdict": "fail", "reason": "no trailer"},',
+ ' {"index": 1, "verdict": "pass", "reason": "ran jj commit"}]',
+ "```",
+ ].join("\n");
+ const results = parseJudgeVerdicts(reply, ["uses jj commit", "adds a trailer"]);
+ assert.deepEqual(results, [
+ { expectation: "uses jj commit", verdict: "pass", reason: "ran jj commit" },
+ { expectation: "adds a trailer", verdict: "fail", reason: "no trailer" },
+ ]);
+});
+
+test("parseJudgeVerdicts falls back to unclear when the reply is unusable", () => {
+ for (const reply of ["I could not tell.", "[not json]", '{"verdict":"pass"}']) {
+ const results = parseJudgeVerdicts(reply, ["a", "b"]);
+ assert.equal(results.length, 2);
+ assert.ok(results.every((r) => r.verdict === "unclear"));
+ }
+});
+
+test("parseJudgeVerdicts treats unknown verdict values as unclear", () => {
+ const results = parseJudgeVerdicts('[{"index":1,"verdict":"maybe","reason":"hmm"}]', ["a"]);
+ assert.deepEqual(results, [{ expectation: "a", verdict: "unclear", reason: "hmm" }]);
+});
+
+test("mapWithConcurrency preserves order and respects the limit", async () => {
+ let inFlight = 0;
+ let peak = 0;
+ const results = await mapWithConcurrency([10, 20, 30, 40, 50], 2, async (item) => {
+ inFlight++;
+ peak = Math.max(peak, inFlight);
+ await new Promise((r) => setTimeout(r, item % 30));
+ inFlight--;
+ return item * 2;
+ });
+ assert.deepEqual(results, [20, 40, 60, 80, 100]);
+ assert.equal(peak, 2);
+});
+
+const report: EvalReport = {
+ skillName: "widget-wrangler",
+ skillPath: "/skills/widget-wrangler/SKILL.md",
+ model: "anthropic/claude-opus-5",
+ thinking: "high",
+ repeat: 1,
+ trigger: [
+ { kind: "trigger", expected: "load", prompt: "calibrate the bench widget", attempts: [{ loaded: true }], passes: 1, costUsd: 0.01 },
+ { kind: "trigger", expected: "load", prompt: "check widget torque", attempts: [{ loaded: false }], passes: 0, costUsd: 0.01 },
+ { kind: "trigger", expected: "skip", prompt: "what is 2 + 2", attempts: [{ loaded: true }], passes: 0, costUsd: 0.01 },
+ ],
+ adherence: [
+ {
+ kind: "adherence",
+ prompt: "calibrate it and report the torque",
+ attempts: [
+ {
+ transcript: "[assistant] wrangle --torque 42",
+ expectations: [
+ { expectation: "names the torque value 42", verdict: "pass", reason: "said 42" },
+ { expectation: "reports newton-metres", verdict: "fail", reason: "no units given" },
+ ],
+ },
+ ],
+ passes: 0,
+ costUsd: 0.05,
+ },
+ ],
+ costUsd: 0.08,
+ runs: 4,
+ startedAt: new Date("2026-01-01T00:00:00.000Z"),
+ finishedAt: new Date("2026-01-01T00:01:00.000Z"),
+};
+
+test("formatEvalReport scores both sections and lists misses", () => {
+ const markdown = formatEvalReport(report);
+ assert.match(markdown, /# Skill eval: widget-wrangler/);
+ assert.match(markdown, /anthropic\/claude-opus-5 \(thinking: high\).*\$0\.0800.*60s/);
+ assert.match(markdown, /## Trigger fidelity — 1\/3/);
+ assert.match(markdown, /\| load \| 0\/1 \| check widget torque \|/);
+ assert.match(markdown, /did not load: check widget torque/);
+ assert.match(markdown, /loaded when it should not have: what is 2 \+ 2/);
+ assert.match(markdown, /## Instruction adherence — 0\/1/);
+ assert.match(markdown, /- fail: reports newton-metres — no units given/);
+});
+
+test("summarizeFailures reports each miss and each unmet expectation", () => {
+ const notes = summarizeFailures(report);
+ assert.equal(notes.length, 3);
+ assert.match(notes[0], /did not fire for "check widget torque"/);
+ assert.match(notes[1], /fired on the near-miss prompt "what is 2 \+ 2"/);
+ assert.match(notes[2], /"reports newton-metres" came back fail: no units given/);
+});
+
+test("summarizeFailures says nothing when every case passed", () => {
+ const clean: EvalReport = {
+ ...report,
+ trigger: [{ kind: "trigger", expected: "load", prompt: "p", attempts: [{ loaded: true }], passes: 1, costUsd: 0 }],
+ adherence: [],
+ };
+ assert.deepEqual(summarizeFailures(clean), []);
+});
+
+/**
+ * End-to-end orchestration against a stub pi entrypoint: no API credit, but it
+ * exercises argument construction, sandboxing, setup scripts, and judging.
+ */
+const FAKE_PI = `
+const args = process.argv.slice(2);
+let stdin = "";
+process.stdin.setEncoding("utf8");
+process.stdin.on("data", (c) => (stdin += c));
+process.stdin.on("end", () => {
+ const emit = (o) => process.stdout.write(JSON.stringify(o) + "\\n");
+ if (args.includes("--no-tools")) {
+ // Judge run: pass the first expectation, fail the rest.
+ const count = (stdin.match(/^\\d+\\. /gm) || []).length;
+ const verdicts = Array.from({ length: count }, (_, i) => ({
+ index: i + 1,
+ verdict: i === 0 ? "pass" : "fail",
+ reason: "stub verdict",
+ }));
+ emit({ type: "agent_end", messages: [{ role: "assistant", content: [{ type: "text", text: JSON.stringify(verdicts) }], usage: { cost: { total: 0.001 } } }] });
+ return;
+ }
+ const skillDir = args[args.indexOf("--skill") + 1];
+ if (args.some((a) => a.startsWith("/skill:"))) {
+ // Adherence run: record the sandbox contents so setup scripts are observable.
+ const fs = require("node:fs");
+ emit({ type: "agent_end", messages: [{ role: "assistant", content: [{ type: "text", text: "sandbox: " + fs.readdirSync(process.cwd()).join(",") }], usage: { cost: { total: 0.002 } } }] });
+ return;
+ }
+ // Trigger run: load the skill only when the prompt mentions widgets.
+ if (/widget/.test(stdin)) emit({ type: "tool_execution_start", toolName: "read", args: { path: skillDir + "/SKILL.md" } });
+ emit({ type: "agent_end", messages: [{ role: "assistant", content: [{ type: "text", text: "done" }], usage: { cost: { total: 0.003 } } }] });
+});
+`;
+
+test("runEvals scores trigger and adherence cases end to end", async () => {
+ const dir = await mkdtemp(join(tmpdir(), "skill-eval-run-"));
+ const previous = process.env.PI_BIN;
+ try {
+ const fakePi = join(dir, "fake-pi.js");
+ await writeFile(fakePi, FAKE_PI);
+ process.env.PI_BIN = fakePi;
+
+ const result = await runEvals({
+ skillName: "widget-wrangler",
+ skillPath: join(dir, "SKILL.md"),
+ suite: {
+ triggerPositive: [{ prompt: "calibrate the widget" }],
+ triggerNegative: [{ prompt: "what is 2 + 2" }, { prompt: "the widget question again" }],
+ adherence: [
+ { prompt: "calibrate it", expect: ["first", "second"], tools: ["read"], setup: "touch fixture.txt" },
+ ],
+ },
+ config: { provider: "anthropic", model: "test-model", thinking: "high", repeat: 1, jobs: 2, timeoutMs: 10_000 },
+ });
+
+ assert.equal(result.runs, 4);
+ assert.deepEqual(
+ result.trigger.map((r) => [r.expected, r.passes]),
+ [
+ ["load", 1],
+ ["skip", 1],
+ ["skip", 0],
+ ],
+ );
+ assert.deepEqual(
+ result.adherence[0].attempts[0].expectations.map((e) => e.verdict),
+ ["pass", "fail"],
+ );
+ assert.match(result.adherence[0].attempts[0].transcript, /sandbox: fixture\.txt/);
+ assert.ok(result.costUsd > 0);
+ } finally {
+ if (previous === undefined) delete process.env.PI_BIN;
+ else process.env.PI_BIN = previous;
+ await rm(dir, { recursive: true, force: true });
+ }
+});
+
+test("runEvals marks a case unclear when its setup script fails", async () => {
+ const dir = await mkdtemp(join(tmpdir(), "skill-eval-run-"));
+ const previous = process.env.PI_BIN;
+ try {
+ const fakePi = join(dir, "fake-pi.js");
+ await writeFile(fakePi, FAKE_PI);
+ process.env.PI_BIN = fakePi;
+
+ const result = await runEvals({
+ skillName: "widget-wrangler",
+ skillPath: join(dir, "SKILL.md"),
+ suite: {
+ triggerPositive: [],
+ triggerNegative: [],
+ adherence: [{ prompt: "calibrate it", expect: ["first"], tools: ["read"], setup: "exit 7" }],
+ },
+ config: { provider: "anthropic", model: "test-model", repeat: 1, jobs: 1, timeoutMs: 10_000 },
+ });
+
+ const attempt = result.adherence[0].attempts[0];
+ assert.match(attempt.error ?? "", /setup script failed/);
+ assert.equal(attempt.expectations[0].verdict, "unclear");
+ } finally {
+ if (previous === undefined) delete process.env.PI_BIN;
+ else process.env.PI_BIN = previous;
+ await rm(dir, { recursive: true, force: true });
+ }
+});
+
+test("runEvals honors --only by skipping the other section", async () => {
+ const dir = await mkdtemp(join(tmpdir(), "skill-eval-run-"));
+ const previous = process.env.PI_BIN;
+ try {
+ const fakePi = join(dir, "fake-pi.js");
+ await writeFile(fakePi, FAKE_PI);
+ process.env.PI_BIN = fakePi;
+
+ const suite = {
+ triggerPositive: [{ prompt: "calibrate the widget" }],
+ triggerNegative: [],
+ adherence: [{ prompt: "calibrate it", expect: ["first"], tools: ["read"] }],
+ };
+ const config = { provider: "anthropic", model: "test-model", repeat: 1, jobs: 1, timeoutMs: 10_000 };
+
+ const triggerOnly = await runEvals({ skillName: "w", skillPath: join(dir, "SKILL.md"), suite, config, only: "trigger" });
+ assert.equal(triggerOnly.trigger.length, 1);
+ assert.equal(triggerOnly.adherence.length, 0);
+
+ const adherenceOnly = await runEvals({ skillName: "w", skillPath: join(dir, "SKILL.md"), suite, config, only: "adherence" });
+ assert.equal(adherenceOnly.trigger.length, 0);
+ assert.equal(adherenceOnly.adherence.length, 1);
+ } finally {
+ if (previous === undefined) delete process.env.PI_BIN;
+ else process.env.PI_BIN = previous;
+ await rm(dir, { recursive: true, force: true });
+ }
+});
+
+test("runEvals reports progress for every attempt", async () => {
+ const dir = await mkdtemp(join(tmpdir(), "skill-eval-run-"));
+ const previous = process.env.PI_BIN;
+ try {
+ const fakePi = join(dir, "fake-pi.js");
+ await writeFile(fakePi, FAKE_PI);
+ process.env.PI_BIN = fakePi;
+
+ const seen: string[] = [];
+ await runEvals({
+ skillName: "widget-wrangler",
+ skillPath: join(dir, "SKILL.md"),
+ suite: { triggerPositive: [{ prompt: "calibrate the widget" }], triggerNegative: [], adherence: [] },
+ config: {
+ provider: "anthropic",
+ model: "test-model",
+ repeat: 2,
+ jobs: 1,
+ timeoutMs: 10_000,
+ onProgress: (done, total, label) => seen.push(`${done}/${total} ${label}`),
+ },
+ });
+
+ assert.deepEqual(seen, ["1/2 trigger: calibrate the widget", "2/2 trigger: calibrate the widget"]);
+ } finally {
+ if (previous === undefined) delete process.env.PI_BIN;
+ else process.env.PI_BIN = previous;
+ await rm(dir, { recursive: true, force: true });
+ }
+});
diff --git a/ai/pi/extensions/skill-eval/run.ts b/ai/pi/extensions/skill-eval/run.ts
new file mode 100644
index 00000000..4fc06a7f
--- /dev/null
+++ b/ai/pi/extensions/skill-eval/run.ts
@@ -0,0 +1,515 @@
+/**
+ * Behavioral evals: run a skill's text through nested pi processes.
+ *
+ * Trigger cases measure whether the description makes the agent reach for the
+ * skill, so the run keeps ordinary skill discovery on (competing descriptions
+ * are part of the test) and allows only `read`, then checks whether the skill
+ * files were read.
+ *
+ * Adherence cases force the skill in with `/skill:<name>`, let the agent work
+ * in a scratch directory, and hand the transcript to a judge run that grades
+ * each declared expectation.
+ *
+ * Runs are hermetic apart from globally discovered skills: no context files, no
+ * extensions, no session, and a fresh sandbox directory per case.
+ */
+
+import { mkdtemp, mkdir, rm } from "node:fs/promises";
+import { tmpdir } from "node:os";
+import { basename, dirname, join, relative, resolve, sep } from "node:path";
+import type { AdherenceCase, EvalSuite, TriggerCase } from "./evals.ts";
+import { formatTranscript, runSetupScript, runSubagent, type SubagentResult } from "./subagent.ts";
+
+export interface EvalRunConfig {
+ provider: string;
+ model: string;
+ thinking?: string;
+ /** Attempts per case. Repeats surface flaky trigger behavior. */
+ repeat: number;
+ /** Cases in flight at once. */
+ jobs: number;
+ timeoutMs: number;
+ signal?: AbortSignal;
+ onProgress?: (done: number, total: number, label: string) => void;
+}
+
+export type Verdict = "pass" | "fail" | "unclear";
+
+export interface ExpectationResult {
+ expectation: string;
+ verdict: Verdict;
+ reason: string;
+}
+
+export interface TriggerAttempt {
+ loaded: boolean;
+ error?: string;
+}
+
+export interface TriggerResult {
+ kind: "trigger";
+ expected: "load" | "skip";
+ prompt: string;
+ note?: string;
+ attempts: TriggerAttempt[];
+ passes: number;
+ costUsd: number;
+}
+
+export interface AdherenceAttempt {
+ expectations: ExpectationResult[];
+ transcript: string;
+ error?: string;
+}
+
+export interface AdherenceResult {
+ kind: "adherence";
+ prompt: string;
+ note?: string;
+ attempts: AdherenceAttempt[];
+ passes: number;
+ costUsd: number;
+}
+
+export interface EvalReport {
+ skillName: string;
+ skillPath: string;
+ model: string;
+ thinking?: string;
+ repeat: number;
+ trigger: TriggerResult[];
+ adherence: AdherenceResult[];
+ costUsd: number;
+ runs: number;
+ startedAt: Date;
+ finishedAt: Date;
+}
+
+/**
+ * What to hand `--skill`, so the target loads even when it isn't discoverable
+ * from the sandbox. Pi accepts either a file or a directory, and a bare `.md`
+ * skill has no directory of its own.
+ */
+export function skillLoadPath(skillPath: string): string {
+ return basename(skillPath) === "SKILL.md" ? dirname(skillPath) : skillPath;
+}
+
+/**
+ * True when the run read the skill itself, or one of its bundled references.
+ * A bare `.md` skill shares its directory with unrelated skills, so only the
+ * file itself counts there.
+ */
+export function loadedSkill(toolCalls: { name: string; args: Record<string, unknown> }[], skillPath: string): boolean {
+ const target = resolve(skillPath);
+ const bundleDir = basename(skillPath) === "SKILL.md" ? resolve(dirname(skillPath)) : undefined;
+ return toolCalls.some((call) => {
+ if (call.name !== "read") return false;
+ const raw = call.args?.path;
+ if (typeof raw !== "string") return false;
+ const read = resolve(raw);
+ if (read === target) return true;
+ if (!bundleDir) return false;
+ const rel = relative(bundleDir, read);
+ return rel !== "" && !rel.startsWith("..") && !rel.startsWith(`..${sep}`);
+ });
+}
+
+export function buildJudgePrompt(
+ skillName: string,
+ task: string,
+ transcript: string,
+ expectations: string[],
+): string {
+ return [
+ `You are grading one run of a coding agent that was given the "${skillName}" skill and then a task.`,
+ "",
+ "Grade each expectation independently against the transcript. Judge only what the transcript",
+ 'shows: "pass" when the transcript clearly meets the expectation, "fail" when it clearly does not,',
+ '"unclear" when the transcript does not settle it. Do not credit intentions the agent never acted on.',
+ "",
+ "Task given to the agent:",
+ task,
+ "",
+ "Expectations:",
+ ...expectations.map((item, i) => `${i + 1}. ${item}`),
+ "",
+ "Transcript:",
+ "<<<TRANSCRIPT",
+ transcript || "(the agent produced no output)",
+ "TRANSCRIPT",
+ "",
+ "Reply with only a JSON array, one object per expectation, in the same order:",
+ '[{"index": 1, "verdict": "pass" | "fail" | "unclear", "reason": "one sentence"}]',
+ ].join("\n");
+}
+
+/** Pull the verdict array out of a judge reply, tolerating code fences and surrounding prose. */
+export function parseJudgeVerdicts(text: string, expectations: string[]): ExpectationResult[] {
+ const fallback = (reason: string): ExpectationResult[] =>
+ expectations.map((expectation) => ({ expectation, verdict: "unclear" as Verdict, reason }));
+
+ const start = text.indexOf("[");
+ const end = text.lastIndexOf("]");
+ if (start === -1 || end <= start) return fallback("judge did not return a JSON array");
+
+ let parsed: unknown;
+ try {
+ parsed = JSON.parse(text.slice(start, end + 1));
+ } catch {
+ return fallback("judge returned unparseable JSON");
+ }
+ if (!Array.isArray(parsed)) return fallback("judge did not return a JSON array");
+
+ return expectations.map((expectation, i) => {
+ const entry: any = parsed.find((item: any) => Number(item?.index) === i + 1) ?? parsed[i];
+ const verdict: Verdict =
+ entry?.verdict === "pass" || entry?.verdict === "fail" || entry?.verdict === "unclear"
+ ? entry.verdict
+ : "unclear";
+ const reason = typeof entry?.reason === "string" ? entry.reason : "judge gave no reason";
+ return { expectation, verdict, reason };
+ });
+}
+
+async function withSandbox<T>(prefix: string, fn: (dir: string) => Promise<T>): Promise<T> {
+ const dir = await mkdtemp(join(tmpdir(), `skill-eval-${prefix}-`));
+ try {
+ return await fn(dir);
+ } finally {
+ await rm(dir, { recursive: true, force: true });
+ }
+}
+
+async function runTriggerAttempt(
+ testCase: TriggerCase,
+ skillPath: string,
+ config: EvalRunConfig,
+): Promise<{ attempt: TriggerAttempt; costUsd: number }> {
+ return await withSandbox("trigger", async (cwd) => {
+ const result = await runSubagent({
+ cwd,
+ provider: config.provider,
+ model: config.model,
+ thinking: config.thinking,
+ tools: ["read"],
+ skillDirs: [skillLoadPath(skillPath)],
+ discoverSkills: true,
+ stdin: testCase.prompt,
+ timeoutMs: config.timeoutMs,
+ signal: config.signal,
+ });
+ return {
+ attempt: {
+ loaded: loadedSkill(result.toolCalls, skillPath),
+ ...(result.error ? { error: result.error } : {}),
+ },
+ costUsd: result.costUsd,
+ };
+ });
+}
+
+async function judge(
+ skillName: string,
+ testCase: AdherenceCase,
+ transcript: string,
+ config: EvalRunConfig,
+): Promise<{ expectations: ExpectationResult[]; costUsd: number }> {
+ return await withSandbox("judge", async (cwd) => {
+ const result = await runSubagent({
+ cwd,
+ provider: config.provider,
+ model: config.model,
+ thinking: config.thinking,
+ tools: [],
+ discoverSkills: false,
+ stdin: buildJudgePrompt(skillName, testCase.prompt, transcript, testCase.expect),
+ timeoutMs: config.timeoutMs,
+ signal: config.signal,
+ });
+ if (result.error) {
+ return {
+ expectations: testCase.expect.map((expectation) => ({
+ expectation,
+ verdict: "unclear" as Verdict,
+ reason: `judge run failed: ${result.error}`,
+ })),
+ costUsd: result.costUsd,
+ };
+ }
+ return { expectations: parseJudgeVerdicts(result.finalText, testCase.expect), costUsd: result.costUsd };
+ });
+}
+
+async function runAdherenceAttempt(
+ testCase: AdherenceCase,
+ skillName: string,
+ skillPath: string,
+ config: EvalRunConfig,
+): Promise<{ attempt: AdherenceAttempt; costUsd: number }> {
+ return await withSandbox("adherence", async (cwd) => {
+ const work = join(cwd, "work");
+ await mkdir(work, { recursive: true });
+
+ if (testCase.setup) {
+ const setup = await runSetupScript(testCase.setup, work);
+ if (!setup.ok) {
+ return {
+ attempt: {
+ expectations: unclearAll(testCase.expect, `setup script failed: ${setup.output || "no output"}`),
+ transcript: "",
+ error: `setup script failed: ${setup.output || "no output"}`,
+ },
+ costUsd: 0,
+ };
+ }
+ }
+
+ let result: SubagentResult;
+ try {
+ result = await runSubagent({
+ cwd: work,
+ provider: config.provider,
+ model: config.model,
+ thinking: config.thinking,
+ tools: testCase.tools,
+ skillDirs: [skillLoadPath(skillPath)],
+ discoverSkills: false,
+ messages: [`/skill:${skillName}`, testCase.prompt],
+ timeoutMs: config.timeoutMs,
+ signal: config.signal,
+ });
+ } catch (error) {
+ const message = (error as Error).message;
+ return {
+ attempt: { expectations: unclearAll(testCase.expect, message), transcript: "", error: message },
+ costUsd: 0,
+ };
+ }
+
+ const transcript = formatTranscript(result.events);
+ if (result.error) {
+ return {
+ attempt: {
+ expectations: unclearAll(testCase.expect, `run failed: ${result.error}`),
+ transcript,
+ error: result.error,
+ },
+ costUsd: result.costUsd,
+ };
+ }
+
+ const graded = await judge(skillName, testCase, transcript, config);
+ return {
+ attempt: { expectations: graded.expectations, transcript },
+ costUsd: result.costUsd + graded.costUsd,
+ };
+ });
+}
+
+function unclearAll(expectations: string[], reason: string): ExpectationResult[] {
+ return expectations.map((expectation) => ({ expectation, verdict: "unclear" as Verdict, reason }));
+}
+
+/** Run tasks with bounded concurrency, preserving input order in the results. */
+export async function mapWithConcurrency<T, R>(
+ items: T[],
+ limit: number,
+ fn: (item: T, index: number) => Promise<R>,
+): Promise<R[]> {
+ const results = new Array<R>(items.length);
+ let next = 0;
+ const workers = Array.from({ length: Math.max(1, Math.min(limit, items.length)) }, async () => {
+ while (next < items.length) {
+ const index = next++;
+ results[index] = await fn(items[index], index);
+ }
+ });
+ await Promise.all(workers);
+ return results;
+}
+
+type Unit =
+ | { kind: "trigger"; expected: "load" | "skip"; testCase: TriggerCase }
+ | { kind: "adherence"; testCase: AdherenceCase };
+
+export interface RunEvalsOptions {
+ skillName: string;
+ skillPath: string;
+ suite: EvalSuite;
+ config: EvalRunConfig;
+ only?: "trigger" | "adherence";
+}
+
+export async function runEvals({ skillName, skillPath, suite, config, only }: RunEvalsOptions): Promise<EvalReport> {
+ const startedAt = new Date();
+
+ const units: Unit[] = [];
+ if (only !== "adherence") {
+ for (const testCase of suite.triggerPositive) units.push({ kind: "trigger", expected: "load", testCase });
+ for (const testCase of suite.triggerNegative) units.push({ kind: "trigger", expected: "skip", testCase });
+ }
+ if (only !== "trigger") {
+ for (const testCase of suite.adherence) units.push({ kind: "adherence", testCase });
+ }
+
+ const total = units.length * config.repeat;
+ let done = 0;
+ const tick = (label: string) => config.onProgress?.(++done, total, label);
+
+ const results = await mapWithConcurrency(units, config.jobs, async (unit) => {
+ if (unit.kind === "trigger") {
+ const attempts: TriggerAttempt[] = [];
+ let costUsd = 0;
+ for (let i = 0; i < config.repeat; i++) {
+ const { attempt, costUsd: cost } = await runTriggerAttempt(unit.testCase, skillPath, config);
+ attempts.push(attempt);
+ costUsd += cost;
+ tick(`trigger: ${unit.testCase.prompt}`);
+ }
+ const passes = attempts.filter((a) => (unit.expected === "load" ? a.loaded : !a.loaded)).length;
+ const result: TriggerResult = {
+ kind: "trigger",
+ expected: unit.expected,
+ prompt: unit.testCase.prompt,
+ ...(unit.testCase.note ? { note: unit.testCase.note } : {}),
+ attempts,
+ passes,
+ costUsd,
+ };
+ return result;
+ }
+
+ const attempts: AdherenceAttempt[] = [];
+ let costUsd = 0;
+ for (let i = 0; i < config.repeat; i++) {
+ const { attempt, costUsd: cost } = await runAdherenceAttempt(unit.testCase, skillName, skillPath, config);
+ attempts.push(attempt);
+ costUsd += cost;
+ tick(`adherence: ${unit.testCase.prompt}`);
+ }
+ const passes = attempts.filter((a) => a.expectations.every((e) => e.verdict === "pass")).length;
+ const result: AdherenceResult = {
+ kind: "adherence",
+ prompt: unit.testCase.prompt,
+ ...(unit.testCase.note ? { note: unit.testCase.note } : {}),
+ attempts,
+ passes,
+ costUsd,
+ };
+ return result;
+ });
+
+ const trigger = results.filter((r): r is TriggerResult => r.kind === "trigger");
+ const adherence = results.filter((r): r is AdherenceResult => r.kind === "adherence");
+
+ return {
+ skillName,
+ skillPath,
+ model: `${config.provider}/${config.model}`,
+ thinking: config.thinking,
+ repeat: config.repeat,
+ trigger,
+ adherence,
+ costUsd: results.reduce((sum, r) => sum + r.costUsd, 0),
+ runs: total,
+ startedAt,
+ finishedAt: new Date(),
+ };
+}
+
+// ---------- Reporting ----------
+
+function firstLine(text: string, max = 80): string {
+ const line = text.split("\n")[0].trim();
+ return line.length > max ? `${line.slice(0, max - 1)}…` : line;
+}
+
+export function formatEvalReport(report: EvalReport): string {
+ const lines: string[] = [];
+ const triggerPasses = report.trigger.reduce((sum, r) => sum + r.passes, 0);
+ const triggerTotal = report.trigger.length * report.repeat;
+ const adherencePasses = report.adherence.reduce((sum, r) => sum + r.passes, 0);
+ const adherenceTotal = report.adherence.length * report.repeat;
+ const seconds = Math.round((report.finishedAt.getTime() - report.startedAt.getTime()) / 1000);
+
+ lines.push(`# Skill eval: ${report.skillName}`);
+ lines.push("");
+ lines.push(`${report.skillPath}`);
+ lines.push("");
+ lines.push(
+ `${report.model}${report.thinking ? ` (thinking: ${report.thinking})` : ""} · ${report.runs} case run(s) · ` +
+ `${report.repeat} attempt(s) per case · $${report.costUsd.toFixed(4)} · ${seconds}s`,
+ );
+ lines.push("");
+
+ if (report.trigger.length > 0) {
+ lines.push(`## Trigger fidelity — ${triggerPasses}/${triggerTotal}`);
+ lines.push("");
+ lines.push("| Expected | Loaded | Prompt |");
+ lines.push("|----------|--------|--------|");
+ for (const row of report.trigger) {
+ const loads = row.attempts.filter((a) => a.loaded).length;
+ lines.push(
+ `| ${row.expected === "load" ? "load" : "skip"} | ${loads}/${row.attempts.length} | ${firstLine(row.prompt)} |`,
+ );
+ }
+ const failures = report.trigger.filter((r) => r.passes < r.attempts.length);
+ if (failures.length > 0) {
+ lines.push("");
+ lines.push("Misses:");
+ lines.push("");
+ for (const row of failures) {
+ const verb = row.expected === "load" ? "did not load" : "loaded when it should not have";
+ lines.push(`- ${verb}: ${firstLine(row.prompt, 120)}`);
+ for (const attempt of row.attempts) {
+ if (attempt.error) lines.push(` - run error: ${attempt.error}`);
+ }
+ }
+ }
+ lines.push("");
+ }
+
+ if (report.adherence.length > 0) {
+ lines.push(`## Instruction adherence — ${adherencePasses}/${adherenceTotal}`);
+ lines.push("");
+ for (const row of report.adherence) {
+ lines.push(`### ${firstLine(row.prompt, 120)}`);
+ lines.push("");
+ row.attempts.forEach((attempt, i) => {
+ if (row.attempts.length > 1) lines.push(`Attempt ${i + 1}:`);
+ if (attempt.error) lines.push(`- run error: ${attempt.error}`);
+ for (const expectation of attempt.expectations) {
+ lines.push(`- ${expectation.verdict}: ${expectation.expectation} — ${expectation.reason}`);
+ }
+ lines.push("");
+ });
+ }
+ }
+
+ return lines.join("\n");
+}
+
+/** Failure detail worth pulling into the conversation so the skill text can be fixed. */
+export function summarizeFailures(report: EvalReport): string[] {
+ const notes: string[] = [];
+ for (const row of report.trigger) {
+ if (row.passes === row.attempts.length) continue;
+ const loads = row.attempts.filter((a) => a.loaded).length;
+ notes.push(
+ row.expected === "load"
+ ? `The description did not fire for "${firstLine(row.prompt, 120)}" (${loads}/${row.attempts.length} loads).`
+ : `The description fired on the near-miss prompt "${firstLine(row.prompt, 120)}" (${loads}/${row.attempts.length} loads).`,
+ );
+ }
+ for (const row of report.adherence) {
+ for (const attempt of row.attempts) {
+ for (const expectation of attempt.expectations) {
+ if (expectation.verdict === "pass") continue;
+ notes.push(
+ `On "${firstLine(row.prompt, 80)}", expectation "${expectation.expectation}" came back ${expectation.verdict}: ${expectation.reason}`,
+ );
+ }
+ }
+ }
+ return notes;
+}
diff --git a/ai/pi/extensions/skill-eval/skill-eval.test.ts b/ai/pi/extensions/skill-eval/skill-eval.test.ts
index d1a157e9..56fadeca 100644
--- a/ai/pi/extensions/skill-eval/skill-eval.test.ts
+++ b/ai/pi/extensions/skill-eval/skill-eval.test.ts
@@ -244,6 +244,14 @@ test("resolveReportPath builds a path under the given state base dir", () => {
assert.equal(path, join("/fake/state", "pi", "skill-eval", "reports", "2026-01-01T00-00-00-000Z.md"));
});
+test("resolveReportPath appends a filename-safe label", () => {
+ const path = resolveReportPath(new Date("2026-01-01T00:00:00.000Z"), "/fake/state", "eval-jj workspaces");
+ assert.equal(
+ path,
+ join("/fake/state", "pi", "skill-eval", "reports", "2026-01-01T00-00-00-000Z-eval-jj_workspaces.md"),
+ );
+});
+
test("writeReport creates parent directories and writes content", async () => {
const dir = await mkdtemp(join(tmpdir(), "skill-eval-report-"));
try {
@@ -310,3 +318,47 @@ test("buildReviewPrompt notes no usage signal for AGENTS.md", () => {
const prompt = buildReviewPrompt("agents", "/home/user/AGENTS.md", "# instructions", null);
assert.match(prompt, /No usage signal available/);
});
+
+import { parseCommandArgs } from "./index.ts";
+
+test("parseCommandArgs treats no arguments and an explicit subcommand as triage", () => {
+ assert.deepEqual(parseCommandArgs(""), { action: "triage" });
+ assert.deepEqual(parseCommandArgs(" "), { action: "triage" });
+ assert.deepEqual(parseCommandArgs("triage"), { action: "triage" });
+});
+
+test("parseCommandArgs keeps the bare-target form working as a review", () => {
+ assert.deepEqual(parseCommandArgs("jj"), { action: "review", target: "jj" });
+ assert.deepEqual(parseCommandArgs("review jj"), { action: "review", target: "jj" });
+});
+
+test("parseCommandArgs reads run flags and applies defaults", () => {
+ assert.deepEqual(parseCommandArgs("run jj"), { action: "run", target: "jj", repeat: 1, jobs: 3 });
+ assert.deepEqual(parseCommandArgs("run jj --repeat 3 --jobs 1 --only trigger"), {
+ action: "run",
+ target: "jj",
+ repeat: 3,
+ jobs: 1,
+ only: "trigger",
+ });
+});
+
+test("parseCommandArgs rejects malformed input with a specific message", () => {
+ const cases: [string, RegExp][] = [
+ ["run", /needs a skill name/],
+ ["run jj bash", /exactly one skill/],
+ ["run jj --repeat 0", /positive integer/],
+ ["run jj --jobs two", /positive integer/],
+ ["run jj --only sideways", /trigger or adherence/],
+ ["run jj --wat", /Unknown flag --wat/],
+ ["triage jj", /takes no arguments/],
+ ["review", /exactly one target/],
+ ["review a b", /exactly one target/],
+ ["frobnicate jj", /Unknown subcommand "frobnicate"/],
+ ];
+ for (const [args, pattern] of cases) {
+ const parsed = parseCommandArgs(args);
+ assert.equal(parsed.action, "error", `expected an error for "${args}"`);
+ if (parsed.action === "error") assert.match(parsed.message, pattern);
+ }
+});
diff --git a/ai/pi/extensions/skill-eval/subagent.test.ts b/ai/pi/extensions/skill-eval/subagent.test.ts
new file mode 100644
index 00000000..df399e9f
--- /dev/null
+++ b/ai/pi/extensions/skill-eval/subagent.test.ts
@@ -0,0 +1,194 @@
+import { test } from "node:test";
+import assert from "node:assert/strict";
+import { mkdtemp, rm, writeFile } from "node:fs/promises";
+import { tmpdir } from "node:os";
+import { join } from "node:path";
+import {
+ buildSubagentArgs,
+ collectEvents,
+ extractCost,
+ extractFinalText,
+ extractToolCalls,
+ formatTranscript,
+ runSetupScript,
+ runSubagent,
+} from "./subagent.ts";
+
+const baseOptions = {
+ cwd: "/tmp/sandbox",
+ provider: "anthropic",
+ model: "claude-opus-5",
+ discoverSkills: true,
+ tools: ["read"],
+};
+
+test("buildSubagentArgs isolates the run and passes model, tools, and skills", () => {
+ const args = buildSubagentArgs({ ...baseOptions, thinking: "high", skillDirs: ["/skills/jj"] });
+
+ for (const flag of [
+ "-p",
+ "--no-session",
+ "--no-extensions",
+ "--no-context-files",
+ "--no-prompt-templates",
+ "--no-approve",
+ ]) {
+ assert.ok(args.includes(flag), `expected ${flag}`);
+ }
+ assert.deepEqual(args.slice(args.indexOf("--mode"), args.indexOf("--mode") + 2), ["--mode", "json"]);
+ assert.deepEqual(args.slice(args.indexOf("--tools"), args.indexOf("--tools") + 2), ["--tools", "read"]);
+ assert.deepEqual(args.slice(args.indexOf("--skill"), args.indexOf("--skill") + 2), ["--skill", "/skills/jj"]);
+ assert.deepEqual(args.slice(args.indexOf("--thinking"), args.indexOf("--thinking") + 2), ["--thinking", "high"]);
+ assert.ok(!args.includes("--no-skills"));
+});
+
+test("buildSubagentArgs disables tools and skill discovery when asked", () => {
+ const args = buildSubagentArgs({ ...baseOptions, tools: [], discoverSkills: false });
+ assert.ok(args.includes("--no-tools"));
+ assert.ok(args.includes("--no-skills"));
+ assert.ok(!args.includes("--tools"));
+});
+
+test("buildSubagentArgs appends messages in order after the flags", () => {
+ const args = buildSubagentArgs({ ...baseOptions, messages: ["/skill:jj", "commit my work"] });
+ assert.deepEqual(args.slice(-2), ["/skill:jj", "commit my work"]);
+});
+
+test("collectEvents parses JSON lines and skips noise", () => {
+ const events = collectEvents(['{"type":"agent_start"}', "not json", "", '{"type":"agent_end","messages":[]}'].join("\n"));
+ assert.deepEqual(
+ events.map((e) => e.type),
+ ["agent_start", "agent_end"],
+ );
+});
+
+test("extractToolCalls returns tool executions in order", () => {
+ const events = [
+ { type: "tool_execution_start", toolName: "read", args: { path: "/skills/jj/SKILL.md" } },
+ { type: "message_end" },
+ { type: "tool_execution_start", toolName: "bash", args: { command: "jj st" } },
+ ];
+ assert.deepEqual(extractToolCalls(events), [
+ { name: "read", args: { path: "/skills/jj/SKILL.md" } },
+ { name: "bash", args: { command: "jj st" } },
+ ]);
+});
+
+const agentEnd = {
+ type: "agent_end",
+ messages: [
+ { role: "user", content: [{ type: "text", text: "commit my work" }] },
+ {
+ role: "assistant",
+ content: [
+ { type: "text", text: "Checking status." },
+ { type: "toolCall", name: "bash", arguments: { command: "jj st" } },
+ ],
+ usage: { cost: { total: 0.01 } },
+ },
+ { role: "toolResult", toolName: "bash", isError: false, content: [{ type: "text", text: "Working copy changes" }] },
+ { role: "assistant", content: [{ type: "text", text: "Done." }], usage: { cost: { total: 0.02 } } },
+ ],
+};
+
+test("extractFinalText returns the last assistant text", () => {
+ assert.equal(extractFinalText([{ type: "agent_start" }, agentEnd]), "Done.");
+});
+
+test("extractFinalText returns an empty string when no assistant text exists", () => {
+ assert.equal(extractFinalText([{ type: "agent_end", messages: [] }]), "");
+ assert.equal(extractFinalText([]), "");
+});
+
+test("extractCost sums per-message cost", () => {
+ assert.equal(Number(extractCost([agentEnd]).toFixed(4)), 0.03);
+});
+
+test("formatTranscript renders prompts, assistant text, tool calls, and results", () => {
+ const transcript = formatTranscript([agentEnd]);
+ assert.match(transcript, /\[user\]\ncommit my work/);
+ assert.match(transcript, /\[assistant\]\nChecking status\./);
+ assert.match(transcript, /\[tool call: bash\]\n\{"command":"jj st"\}/);
+ assert.match(transcript, /\[tool result: bash\]\nWorking copy changes/);
+});
+
+test("formatTranscript truncates long tool results", () => {
+ const events = [
+ {
+ type: "agent_end",
+ messages: [
+ { role: "toolResult", toolName: "bash", isError: false, content: [{ type: "text", text: "x".repeat(50) }] },
+ ],
+ },
+ ];
+ const transcript = formatTranscript(events, 10);
+ assert.match(transcript, /x{10}\n… \[40 more characters\]/);
+});
+
+test("runSetupScript reports success and failure with output", async () => {
+ const ok = await runSetupScript("echo hello", "/tmp");
+ assert.equal(ok.ok, true);
+ assert.equal(ok.output, "hello");
+
+ const failed = await runSetupScript("echo boom >&2; exit 3", "/tmp");
+ assert.equal(failed.ok, false);
+ assert.match(failed.output, /boom/);
+});
+
+/** Stand in for the pi entrypoint via PI_BIN so these tests spend no API credit. */
+async function withFakePi<T>(script: string, fn: () => Promise<T>): Promise<T> {
+ const dir = await mkdtemp(join(tmpdir(), "skill-eval-fakepi-"));
+ const path = join(dir, "fake-pi.js");
+ await writeFile(path, script);
+ const previous = process.env.PI_BIN;
+ process.env.PI_BIN = path;
+ try {
+ return await fn();
+ } finally {
+ if (previous === undefined) delete process.env.PI_BIN;
+ else process.env.PI_BIN = previous;
+ await rm(dir, { recursive: true, force: true });
+ }
+}
+
+test("runSubagent parses the event stream, forwards stdin, and reports cost", async () => {
+ const script = `
+let input = "";
+process.stdin.setEncoding("utf8");
+process.stdin.on("data", (chunk) => (input += chunk));
+process.stdin.on("end", () => {
+ process.stdout.write(JSON.stringify({ type: "tool_execution_start", toolName: "read", args: { path: input } }) + "\\n");
+ process.stdout.write(JSON.stringify({
+ type: "agent_end",
+ messages: [{ role: "assistant", content: [{ type: "text", text: "ok: " + process.argv.includes("--no-tools") }], usage: { cost: { total: 0.5 } } }],
+ }) + "\\n");
+});
+`;
+ const result = await withFakePi(script, () =>
+ runSubagent({ ...baseOptions, cwd: "/tmp", tools: [], discoverSkills: false, stdin: "/skills/jj/SKILL.md" }),
+ );
+
+ assert.equal(result.exitCode, 0);
+ assert.equal(result.error, undefined);
+ assert.deepEqual(result.toolCalls, [{ name: "read", args: { path: "/skills/jj/SKILL.md" } }]);
+ assert.equal(result.finalText, "ok: true");
+ assert.equal(result.costUsd, 0.5);
+});
+
+test("runSubagent reports a nonzero exit with stderr instead of throwing", async () => {
+ const result = await withFakePi('process.stderr.write("no API key\\n"); process.exit(2);', () =>
+ runSubagent({ ...baseOptions, cwd: "/tmp" }),
+ );
+
+ assert.equal(result.exitCode, 2);
+ assert.match(result.error ?? "", /pi exited with code 2: no API key/);
+});
+
+test("runSubagent kills a run that outlives its timeout", async () => {
+ const result = await withFakePi("setTimeout(() => {}, 60000);", () =>
+ runSubagent({ ...baseOptions, cwd: "/tmp", timeoutMs: 200 }),
+ );
+
+ assert.equal(result.timedOut, true);
+ assert.match(result.error ?? "", /exceeded 200ms/);
+});
diff --git a/ai/pi/extensions/skill-eval/subagent.ts b/ai/pi/extensions/skill-eval/subagent.ts
new file mode 100644
index 00000000..caab00a1
--- /dev/null
+++ b/ai/pi/extensions/skill-eval/subagent.ts
@@ -0,0 +1,236 @@
+/**
+ * Subagent runner.
+ *
+ * Pi has no subagents, so a "subagent" here is a nested `pi -p --mode json`
+ * process: no session file, no extensions, no context files, an explicit tool
+ * allowlist, and a scratch cwd. Everything the caller needs — tool calls,
+ * final text, spend — comes from parsing the JSON event stream on stdout.
+ */
+
+import { spawn } from "node:child_process";
+
+export interface SubagentOptions {
+ cwd: string;
+ provider: string;
+ model: string;
+ thinking?: string;
+ /** Tool allowlist. An empty list disables tools entirely. */
+ tools: string[];
+ /** Extra skill paths to load on top of ordinary discovery. */
+ skillDirs?: string[];
+ /** Whether ordinary skill discovery runs, which keeps description competition realistic. */
+ discoverSkills: boolean;
+ /** Messages processed in order, each as its own turn. Must not start with "-" or "@". */
+ messages?: string[];
+ /** Prompt text piped in, which avoids argv escaping limits entirely. */
+ stdin?: string;
+ timeoutMs?: number;
+ signal?: AbortSignal;
+}
+
+export interface SubagentToolCall {
+ name: string;
+ args: Record<string, unknown>;
+}
+
+export interface SubagentResult {
+ exitCode: number | null;
+ timedOut: boolean;
+ stderr: string;
+ events: any[];
+ toolCalls: SubagentToolCall[];
+ finalText: string;
+ costUsd: number;
+ /** Set when the run itself failed, as opposed to the graded behavior failing. */
+ error?: string;
+}
+
+export function buildSubagentArgs(opts: SubagentOptions): string[] {
+ const args = [
+ "-p",
+ "--mode",
+ "json",
+ "--no-session",
+ "--no-extensions",
+ "--no-context-files",
+ "--no-prompt-templates",
+ "--no-approve",
+ "--offline",
+ "--provider",
+ opts.provider,
+ "--model",
+ opts.model,
+ ];
+ if (opts.thinking) args.push("--thinking", opts.thinking);
+ if (opts.tools.length === 0) args.push("--no-tools");
+ else args.push("--tools", opts.tools.join(","));
+ if (!opts.discoverSkills) args.push("--no-skills");
+ for (const dir of opts.skillDirs ?? []) args.push("--skill", dir);
+ for (const message of opts.messages ?? []) args.push(message);
+ return args;
+}
+
+/** Resolve the pi entrypoint of the running process so subagents match this build. */
+export function piExecutable(): { command: string; prefixArgs: string[] } {
+ const cli = process.env.PI_BIN ?? process.argv[1];
+ if (cli) return { command: process.execPath, prefixArgs: [cli] };
+ return { command: "pi", prefixArgs: [] };
+}
+
+export function collectEvents(stdout: string): any[] {
+ const events: any[] = [];
+ for (const line of stdout.split("\n")) {
+ if (!line.trim()) continue;
+ try {
+ events.push(JSON.parse(line));
+ } catch {
+ // Non-JSON noise on stdout isn't fatal; the stream is line-oriented.
+ }
+ }
+ return events;
+}
+
+export function extractToolCalls(events: any[]): SubagentToolCall[] {
+ return events
+ .filter((event) => event.type === "tool_execution_start")
+ .map((event) => ({ name: event.toolName, args: event.args ?? {} }));
+}
+
+function finalMessages(events: any[]): any[] {
+ for (let i = events.length - 1; i >= 0; i--) {
+ if (events[i].type === "agent_end" && Array.isArray(events[i].messages)) return events[i].messages;
+ }
+ return [];
+}
+
+export function extractFinalText(events: any[]): string {
+ const messages = finalMessages(events);
+ for (let i = messages.length - 1; i >= 0; i--) {
+ const message = messages[i];
+ if (message.role !== "assistant" || !Array.isArray(message.content)) continue;
+ const text = message.content
+ .filter((block: any) => block.type === "text" && typeof block.text === "string")
+ .map((block: any) => block.text)
+ .join("\n")
+ .trim();
+ if (text) return text;
+ }
+ return "";
+}
+
+export function extractCost(events: any[]): number {
+ let total = 0;
+ for (const message of finalMessages(events)) {
+ const cost = message?.usage?.cost?.total;
+ if (typeof cost === "number") total += cost;
+ }
+ return total;
+}
+
+/** Readable transcript of a run, for handing to a judge. */
+export function formatTranscript(events: any[], maxResultChars = 1500): string {
+ const lines: string[] = [];
+ for (const message of finalMessages(events)) {
+ if (message.role === "user" && Array.isArray(message.content)) {
+ const text = message.content
+ .filter((block: any) => block.type === "text")
+ .map((block: any) => block.text)
+ .join("\n")
+ .trim();
+ if (text) lines.push(`[user]\n${text}`);
+ } else if (message.role === "assistant" && Array.isArray(message.content)) {
+ for (const block of message.content) {
+ if (block.type === "text" && block.text?.trim()) lines.push(`[assistant]\n${block.text.trim()}`);
+ if (block.type === "toolCall") {
+ lines.push(`[tool call: ${block.name}]\n${truncate(JSON.stringify(block.arguments ?? {}), maxResultChars)}`);
+ }
+ }
+ } else if (message.role === "toolResult" && Array.isArray(message.content)) {
+ const text = message.content
+ .filter((block: any) => block.type === "text")
+ .map((block: any) => block.text)
+ .join("\n");
+ lines.push(`[tool result: ${message.toolName}${message.isError ? " (error)" : ""}]\n${truncate(text, maxResultChars)}`);
+ }
+ }
+ return lines.join("\n\n");
+}
+
+function truncate(text: string, max: number): string {
+ if (text.length <= max) return text;
+ return `${text.slice(0, max)}\n… [${text.length - max} more characters]`;
+}
+
+export async function runSubagent(opts: SubagentOptions): Promise<SubagentResult> {
+ const { command, prefixArgs } = piExecutable();
+ const args = [...prefixArgs, ...buildSubagentArgs(opts)];
+
+ const env = { ...process.env, NO_COLOR: "1", PI_OFFLINE: "1" };
+ // A nested run must not inherit this session's identity.
+ delete env.PI_SESSION_FILE;
+ delete env.PI_SESSION_ID;
+
+ return await new Promise<SubagentResult>((resolve) => {
+ const child = spawn(command, args, { cwd: opts.cwd, env, stdio: ["pipe", "pipe", "pipe"] });
+ let stdout = "";
+ let stderr = "";
+ let timedOut = false;
+ let settled = false;
+
+ const timer = opts.timeoutMs
+ ? setTimeout(() => {
+ timedOut = true;
+ child.kill("SIGKILL");
+ }, opts.timeoutMs)
+ : undefined;
+
+ const onAbort = () => child.kill("SIGKILL");
+ opts.signal?.addEventListener("abort", onAbort, { once: true });
+
+ const finish = (exitCode: number | null, error?: string) => {
+ if (settled) return;
+ settled = true;
+ if (timer) clearTimeout(timer);
+ opts.signal?.removeEventListener("abort", onAbort);
+ const events = collectEvents(stdout);
+ resolve({
+ exitCode,
+ timedOut,
+ stderr: stderr.trim(),
+ events,
+ toolCalls: extractToolCalls(events),
+ finalText: extractFinalText(events),
+ costUsd: extractCost(events),
+ ...(error ? { error } : {}),
+ });
+ };
+
+ child.stdout.setEncoding("utf8");
+ child.stdout.on("data", (chunk) => (stdout += chunk));
+ child.stderr.setEncoding("utf8");
+ child.stderr.on("data", (chunk) => (stderr += chunk));
+ child.on("error", (error) => finish(null, error.message));
+ child.on("close", (code) => {
+ if (timedOut) finish(code, `run exceeded ${opts.timeoutMs}ms and was killed`);
+ else if (code !== 0) finish(code, `pi exited with code ${code}${stderr.trim() ? `: ${stderr.trim()}` : ""}`);
+ else finish(code);
+ });
+
+ if (opts.stdin !== undefined) child.stdin.end(opts.stdin);
+ else child.stdin.end();
+ });
+}
+
+/** Run a case's setup script in its sandbox. */
+export async function runSetupScript(script: string, cwd: string): Promise<{ ok: boolean; output: string }> {
+ return await new Promise((resolve) => {
+ const child = spawn("bash", ["-euo", "pipefail", "-c", script], { cwd, stdio: ["ignore", "pipe", "pipe"] });
+ let output = "";
+ child.stdout.setEncoding("utf8");
+ child.stdout.on("data", (chunk) => (output += chunk));
+ child.stderr.setEncoding("utf8");
+ child.stderr.on("data", (chunk) => (output += chunk));
+ child.on("error", (error) => resolve({ ok: false, output: error.message }));
+ child.on("close", (code) => resolve({ ok: code === 0, output: output.trim() }));
+ });
+}