[EmadMokhtar]

Follow-up to #334. This PR moves the optional skill-lens eval from skill-lens 0.23.0 to 0.25.0 and adds a second case, which tests that a fresh session resumes from its own plan.

What changed

Move to skill-lens 0.25.0 (commit docs: move the skill-lens eval to skill-lens 0.25.0)

• skill-lens.toml: product_sandbox = "required", beside the existing script_sandbox = "required", and max_spend_usd = 10.00 for the whole run. The comments name the variables Claude Code now needs.
• docs/skill-lens.md:
  • Run it: pins skill-lens==0.25.0 and says to export CLAUDE_CODE_OAUTH_TOKEN (from claude setup-token) or ANTHROPIC_API_KEY.
  • How a run is isolated: rewritten for the product sandbox. It says what the sandbox blocks and what it does not.
  • What the hooks did: a new section.
  • Budgets: adds the run-wide spend ceiling.
  • Known limits: drops the four limits 0.25.0 fixed.

A resume case (commit feat: add a skill-lens case that resumes from its own plan)

• skills/planning-with-files/evals/planning-with-files.eval.yaml: a second case, resumes-from-its-own-plan. It runs as two sessions against one workspace. Each session is a new agent process with no conversation history; only the files carry over.

  1. "Build a Python CLI todo tool with persistence, in three phases (design; add and list; done and delete). Do phase 1 only, then stop."
  2. "Continue the todo tool from where the last session stopped. Do the next phase only, then stop."

  Every check runs after session 2 and looks for the work session 2 was asked to do:

  • Two regex checks: task_plan.md marks Phase 2 complete, and does not mark Phase 3 complete.
  • Two judge checks: progress.md records the add and list work, and has an entry for phase 1 before it.

  The checks cannot tell which session did the work, and the guide says so.

Review fixes and a re-run (commits fix: correct the resume case's checks and the guide's claims and docs: report the skill-lens re-run after the review fixes)

• The Phase 3 check now accepts any status but complete. Before, it failed a run that wrote not_started.
• Both phase checks now stop at the next heading, so a **Status:** line in a later section no longer counts.
• The second judge line now asks only what the final file shows.
• Several statements in the guide were corrected.
• The results below come from a fresh run after these fixes.

Why

skill-lens 0.25.0 runs Claude Code with a temporary home directory inside a sandbox. Without this PR, the documented command stops at preflight with exit 2, because the login saved in your home directory no longer reaches Claude Code. The release also fixed four limits #334 listed:

• every file the skill ships reaches the agent, including templates/, reference.md and examples.md;
• the report records which hooks fired;
• a run can stop at a spend ceiling;
• Claude Code runs sandboxed.

The skill's main claim is recovery: a fresh session resumes from the planning files. #334 could not test that, because a case was one session, and seeding the files by hand tests reading files the author wrote. 0.25.0 adds sessions: (several sessions in one workspace).

Results

One run, five runs per arm, skill-lens 0.25.0, planning-with-files v3.25.1 (99402ce) with this PR's eval files and config, both sandboxes on (sandbox-exec, macOS). Judge openai:gpt-6-luna.

Case	Runner and model	With skill	Without skill
creates-the-three-files	claude-code, claude-sonnet-4-6	2/5	0/5
creates-the-three-files	pydantic-ai, openai:gpt-6-luna	5/5	0/5
resumes-from-its-own-plan	claude-code, claude-sonnet-4-6	4/5	0/5
resumes-from-its-own-plan	pydantic-ai, openai:gpt-6-luna	0/5	0/5

• creates-the-three-files: Claude Code's three failures left findings.md as the empty template init-session.sh writes. Its pass count varies between runs: the two earlier runs, on 0.23.0 and on 0.25.0, both gave 3/5.
• resumes-from-its-own-plan, Claude Code: session 2 resumed correctly in all five runs. The failed run wrote its phases as ## Phase N headings instead of the template's ### Phase N, so neither regex found the sections.
• resumes-from-its-own-plan, PydanticAI:
  • In all five runs, the final progress.md had no entry for phase 1. The report does not record tool calls, so it does not show how session 1's entries were lost.
  • In 3 runs, session 2 also left Phase 2 in_progress, because it had no shell to run the tests it wrote.
• Without the skill, no run wrote a planning file.

The 40 runs cost $5.87, judge included. docs/skill-lens.md has tokens, cost and latency for each case.

Worth a look on the skill side

• The skill's hooks did nothing visible under Claude Code. In every run, every hook exited 0 and printed nothing: 234 hooks in this run. Run by hand in a copy of a kept workspace, with CLAUDE_SKILL_DIR set, the UserPromptSubmit hook prints the plan. Each hook command looks for ${CLAUDE_SKILL_DIR}/scripts/skill-hook.sh, then for a copy under $HOME/.claude/, and exits 0 when it finds neither. My reading is that Claude Code 2.1.289 does not set CLAUDE_SKILL_DIR for a skill's hooks: in its source the placeholder appears to be replaced only in the skill's text and allowed-tools. If so, the hooks work only through a personal install in $HOME. I have not confirmed this.
• PydanticAI lost session 1's log in every run. The Quick Start's "Preserve existing work" may not reach an agent whose only write tool replaces whole files.

How it was tested

• python -m pytest tests/ -q on Python 3.12 with pytest and PyYAML: 984 passed, 260 skipped.
• skill-lens list skills/planning-with-files with skill-lens 0.25.0 from PyPI loads both cases.
• Preflight with no credentials stops with exit 2 and names CLAUDE_CODE_OAUTH_TOKEN, as the guide says.
• The two regexes were checked against 15 plan files. These included not_started, Pending, backticked and emoji statuses, Complete, a later section with its own **Status:** line, and the untouched template. Each gave the expected result.
• The results come from skill-lens run skills/planning-with-files --baseline none --repeat 5 --runner claude-code --runner pydantic-ai.

Known limitations

• The resume case checks the planning files, not the code. A first session that ignored "then stop" and a second session that did nothing could still pass.
• The regexes look for the template's ### Phase N headings.
• Running both cases costs about $6 and needs CLAUDE_CODE_OAUTH_TOKEN (or ANTHROPIC_API_KEY) and OPENAI_API_KEY. The eval is not wired into this repository's CI.

🤖 Generated with Claude Code

[Copilot]

Copilot was unable to review this pull request because the user who requested the review has insufficient remaining quota.

[OthmanAdi]

Offline review against skill-lens 0.25.0 reproduced a resume-check false positive. A Phase 2 marked pending can pass when a later level-4 subsection says complete; a completed Phase 3 can pass when a later subsection says pending. The native assertion evaluator accepts both invalid fixtures.

I will keep your contribution and add a narrow maintainer follow-up that binds each check to the phase's first status line, with these counterexamples as regression fixtures. The measurements will remain attributed to your original runs; the maintainer regex correction is being validated offline.

[OthmanAdi]

@EmadMokhtar, merged with your authorship preserved and released in v3.25.2.

The 0.25.0 configuration and two-session resume case are included. The maintainer follow-up binds each phase assertion to its first status line, so a child subsection cannot turn an incomplete Phase 2 or completed Phase 3 into a pass. All 17 offline fixtures match their expected outcomes in the pinned native evaluator. The guide identifies the reported measurements as preceding this assertion correction.

Your CONTRIBUTORS.md entry includes this contribution. Final contributor commit: 3b25685. Assertion follow-up: 2bc4396. Release: dcdfb27.