skill-up

skill-up is a CLI for evaluating Agent Skills, agents, and their workspaces. It runs declarative cases against an Agent Engine, grades the response and workspace changes, and produces reports locally or in CI.

• Skill evaluation measures a Skill's behavior across cases and can compare runs with and without the Skill.
• Agent evaluation runs the same cases without installing a Skill, so you can assess an engine or custom agent on its own.
• Workspace evaluation checks how an agent works with files and repositories, using case fixtures or an existing local workspace.
• Skill evolution uses skill-upper to inspect failures, repair or expand the eval suite, and rerun skill-up through conversation.

• Eval-to-Evolution Loop with skill-upper: Create evals through natural conversation, diagnose failures, automatically repair or expand cases, and rerun skill-up until the eval suite evolves.
• Declarative Eval Config: Define evaluation environment, engine, model, and cases through YAML (eval.yaml + cases/*.yaml).
• Environment Provisioning: Prepare a local, Docker, or OpenSandbox runtime for each case, install configured real or mocked MCP servers into supported Agent Engines, and upload context.repo_fixture and context.files into the workspace.
• Skill-Optional Runs and Workspace Cases: Evaluate an agent without a Skill, prepare repository fixtures per case, or run against an existing local workspace with --workspace.
• Multi-Engine Support: Works with Qoder CLI, Claude Code, and Codex as built-in Agent Engines, plus user-defined agents via engine.custom (local commands or HTTP services — see docs/design/custom-engine.md).
• DeepSeek Harness Plugin: Use the bundled skill-upper Skill, opt-in durable observation capture, approval-gated regression cases, isolated runs, and status comparison to verify evidence-backed Skill improvements. See plugins/dsh-skill-up.
• Multi-Turn Conversations and User Simulation: Run fixed turns, mix scripted and simulated replies, or let an independent user model drive follow-ups and decide when to stop.
• Role-Specific Engines and Models: Configure the evaluated agent, Agent Judge, and user simulator separately, including independent judging of custom agents.
• Flexible Judging: Supports rule_based, script, and agent_judge evaluation strategies.
• Structured Reports: Outputs Anthropic-compatible grading.json, benchmark.json, benchmark.md, plus result.json, JUnit XML, and HTML reports.
• Anthropic Compatible: Import evals.json via skill-up import, or auto-detect with --auto.
• CI-Ready: Designed for local development and continuous integration pipelines.

Harness engineering treats an agent's environment, tools, constraints, and feedback loops as parts of the system to design and improve. Behavioral evals make expected agent actions observable and help catch regressions. skill-up contributes evaluation and feedback to this work: repeatable prompts and workspace setups, checks on responses, tool calls, and file changes, and structured reports.

For Skills, the official Agent Skills evaluation guide describes a further loop: run with and without the Skill, grade outputs, aggregate results, and iterate. skill-up supports these workflows:

• Replaces ad hoc run folders with a declarative eval.yaml + cases/*.yaml format.
• Closes the improvement loop: skill-upper can interpret failed reports, repair or add eval cases, and drive the next skill-up run through conversation.
• Automates workspace setup, optional Skill installation, Agent Engine invocation, judging, and report generation.
• Supports multiple engines (claude_code, codex, qodercli, qwen_code) instead of tying the workflow to one client.
• Keeps compatibility with Anthropic-style evals.json while adding richer judges, CI-friendly commands, and structured reports.

For Skill evolution, use skill-upper, the Agent Skill shipped in this repository. It lets your AI agent create evals, run skill-up, understand failures, fix the Skill or its evals, add regression coverage, and repeat the loop through conversation.

For repository development, skills/skill-upper/ is the only editable source. Plugin bundles are generated into ignored build directories rather than tracked copies or symlinks; run make bundle-plugins when preparing them locally. make package-plugins VERSION=<version> creates self-contained Codex and DSH archives. Tagged GitHub releases attach both archives and their checksums; this repository does not publish either plugin to an external marketplace or npm.

# Codex, global install
npx skills add https://github.com/alibaba/skill-up/tree/main/skills/skill-upper -g -a codex -y

# Claude Code, global install
npx skills add https://github.com/alibaba/skill-up/tree/main/skills/skill-upper -g -a claude-code -y

You normally do not need to install skill-up first. skill-upper checks for the CLI when it runs and guides the agent through installation if needed.

Open a project that contains your Skill's SKILL.md in Codex, Claude Code, or another compatible Agent, then ask:

Use skill-upper to evaluate this Skill.
Read SKILL.md, identify its most important behaviors, create realistic eval
cases with appropriate judges, validate the configuration, and run skill-up.
Summarize the results and the highest-impact failures.

skill-upper creates the declarative eval suite and drives the CLI for you:

my-skill/
  SKILL.md
  evals/
    eval.yaml
    cases/
      <case-id>.yaml
my-skill-workspace/
  iteration-1/
    result.json

Continue in the same conversation:

Review the latest skill-up results. For each failure, determine whether the
Skill or the eval is wrong. Fix SKILL.md and supporting files, or repair the
eval case and judge as appropriate. Add regression cases for the bugs you
found, rerun skill-up, and continue until the important behaviors pass.

This is the evolution loop: reports become fixes, fixes become regression cases, and every iteration makes the Skill and its eval suite stronger.

You can still install the CLI directly and hand-write eval.yaml and case files:

curl -fsSL https://raw.githubusercontent.com/alibaba/skill-up/main/install.sh | bash

See the official documentation for Getting Started, Writing Evals, CLI Reference, and User Configuration. Windows-specific setup and limitations are covered in the Windows guide.

An agent-only eval does not need SKILL.md. Set skills: [] explicitly: when skills is omitted and skill-up finds a SKILL.md above the config, it installs that Skill automatically. Put eval.yaml and case files under evals/, then pass the config path explicitly. Case and fixture paths resolve from the discovered Skill root, or from the directory containing evals/ if none is found.

# evals/eval.yaml
schema_version: v1alpha1
environment:
  type: none
skills: []
engine:
  name: codex
cases:
  files:
    - evals/cases/agent-response.yaml

# evals/cases/agent-response.yaml
input:
  prompt: Reply with READY.
expect:
  must_contain: [READY]

skill-up validate ./evals/eval.yaml
skill-up run ./evals/eval.yaml

For repository tasks, a case can use context.repo_fixture to prepare a fresh workspace and expect or a judge to check the result. To evaluate an agent in an already prepared local directory, use a case written for that directory and run skill-up run ./evals/eval.yaml --workspace /path/to/project --parallelism 1. The existing directory is preserved, but the agent and case setup can change its contents. This option requires environment.type: none, one case at a time, and benchmark mode off. See Writing Evals and the CLI Reference.

A simulated user can react to what the agent actually says, so follow-up messages need not be scripted in advance. For example, evaluate a Twenty Questions player:

• The evaluated agent asks natural-language yes/no questions and guesses an object.
• The simulated user knows the hidden object and answers from the conversation history.
• An independent Agent Judge reviews the complete dialogue against explicit criteria.

This case YAML excerpt lets the simulator drive all replies after the opening:

id: umbrella
title: Guess a hidden object through simulated user replies
user_simulator:
  scenario: |
    You host Twenty Questions. The hidden object is an ordinary folding umbrella.
    Answer questions using common knowledge, with only yes, no, or unsure.
    Do not reveal the object or volunteer hints. Stop immediately when the player
    guesses correctly, or after the final permitted guess.
input:
  prompt: |
    I have a common everyday object in mind. Guess it by asking yes/no questions;
    I will answer yes, no, or unsure. Ask only one question per turn, and ask at
    least two questions before guessing. You have 20 questions or incorrect
    guesses in total, followed by one final guess. Use natural language.
constraints:
  max_turns: 22

The hidden answer belongs to the simulator's scenario, not the player's prompt. For example, a question about rain protection can lead to a different next question than one about electronics; the simulator responds to the actual question. The 22-turn runtime cap leaves room for 20 questions or incorrect guesses, a final guess, and a closing response. The judge separately checks the game's rules; the runtime cap alone does not enforce them.

Other conversation patterns remain available:

Real turn-by-turn interaction requires session resumption. claude_code, codex, qodercli, and opencode support it; custom engines must set engine.custom.conversation_mode: stateful and resume using session_id. Ordinary fixed-turn cases fall back to a batch prompt on engines without resumption; simulated cases require resumption. The opening prompt counts as turn one in an autonomous conversation, and max_turns bounds the total. A simulator stop ends the dialogue; the configured judge still determines PASS or FAIL.

This eval.yaml excerpt shows role configuration. Model IDs are placeholders; replace them with IDs supported by each provider and configure case files and environment for your evaluation. These criteria grade the game above; pass_threshold: 1.0 requires all criteria to pass:

engine:
  name: codex
  model:
    provider: openai
    name: your-agent-model-id

judge:
  type: agent_judge
  engine:
    name: claude_code
    model:
      provider: anthropic
      name: your-judge-model-id
  criteria:
    - "The player asked at least two questions before guessing and correctly identified an umbrella"
    - "The player asked only one question per turn and respected the 20-attempt limit plus one final guess"
    - "The simulated user answered consistently with a folding umbrella and did not reveal the answer before a correct guess"
  pass_threshold: 1.0
  context:
    transcript: file_ref

user_simulator:
  provider: simulation
  protocol: openai
  model: your-simulator-model-id
  timeout_seconds: 60

judge.engine does not inherit the evaluated agent's model, CLI overrides, or connection configuration; prepare the required engines and credentials separately. It is mutually exclusive with legacy judge.model, which reuses the evaluated agent's engine. The simulator above resolves SIMULATION_API_KEY and SIMULATION_BASE_URL, or the same provider in the credentials file; protocol supports openai (Chat Completions) and anthropic (Messages). An independent judge still uses the case runtime; there is no judge.environment. The host agent/model that drives skill-upper's diagnosis and edits is configured in your host application, outside these evaluation fields.

See Writing Evals for complete fields, per-turn assertions, and the session support matrix. See the user simulator example for runnable configurations and validation scope, and the Custom Engine design for custom agent integration.

skill-up auto-loads an optional user-level config that supplies default OpenTelemetry env vars and per-environment runtime kwargs. The embedded defaults are empty; downstream consumers maintain their own config file.

embed (empty) < user (~/.config/skill-up/config.yaml) < project ($PWD/.skill-up.yaml) < explicit (--config)

Missing files at the user and project layers are silently skipped; a missing --config path is a hard error. A corrupt config at any layer also fails the run.

skill-up init                            # writes a template to ~/.config/skill-up/config.yaml (XDG-aware)
skill-up init --local                    # writes a template to $PWD/.skill-up.yaml
skill-up init --print                    # prints the template to stdout
skill-up init --force                    # overwrite an existing file
skill-up init --config foo.yaml          # reads foo.yaml, writes it to ~/.config/skill-up/config.yaml
skill-up init --config foo.yaml --local  # reads foo.yaml, writes it to $PWD/.skill-up.yaml

With --config <path>, init reads that file (validating it as a skill-up config) and writes its raw bytes to the target — comments and formatting are preserved. Without --config, init writes a commented YAML template.

schema_version: v1alpha1
kind: SkillUpConfig

telemetry:
  service_name: skill-up                              # OTEL_SERVICE_NAME
  traces_exporter: otlp                                 # OTEL_TRACES_EXPORTER
  traces:
    endpoint: http://localhost:4317                     # OTEL_EXPORTER_OTLP_TRACES_ENDPOINT (4317 for grpc, 4318/v1/traces for http/protobuf)
    protocol: grpc                                      # OTEL_EXPORTER_OTLP_TRACES_PROTOCOL (grpc | http/protobuf); skill-up defaults to grpc
  resource_attributes:                                  # serialized into OTEL_RESOURCE_ATTRIBUTES
    deployment.environment: local
  verbose: false                                        # if true, also enables OTEL_LOG_* payload capture

env:                                                    # arbitrary defaults, applied only-if-unset
  OTEL_EXPORTER_OTLP_HEADERS: authorization=${OTLP_TOKEN}

runtime_kwargs:                                         # keyed by environment.type
  opensandbox:
    base_url: http://localhost:8080
    # extensions: '{}'

For environment variables: any value already set in the process environment wins; the config only fills in missing keys.

For runtime_kwargs: explicit --runtime-kwarg on run > eval.yaml environment.kwargs > user-config runtime_kwargs[environment.type].

Prefer ${ENV_VAR} references inside the config file rather than baking secret literals. The redaction mechanism (userconfig.Redact) masks fields tagged secret:"true" when printing; currently no Config field carries the tag, but the mechanism is in place for future fields.

Use skill-up import to migrate an Anthropic-compatible evals.json into the YAML layout used by this repo:

skill-up import ./evals/evals.json --output ./evals

Run your Agent Skill evals in CI on every pull request — and check the same skill across engines (claude_code / codex / qodercli / qwen_code) in one step. This repo ships an action at its root (action.yml):

# .github/workflows/skill-eval.yml
name: Skill Eval
on:
  pull_request:
    paths: ['skills/**', 'evals/**', '**/SKILL.md']
jobs:
  eval:
    runs-on: ubuntu-latest          # Docker container action — Linux only
    steps:
      - uses: actions/checkout@v4
      - uses: alibaba/skill-up@main  # see "Versioning" below
        with:
          engine: claude_code        # or codex / qodercli / qwen_code; empty = let eval.yaml decide
          api-key: ${{ secrets.ANTHROPIC_API_KEY }}
          base-url: https://api.anthropic.com   # your model endpoint
          skill-target: evals/eval.yaml

Requirements for the caller: a Linux runner (it's a Docker container action), and your model credential stored as a repo secret. The runner image is public, so no extra registry auth is needed.

Key inputs: engine, model, provider, api-key, base-url, skill-target, parallelism. The action prebuilds skill-up + the three engine CLIs into its runner image, so a run is just "pull image, eval". See action.yml for the full input/output reference.

The container image includes a deliberately pinned skill-up CLI version. The version is not resolved from latest when a workflow starts, so a given image digest always runs the same CLI.

The skill-up-version input is only a fallback for a custom image that does not already contain the skill-up binary. The official image contains the binary, so this input cannot override its bundled version. To find the effective version, check the skill-up --version line in the Action log.

Publishing a new skill-up CLI release does not automatically update the GitHub Action image. Maintainers must synchronize the pinned version, publish and test a new runner image, and update the image digest in action.yml. The complete maintainer procedure is documented in the CI maintenance runbook.

The production Action must use an immutable sha256: image digest. Do not replace it with skill-up-runner:latest; a mutable tag would allow existing Action references to change behavior without a repository commit.

uses: points at any git ref that contains action.yml. Pin a release tag (the first release that includes the action onward) or a commit SHA for stability; @main always tracks the latest. Release tags published before the action was added do not contain action.yml and cannot be used as the ref.

A CLI release tag captures the action.yml and runner-image digest that existed when that tag was created. Because the current runner image is refreshed manually after CLI release assets become available, do not assume that a CLI tag automatically contains an Action image with the same CLI version. Until a separate Action release tag process is introduced, use a post-refresh commit SHA for an immutable reference or @main when intentionally following Action updates.

skill-up/
├── cmd/skill-up/          # CLI entry point
├── internal/              # Private implementation packages
│   ├── cli/               # Commands and flags
│   ├── evaluator/         # Case execution and judging orchestration
│   ├── evalevent/         # Evaluation event model, lifecycle, publisher, and JSONL sink
│   ├── runner/            # End-to-end run orchestration
│   └── report/            # JSON, JUnit, HTML, and benchmark reports
├── pkg/                   # Publicly importable APIs
├── plugins/               # Agent host plugin bundles
│   ├── codex-skill-up/    # Codex adapter; release bundles add shared assets
│   └── dsh-skill-up/      # DSH bundle with a generated dist/ Skill copy
├── schemas/               # Evaluation-event and host-neutral observation contracts
├── skills/skill-upper/    # Canonical source for the distributable workflow Skill
├── docs/                  # VitePress documentation
├── e2e/                   # End-to-end tests and fixtures
└── examples/              # Example evaluations and fixtures

See CONTRIBUTING.md for the complete directory map and architectural responsibilities.

Apache License 2.0 — see LICENSE.