Observation 01
“$9,960 of last month’s model spend went to a bigger model than the work needed.”
Model spend that buys nothing.
Open source. Built by Eversynced. Measure how your team actually works with AI coding agents.
An open-source AI telemetry tool created by Eversynced. It sits quietly inside the AI coding agents Claude Code, Cursor, and Codex via hooks, capturing lightweight interaction signals: the shape of the session, not its contents.
npm install -g pheebs Works with Claude CodeCursorCodex
pheebs · claude_code - session_started codebase=acme/checkoutmodel=opus
- prompt_submitted prompt_length=64intent=task
- tool_use_completed tool=Edit840ms
- tool_use_completed tool=Bashtool_intent=test_run
- context_compacted trigger_type=auto
- turn_ended background_task_count=1
- 3 coding agents Claude Code, Cursor, Codex
- 17 event types session_started to artifact_found
- 0 lines of code stored shape of the session, never its contents
- 4 routes to any backend self-hosted, or managed by us
What you did, and what Pheebs recorded.
Pheebs records the shape of the session, never its contents.
What you did, on your machine
- you /model sonnet
- you Fix the failing checkout test
- claude Read
src/billing/checkout.test.ts - claude Edit
src/billing/session.ts - you That changes the token TTL. Is that safe?
- claude Bash
npm test -- checkout - claude Passing. The TTL is read from config, so no.
What Pheebs recorded, interaction signals
- model_switched from_model=opusto_model=sonnet
- prompt_submitted prompt_length=29intent=debug
- tool_use_completed tool=Read412ms
- tool_use_completed tool=Edit1.2s
- prompt_submitted prompt_length=41requests_verification=true
- tool_use_completed tool=Bashtool_intent=test_run
- turn_ended background_task_count=0
Then it hands things back.
The events roll up into observations about your own work, the kind a colleague might make if they had been sitting next to you for a month and could remember all of it.
Observation 02
“61% of AI-written lines in the payments service shipped with no check.”
Where AI code ships unchallenged.
Observation 03
“Half of your team’s AI edits never had a test, typecheck, or build run behind them.”
Whether AI output gets verified.
Observation 04
“One in three follow-up prompts was fixing something the AI broke, not moving the work forward.”
Rework hiding inside the speedup.
Observation 05
“The review skill you shipped is used weekly by 78% of engineers. The migration skill never caught on.”
Whether the enablement investment landed.
Observation 06
“Nobody on the team runs tests inside the agent loop. That’s a missing harness, not a skills gap.”
Fix the setup, or coach the people.
It also puts a price on the model you never switch.
Most teams run the largest model for everything, because nothing tells them what the work needed. Pheebs sizes the work in every session and compares it with the model that ran. The gap between the two is a savings opportunity, priced in dollars from your team's real sessions.
-
Tasks are sized
Every task prompt gets a scope, from a one-file change to open-ended design. A session is judged on its hardest prompt.
-
Misses count both ways
An over-provisioned session burns budget silently. An under-powered one shows up as repair prompts.
-
An audit, not a router
Pheebs never intercepts a prompt or switches a model on anyone's behalf. It reads the gap and prices it. The decision stays yours.
Dollar savings Team · trailing 30 days Illustrative data
Savings opportunity $9,960 of $32,400 list-price spend 31% API list-price equivalent, estimated upper bound
Models used 67 · 19 · 12 · 2%
Work, as sized 37 · 25 · 33 · 5%
Coverage complete 1,912incomplete 131no telemetry 58unpriced 14 Decisions and figures come from complete sessions only.
Hooks in, data out.
Pheebs registers lifecycle hooks in the agent's own config, plus OpenTelemetry export where the tool supports it. From then on it fires in the background.
-
A hook fires
Session starts and ends, prompts, skill and slash-command expansions, sub-agent spawns, tool calls and failures, compaction, background tasks.
-
Lightweight fields are extracted
Event type, durations, counts, models, trigger types. A prompt becomes a character count and, when the prompt intent classifier is enabled, an intent label. The text itself is never stored.
-
Identity and repo are resolved
The developer is the id behind your Pheebs token, stamped by the backend, or a truncated hash of your git email when no token is set. The codebase is org/repo from the git remote.
-
Logged locally, then sent
Every event is stored in a local JSONL log. With a token set, it also goes to the backend.
-
OpenTelemetry rides along
Claude Code and Codex export native OTel metrics and logs through the Pheebs proxy.
- Claude Code Hooks + OpenTelemetry
- Cursor Hooks
- Codex Hooks + OpenTelemetry
What it never stores.
A fresh install ships with no backend endpoint, so it writes only to ~/.pheebs/logs/.
- Source code and file contents.
- File paths and directory structures. A repo is
org/repofrom the git remote. - Prompt text. A prompt becomes a character count. See the exception alongside.
- Command strings.
npm testis read in process and recorded astool_intent: test_run. - Your name or your email. You are the id behind your token, or a truncated hash of your git email. Your GitHub handle is never looked up.
Pheebs captures interaction patterns.
Then you decide where it goes.
Pheebs holds a base URL and a token, and nothing else. Give it both and the events are posted to your backend as well.
# Where to send pheebs config set base-url https://pheebs.acme.dev # Who you are pheebs config set token <token>
Both need to be set, or nothing is posted. Unset either one and sending stops. The local JSONL stays the durable copy either way.
You can find a reference backend in the Pheebs repo.
What your endpoint answers
- POST /ingest One event envelope per request.
- POST /validate-token Resolves a token to an identity and its consent flags.
- POST /classify-prompt One prompt in, one label out. The only route that receives raw text.
- POST /otel/v1/{signal} An OTLP passthrough, so no observability credential ever ships in the client.
- GET /insights Optional. What one developer can see about their own work.
The AI Proficiency Model
Proficiency = repertoire + judgement signals.
Pheebs proposes a specific way of looking at the data it captures: the AI Proficiency Model. The model defines the practices and signals that matter when working with AI agents, and gives structure to what would otherwise be a stream of raw events.
Layer 1 reads repertoire: which AI harness capabilities show up in an engineer's work. Layer 2 reads judgement: what happens to AI output before it ships, and whether the model that ran fit the work.
AI harness repertoire
Twenty-four practices across six competencies. Each practice is detectable from telemetry: either its detector fired or it didn't.
Models
Which models are in play: model choice, effort settings, plan mode, and autonomy modes.
Artifacts
The reusable config that shapes the agent: skills, sub-agents, slash commands, and context files.
MCP
Live connections to external systems: tickets, databases, browsers, and documentation.
Evals
Verification wired into the agent loop: tests, typecheck, lint, build, and review passes.
Context management
Deliberate use of the context window: compaction and the save, resume, clear lifecycle.
Orchestration
More than one agent at a time: sub-agents, parallel work, worktrees, hooks, and plugins.
AI judgement signals
Judgement on both sides of the AI loop: whether output gets verified, challenged, and refined before it ships (inspired by the Discernment competency from Anthropic's AI Fluency Index), and whether the model chosen fit the work. Continuous rates computed from session telemetry.
Verification coverage
The share of AI edits followed by a verification action: a test run, typecheck, lint, build, or a check against a spec.
Pushback rate
How often the engineer challenges AI output instead of accepting it. Questioning collapses exactly when output looks polished.
Refinement-to-repair ratio
Whether follow-up prompts refine intent (healthy iteration) or repair breakage (rework).
Wholesale-accept rate
Sessions with no pushback, no repair, and no verification, weighted by lines changed. The composite red flag: polished output, no questions asked.
Model-fit rate
The share of sessions whose model class matched the size of the work. Misses count both ways: over-provisioned burns budget, under-powered shows up as repairs.
The model, rendered. Sample views from the reporting built on top of Pheebs:
Practice adoption funnel Team · trailing 4 weeks Illustrative data
Models 67% recurring
Artifacts 50% recurring
MCP 50% recurring
Context management 33% recurring
Orchestration 33% recurring
Evals 0% recurring
Practice heatmap Team · trailing 4 weeks Illustrative data
Engineer view Pam · Trailing 4 weeks Illustrative data
Models Recurring 67%
Artifacts Recurring 33%
MCP Recurring 100%
Evals Adopted 20%
Context management Recurring 50%
Orchestration Recurring 40%
Coverage index 57%
Signals by engineer Team · trailing 4 weeks Illustrative data
| Engineer | Verification | Pushback | Refine : repair | Wholesale | Model-fit |
|---|---|---|---|---|---|
| Michael | 39% | 9% | 1.1 : 1 | 41% | 58% |
| Dwight | 55% | 17% | 1.6 : 1 | 26% | 63% |
| Jim | 44% | 12% | 1.2 : 1 | 38% | 51% |
| Pam | 71% | 22% | 2.4 : 1 | 15% | 35% |
| Angela | 62% | 15% | 1.9 : 1 | 21% | 66% |
| Kevin | 26% | 4% | 0.7 : 1 | 55% | 41% |
| Team median | 50% | 14% | 1.4 : 1 | 33% | 55% |
Two ways to run it.
Either run the backend, hold the data, and set up the reporting or hire us to do it for you.
Self-hosted
You run the backend, you hold the data
Stand up backend. Telemetry goes from your developers' machines to your infrastructure. We never see it.
- The full client, all three agents, Apache-2.0.
- A documented contract and a reference backend in the repo.
- Raw JSONL you can query with whatever you already use.
- No account, no key from us, no requests.
Managed
We run it, and the reporting on top
The same open-source client, pointed at a backend we operate, with the proficiency model rendered as reports and dashboards. That is our AI Enablement Assessment service: a 30-day telemetry sprint that ends in an executive debrief and a plan for the gaps.
- The report views above, maintained rather than rebuilt by you.
- Judgement signals generated by our prompt classifier.
- The model-fit gap priced in dollars, from your team's real sessions.
- Insights tracked over time.
Three commands, then forget about it.
init detects the agents you already have, configures the ones you pick, and asks where to send events. Leave that blank and Pheebs stays local.
npm install -g pheebs # Install pheebs init # Interactive setup, all three agents pheebs doctor # Check the wiring
We run it on ourselves
Pheebs is how we back our own claim.
Every Eversynced engineer is instrumented with Pheebs. It powers the measurement layer of our AI delivery framework, and the reporting built on top of it ships with the AI Enablement Assessment we run for client teams.
