jev-harness

A small TypeScript library that turns TypeSafe Jev answers into actions you can ship.

Jev returns structured decisions (choice / score / noul) with confidence scores. This harness adds the missing pieces around that API: a policy, a confidence gate, shadow mode, reusable recipes, and an offline eval CLI.

It is not affiliated with TypeSafe. You need a TypeSafe API key to call Jev.

Calling Jev once is easy. Running it in production usually needs more:

• Policy — map answers to an action (notify, suppress, place, skip, …)
• Confidence gate — if confidence is low, review / escalate / suppress instead of guessing
• Shadow mode — log what you would do without changing live behavior
• Recipes — copy-paste patterns for common jobs (alerts, routing, row filters, agent handoffs, …)
• Evals — replay fixtures and assert on the action, not on free text

Same job: filter 24 people rows with a natural-language predicate.

npm install jev-harness
export TYPESAFE_API_KEY=tsk_...

Or clone and npm install && npm run build. Requires Node 20+.

import { DecisionHarness, choice, noul, score } from "jev-harness";

const harness = new DecisionHarness();

const result = await harness.run({
  id: "alert-42",
  state: { title: "Disk 92% on db-3", service: "payments" },
  questions: {
    disposition: choice("What should we do?", {
      notify: "Page someone now",
      queue: "Put in the review queue",
      suppress: "Ignore as noise",
    }),
    severity: score("How bad is this?", [
      "Low",
      "Medium",
      "High",
      "Critical",
    ]),
    needs_human: noul("Does a human need to look at this?"),
  },
  policy: {
    minConfidence: 0.55,
    onLowConfidence: "review",
    decide: ({ answers }) => {
      if (answers.disposition.choice === "suppress") return "suppress";
      if (answers.needs_human.noul >= 0.6 || answers.severity.score >= 2) {
        return "notify";
      }
      return "queue";
    },
  },
});

console.log(result.action, result.confidence, result.reason);

Shadow mode (no live side effects):

await harness.run({
  /* same as above */
  mode: "shadow",
});
// result.action === "shadow_noop"
// result.intendedAction === what the policy wanted

The harness talks to anything that implements DecisionBackend (systemOne({ state, questions, model }) → { model, answers, usage }):

• Self-hosted /v1/systemone server (e.g. an open Jev-style model): set TYPESAFE_BASE_URL=http://localhost:8000. No code change.
• In-process model or rules: pass your own backend.

import { DecisionHarness, type DecisionBackend } from "jev-harness";

const local: DecisionBackend = {
  async systemOne({ state, questions }) {
    // run your BERT / SLM / rules here; return one typed answer per question,
    // each with a calibrated `confidence`
    return { model: "my-local", answers, usage: { input_tokens: 0, output_tokens: 0 } };
  },
};
const harness = new DecisionHarness({ client: local });

Compare a backend against the recipe fixtures (pass/fail per case):

npm run eval -- --live --backend=./my-backend.ts --all   # default export = DecisionBackend

The confidence gate is only as good as the backend's calibration, so check the low-confidence cases too.

Common decision shapes live under recipes/. A few:

Full list: research/crazy-fast-decisions.md.

npm run eval -- eval/fixtures/alert-gate.jsonl

Fixtures assert on the action (and optional confidence band), so you can test policy changes offline.

SHADOW=1 TYPESAFE_API_KEY=tsk_... npx tsx examples/alert-gate.ts
npx tsx examples/model-router.ts

UI mock cards for the common jobs (not the measured terminal proof above):

See demos/marketing/ and paste prompts in demos/marketing/PROMPTS.md.

This is not a Claude Code / Codex /compact replacement. Line-by-line keep/drop on agent history (probability filter, incomplete tool results, mid-thread deletes) is a different problem: it can bust prompt cache, drop encrypted reasoning traces, and push the model into retry loops. Prefer the lab defaults for transcript compaction. Use this harness when your app already has structured state and needs a policy + confidence gate on a decision.

• TypeSafe Jev docs — the model this wraps

MIT © Antonio Coppe