Package akka.javasdk.testkit.eval

Interface Judge

All Known Implementing Classes:
ModelBasedJudge
public interface Judge
Decides how well a reply meets a criterion given as a sentence. shouldSatisfy(java.lang.String) and shouldScoreAtLeast(java.lang.String, double) turn the score into an Evaluator. 

 var judge = Judge.modelBased(testKit);

 EvalCase.of(
     "tier",
     "Which tier is cust_1 on?",
     Evaluators.shouldCallTool("getCustomer"),
     judge.shouldSatisfy("the reply states the customer's tier and invents nothing"));

modelBased(akka.javasdk.testkit.TestKit) asks a model through JudgeAgent, which uses the model provider of the test configuration. A test can also return a Judge.Verdict directly, or mock that agent's model.

A judged score can differ between runs of the same reply. Use a judge for criteria that have no exact answer to compare against, and gate a batch on the rate instead of asserting per case.

  • Method Details

    • decide

      Judge.Verdict decide(String criterion, Interaction interaction)
      Parameters:
      criterion - what a good reply does, as one sentence
      interaction - the input, the reply and the evidence the case produced

    • modelBased

      static ModelBasedJudge modelBased(TestKit testKit)
      A judge that asks a model through JudgeAgent, using the TestKit component client.

    • shouldScoreAtLeast

      default Evaluator shouldScoreAtLeast(String criterion, double threshold)
      The criterion must score at least threshold. Inconclusive when there is no reply. Fails when the judge throws or gives no verdict, or when the score is not between 0 and 1.

    • shouldSatisfy

      default Evaluator shouldSatisfy(String criterion)
      Same as shouldScoreAtLeast(java.lang.String, double) with a threshold of 0.5.