feat(evals): compare student agreement with repeated teacher decisions

[xinpingzh]

What

Add an offline teacher/student agreement workflow to laya.evals:

• compare_agreement compares saved reference-teacher, repeat-teacher and student decisions. evaluate_agreement runs a local student against the same records.
• laya-evals agreement replays the saved runs without loading weights.
• Reports include paired decisions, provenance, coverage, slices and a case-clustered bootstrap interval for the student-minus-repeat agreement gap.

Why

Following the measurement discussion in #919, this makes teacher repeatability and student agreement measurable on the same inputs. Teacher observations stay separate from the labelled harness's accuracy and calibration metrics.

The runs must have matching case IDs, states, ordered questions and teacher settings. Missing or malformed observations fail validation. Student abstentions remain in the denominator and lower coverage. The bootstrap keeps all questions from one case together and weights the result by decision count, including when cases have different numbers of questions.

This PR covers the measurement workflow only. It does not add the store/signature routing discussed in that issue, or treat repeat agreement as a performance ceiling or replacement gate.

How it was verified

The new tests cover hand-calculated results, unequal-size case clusters, abstentions, input drift, runner failures, pre-inference validation and CLI replay/error handling. The evals suites passed 202 tests. The existing router, criteria, hooks, hook API and eval API checks also passed.

python -m pytest tests/test_evals.py tests/test_evals_agreement.py -q
python tests/test_evals_api.py
python tests/test_router.py
python tests/test_criteria.py
python tests/test_hooks.py
python tests/test_hooks_api.py
ruff check laya/ --select=E9,F63,F7,F82,F401,F811 --line-length=120
python -m compileall -q laya/ tests/
zensical build --strict --clean

The Windows pytest run used a writable --basetemp; the strict docs build reported no issues. An independent review checked the API paths and reproduced the bootstrap against a separate implementation on 150 panels.

For a real checkpoint check, examples/evals/teacher_agreement.py uses the first 32 rows of the pinned localgate/mmlu-pro-closed test set, teacher repetitions 0/1, and convaiinnovations/laya@55cf4c4ebb4ebe31b2550e8bdf3bd21b99753851 on CPU. The sample and repetition selection are fixed in the script.

python examples/evals/teacher_agreement.py --output-dir /tmp/laya-agreement
laya-evals agreement /tmp/laya-agreement/reference.json \
    /tmp/laya-agreement/repeat.json /tmp/laya-agreement/student.json \
    --json /tmp/laya-agreement/replayed.json

Measured teacher repeat agreement was 26/32, student agreement 1/32, coverage 32/32, and the paired gap -0.78125 (95% interval [-0.90625, -0.625]). The saved runs reproduced the full live report through the CLI. These are pipeline-validation results on a small MCQ panel, not a quality improvement claim. The teacher's exact model revision was not published; it remains null and is flagged in the report. The example also documents the model-specific prompt adaptation.

Checklist

• Focused on one change
• Based on the latest main (1adc59f7)
• Tests pass locally
• Docs and a runnable example cover the new API

Refs #919

[NandhaKishorM]

Merged.

This is the measurement I said in the #919 thread I wanted to have a home in laya.evals rather than living in an issue comment: comparing a student's agreement with a teacher against the teacher's agreement with itself. The ceiling matters because it is computable without a single human label, and without it an agreement number has nothing to be judged against.

I read the network path carefully before taking it, since "teacher" plus urlopen is the shape I check hardest here. It holds up:

• The shipped package makes no network or subprocess calls. laya/_agreement.py, laya/evals.py and laya/evals_cli.py add zero lines matching urlopen, urllib, requests, httpx, socket or subprocess.
• The one urlopen is in examples/evals/teacher_agreement.py and fetches a pinned public dataset parquet (DATASET_REVISION is a commit sha), not a model API.
• Your own docstring says it plainly: "The pinned English Laya checkpoint runs locally on CPU; no teacher API is called." The teacher observations are published repeats in that dataset, five per question, not something this generates.
• PROMPT_TEMPLATE is recorded as provenance of how the publisher produced those observations, and the example says so. It is metadata, not something laya runs.

That matters because this project is local-only inference, and an evals helper that quietly called a hosted model would cross that line. This one does not.

The honesty details I want to credit: fixing the first 32 rows and teacher repeats 0/1 before student inference, so the comparison cannot be tuned after seeing the result; and recording teacher_revision_note: "not recorded in the publisher's manifest" as an explicit null rather than inventing a revision. A provenance field that says "unknown" is worth more than one that guesses.

Registering the suite in scripts/test_suites.py in the same PR is the thing most people miss, and tests/test_packaging.py would have caught it.

All 86 suites pass on the merged tree.

[xinpingzh]

Thanks for the careful review, especially the check on the network boundary. Glad this fits what you had in mind for #919.