Skip to content

MT-Bench single-answer evaluation for OpenAI-compatible endpoints with EvalScope #3949

Open

Description

@Yunnglin

Hi FastChat maintainers, for users who already serve a chat model through an OpenAI-compatible endpoint, EvalScope can run the two-turn MT-Bench conversation and single-answer judging in one evaluation run, with saved generations and scores.

pip install 'evalscope==1.14.1'
evalscope eval --model YOUR_MODEL --api-url MODEL_API_URL --api-key MODEL_API_KEY \
  --datasets mt_bench \
  --judge '{"strategy":"llm","models":[{"model_id":"YOUR_JUDGE_MODEL","api_url":"JUDGE_API_URL","api_key":"JUDGE_API_KEY"}]}' \
  --limit 3

This covers single-answer judging, not FastChat's pairwise modes. Use the intended GPT-4 judge setup when comparing with historical GPT-4-judged scores; a different judge should be reported separately. MT-Bench usage guide. Would a short link in the MT-Bench usage docs be useful?

Activity

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    No labels
    No labels

    Type

    No type

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions