You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
{{ message }}
Repository navigation
MT-Bench single-answer evaluation for OpenAI-compatible endpoints with EvalScope #3949
Hi FastChat maintainers, for users who already serve a chat model through an OpenAI-compatible endpoint, EvalScope can run the two-turn MT-Bench conversation and single-answer judging in one evaluation run, with saved generations and scores.
This covers single-answer judging, not FastChat's pairwise modes. Use the intended GPT-4 judge setup when comparing with historical GPT-4-judged scores; a different judge should be reported separately. MT-Bench usage guide. Would a short link in the MT-Bench usage docs be useful?
Activity