Skip to content

Route output facts should outrank uncatalogued compatibility guesses #5460

Closed

Description

@h3c-hexin

Problem

effective_max_output_tokens_for_route always intersects an uncatalogued
model's conservative compatibility guess (8192) before a concrete
RouteLimits.output_tokens fact. As a result, an unknown OpenAI-compatible
wire alias with a resolved route limit of 24576 is still sent as 8192.

The fail-closed rule is sound when the route publishes nothing: absence is not
permission. But once the resolved route carries a positive output maximum,
that exact route fact is stronger evidence than a catalogue-absence guess.
Today it can narrow the guess but can never replace it.

This affects hosts and route resolvers that already provide an explicit output
limit for a custom or gateway model that is not in Codewhale's static model
catalogue.

Proposed behavior

  • Unknown model, no route output fact: keep the conservative 8192 ceiling.
  • Unknown model, route output limit 24576: use min(requested_cap, 24576).
  • Unknown model, route output limit 64000: use min(requested_cap, 64000).
  • Known model: its documented compatibility ceiling remains authoritative;
    a route limit may only narrow it.
  • A route limit must never raise the configured/requested output cap.

This keeps "absence is not permission" intact while treating a concrete route
fact as permission for that exact route.

Acceptance

  • Regression tests cover unknown Openai and Custom routes with and
    without explicit output limits.
  • Documented model ceilings remain authoritative.
  • Formatting, focused route_budget tests, and clippy pass.

Metadata

Metadata

Assignees

No one assigned

    Labels

    No labels
    No labels

    Type

    No type

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions