Skip to content

Fix compaction using actual context usage - #684

Merged
Blaizzy merged 3 commits into
mainfrom
pc/fix-chat-compaction-output-budget
Oct 13, 2026
Merged

Blaizzy merged 3 commits into
mainfrom
pc/fix-chat-compaction-output-budget

Conversation

@Blaizzy

@Blaizzy Blaizzy commented Oct 7, 2026 •

Copy link
Copy Markdown
Owner

A short message can fail with “Lower Max output to leave room for conversation compaction” when a saved Max output setting is equal to or larger than the effective context window. For example, context 4,096 and Max output 4,096 should allow “Hello” to generate a short answer.

Trigger compaction from actual input usage at the selected percentage (75% by default), without subtracting Max output or changing its saved value. Document fitting also stops reserving unused output tokens when Auto compact is enabled. The server counts and, when necessary, compacts the input before limiting generation to the smaller of Max output and the remaining context capacity.

Pin the bundled mlx-vlm server to the companion fix in Blaizzy/mlx-vlm#2458 (0a46548aab8fd034a68f54f5cb49cc6ce113728a). Preserve MLX_VLM_SOURCE_PATH overrides when resolving the pinned requirements. External servers need the companion fix as well. Compaction still happens between generation requests, including tool rounds; reaching the context limit ends a response with a length result.

Validation:

  • Xcode app build and 36 focused compaction/document tests passed. Used a filtered test target because the repository's full test target has an existing SystemMonitorIdentityTests import/configuration failure.
  • Bundled-server source override regression test passed; pip resolved/downloaded the exact VCS pin with the bundle's --only-binary=:all: option.
  • Companion server suite: 541 passed, 6 opt-in integration tests skipped.
  • Separate live MiniCPM5-2B HTTP checks with context and Max output both 4,096: streaming and non-streaming short prompts succeeded (19 input + 3 output); a 4,064-token prompt generated exactly 32 tokens and returned response.incomplete.

Addresses #670.

@Blaizzy Blaizzy changed the title Fix chat compaction with oversized output limits Fix compaction using actual context usage Oct 7, 2026
@Blaizzy
Blaizzy merged commit 2623f6f into main Oct 13, 2026
5 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant