Skip to content

reduce system prompt overhead - #654

Merged
Lazarus-931 merged 4 commits into
Blaizzy:mainfrom
Lazarus-931:optimize/system-prompt-reduction
Oct 7, 2026
Merged

Lazarus-931 merged 4 commits into
Blaizzy:mainfrom
Lazarus-931:optimize/system-prompt-reduction

Conversation

@Lazarus-931

@Lazarus-931 Lazarus-931 commented Oct 2, 2026 •

Copy link
Copy Markdown
Contributor

this shortens the built-in system prompt so nativ sends fewer prompt tokens during prefill while keeping the same profile, style, tool, browser, terminal, and safety instructions.

i tested the original and shorter prompts on five model sizes: qwen3-4b, llama-3.1-8b, glm-4.7-flash, qwen3.5-122b-a10b, and deepseek-v4-flash optiq. the shorter prompt matched the original semantic score on every model in this test suite, used around 46% fewer prompt tokens, and reduced median uncached prefill by 42.5% in this benchmark.

i also compared the original pr 635 chat_work prompt with the rewritten routing-focused version using exact tool-name, action, argument, generated-content, and plain-text assertions. the original and rewritten scores were:

model original rewritten
llama-3.2-1b 0/7 0/7
qwen2.5-1.5b 1/7 2/7
llama-3.2-3b 4/7 4/7
qwen3-4b 6/7 6/7
mistral-7b-v0.3 2/7 2/7
llama-3.1-8b 4/7 5/7
gemma-3-12b 1/7 5/7
gemma-3-27b 1/7 4/7
glm-4.7-flash 7/7 7/7
llama-3.3-70b 5/7 5/7
qwen3.5-122b-a10b 6/7 6/7
deepseek-v4-flash optiq 1/7 1/7

deepseek loaded and passed the plain-text case, while its six tool cases hit the same engine keyerror in both arms.

across that matrix, the rewritten chat_work prompt scored 47/84 versus 38/84 for the original, used 13.7% fewer prompt tokens on average, and reduced estimated prefill by 15.1%. it matched or improved the original score on every model in the matrix.

llama 8b scored 26/27 with both prompts because it labeled one meta-question incorrectly, but its response still followed the requested terse, no-emoji, unformatted style. these numbers are from these model benchmarks and are not meant as a universal speed claim.

the nativ app builds and 78 focused NativSettingsTests and ChatWorkTests pass.

@Lazarus-931
Lazarus-931 marked this pull request as ready for review October 3, 2026 00:15
@lucasnewman

Copy link
Copy Markdown
Collaborator

@Lazarus-931 Can you file an issue for the DeepSeek model failure? Otherwise this looks fine to me.

@Lazarus-931

Copy link
Copy Markdown
Contributor Author

@Lazarus-931 Can you file an issue for the DeepSeek model failure? Otherwise this looks fine to me.

I'll follow up with that!

@Lazarus-931
Lazarus-931 merged commit ad1d0e2 into Blaizzy:main Oct 7, 2026
3 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants