feat: token optimization - #3
Merged
Merged
Conversation
1. Lazy PREVIEW_GUIDE injection: full design system guide (~4500 tokens) only sent when user message contains visual keywords (chart, diagram, etc). Regular chat gets a 50-token mini hint instead. 2. Anthropic prompt caching: system prompt and tools marked with cache_control ephemeral, plus anthropic-beta header. Subsequent requests in same session pay ~10% of normal input cost. 3. Tool result truncation: large tool outputs (file reads, command results) capped at 3000 chars with head+tail preservation to prevent token bloat across ReAct loop iterations.
… savings - Add 'chat' mode as default agent type — routes to send_message (no tools, no agent system prompt) instead of run_agent - Chat mode: ~200 tokens vs orchestrator/planner: ~19,000 tokens - Apply lazy visual guide injection to chat_service.rs (both OpenAI and Anthropic paths) — full preview instructions only sent when user message contains visual keywords - Add prompt caching to Anthropic path in chat_service.rs - Default selectedAgentType changed from 'orchestrator' to 'chat' - Users can still switch to Orchestrator/Planner when needed Token impact for casual chat: 19,000 -> ~200 tokens (99% reduction)
… prompt caching Root cause: enowX Coder was sending OpenAI-compatible format to enowxlabs proxy, while Claude Desktop sends Anthropic native format with cache_control. Same proxy, same API key, but caching only works with Anthropic native format. Changes in runner.rs (agent mode): - Route enowxlabs to send_anthropic_with_tools (not OpenAI-compatible) - Add base_url + provider_type params for dynamic endpoint resolution - Mark last user message with cache_control ephemeral - Auth: Bearer for gateways, x-api-key for Anthropic direct Changes in chat_service.rs (chat mode): - Route enowxlabs to send_anthropic (not send_openai_compatible) - Build messages as Anthropic content-block format with cache_control - Dynamic endpoint: anthropic direct vs enowxlabs gateway - System prompt as cached content block array Expected result: enowxlabs proxy now receives identical format to Claude Desktop, enabling prompt caching (input_tokens: 0 on cache hit)
… gateways - Add api_format column to providers table (openai/anthropic) - Route chat/agent/title-gen through Anthropic format when api_format=anthropic - Add non-streaming fallback when gateway returns empty SSE stream - Add SSE event-name fallback for gateways that strip JSON type field - Trim chat history with sliding window to reduce token usage - Strip html:preview blocks from assistant context - Add API Format toggle in ProvidersTab UI (new + edit) - Unify model listing to use Provider struct directly
… 99.87% reduction
|
Sanitizing development flow - moving to enowX-Coder-verversion fork first |
4 tasks
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
3 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
api_formatfield, chat history sliding window (20 pairs, 32K char budget), lazy prompt injection,html:previewblock strippingapi_formatselector (OpenAI/Anthropic) for custom gateways, enable/disable toggles, SSE event-line fallback for third-party proxies, non-streaming auto-retry fallbackToken Usage — Before vs After
Screenshots
Screenshots included in
screenshots/folder: enowX Flux, Light Mode, Excalidraw Canvas, Provider Settings, and Before/After token optimization comparison.