Token Budgeting for LLM Apps

The two pressures

Every request has a context limit (how much fits) and a cost (you pay per token, in and out). Long chats and big RAG contexts push against both.

Tactics

• Count tokens before sending (tiktoken) and trim if needed.
• Trim history — keep the system prompt plus the last N turns.
• Summarize older turns into a compact note.
• Cap retrieval — top-k chunks, not everything you found.
• Right-size the model — small models for simple steps.

Mindset

Treat the context window as a scarce budget you allocate deliberately. The best apps send the least context that still produces a great answer.