Optimizing Local Models for Agentic Workflows: Balancing Context, Performance, and Cost

As enterprises increasingly evaluate where local AI fits alongside cloud-based frontier models, understanding the tradeoffs between cost, performance, and capability is becoming an important architectural consideration. While cloud models remain the right choice for many workloads, local inference continues to mature as a practical option for agentic workflows, particularly where data privacy, latency, predictable costs, or specialized workloads are priorities. Knowing how to properly optimize local models can make the difference between a proof of concept and a genuinely useful AI capability.

Optimizing Local Models for Agentic Workflows

As the cost of frontier AI usage continues to grow, interest in using local models for agentic workflows is rising. I was recently having a conversation with a coworker who was having trouble getting useful output from their local inference server.

Like many people out there, I don’t have the funds to buy a Spark or HP Z Nano. Some of us have decent GPUs left over from our video game days or, cough, build inference servers on the cheap. In this day and age, 24GB of dedicated VRAM is just enough to get a capable model running with, and this is the crucial part, a reasonably sized context window.

Here are some of the gotchas that might help if you’re struggling to get useful agentic work done with a local model.

Choose the Right Local Model

This is more nuanced than a blanket “use this model.”

From my experience, Qwen3.6 is a great choice. I’ve also had decent luck with Gemma 4 27B, but I tend to stick with Qwen3.6. The combination of good inference speed and a strong knowledge base makes it an excellent general-purpose model for agentic workflows.

This recommendation is valid as of July 2026. After that, all bets are off.

Model Quantization

This matters far more than people realize.

If you ask a frontier model which quantization level to use, it’ll usually tell you that higher precision results in greater accuracy. That’s absolutely true, but it often misses the nuance that higher precision comes at the expense of your available context window, which we’ll discuss next.

My preferred quantization for Qwen3.6 is Q3. The model remains more than capable for the agentic work I need while freeing up significantly more VRAM for context and KV cache.

One caveat: some models are much more sensitive to quantization than others. Qwen3.6 doesn’t seem to mind dropping to Q3, while I’ve heard from others experimenting with Gemma 4 that it degrades much more noticeably when heavily quantized.

Context Window

This is where the rubber meets the road.

If you’re like me, when you first started experimenting with local models, you probably grabbed the biggest model you could find because bigger must be better. Then you loaded it into your system and wondered why performance was terrible.

Here’s the secret: your context window is just as important as the model itself, and it’s separate from the model size.

If a model takes 11GB of VRAM, that’s only the model weights. It doesn’t include the memory required for your context window.

If you only have 12GB of VRAM and load an 11GB model, you’ve left less than 1GB for context. That creates two problems:

• Your available context window becomes extremely small.
• The remaining context spills into system memory, dramatically slowing inference.

How does this affect real-world performance?

With a 4,096-token context window (which is a common default in Ollama when a model barely fits) you often can’t even load most agentic toolsets into memory. The model loads the tooling, reads a file or two, runs out of context, resets, and suddenly asks what it’s supposed to be doing. If you’re not watching closely, you may not even realize that’s what’s happening.

Context Quantization

Most inference systems, including llama.cpp, Ollama, and LM Studio, default to keeping the context (KV cache) at full precision.

That’s great for accuracy, but it consumes a tremendous amount of VRAM.

Fortunately, all three platforms allow you to quantize the KV cache. While this introduces a small reduction in accuracy, it can dramatically increase your available context window.

My go-to model is:

unsloth/Qwen3.6-27B-MTP-GGUF at Q3_K_M.

It occupies roughly 13.8GB of VRAM.

On my 24GB Nvidia P40, that leaves about 10GB available for KV cache and context.

With a full-precision cache, I end up with roughly an 88K-token context window.

By switching to a Q8 quantized KV cache, I can increase that to approximately 169K tokens, nearly doubling the available context while taking only a modest hit in accuracy.

For agentic workflows, that’s often a tradeoff well worth making.

Tradeoffs

This all sounds great, but what’s the catch?

Accuracy.

The reality is we’re entering the era of “good enough AI.”

Not every task requires a frontier model. If you’re trying to build practical agentic workflows on a reasonable budget, these are exactly the kinds of engineering tradeoffs you’ll find yourself making.

For my use case, the choice is simple.

I can either run a model that’s technically more accurate but doesn’t have enough context to complete meaningful agentic tasks, or I can run a model that’s slightly less accurate but has enough working memory to actually finish the job.

I’ll take the second option every day of the week, and twice on Sunday.

As local AI capabilities continue to improve, organizations will increasingly need to evaluate where local inference complements cloud-based AI services rather than viewing them as competing approaches. Understanding the tradeoffs between model quality, context windows, hardware constraints, and cost helps teams build AI architectures that are both practical and scalable. At RBA, we help organizations assess these decisions as part of broader AI, infrastructure, and enterprise architecture strategies, ensuring the technology aligns with real business outcomes instead of simply chasing the latest model.