As enterprises continue evaluating where AI workloads should run, local inference is becoming a more practical option for specific use cases. Organizations are looking for ways to maximize existing hardware, reduce inference costs, and keep sensitive workloads on-premises. While most production deployments still rely on dedicated infrastructure, projects like Exo demonstrate that distributed inference is becoming increasingly accessible, and may offer an interesting path for organizations with underutilized hardware.

As I’ve mentioned in my previous blogs, I built a budget inference machine using a 10-year-old NVIDIA GPU. This machine does its job, but the P40 is a 250W card and doubles as a space heater. In another blog, I talked about using multiple machines as a distributed Ollama cluster. The limitation there was that each machine still had to fit the entire model and context before returning a response. That worked well for specific use cases (like repetitive tasks with consistent inputs and prompts) but it definitely had its limits.

I recently stumbled across another fun solution for generating organic, home-grown tokens.

Exo from Exo Labs

Exo Labs created Exo, which you can find at https://exolabs.net. The tool is primarily designed to wire multiple Macs together and shard a model across all of them. Since I had a couple of MacBooks sitting around, I decided to give it a shot.

They give you the option to run over TCP/IP, but I wouldn’t recommend it. A much better approach is using a Thunderbolt Bridge with a Thunderbolt 4 or Thunderbolt 5 cable. If the machines are dedicated to this purpose, you can even enable RDMA over Thunderbolt 5 for some pretty ridiculous (relatively speaking) transfer speeds. That does require RDMA support and currently only works on M4 and newer Macs.

Would I go buy a stack of Mac Pros specifically to run this? Probably not.

But I do think this fills an interesting niche for organizations that already have older (but perfectly usable) Apple hardware sitting around. I think back to my days working in device management. I had a stack… yes, an actual stack… of Mac minis on my desk for testing, and I certainly wasn’t the only one.

Between a 16GB M1 Pro MacBook and a 16GB M3 MacBook, I was able to load a 9B model with a sizeable context window. For reference, either of those machines by themselves turns into a PowerPoint presentation trying to run even a 7B model.

Mixing Apple Silicon and NVIDIA

One downside I noticed is that the Exo project doesn’t officially support NVIDIA GPU servers as part of the cluster. Thanks to Copilot, I was able to patch the project and get it working.

It shouldn’t surprise anyone that generation speeds were…really, really slow.

The upside is that with 24GB + 16GB + 16GB of combined memory, I was able to run a 70B Q4 model, something that’s impossible on any of my individual systems.

My theory is that most of the performance bottleneck comes from using TCP/IP on a standard home network. It simply can’t provide the bandwidth needed to move model data efficiently. On top of that, all my network interfaces are limited to 1 GbE, while RDMA over Thunderbolt can deliver somewhere in the neighborhood of 40–50 GB/s. I’m not going to compete with that.

Parallel Compute Matters

Exo also supports multiple sharding strategies.

One option allows multiple machines to actively participate in inference simultaneously, increasing overall tokens per second. Other approaches (including Exo’s default configuration and the Llama.cpp RPC implementation) don’t distribute compute in parallel the same way, so throughput tends to be lower.

It’s an interesting reminder that distributed inference isn’t just about fitting a larger model into memory. How the workload is divided across machines has a significant impact on real-world performance.

Final Thoughts

With the great token crisis of 2026 (kidding), I think we’ll continue to see growing interest in running models locally. As smaller models become increasingly capable, the demand for larger context windows continues to grow alongside them.

That makes tools like Exo and Llama.cpp’s RPC implementation increasingly interesting. They’re giving enthusiasts, and potentially enterprise engineering teams, a way to squeeze more capability out of hardware they already own instead of constantly buying bigger, more expensive systems.

It’s an exciting time to be part of the local inference scene.

 

The RBA Perspective

Exploring technologies like distributed local inference helps organizations understand where AI infrastructure is heading and where existing investments can be extended. At RBA, we help enterprises evaluate emerging AI architectures, balance performance with cost, and identify where local inference, cloud AI, or hybrid approaches make the most strategic sense as AI adoption continues to evolve.