Running a Local LLM Inference Server on a Budget: Lessons From Building With a Used NVIDIA Tesla P40

Running a Local Inference Server on a Budget: Is It Actually Possible?

After blowing through 25 million tokens in just two weeks on a project using Factory.ai’s Droid tool, not including the 0.25x multiplier, I started seriously questioning the economics of cloud-based inference. The cost curves simply did not make sense.

I explored the usual suspects. Copilot subscriptions quickly ran into timeouts and usage limits. Extrapolating my token consumption against the Claude API landed me squarely in the “thousands of dollars per month” range. Renting GPUs in the cloud introduced its own problems, including inconsistent availability, performance variability, and long-term cost uncertainty.

At some point, I was venting about this to a friend, lamenting the lack of a cost-effective way to generate tokens at scale. That conversation sent me down a different path, straight to eBay.

The $200 GPU That Changed Everything

That is where I found it. A used NVIDIA Tesla P40 with 24GB of GDDR5 VRAM for about $200. 

Yes, the P40 is an older datacenter card. It cannot run the largest cutting-edge models and comes with architectural limitations. That said, it is still very capable. With the right setup, it can comfortably run 30B parameter models with usable context windows.

In my case, the P40 runs Qwen3 Coder 30B at roughly 50 tokens per second. That is more than acceptable for chat-based workflows and even viable for agentic use cases when you are mindful of context size.

Hardware Reality Checks and False Starts

Naturally, I assumed I could drop the card into an old workstation or gaming rig that had been collecting dust in a closet. That assumption was wrong.

The older chipset in that machine simply could not handle the amount of VRAM on the P40. Back to the drawing board.

After some back-and-forth with Claude, I landed on a Dell T3600 workstation. It was newer than my old rig, widely available, and inexpensive at around $120. On paper, it looked promising. In practice, Dell’s proprietary power supply became the blocker. It lacked both the power and the correct connectors to properly drive the P40.

I even tried using an Add2PSU to pair the system with a 1600W PSU from a previous build. No luck. The T3600 was extremely finicky about power and refused to POST if the Add2PSU was connected. It booted fine otherwise, which somehow made this more frustrating.

Accepting the Inevitable: A New Build

At that point, I finally gave in and built a new machine. Fortunately, I did not need to start from scratch. I reused the case, SSD, and that overkill power supply.

With help from Claude and Rufus, Amazon’s shopping assistant, I settled on the following setup:

• MSI B550 Pro V1 motherboard
• AMD Ryzen 5500
• 16GB DDR4 RAM

Nothing exotic here, but it was a meaningful upgrade from the previous attempts. It stayed within budget and, most importantly, supported the P40 without drama.

Bringing the Inference Server Online

I installed Ubuntu Server 24.04, added the NVIDIA drivers, and ran nvidia-smi with my fingers crossed.

Success. The P40 showed up immediately.

Next came Ollama and a test run of Qwen3 Coder 30B. My previous CPU-based attempts on a Ryzen laptop struggled to hit 10 tokens per second. Now I was consistently seeing around 50 tokens per second. That is the difference between theoretical and usable.

Is a 30B Model Actually Practical?

Yes, absolutely.

Qwen3 Coder 30B punches well above its weight. At 50 tokens per second, agentic workflows are viable as long as you are disciplined about context. You need to be explicit about which files the model should see and edit, but that tradeoff is reasonable.

Compared to cloud offerings, there are some minor restrictions. The upside is massive.

There are no request caps. No monthly token quotas. No surprise bills. I can run this system as much or as little as I want. For anyone doing sustained development, research, or experimentation, that freedom matters.

Practical Lessons From the P40 Route

If you decide to explore this approach, there are a few important caveats worth calling out.

First, the Tesla P40 is a datacenter card with no built in active cooling. It is designed for servers with strong front to back airflow. If your case does not provide that, you will need supplemental cooling.

Fortunately, several community members have published 3D printable blower shroud designs. Pairing one of these with a radial blower works extremely well and is what I am using today.

Second, and this is critical, the P40 uses a CPU EPS power connector. It does not use a standard PCIe power connector. Verify this against official documentation for your specific card. Do not rely on generic advice, including from ChatGPT. I take no responsibility for any damage to your hardware.

Final Thoughts

People joke about “organic, homegrown tokens” on LinkedIn, but there is something deeply satisfying about finally getting a setup like this running after weeks of trial and error.

A repurposed, budget-friendly datacenter GPU paired with modern open source tooling can absolutely support serious local inference workloads. For developers and teams looking to reduce cloud dependency, avoid vendor lock-in, or simply experiment freely, this approach is worth considering.

If you decide to go down this path, let us know how it goes. At RBA, we are actively exploring where local AI infrastructure fits alongside cloud and enterprise platforms, and we are always interested in what the community is building next.