Why isn't local AI always free?

An API model bills by tokens, so every reply it reads and writes adds to the meter. Running the model yourself replaces that per-token charge with the cost of the machine doing the work. For high volume or private data, that shift can be the whole reason to go local. It is a real benefit, but not the same thing as no cost.

• Hardware, up front. A capable GPU or a machine with enough memory is a real purchase. If you already own it, the cost is sunk rather than absent. It still counts when you plan.
• Power. A machine running inference draws electricity, and more so under load. Left on all day, that shows up on the bill.
• Setup and maintenance time. You install and update the model runner, manage models and storage, and fix things when they break. That time is a cost even when no money changes hands.
• Heat and noise. Sustained load means fans, warmth, and noise in whatever room the machine sits in.
• Reliability. A home machine is not built to stay up like a server. Sleep, restarts, and dropped connections all interrupt it.
• Speed. Depending on the hardware and the model, responses can be slower than an API, which is a cost in patience if not in dollars.

A model can fit in memory and still be painful to use. Fit only means the machine may load and run it. How usable it feels depends on the hardware, memory bandwidth, quantization, the runtime, how long your context grows, and what else the machine is doing. If you are unsure a model will fit at all, start with can my computer run a local LLM.

A local model is only reachable while the machine stays awake, connected, and reachable. If it sleeps, disconnects, or restarts, the model stops answering until the machine is back. For a chat you open when you need it, that is fine. For an agent that should respond at any hour, it is a real limitation unless you design around it, and that design work is another cost.

Local tends to make sense for high, steady volume, or when keeping data on your own hardware matters more than convenience. But whether it is actually cheaper than an API depends on the model, your usage, what the hardware cost, your power price, and which API it would replace. Self-hosting is not always cheaper. It can be, when utilization is high enough to justify the hardware and the upkeep. To weigh it against paying per token, compare local LLM vs cloud API, or work through the parts in the cost to run an LLM.

• Cost to run an LLM
• Cost to run an AI agent
• Local LLM vs cloud API
• Run a local LLM
• GPU hosting vs model API
• Browse all stack parts
• Read the FAQ