The True Cost of AI Compute

That single number decides more infrastructure budgets than any benchmark and it quietly misleads most of the people who lean on it. An hourly rate tells you what a machine costs to hold and it says nothing about how much useful work that machine finishes while you are holding it, which is the only thing your users and your invoice actually care about.

Comparing accelerators by GPU-hour alone works about as well as comparing delivery vans by their daily rental rate. Renting the cheaper van feels like the disciplined choice right up until you count the extra trips it takes, the extra fuel it burns and the extra hours it spends finishing the same route. The rental rate was never the number that mattered, because the job was always to move the cargo, not to hold the van.

For an AI workload the cargo is tokens and the honest unit of cost is what it takes to move a million of them. A GPU that rents for more per hour can still generate tokens more cheaply if it produces enough additional throughput during that hour, so a machine at $0.22/1M per output token can undercut one running at $0.28/1M even while its hourly price looks alarming by comparison.

None of this means the newest and fastest accelerator always wins, because a faster chip loses the moment its extra speed goes unused. Set the mature workhorse against the current flagship in the explorer below, then move the demand slider and watch the cheaper option flip.

The unit that matters is cost per million tokens, reported separately for the tokens you feed in and the tokens the model writes back, because generating output is usually the heavier and more expensive step and a single blended figure hides that difference.

Around that central number sit the measures that turn a benchmark into a budget. Tokens per watt tells you how much work you get from each unit of power, which becomes the deciding metric the moment a facility runs out of megawatts before it runs out of money. Tokens per rack tells you how much a fixed footprint can produce. Time to first token and the delay between tokens tell you whether a system that looks fast on a throughput chart will still feel fast to one person waiting for an answer.

This is the trade every provider is quietly making on your behalf. When many requests are batched together the chip stays busy and the cost of each token falls, while every individual waits a little longer for a turn. When the provider protects the speed of a single user the chip spends part of its time idle and the cost of each token climbs. The same hardware can look several times cheaper in a benchmark that batches hundreds of requests than it does inside a live product answering one person at a time and both of those numbers can be completely honest.

The amount of high-bandwidth memory on an accelerator decides how many of them a model needs and the number of GPUs is usually the largest line on the bill.

A model in the 70-billion-parameter range at FP8 precision needs roughly 112GB once its weights and its working memory are counted. That fits inside a single accelerator carrying 192GB and it does not fit inside one carrying 80 GB, so the smaller-memory chip is forced onto two devices to hold the same model. A GPU that looked cheaper on the hourly line can therefore double your cost the instant memory, not speed, becomes the binding constraint and this is exactly why a higher-priced accelerator with more memory sometimes produces the lower monthly bill.

The pressure grows with the length of the conversation. While a model generates a reply it keeps a running memory of everything it has already read and written and that memory grows with the length of the context and with the number of people being served at once. Double the context and it roughly doubles and serve twice as many users and it roughly doubles again, which is why a long-document workload can run out of memory long before it runs out of computing power. For that kind of work, memory capacity is the specification that decides the whole design.

The metric that decides the winner is set by the job and a chip that dominates one workload can be the wrong choice for the next one down the list.

For high-volume inference, where a service answers millions of requests a day, the number that governs the budget is cost per output token at a stated latency, closely followed by tokens per watt once power becomes the ceiling. For a low-latency chat or coding assistant the priority shifts to time to first token and the smoothness of the stream and the highest-throughput accelerator can actually be the wrong buy here, because it sits underused and therefore expensive per useful request.

Long-context and document work is governed by memory rather than speed, so the deciding question is how many concurrent long conversations a device can hold before it needs a second one. Training changes the frame again, because the honest measure there is the total time and cost required to reach a target model quality and a single-GPU specification tells you almost nothing about how a cluster of them will behave at scale.

On-demand rental buys flexibility at the highest hourly rate, reserved contracts trade that flexibility for a lower effective cost and interruptible spot capacity is cheaper still for work that can tolerate being paused.

Ownership is its own calculation, because a purchased server carries financing, electricity, cooling, networking, maintenance and depreciation long after the sticker price is paid and the only honest way to compare it against rental is to divide the full cost of ownership by the hours the machine is actually used rather than by the hours it theoretically could be. A box that runs at a quarter of its capacity costs four times its apparent hourly rate, which is how an owned asset that looked cheap on paper becomes the most expensive option in practice.

Start asking which system finishes your workload at the lowest practical cost while meeting your latency, quality, memory and infrastructure constraints, because that is the only version of the question with a useful answer.

The market usually describes compute in the units that sellers find convenient, which are GPU-hours, server prices, accelerator counts and peak performance figures. Buyers need a different set of units entirely, measured in cost per completed task, cost per useful token, tokens per watt, tokens per rack and total cost under the utilization they will realistically achieve.

None of this can be settled by reading a specification and all of it can be settled by running your own numbers against a consistent method. That is what the companion calculator is built to do, taking your prices, your model, your demand and your latency target and returning the option that fits.