The execution gap.

What we built, and what we use

The supply side of this story is clear. Within one NVIDIA generation, dense FP16 throughput climbed from 125 TFLOPS on a V100 to 312 on an A100, 989 on an H100, 2,250 on a B200, and 2,500 on a GB300: roughly a twentyfold gain in peak compute per chip.¹²³⁴

Delivered performance tells the opposite story. Even a well-tuned training run reaches only 35 to 45 percent of theoretical peak, and that is the ceiling, not the floor. Meta's Llama 3 run, executed across sixteen thousand H100s with careful optimization, landed at 38 to 43 percent.⁵⁶

Mixed production workloads fall far below even that. Direct measurement across tens of thousands of clusters in 2025 and 2026 puts average GPU use near 5 percent, which means organizations were paying for roughly twenty times the capacity they actually used.⁷⁸ Inference does better but is still low, usually 20 to 40 percent before optimization, and the pattern is not limited to accelerators: ordinary server use has sat in the low double digits for years, with a meaningful share of installed machines doing no useful work at all.^91011171819

The conclusion holds at every layer of the stack. Peak rose by a factor of twenty; the share we use barely moved.

The gap is the point

A low rate of use is easy to forgive when peak is small, because the waste it represents is small too. That stops being true once peak explodes. Hold the share you use flat and multiply it against a peak that has grown twentyfold, and the raw amount of wasted compute per chip grows almost in step with the peak itself.

This is what we call the execution gap. It is not a percentage that looks reassuringly stable on a chart; it is a growing pool of capacity that was bought, powered, and cooled, and never turned into useful work. And it is widest exactly where the work is most valuable.

Why the gap exists

The gap exists because the workload changed shape while the execution model stayed still.

Infrastructure was designed for predictable work: web applications, APIs, databases, services, all built on the assumption that a job could be placed on a machine and left there. Modern workloads break that assumption at every turn. They are larger and more dynamic, they move between CPU-heavy and GPU-heavy stages, and they span clouds, regions, clusters, networks, and edge environments, placing demands on the system underneath them that it was never designed to handle.¹²

When work does not fit the machine, teams compensate. They layer on orchestration, placement logic, and manual coordination; they reserve peak headroom and let it sit idle to protect a latency target; they hold whole accelerators against bursts that rarely arrive at the assumed scale.¹³²⁰ Each of these is a reasonable local decision, and together they add up to the industry's standing workaround for a gap it has not closed.

Low use, high cost, and rising complexity are not three problems. They are one mismatch seen from three angles.

Orchestration is compensation, not a fix

Orchestration manages the symptom, not the cause. It decides where work is placed and when it moves, but it changes neither the unit of work itself nor the path that work takes to reach the resource best able to run it.

The evidence is hard to argue with: use has not improved as orchestration has matured. In the very environments where orchestration has become the default foundation for AI, measured use has gone backwards.⁷¹⁴ More scheduling on top of the same execution model does not close the gap; it manages it.

What is worth noting is where the rest of the field has landed. Vendors and analysts arriving at the same diagnosis now describe the fix the same way: break execution apart and separate the stages so each one runs on the resource it actually needs.^9122122 That is a description of the problem TAHO was built to solve.

What TAHO changes

TAHO changes the unit of execution. It breaks a workload into smaller units, sends each unit to the resource best suited to run it, and runs it there.

This is not a replacement for orchestration but a layer beneath it and above the hardware. We do not compete with the scheduler; we change what the scheduler is scheduling.

The thesis is simple to say and hard to build: capacity is no longer the constraint, fit is. Teams that fit work to machines better will get more out of the hardware they already own, spend less to deliver the same result, and start to measure how well they execute rather than how much they spend. That efficiency is the value the execution gap has been hiding, and closing it is the opportunity.

A note on the numbers

Every peak figure on this page is dense FP16, taken from primary NVIDIA datasheets and held to a consistent precision so the generations compare cleanly. Sparse and lower-precision figures run higher; using them would make the gap look larger, not smaller, so we chose the conservative number on purpose. The Vera Rubin figure is a projection, not a published spec, and is labeled as such wherever it appears.¹⁵¹⁶

Use figures are reported with their source and what they measure, because fleet utilization, model FLOPs utilization, and server capacity utilization are three different things and should not be mixed. The argument does not rest on any single number. It rests on the shape of all of them together.