The Cloud Tax on Inference

The $100,000 That Vanished in Three Months

We secured $100,000 in cloud GPU credits. It felt like a windfall: enough runway to build, train, and deploy our medical imaging models without worrying about infrastructure costs.

Three months later, the credits were gone.

Not because we'd been careless, but because we'd been successful. Our inference workloads scaled faster than anticipated. Processing that couldn't stop, couldn't slow down, couldn't be interrupted consumed compute at a rate we hadn't fully modelled.

The question that followed seemed straightforward: should we buy our own hardware? The answer depends on utilisation patterns, regulatory requirements, technology obsolescence, the opportunity cost of capital, and a dozen other factors that most cost calculators ignore. Six months of analysis, procurement and deployment later, we had our answer, and a framework that will save us $1.9 million over three years.¹ The variables matter more than the verdict: by the end you will be able to run the same calculation for your own workloads.

The Trillion-Dollar Context

The macroeconomic context first, because it sets the prices of everything below. Global AI spending will reach $1.5 trillion in 2025 and exceed $2 trillion by 2026. Organisations increased spending on AI infrastructure by 166% year-over-year in Q2 2025, reaching $82 billion in a single quarter. The AI infrastructure market is projected to reach $758 billion by 2029.²

The hyperscalers are leading this charge. In 2024, Alphabet, Microsoft, Amazon, and Meta invested nearly $200 billion in capital expenditure, a figure expected to climb by over 40% in 2025. Hyperscaler CapEx is projected to reach $315 billion in 2025 and exceed $600 billion in 2026. For scale: tech CapEx as a percentage of GDP in 2025 nearly matched the largest capital projects of the twentieth century: the interstate highway system, the Apollo programme, and nationwide broadband development combined.

It is driving two effects at once: aggressive price competition in cloud GPUs, and sustained demand that keeps the hardware itself expensive. Renting gets cheaper while owning stays dear; the whole decision turns on that asymmetry.

What the Cloud Actually Costs

Cloud GPU prices are falling fast. AWS cut H100 prices by 44% in June 2025, dropping from approximately $7 per hour to $3.90 per hour. Specialised providers like GMI Cloud offer H100s at $2.10 per hour, and Hyperbolic at $1.49 per hour. The market for H200 GPUs ranges from $2.15 to $6.00 per hour depending on provider.³

But hourly rates tell only part of the story. The on-demand cost for an AWS p5.48xlarge instance (8x H100 GPUs) is $98.32 per hour.⁴ Running this configuration continuously for one year costs $861,283. Over five years, continuous AWS usage would cost over $4.3 million. Even with three-year reserved instances offering significant discounts, the cost burden remains $2.4 to $2.8 million, at least $1.5 million more than equivalent on-premise infrastructure.

Hidden costs compound the problem. Data transfer egress fees range from $0.08 to $0.12 per gigabyte on hyperscale platforms, costs that can add 20–40% to monthly bills. Storage, networking, and API calls accumulate. Enterprise AI budgets reached an average of $85,521 per month in 2025, a 36% increase from 2024. The proportion of organisations planning to spend over $100,000 monthly on AI more than doubled, from 20% in 2024 to 45% in 2025.

Inference workloads are the real cloud tax. Companies regularly jump from $5,000 to $50,000 per month overnight when inference scales. Enterprises processing 5 to 50 billion tokens monthly face API costs ranging from $45,000 to $1,000,000 per month.

What Owning Actually Costs

On-premise costs are front-loaded and predictable, but the sticker price is just the beginning. A single NVIDIA H100 GPU costs between $25,000 and $40,000 depending on configuration (PCIe versus SXM) and vendor relationships. Complete 8-GPU server systems range from $200,000 to $400,000 including all necessary components. An 8-GPU DGX H100 system can exceed $300,000.

Infrastructure costs often match or exceed hardware costs. Power infrastructure for H100 GPUs (each requiring up to 700W under load) can add $10,000 to $50,000 for dedicated power distribution units and facility upgrades. Cooling systems for dense GPU clusters add $15,000 to $100,000 depending on scale, with water-cooling or enhanced HVAC often required. Rack infrastructure, cable management, and monitoring systems add another $5,000 to $15,000 per rack. Annual cooling infrastructure costs in data centres range from $1,000 to $2,000 per kilowatt.

Operational expenses continue indefinitely. Power costs average $0.10 per kWh, translating to approximately $35,000 annually for a 40kW GPU rack. Cooling adds 30% to power costs. Maintenance contracts run 10–15% of hardware costs annually. Dedicated IT staff for setup, maintenance, and security represent ongoing personnel costs. These operational costs often exceed 30–40% of hardware costs annually.

Lead times create planning challenges. On-premise deployment typically requires 5–6 months for hardware delivery and setup. Cloud providers offer access in minutes to hours. For organisations that can't predict demand or need to move quickly, this flexibility has real economic value.

The Break-Even Calculation

The critical question is: at what point does on-premise infrastructure become more cost-effective than cloud? Lenovo's TCO analysis provides concrete numbers. For an 8x H100 server configuration, the break-even point is reached at approximately 8,556 hours, about 11.9 months of continuous usage. Beyond this point, operating on-premise infrastructure becomes more cost-effective than cloud services.

The consensus across analysts: on-premise becomes economically viable when its total costs reach 60–70% of equivalent cloud spending, achievable at scale.⁵ And the bar is drifting: as cloud APIs get more competitive, break-even will shift towards 50–60% of cloud cost, meaning higher usage thresholds to justify owning at all.

Utilisation is the key variable. Cloud rental wins on economics unless GPU utilisation genuinely exceeds 60–70% continuously, a threshold few organisations actually achieve. For usage under 100 hours per month, cloud typically wins. Over 200 hours per month, local hardware typically wins. The messy middle (100 to 200 hours) requires careful analysis of your specific workload patterns.

A concrete example: a 100-GPU cluster costs approximately $3 million to build but would accumulate $4.2 million in annual cloud costs. After three years, the on-premise deployment saves $9.6 million while providing complete control over hardware, software, and data. Even accounting for operational expenses, on-premise deployments cost 65% less than cloud equivalents over five years for continuously utilised workloads.

When Cloud Is the Right Answer

Cloud wins when demand is a guess. Variable or intermittent workloads benefit from pay-as-you-go pricing that scales with actual usage. Development, testing, and proof-of-concept projects avoid the risk of underutilised hardware investments. Organisations requiring access to the latest hardware without maintenance overhead can leverage cloud providers' continuous infrastructure updates.

Rapid scaling requirements favour cloud. Spinning up additional GPU clusters for training spikes takes minutes in the cloud versus months for on-premise procurement. Organisations with unpredictable growth patterns benefit from cloud flexibility. A mid-size e-commerce platform processing 3 billion tokens monthly with highly variable seasonal demand found that the flexibility premium justified higher per-unit costs.

Access to frontier models provides another cloud advantage. Cloud APIs maintain a 6–12 month lead in model quality compared to models that can be self-hosted. For applications requiring the absolute latest capabilities, this gap may justify the cost premium. The trade-off is dependency on provider roadmaps and pricing decisions.

Opportunity cost of capital matters. That $300,000 H100 purchase could alternatively fund 2–3 experienced ML engineers for a year, develop new product features, or extend runway for a startup. Capital locked in depreciating hardware generates zero returns beyond its direct utility. For startups and capital-constrained organisations, preserving flexibility often outweighs long-term cost savings.

When On-Premise Is the Right Answer

On-premise wins when demand is a heartbeat: sustained, predictable, always on. SaaS providers whose databases, caching systems, or ML functions must be permanently available benefit from fixed costs. Medical research institutions analysing large image datasets with deep learning models achieve predictable economics. Financial service providers using GPU-based real-time scoring, risk analysis, or fraud detection see operational expense stability. Media companies continuously transcoding or evaluating video formats avoid usage-based cost escalation.

Latency-sensitive applications often require on-premise deployment. On-premise achieves 2–5x lower latency for real-time applications compared to cloud alternatives. Direct GPU access eliminates network delays and connection issues. For applications where milliseconds matter (autonomous systems, real-time trading, interactive AI assistants), this performance difference is decisive.

Data sovereignty and regulatory compliance drive many on-premise decisions. The EU's General Data Protection Regulation requires strict controls on personal data processing. The Digital Operational Resilience Act (DORA) sets uptime standards and data protection regulations for financial institutions. Healthcare organisations must comply with HIPAA in the US and similar regulations globally. Under the US CLOUD Act, AI solutions hosted in hyperscalers' public cloud environments may fall under foreign government access rules. For organisations in regulated industries, on-premise may be the only compliant option.

A large healthcare system processing 15 billion tokens monthly for medical record analysis and clinical decision support initially used cloud APIs through a HIPAA-compliant provider at $135,000 per month ($1.62 million annually). Migration to on-premise infrastructure achieved 39% savings ($1.9 million against a $4.86 million three-year cloud bill) while ensuring complete data control and regulatory compliance.

The Hybrid Approach

The choice between cloud and on-premise is no longer binary. Organisations achieving the best outcomes combine both strategically: using on-premise for high-volume, predictable workloads and cloud for peaks, experiments, and access to frontier models.

The optimal pattern: train in the cloud for scale, deploy at the edge for speed. Cloud infrastructure remains indispensable for the heavy computational lift of AI model training. Training requires massive, elastic GPU clusters and petabytes of data for high accuracy. Once trained in the cloud, models are optimised, compressed, and deployed to local infrastructure for real-world application. This ensures sub-second decision-making, minimal data transfer, and continuous operation where value is delivered.

A mixed approach (using cloud-based models for 60% of image editing workloads and on-premise GPUs for the remaining 40%) can reduce total GPU spend by approximately 18% while maintaining quality standards. The key is matching infrastructure to workload characteristics: baseline capacity on owned hardware, burst capacity from cloud.

Technology obsolescence risk favours hybrid strategies. NVIDIA's Blackwell architecture (B100/B200 GPUs) launched in 2025 with significant performance improvements over H100. Purchased hardware locks you into current-generation technology while competitors using cloud providers can instantly access superior hardware. Maintaining diversified GPU portfolios, some owned, some rented, provides hedging against both price increases and technology shifts.

Cost Optimisation Beyond Infrastructure

Software optimisation delivers order-of-magnitude better efficiency gains than hardware improvements alone. Systems delivering GPT-3.5-equivalent performance saw inference costs decline 280-fold between November 2022 and October 2024.⁶ This dramatic improvement stems from algorithmic advances, quantisation techniques, and efficient batching strategies, not hardware upgrades.

Model quantisation reduces the precision of a model's numerical values from 16-bit to 8-bit or 4-bit formats. This can dramatically shrink memory footprint and accelerate inference with minimal accuracy loss. Organisations implementing systematic model compression strategies report 70% reduction in inference costs and 10x improvement in deployment speed. Post-training quantisation to 8-bit achieves 2x compression with 99.9% accuracy retention. 4-bit quantisation delivers 3.5x model size reduction with 2.4x speedup.

Model distillation compresses large teacher models into smaller student models, retaining up to 97% of the original performance at a fraction of the size. Distilled students have matched teacher-class accuracy at a quarter of the training cost and a small fraction of the runtime cost. Adoption is surging: distillation is becoming a standard step in production model pipelines.

Optimised serving infrastructure maximises hardware utilisation. vLLM improved throughput by 2.7x and latency by 5x on Llama-8B compared to previous versions through continuous batching and optimised memory management. TensorRT-LLM with INT8 quantisation cuts inference GPU hours by 45%. For a high-volume API processing 10,000 requests per day, this equates to approximately $12,000 in annual savings at $2.00 per GPU-hour.

The Hidden Problem: GPU Utilisation

Most cost analyses assume reasonable utilisation rates. The reality: most colocation providers operate between 30–50% utilisation. Even best-in-class hyperscalers struggle to sustain utilisation rates above 60–70%. Energy consumption is rarely linked to compute capacity, system configuration, or workload type in any consistent way, and few providers disclose actual utilisation metrics at all.⁷

GPUs are now the bulk of AI data centre build costs. In the 2010s, traditional data centres were built with hardware (GPUs/accelerators) as 30–40% of all-in build cost and infrastructure as 60–70%. Today, modern AI clusters see this ratio inverted. With GPUs representing the majority of capital expenditure, utilisation efficiency directly impacts return on investment.

GPU power fluctuations compound utilisation challenges. During AI training cycles, instantaneous power draws can surge or decline by 30–60% within milliseconds as GPUs move between phases: matrix multiplication, memory transfer, synchronisation. Managing these surges requires oversized power-delivery networks, harmonic filtering, and fast-response uninterruptible power systems to prevent voltage dips. This infrastructure overhead further increases the true cost of on-premise deployment.

Workload orchestration is emerging as a critical optimisation lever. Platforms like Run:AI and CoreWeave aim to pack jobs more tightly so GPUs don't sit idle. Dynamic scheduling systems and power-aware workload balancing can dramatically improve utilisation, but require operational sophistication that many organisations lack. The infrastructure choice (cloud versus on-premise) matters less than operational excellence in either environment.

The Edge AI Shift

The compute mix itself is shifting. Inference workloads will account for roughly two-thirds of all compute in 2026, up from one-third in 2023 and half in 2025. The market for inference-optimised chips will grow to over $50 billion in 2026. This shift changes the economics of GPU deployment.

Enterprises spent roughly $40 billion on cloud AI inference in 2024: every API call to OpenAI, every image processed through AWS, every voice command routed to cloud services. By 2026, organisations are recognising that many of these costs are avoidable. Modern smartphones run 7-billion-parameter models locally. Edge servers process complex AI workloads without touching the internet. The same inference that costs $0.50 in the cloud can cost $0.05 on-device, a 90% cost reduction in production across retail, healthcare, manufacturing, and financial services.

European regulators have issued billions in GDPR fines, and data transmitted to cloud providers for processing is a recurring trigger. Edge AI eliminates this risk by keeping data local. The same regulatory pressure driving data sovereignty concerns is accelerating edge deployment.

Model compression enables edge deployment. TensorFlow Lite, ONNX Runtime, and PyTorch Mobile provide tools for deploying models on resource-constrained devices. Quantisation reduces precision to enable deployment on mobile phones, embedded systems, and IoT devices. Knowledge distillation creates compact architectures that retain the knowledge of much larger systems. The technical barriers to edge AI deployment are falling rapidly.

A Practical Decision Framework

The framework below is what those six months taught us.

Start with Workload Characterisation

Identify whether workloads are predominantly training (high GPU utilisation, bursty) or inference (latency-critical, continuous). Training workloads tolerate delays of up to 100 milliseconds between regions and can be sited in power-rich areas where capacity is available. Inference workloads power real-time applications and require 30–150 kW per rack with low latency. The deployment strategy differs fundamentally.

Calculate Honest Utilisation

Under 100 hours per month: cloud wins definitively. 100–200 hours per month: requires detailed analysis. Over 200 hours per month: local hardware typically wins. Under 12 months to break-even: strong case for local. Over 24 months to break-even: cloud likely better. Be honest about actual versus projected utilisation; most organisations overestimate.

Factor Regulatory Requirements

Healthcare, finance, government, and defence often require on-premise for data control regardless of cost analysis. GDPR, HIPAA, DORA, and sector-specific regulations may mandate data residency. The regulatory landscape is tightening: what's optional today may be mandatory tomorrow. Build compliance into infrastructure decisions from the start.

Consider Technology Trajectory

GPU technology advances rapidly. H100 is being superseded by H200 (available now) and Blackwell architecture (2025). Purchased hardware loses value quickly; cloud rental provides automatic access to latest hardware without additional investment. Balance long-term cost savings against technology obsolescence risk.

Invest in Optimisation Before Infrastructure

Software optimisation delivers order-of-magnitude gains at the model level; utilisation tuning delivers percentage points. Organisations focusing exclusively on infrastructure procurement miss the larger opportunity. Quantisation, distillation, and serving optimisation should precede infrastructure expansion decisions.

Key Statistics Reference

Figures as compiled July 2026.

Market Scale: Global AI spending $1.5 trillion (2025), $2+ trillion (2026). AI infrastructure spending grew 166% YoY in Q2 2025. AI infrastructure market projected to reach $758 billion by 2029.

GPU Hardware Costs: Single H100: $25,000–$40,000. 8-GPU server systems: $200,000–$400,000. Infrastructure (power, cooling, racks): often equals hardware cost.

Cloud GPU Costs: H100 hourly: $1.49–$6.98 depending on provider. AWS p5.48xlarge (8x H100): $98.32/hour. Annual continuous usage: $861,283. Five-year continuous: $4.3+ million.

Break-Even Analysis: On-premise break-even: 8,556 hours (11.9 months) for 8x H100 configuration. On-premise viable at 60–70% of cloud cost. Utilisation threshold: 60–70% continuous utilisation required.

Optimisation Impact: Inference costs declined 280-fold (Nov 2022 to Oct 2024). Quantisation: 70% inference cost reduction. Model compression: 10x deployment speed improvement. TensorRT-LLM: 45% GPU hour reduction.

Utilisation Reality: Colocation providers: 30–50% utilisation. Best-in-class hyperscalers: 60–70% utilisation ceiling. GPU power fluctuations: 30–60% within milliseconds during training.

Edge AI Economics: Cloud inference 2024: $40 billion enterprise spend. Edge inference cost: 90% reduction versus cloud. Inference share of compute: 67% by 2026 (up from 33% in 2023).

Enterprise Spending: Average monthly AI spend: $85,521 (2025), up 36% from 2024. Organisations spending $100K+/month: 45% (2025), up from 20% (2024).

The Decision We Made

After six months of analysis, we deployed a hybrid architecture. On-premise infrastructure handles our continuous medical image analysis workloads: the 24/7 processing that was driving $35,000 monthly cloud bills. Cloud capacity handles training runs, experimentation, and burst capacity during research sprints.

The economics worked because our utilisation pattern matched the on-premise sweet spot: continuous, predictable, high-volume inference. We process approximately 12 billion tokens monthly, enough volume to justify capital investment. Healthcare regulatory requirements meant we needed data sovereignty regardless of cost analysis. The 11.9-month break-even point meant we'd see returns within a year.

But we also invested heavily in optimisation before infrastructure. Model quantisation reduced our inference requirements by 70%. Optimised serving infrastructure improved throughput 2.5x. These software improvements delivered more cost reduction than the infrastructure migration itself. The hardware decision was the last optimisation, not the first.

The $1.9 million in projected savings over three years isn't theoretical; it's the difference between our current operational costs and what we would have paid continuing on cloud infrastructure at scale. But the number that matters more is the capability we've built: the operational sophistication to manage GPU infrastructure efficiently, the optimisation expertise to maximise utilisation, and the flexibility to adapt as technology and economics evolve.

The GPU infrastructure decision isn't a one-time choice. It's an ongoing strategic capability. The organisations that win in AI will be those that understand the economics deeply enough to make the right call for each workload, each phase of growth, and each shift in the technology landscape. The $1.5 trillion being invested in AI infrastructure in 2025 will separate companies that deploy capital wisely from those that simply spend.

The question isn't cloud versus on-premise. It's whether you understand your workloads well enough to choose.

Footnotes

1. Projected against July 2026 cloud list prices for our workload mix. The absolute number will age as prices fall; the framework is the durable part. ↩

2. Gartner spending forecasts and IDC's quarterly infrastructure tracker, both published in 2025. Forecasts at this scale are revised quarterly; the direction has been consistent. ↩

3. Prices as quoted July 2026. Spot and on-demand GPU rates move monthly, and specialised providers reprice or vanish faster than essays get revised: treat every figure in this section as a snapshot, not a menu. ↩

4. Lenovo total-cost-of-ownership analysis, 2025. The same analysis supplies the break-even figures later in this essay. ↩

5. Deloitte infrastructure research, 2025. The 50–60% figure is a forward projection for 2026, not a measurement. ↩

6. Stanford HAI, AI Index Report 2025. ↩

7. The disclosure gap was documented in a 2025 Association for Computing Machinery study of data-centre energy reporting. ↩