June 22, 2026: U.S. chip stocks surged across the board—The Philadelphia Semiconductor Index jumped 6.42% in a single day. Intel soared over 10% on news of a chip manufacturing partnership with Apple. TSMC’s ADR climbed 6.94% to close at $462.12, and Nvidia rose nearly 3%. This market rally reflects an accelerating industry shift: AI computing demand is moving from training-driven to inference-driven.
Industry analysis shows inference now accounts for two-thirds of total AI compute demand, up from about one-third in 2023, and is expected to reach 70%–85% by 2028–2030. This structural change is redefining the main battleground for chip competition—from "who has the fastest GPU for training" to "whose chip delivers the lowest total inference cost and highest throughput."
The global AI inference chip market is valued at $85.4 billion in 2024 and is projected to grow from $105.47 billion in 2025 to $570.77 billion by 2033, with a compound annual growth rate (CAGR) of 23.5% over the forecast period. The cloud AI inference chip market alone is estimated at $102.19 billion in 2025, expected to reach $118.9 billion in 2026, and could hit $320.98 billion by 2032. Meanwhile, the global edge AI chipset market (including both inference and training) is forecast to expand from $34.4 billion in 2026 to $96 billion by 2031.
During this expansion cycle, the balance of power among chip types is shifting in subtle but profound ways. GPUs remain the dominant market player, supported by both training and inference demand, and are expected to maintain a 20% CAGR through 2031. However, AI ASICs are viewed by many institutions as the fastest-growing segment. JPMorgan analysts estimate the digital AI ASIC market will reach $60–$70 billion by 2026, with a CAGR of over 40–50% in the coming years.
Even more noteworthy is the resurgence of CPUs. For the past three years, CPUs have played a peripheral role in AI narratives, but the explosion in inference demand is reshaping this landscape.
Why CPUs Are Returning to Center Stage
AI inference and training differ fundamentally in their computational logic. Training involves massive parallel matrix operations—trillions of floating-point calculations executed simultaneously across thousands of GPU cores, the domain where GPUs excel. Inference, especially for agentic AI, involves task orchestration, tool invocation, multi-step logical reasoning, and sequential decision-making. These workloads rely heavily on complex logic control and serial processing, areas where CPUs shine.
A joint study by Georgia Tech and Intel found that in agentic AI scenarios, 50%–90% of latency comes from the CPU, not the compute accelerator—because large models must call plugins, perform web searches, and handle multi-step logic, all managed by the CPU. Nvidia itself acknowledged this reality in March 2026: executive Dion Harris publicly stated, "The CPU is becoming the bottleneck in AI workflows"—a striking admission from a company built on the belief that "GPUs are the only chips AI needs."
Changes in configuration ratios offer a clear view of this trend. In AI training, CPU-to-GPU ratios typically sit at an extreme 1:8, with GPUs bearing most of the computational load. But in the inference era, TrendForce reports this ratio is rapidly narrowing to between 1:1 and 1:2. Intel CEO Pat Gelsinger noted in the Q1 2026 earnings call that training workloads usually require 7–8 GPUs per CPU, but inference workloads have tightened to 3–4 GPUs per CPU, with the prospect of moving toward a 1:1 balance.
Referencing Nvidia CEO Jensen Huang’s estimates: each GW-scale data center requires about 300,000 Rubin GPUs, and, based on 136 cores per ARM CPU, about 221,000 CPUs per GW. This sets the new CPU-to-GPU ratio at roughly 1:1.4. Compared to the GPU-dominated era, the CPU’s status has risen significantly.
The GPU Moat and Challenges of Inference Workloads
Despite CPUs regaining ground, GPUs still hold an irreplaceable position in AI inference, thanks to their advantages in memory bandwidth and parallel throughput.
During LLM inference, generating each token requires reading hundreds of millions to tens of billions of parameters—a classic memory-intensive task. CPUs rely on system DDR memory, typically offering 50–100 GB/s bandwidth. GPUs use GDDR6X or HBM memory, with bandwidth exceeding 800 GB/s; high-end GPUs with HBM2e can reach 1.5 TB/s, 20 times that of CPUs. In Llama 3.1 8B model inference, CPU solutions deliver just 819 tokens/s per task, while an 8-GPU cluster achieves 46,841 tokens/s. As concurrent requests increase, CPU performance drops sharply from 819 tokens/s to 257 tokens/s, while the 8-GPU cluster sees almost no degradation.
In terms of compute density, GPUs offer thousands of CUDA cores for parallelization, support low-precision formats like FP4/FP8, and deliver hundreds of TFLOPS. CPUs typically provide FP32 compute in the 1–10 TFLOPS range.
These figures show that for high-throughput, high-concurrency inference scenarios—such as large-scale cloud AI services—GPUs remain the optimal solution. Nvidia’s dominance in this field is unchallenged. According to SemiAnalysis, Nvidia held a 92% share of the AI training chip market and 78% of the inference chip market in Q1 2026. IDC estimates Nvidia controls about 81% of the AI chip market. The AI accelerator market is expected to reach $160 billion in 2025 and over $200 billion in 2026, with inference spending accounting for two-thirds.
However, GPU market share in inference is facing multiple pressures—from the CPU’s comeback, specialized ASIC competition, and practical cost considerations.
CPU Vendors’ Inference Counteroffensive
The revaluation of CPUs in inference has translated into measurable market momentum.
The data center processor market is experiencing rapid growth, fueled by surging demand for generative AI workloads. Market size is projected to expand from $215 billion in 2025 to $656 billion by 2031. Guohai Securities notes that hyperscale data centers are entering an "upgrade cycle," with server CPU shipments expected to grow 25% in 2026.
AMD is a standout beneficiary of this trend. AI server demand has boosted EPYC CPU shipments, with the fifth-generation Turin capturing a significant share of the server CPU market. AMD’s server CPU business is expected to grow at least 50% in 2026. Bernstein analysts forecast that flagship EPYC processor sales could jump 30% in 2026. As of early 2026, Intel holds about 60% of the data center CPU market, AMD about 24%, and Nvidia about 6%. AMD also competes in the AI GPU market with its Instinct accelerators, giving it a unique dual positioning in the inference era.
Intel is also actively adjusting its strategy. At Computex in June 2026, new CEO Pat Gelsinger announced the return of CPUs to prominence in the inference era, leveraging 18A process technology and rack-scale decoupled architectures. AI infrastructure is moving from "one-stop shopping" to "Lego-style assembly." Intel’s Xeon processors feature Advanced Matrix Extensions (AMX), which accelerate inference for large language models with small to medium parameter sizes, even without GPUs or other AI accelerators.
The most symbolic shift comes from Nvidia itself. The company that defined the AI era with GPUs launched its Grace and Vera CPU product lines in 2026, with Vera CPUs specifically designed for inference and agentic AI workloads. Nvidia expects its CPU business revenue to reach $20 billion in 2026. Nvidia and Arm also released standalone CPU products in 2026, marking the GPU giant’s official entry into the CPU arena.
ASICs and Dedicated Chips: The Rise of a Third Path
Beyond the GPU-CPU binary, ASICs (application-specific integrated circuits) are emerging as the fastest-growing variable in the inference market.
TD Cowen forecasts that commercial accelerator market share will drop from about 91% in 2025 to 75% in 2030, while custom ASICs will rise from 9% to 25%. ASIC server shipments are expected to grow 44.6% in 2026, compared to GPU server shipment growth of 16.1%—just one-third that of ASICs.
Hyperscale cloud providers are accelerating their development of custom inference chips. Google TPU, AWS Inferentia, Meta MTIA, and Groq’s LPU (Language Processing Unit) are all ASIC chips optimized for inference. Broadcom’s AI revenue hit $10.8 billion in Q2 2026, up 143% year-over-year, with full-year AI guidance at $56 billion, up 180%. Broadcom is expected to capture about 60% of the custom AI chip market.
This trend signals a shift in the inference chip market from "general-purpose GPU dominance" to a diversified landscape of "GPU + CPU + ASIC." GPUs handle intensive training and large-scale inference, CPUs manage task orchestration and system control, and ASICs deliver extreme energy efficiency for specific inference workloads.
Cost Structure and the Reshaping of Inference Economics
Ultimately, chip selection for inference comes down to a central question: the cost per million tokens.
During training, model accuracy and training time are the primary metrics, and cost tolerance is higher. Inference, however, is a continuous, high-frequency production activity—every API call and user request incurs direct costs. This shifts chip competition from "absolute performance" to "effective throughput per unit cost."
GPU solutions require higher upfront hardware investment. For example, the AMD MI300X sells for $10,000–$15,000, while Nvidia’s H100 ranges from $25,000–$40,000. Yet GPUs deliver lower per-unit compute costs—on-demand GPU instances from cloud providers generate tokens at 40%–60% lower cost per second than CPU instances. CPUs are advantageous for single-task, low-concurrency, low-latency scenarios, as they require no additional hardware investment.
However, as inference scales up, CPU solutions face rapidly rising marginal costs. When concurrent requests increase, CPUs must schedule tasks via time-slice rotation, with context-switching overhead growing exponentially. This means that for large-scale inference deployments, the initial high investment in GPU or ASIC solutions often delivers superior long-term ROI through higher throughput and lower per-unit costs.
Conclusion
The rise of inference demand from one-third to two-thirds of AI compute reflects a fundamental shift in chip industry competition.
For Nvidia, its near-monopoly in the training market (about 90% share) is unlikely to be challenged in the short term, but the battle for incremental inference market share will intensify. New Street Research offers the most aggressive forecast: Nvidia’s inference share could drop to 20%–30% by 2028. Even Bloomberg Intelligence’s more conservative prediction—that Nvidia will retain 70%–75% share by 2030—acknowledges the fact that ASIC shipment growth far outpaces that of GPUs.
For AMD and Intel, the resurgence in CPU demand during the inference era is a structural opportunity. AMD’s dual strategy with EPYC CPUs and Instinct GPUs, and Intel’s ongoing Xeon processor iterations with 18A process technology, both aim to capture this window.
For cloud providers and AI application developers, more chip options mean greater opportunities for cost optimization. From general-purpose GPUs to custom ASICs, and from CPU inference to GPU acceleration, hardware selection will increasingly depend on the specifics of each workload—model size, latency requirements, concurrency, and budget.
AI inference compute demand is growing faster than training. This shift from training to inference is reshaping the entire industry chain, from chip design to data center architecture. GPUs will not lose their place, but they are no longer the only answer.
