Google TPUv7 Shows Competitive Economics on Long-Context Agentic Inference
AgentX results put TPUv7 near or ahead of GPUs on several workloads, while model shape, precision and serving design still decide the winner.

Google’s TPUv7 Ironwood is now being measured on inference workloads that look more like production agent sessions than short chatbot exchanges. The first AgentX submissions cover four models and show TPUv7 competing on tokens per dollar across several configurations. In some comparisons it leads; in others, GPU systems retain an advantage through lower precision or stronger end-to-end performance.

The results also test a wider claim: that Google’s accelerator can attract users beyond its own services. A chip’s performance matters, but so do the software tools, open-source runtimes and serving infrastructure teams already use. TPUv7 submissions from developers working with SGLang, vLLM and llm-d suggest that ecosystem support is becoming part of the comparison.
What AgentX measures
AgentX is designed around multi-turn, long-context sessions. A request can carry forward a growing conversation and reuse much of its prefix, while also launching short-lived sub-agents with fresh context. That combination creates different demands from a single prompt followed by one response: cache capacity and reuse matter, and a serving system must balance prompt processing against token generation.

InferenceX reports results across different interactivity targets, including tokens per second per user. The submissions described here compare systems at stated model and precision settings. Those details matter: a result for FP8 weights does not directly settle a comparison with FP4, particularly when quantization changes output quality.
The benchmark has submissions from TPUv7, NVIDIA and AMD systems, with additional platforms listed as planned. Its contributors include developers from vLLM, SGLang and llm-d, among other machine-learning projects. The benchmark’s results are useful as a common reference, but the configurations and submitters still need to be considered when interpreting each chart.


The new tests build on earlier single-turn comparisons. In an FP8 test on Qwen 3.5 397B, TPUv7 was reported to deliver up to 50% better performance per dollar than B200 and B300. AgentX expands the scope to workloads with multiple turns and long-lived context, where routing and memory management can change the outcome.

Four models, four different trade-offs
The initial set covers Muse Spark Open, GLM-5.2, Qwen3.5 397B and Kimi K3. Each result uses a different combination of model architecture, numerical format and serving stack. The pattern is less “one accelerator wins” than “the software and workload shape determine where each system is strong.”

Muse Spark Open: RadixArk submitted an SGLang AgentX result using NVFP4 weights. TPUv7’s decode performance per dollar is competitive using the stated TPU cost basis, despite the model’s architecture being tuned for GPUs.
GLM-5.2: In vLLM at FP8, TPUv7 is the lowest-cost option in the reported comparisons from 60 to 300 tokens per second per user. It delivers 1.2 to 2.7 times the tokens per dollar of B200, B300 and MI355X. NVIDIA NVFP4 produces more tokens per dollar in some comparisons, with an associated accuracy or quality reduction.

Qwen3.5 397B: With llm-d disaggregated serving, TPUv7 FP8 beats GB300 FP8 across the tested interactivity range. The submission also reports parity with GB300 FP4 at higher output quality.
Kimi K3: Inferact submitted a vLLM result using MXFP4 weights. TPUv7 was competitive with GB300 NVL72 at launch and delivers 1.15 to 2.5 times the tokens per dollar of GB200, B200, B300 and MI355X in the reported comparisons.


These figures need their precision labels attached. TPUv7 does not have native FP4 support, so comparisons against FP4 GPU runs combine hardware differences with a change in weight representation. For GLM-5.3, which was not trained with block-scaled FP4 quantization, FP4 can reduce accuracy relative to FP8. At 100 tokens per second per user, the reported GB300 and GB200 FP4 results deliver 3.6 and 3.5 times TPUv7’s tokens per dollar, respectively.
That gap reflects a familiar inference trade-off. During small-batch decoding, weight traffic can dominate. FP4 halves the bytes required per weight compared with FP8, which can raise throughput per unit cost when quality remains acceptable. TPUv8, expected to add FP4 support in Q1 2027 according to the shipment estimates cited by the benchmark publisher, could change that comparison; the current TPUv7 results cannot establish how it will perform.

Muse Spark Open tests TPU flexibility
Meta’s Muse Spark Open is an 816-billion-parameter model and one of the launch models for SGLang-TorchTPU. Meta is already deploying TPUv7 with that stack, while the benchmark submission came from RadixArk, a community developer and SGLang maintainer. The result suggests competitive decode economics even though the model’s attention shape is an awkward fit for TPU hardware.
Muse Spark Open uses an attention head dimension of 64. On TPU, long-context attention with this shape leaves 25% of the systolic array padded, according to the benchmark analysis. GPU matrix cores handle smaller tiles, so a head dimension of 64 can use hardware efficiently on H100 and B200 systems. Model researchers can therefore choose the smaller dimension without paying the same inference penalty on those GPUs.

TPUv7’s decode implementation uses a fused megakernel, combining work that would otherwise require many kernel launches and reducing launch overhead. At high concurrency, the submission also uses SGLang HiCache to offload data to CPU DRAM through Google’s open-source TPU-Sync library, aiming to improve prefix-cache hit rates.
The decode result is promising, but the submission still needs better end-to-end latency to compete, including further work on prefill kernels. That distinction matters in serving: improving token generation alone does not guarantee faster responses if processing a long input remains the bottleneck. RadixArk said it was preparing documentation for the optimized configuration.


GLM-5.3: cache layout and sparse attention
GLM-5.2 and GLM-5.3 are mixture-of-experts transformer models with 744 billion parameters and about 39 billion active per token. They support a one-million-token context using DeepSeek Sparse Attention. An indexer selects 2,048 prior tokens for full attention, reducing how much of the context each query must read.
For these long sessions, the KV cache limits how many users can be served at once. In the AgentX traces, more than 90% of each request is already cached. Google’s optimizations address both the distribution of that cache across TPU chips and the cost of moving selected entries into the attention calculation.


First, decode context parallelism splits the cache by sequence position. GLM’s attention reads a shared compressed KV entry per token; splitting attention heads across devices would copy the full cache to each device. With decode context parallelism, devices process separate portions of the sequence and merge partial attention results at each layer.
Second, TPUv7 SparseCores handle the indexer’s scattered memory accesses. They select and gather the 2,048 entries while TensorCores perform the attention calculation. This allows sparse retrieval and matrix work to proceed on hardware suited to each task.

A third change improves how indexer data is packed into TPU registers. Each register has eight rows and 128 lanes. In the previous layout, a 128-byte key plus a scale byte made 129 bytes, so the row had to be padded to 256 bytes. The revised layout runs tokens across rows and each token’s key down 32 four-byte rows, with another row for the scale. That occupies 132 bytes, leaving only three bytes unused.
Finally, a fused kernel combines routing, expert computation, all-reduce and attention merging into a collective operation. TPU programs are compiled for fixed shapes; small decode steps had been padded to 16 tokens, so Google added four- and eight-token shapes to reduce that wasted work.


Qwen3.5 and disaggregated serving
Qwen3.5-397B combines 45 linear-attention layers, which keep a fixed-size state per request, with 15 full-attention layers. Its AgentX result was Google’s first TPUv7 submission to the benchmark. In the reported comparison, TPUv7 FP8 delivered 1.67 times the performance per dollar of GB300 NVL72 FP4.
The serving stack uses a decode megakernel that runs the decode step in one kernel and prefetches the next layer’s weights while the current layer computes. KV cache and linear-attention state can spill to host DRAM through TPU-Sync. llm-d routes each turn to a prefill host that already holds the session prefix, then keeps the session on its decode host unless it reaches capacity. Google reported that routing to the exact data-parallel rank raised decode cache hit rates from 56% to 76%.

Disaggregated serving separates prefill and decode onto different hosts so a long prompt does not stall token generation for other requests. In the tested llm-d setup, each host contains four TPUv7 chips and each result uses 24 to 32 chips. Decode hosts cap the number of active requests, from one at the low-latency end up to 112 at the high-throughput end. Requests beyond the cap wait for an available host, increasing time to first token.

At the fastest reported point, p90 time to first token was 8.6 seconds for TPUv7, compared with 11.1 seconds for GB300 FP4. Across FP8 interactivity settings, TPUv7 with llm-d beat GB300 FP8, with more than twice the tokens per dollar at high throughput and over 1,000 tokens per second per user in the ultrafast region. The turn-based routing method nevertheless produced weaker end-to-end performance than GB300 TensorRT-LLM FP4. Against GB300 FP8 and FP4 using SGLang, TPUv7 performed better in the reported comparisons.

On the initial Qwen3.5 FP4 counter-submission, NVIDIA’s decode performance exceeded TPUv7 FP8, but the configuration was tuned at the expense of prefill. The end-to-end result still favored TPUv7 FP8. Later NVIDIA refreshes changed some comparisons, so results should be read alongside the submission date and serving configuration.
Kimi K3 and megakernels
Kimi K3 uses multi-head latent attention, which stores KV data as a shared latent across heads. Tensor parallelism cannot split that latent in the usual way, leaving each tensor-parallel rank with a full copy. The submitted setup instead uses context parallelism to assign different token slices to ranks. Using context parallelism for prefill would have required a lower tensor-parallel degree, copying weights across more chips and leaving less high-bandwidth memory for the already constrained KV cache.

Megakernels are central to the decode work. A conventional step can launch hundreds or thousands of kernels, leaving gaps between operations. A megakernel combines more of the work into a long-running kernel, prefetching upcoming weights as the current layer computes. AMD calls its implementation MonoKernel. The benchmark analysis cites AMD’s report that, on eight MI350X GPUs running GLM-5.2, its approach reduced launches from about 2,459 to 210 and ran 2.31 times faster at batch size one.

Inferact, founded by vLLM’s creators and maintainers, developed the Kimi K3 megakernels with Google and submitted the result jointly. The implementation plugs into vLLM as a model runner that replaces the forward pass while retaining vLLM scheduling, prefix caching and KV transfer. The code is open source, and Inferact provides a reproduction guide using a Docker image. NVIDIA added its own megakernel support within a week of the earlier open-source implementations.

At its October 17 launch, TPUv7 delivered 0.77 to 0.91 times GB300’s tokens per dollar from 50 to 200 tokens per second per user, while reaching 391 tokens per second per user for a single user against GB300’s 251. At Google’s internal cost basis, TPUv7 was ahead at 100 and 200 tokens per second per user. After four NVIDIA refreshes, GB300 FP4 leads TPUv7 by 1.5 to 2.6 times. TPUv7 still delivers 1.15 to 2.5 times the tokens per dollar of GB200, B200 and B300 in the reported range.
The comparison with Rubin NVL72 is less settled. Preview submissions show Rubin up to six times better on performance per dollar than TPUv7, but the systems are from different generations; the benchmark analysis says TPUv8i is the more appropriate comparison and that direct access is needed to reproduce Rubin results across the full suite.

Software support is part of the hardware result
TPU’s progress depends on getting its stack into the tools developers already use. Google has worked with maintainers across PyTorch, vLLM, SGLang, llm-d and Ray. Meta’s Muse Spark Open has TPU support in SGLang-TorchTPU, while llm-d added TPUv7 support for Kimi K3.1 Pro at launch. These integrations make it easier to evaluate TPU systems within familiar serving workflows.
The comparison also puts pressure on NVIDIA’s developer relations and response times. The benchmark publisher describes customer and maintainer complaints about delays to driver features and infrastructure changes. It also points to faster work by AMD and Google on open-source inference features. Those are reported experiences, not a measure of chip performance, but they affect how quickly a platform can be tuned and adopted.
TPUv7’s AgentX results show that a credible alternative can emerge when hardware, runtime work and serving design are developed together. They also leave clear limits: FP4 remains an advantage for GPU systems in some memory-bound cases, model dimensions can favor one architecture, and a decode win may not translate into better end-to-end latency. For teams comparing accelerators, the useful question is which combination of model, precision, cache strategy and interactivity target matches their own workload.
AMD Venice Uses 2.5D Packaging to Feed More Bandwidth Into EPYC Zen 6
CNC Controller Board Design: Interfaces, PCB Specs and Assembly Testing
Stratasys and Nano Dimension End $42.5 Million Markforged Sale Agreement