Table of Contents
Nvidia’s strongest advantage may not be its chips alone, but something less visible. It may be the software that developers already know. This week, two of the biggest cloud companies went after that advantage. Google released TorchTPU as open source, one day after Amazon unveiled a similar tool, and both aim to make it far easier to run AI code on hardware that is not made by Nvidia.
What Google Announced With TorchTPU
Google said on Tuesday, 20 October, that TorchTPU, its native PyTorch backend for its Tensor Processing Units, is now open source. Amazon had unveiled a similar tool, TorchNeuron, a day earlier. Both companies are targeting the software advantage that keeps most AI apps tied to Nvidia graphics chips.
PyTorch is the open-source framework most AI developers use to build and run models, and many of its features were built around Nvidia hardware. Each new tool lets developers move existing code to a rival chip with a configuration change instead of a rewrite.
In practical terms, a backend is the layer that translates a framework’s instructions into work a specific chip can carry out. If that layer works well, a developer sees familiar PyTorch behaviour while the hardware underneath changes.
That promise is the heart of the story. A developer who can switch hardware without rebuilding an application faces far less risk when trying a different chip.
Google’s TorchTPU Goes Public
Built With Meta and the Wider Community
Google built TorchTPU with the PyTorch team at Meta and made it public at PyTorch Conference North America in San Jose. The code is on GitHub under the Apache 2.0 licence.
Google said it worked with Meta, Hugging Face, Inferact (the maker of vLLM), and RadixArk (the maker of SGLang) so that common tools behave on TPUs as they do in standard PyTorch.
The partner list matters because tools such as vLLM and SGLang are used to serve models. If they behave on TPUs as they do elsewhere, teams can keep much of their existing serving setup.
Agents, Engines, and Early Use
Google also launched a TPU agent suite to help developers migrate apps and fix errors. The SGLang team said its TorchTPU serving engine is already running in production at Meta, and Google said its new vLLM and SGLang engines are serving preview customers at scale.
Google first described TorchTPU in April and had been testing it with select customers. The public release therefore follows months of earlier work rather than arriving from nowhere.
Amazon’s TorchNeuron Targets Trainium
Amazon announced TorchNeuron on Monday, 19 October, and released it as an open-source public beta on Tuesday. The tool runs on Trainium2 and Trainium3 instances, and developers enable it by changing the PyTorch device setting to “neuron”.
Amazon is also submitting changes to the main PyTorch project that remove code tying the framework to specific chips. Before TorchNeuron, developers had to rewrite PyTorch code to call Trainium-specific interfaces, said Colin Brace, an executive at Amazon’s Annapurna Labs chip unit. Removing chip-specific code from the main project could also help other hardware makers, since the framework would depend less on any single vendor’s design.
Seen side by side, the two efforts share one strategy. Instead of asking developers to learn a new system, each company is meeting them inside the framework they already use.
Gaps Remain
Limits in the First Release
The first release has limits. TorchNeuron does not yet support some features, such as dynamic input shapes, and about 22,000 PyTorch tests are skipped. Google has also said that developers may still need to adjust model design or write custom code to get the best performance from TPUs.
So a configuration change may get an application running, but it does not always deliver the best speed straight away. Skipped tests are a reminder that compatibility is a moving target, and each missing feature can block a real project from moving over.
Why Nvidia Still Holds an Edge
Nvidia’s lead rests partly on CUDA, its software platform for running models on its GPUs. Its libraries and debugging tools form an ecosystem that rival chipmakers also have to compete with.
Being able to switch chips easily does not mean developers will leave Nvidia hardware in large numbers. Still, Amazon and Google are clearly working to end Nvidia’s advantage.
What This Means for Everyday Users
Most people will never write PyTorch code, yet the outcome could still reach them. If developers can run the same models on more kinds of chips, cloud providers may compete harder on availability and price. That could shape how quickly new AI features arrive in the apps people use every day.
For developers and teams, the sensible approach is to test rather than assume. Try TorchTPU or TorchNeuron on a small workload, compare results against your current setup, and note which features are missing. Start with a model you already run, because migration claims matter most when measured on your own code.
Frequently Asked Questions
1. What is TorchTPU?
Google’s native PyTorch backend for its Tensor Processing Units, now open source under the Apache 2.0 licence.
2. What is TorchNeuron?
Amazon’s similar tool for Trainium2 and Trainium3 instances, released as an open-source public beta.
3. How do developers use these tools?
Each lets developers move existing code to a rival chip with a configuration change rather than a rewrite. For TorchNeuron, that means setting the PyTorch device to “neuron”.
4. Will developers abandon Nvidia?
Not necessarily. Nvidia’s CUDA ecosystem, libraries, and debugging tools remain a strong advantage, and switching easily does not guarantee large-scale moves.
Conclusion
TorchTPU and TorchNeuron show that Google and Amazon see software, not just silicon, as the way to compete with Nvidia. The tools are promising, but early limits and CUDA’s deep ecosystem mean the real test is still ahead. For developers, the practical question is no longer whether alternatives exist, but whether they work well enough on real workloads to justify the effort of moving.