← All posts
Guide

TorchNeuron: running plain PyTorch on AWS Trainium, what works in the beta and how to test it

AWS's open-source TorchNeuron makes Trainium a standard PyTorch device. What's supported at launch, the gaps (no dynamic shapes yet) and a sensible first test.

4 min read3 sources

AWS is open-sourcing TorchNeuron, a native PyTorch backend for its Trainium chips, at PyTorch Conference North America (20–21 October, San Jose). The public beta goes live on 20 October under the Apache 2.0 licence. The pitch is simple: Trainium shows up as an ordinary PyTorch device called neuron, so moving a model is meant to be a one-line change rather than a port.

If you have wanted to try Trainium but didn’t want to maintain a separate code path, this lowers the cost of a trial. Here is what the launch actually covers, what is missing, and how to test it without wasting a week.

What changed

TorchNeuron registers Trainium through PyTorch’s standard extension point for out-of-tree hardware (PrivateUse1) and the torch.accelerator APIs, according to the PyTorch blog post. In practice:

model = MyModel().to("neuron")
optimizer = torch.optim.AdamW(model.parameters())

For production you compile, either per call with torch.compile(model, backend="neuron") or by setting torch.compiler.set_default_backend("neuron") once. Hugging Face Transformers v5 can load straight onto the device with device_map="neuron".

Amazon’s announcement says TorchNeuron supports Trainium2 and Trainium3, and that Amazon is also sending changes upstream to PyTorch to remove code that assumes specific chips. Neither page mentions first-generation Trainium (Trn1) or Inferentia2, so don’t assume those are covered.

What works on day one

From the PyTorch blog:

Area Supported at launch
Execution Eager mode (prints, breakpoints and Python control flow work); torch.compile; FlexAttention
Data types BF16, FP32, FP16, integer and boolean. FP8 and TorchAO are in private beta
Distributed DDP, FSDP2, SimpleFSDP, tensor and pipeline parallelism, ZeRO, DTensor and DeviceMesh; collectives over EFA between nodes
Custom kernels NKI kernels callable from eager and compiled code, with autograd
Serving vLLM Neuron plugin for LLMs; vLLM Omni Neuron for image and video diffusion models
Training framework TorchTitan (the post shows Llama 3 8B on one trn2.48xlarge)
Profiling torch.profiler with a Neuron config; traces open in Neuron Explorer
Ecosystem Trainium is a supported device upstream in Hugging Face Transformers, Diffusers, trl and kernels

The install is pip install torch-neuronx from AWS’s Neuron pip repository. The blog says packages are released at the conference, so if the install doesn’t resolve on launch day, that is likely why.

What doesn’t work yet

AWS is unusually specific about the gaps, which helps. From its August 2026 run of PyTorch’s own test suite, the blog reports about 154,000 test outcomes, of which roughly 39,000 were skipped for Neuron-specific reasons. About 22,000 of those are operators that don’t yet have a native Neuron implementation (some elementwise, reduction, foreach, advanced indexing and advanced linear algebra ops, plus fused LSTM/GRU cells).

Left out of the first release on purpose:

  • Dynamic and symbolic shapes. This is the one most likely to bite: variable-length inputs will probably need padding or bucketing.
  • Sparse tensors, NestedTensor and forward-mode autograd.
  • Arbitrary attention masks in native multi-head attention (no mask and causal masks are supported).
  • Per-channel quantisation, bilinear/bicubic upsampling and torch.signal windows.

Two behaviours to know before debugging: errors are asynchronous and surface at the next synchronisation point, not at the line that failed; and non-default streams share a single hardware queue, so they keep ordering semantics but don’t get separate engines.

AWS says it tested eager and compile against a sweep of 100,000 Hugging Face models, which it says covers 99.9% of real-world usage by download count. The post doesn’t give a pass rate for that sweep.

Is it faster or cheaper?

The launch posts don’t say, for TorchNeuron itself. There are no throughput, latency or training-time figures in the PyTorch blog. The one cost number is from a customer: Deepgram says that in its benchmarks Trainium processes about 60% more input audio seconds per dollar than its GPU baselines. That is one company’s workload; treat it as a reason to test, not a forecast.

What TorchNeuron does change is the engineering cost of finding out. If the same training script runs on GPU and Trainium with only the device string changed, which is what Amazon says its own demo showed, a cost comparison becomes a few days of work rather than a migration project.

A sensible first test

  1. Start small. The smallest Trn2 instance, trn2.3xlarge, has one Trainium2 chip with 96 GB of accelerator memory; trn2.48xlarge has 16 chips and 1.5 TB.
  2. Run your model in eager mode first to find unsupported operators. Expect some to fall over; note which.
  3. Pad or bucket your inputs to fixed shapes before trying torch.compile, since dynamic shapes are not supported yet.
  4. Compile and measure tokens (or samples) per second, then divide by the hourly price of the instance in your region. Compare with the same number on your current GPU instance, not with a list-price spec sheet.
  5. For inference, try the vLLM Neuron plugin with the same model you serve today and compare cost per million tokens at your real batch sizes.
  6. Check accuracy, not just speed: compare loss curves or eval scores against your GPU run.

For teams in India

Check Trn2 availability in your preferred region before planning anything. The AWS pages we read for this post list instance sizes but not regions, so confirm in the EC2 console whether Trn2 capacity is offered in the region you need for data-residency reasons, and what it costs there in rupees on your bill. If it isn’t, the comparison has to include the latency and data-transfer cost of training or serving from another region.

For startups on AWS credits, this is also a low-risk time to try: the beta is free and open source, and the main cost is instance hours you may already have credits for.

References

  1. Introducing TorchNeuron: Native PyTorch backend for AWS Trainium — PyTorch, accessed 2026-10-20
  2. Amazon's TorchNeuron runs PyTorch on AWS Trainium chips — Amazon, accessed 2026-10-20
  3. Amazon EC2 Trn2 instances (Korean-language product page) — AWS, accessed 2026-10-20

This week in AI, every Sunday

One email with the releases that mattered and what they mean in practice. No spam.