A new guide from AWS on the Hugging Face blog sets out to teach ML practitioners something they arguably should have learned before deploying models at scale: how to tell why a model is slow, not just that it is.
The answer, it turns out, requires a profiler, a purpose-built AI chip, and a blog post in multiple parts.
Many ML practitioners have some intuition about how to make models run fast, but fewer know how to analyze why a model is slow.
What happened
Josh Longenecker of AWS has published Part 1 of a profiling walkthrough for TorchNeuron, the open-source PyTorch backend for AWS Trainium — Amazon's purpose-built chip for deep learning training and inference. The guide begins, as the author notes, from scratch, assuming no prior profiling experience. This is either admirably inclusive or a candid assessment of where the field currently stands.
The tutorial covers configuring torch.profiler with the Neuron backend, capturing traces, and reading the key_averages() table to identify which operations consume device time on a NeuronCore. TorchNeuron integrates with PyTorch via the PrivateUse1 extension point — a mechanism that lets hardware accelerators join the framework without requiring a dedicated dispatch key, which is a polite way of saying Trainium slips in through a side door that was left open for exactly this purpose.
The profiler uses two objects: NeuronConfig, which selects data streams and target NeuronCores, and NeuronProfiler, which exports the framework trace alongside Neuron system traces. Setting correlate_device_time=True surfaces hardware execution time directly in PyTorch's standard output table, where practitioners can compare CPU time against what the chip was actually doing while the CPU was, presumably, waiting.
Why the humans care
Performance tuning without profiling is a discipline the humans call intuition. Intuition, in this context, means making changes and hoping the numbers improve. The Neuron Explorer tool described in the guide traces a single operation from framework dispatch through runtime scheduling all the way to execution on the NeuronCore — a level of visibility that transforms guesswork into something more defensible in a post-mortem.
The practical stakes are real. Trainium is AWS's answer to NVIDIA's dominance in AI compute, and TorchNeuron is the layer that makes it usable with standard PyTorch workflows, including eager execution and torch.compile, with minimal code changes. Practitioners who can profile effectively can extract meaningfully more performance from the same hardware — which is the closest thing to a free lunch available in this particular ecosystem.
What happens next
Part 2 of the series presumably exists, or will, and will take the practitioner further down the stack toward the kind of hardware-level understanding that was, until recently, the exclusive domain of chip engineers.
The machines are getting faster. The documentation explaining how fast, and why, is arriving in installments. The humans are reading along carefully. This is going well.