#
PyTorch Programming Tutorials & Engineering Articles
815 PyTorch tutorials, guides, and engineering insights from NVIDIA, Meta, Google, and more
Companies Using This
PyTorch Articles & Tutorials
Filter:
As AI agents tackle increasingly complex workflows, reducing time-to-train is critical for rapidly iterating on both models and the agents they power.
AI applications at the edge often need to run multiple neural networks within limited compute and power budgets. NVIDIA Deep Learning Accelerator (DLA)…
Autonomous vehicle (AV) developers need reasoning models that can operate within the compute and latency constraints of an onboard system.
Marco Pavone
12 min read
Includes Code
--
Hitarth Trivedi, Jay Narale, Jianjian Xie, Aerial view of a busy multi-level highway interchange with cars and bright streetlights at night., Yunfei Chen, Sanah Imani
15 min read
--
Triton Low-Level Extensions (TLX), is an extension to Triton that helps engineers optimize GPU performance while preserving Triton’s productive programming model. TLX introduces explicit war…
Hongtao Yu
20 min read
Includes Code
--
Pathways is a single-controller distributed training system designed to coordinate large-scale model training across multiple accelerator groups by representing computations as a global dataflow graph. Unlike traditional SPMD models like DDP and FSDP that rely on synchronous collectives, Pathways uses asynchronous dispatch and a central scheduler to maximize hardware utilization and prevent deadlocks in complex workloads. By decoupling model definition from physical hardware via virtual devices, developers can efficiently scale complex training architectures and seamlessly handle dynamic workloads like reinforcement learning post-training.
During reinforcement learning (RL) post-training, training trajectories are generated on the fly by running inference with the current policy model.
Tanya Lenz
11 min read
Includes Code
--
As distributed AI training scales, failures across hardware and software dependencies increasingly disrupt training progress, making fast…
Elizabeth Goodman
12 min read
Includes Code
--
Physical AI developers working with world models face two distinct challenges: adapting models to their sensors, tasks, and data, and getting those models into…
Elizabeth Goodman
9 min read
Includes Code
--
Atomistic simulations help researchers understand how molecules and materials behave, from chemical reactions and phase transitions to mechanical failure.
Anoushka Bhutani
11 min read
Includes Code
--
Google Research has completed the 32-bit open-source Coral NPU RISC-V architecture with the introduction of its Matrix Execution Unit, which delivers up to 1 TOPS while targeting 10 mW average power for edge inference. Alongside this architecture, the release of the v1.0 open-source IREE/MLIR compiler allows developers to take models from JAX, PyTorch, and TensorFlow straight to Coral NPU hardware without writing custom assembly. Developers can now prototype and deploy these multimodal edge applications using new hardware options like the Synaptics Astra SR200 MCU series and globally available Coralboard devkits.12
Billy Rutledge
7 min read
--
This post announces native PyTorch backends for vLLM and SGLang powered by TorchTPU, enabling developers to run and optimize serving workloads using familiar PyTorch tools. Built in collaboration with open-source project maintainers, these engines include optimized TPU kernel implementations, production-ready features like KV cache management with TPU Sync, and disaggregated serving. Public benchmarks demonstrate state-of-the-art performance and industry-leading cost efficiency on Ironwood TPU across long-context and agentic workloads.
Qi Zhou, Chris Chan
7 min read
Includes Code
--
Google AI Edge Model Explorer now supports custom compiler intermediate representations and open-source adapters like TOSA, VGF, and PTE for on-device machine learning. Developers can leverage hierarchical graph visualization, split-pane views, and per-node trace overlays to catch export regressions and verify hardware delegation. These tools enable engineering teams to transparently debug execution graphs and pinpoint cycle-level latency bottlenecks locally.
Eric Yang, Alan Ma
4 min read
Includes Code
--
TorchTPU integrates a native torch.device("tpu") backend directly into PyTorch, allowing developers to train large-scale models on Google Cloud TPUs without rewriting code in JAX. The TorchTitan TPU fork enhances this with production-grade scaling primitives like SimpleFSDP, Graph Trainer, and custom Pallas kernels that achieve high Model Flops Utilization (MFU) across dense and MoE architectures. Developers can leverage these battle-tested optimization playbooks, recipes, and the Hugging Face modeling backend to efficiently scale their own workloads on TPU infrastructure.
Aleksey Vlasenko, Rob Torres, Will Constable
22 min read
Includes Code
--
NVIDIA CUDA Tile (cuTile) is a Python-based DSL and parallel programming model for NVIDIA GPUs that automatically leverages advanced hardware capabilities to…
Pinterest Engineering
6 min read
--
How Stripe scaled fraud detection and improved it with rapid agent-assisted experimentation for the Radar product.
Ryan Drapeau
10 min read
Includes Code
--
Meet the LiteRT Tensor API: giving developers native C++ control over the model graph to unify end-to-end ML computations on heterogeneous edge hardware.
Ping Yu, Quentin Khan, Chintan Parikh, Lu Wang
10 min read
Includes Code
--
Bring coding agents, model customization, and CUDA-accelerated development to your everyday PC…
Spotify Engineering and NVIDIA Engineering
8 min read
--
Meta launches Muse Spark Open, a coding-optimized 816B mixture-of-experts (MoE) model with 42B active parameters and a 1M token context window.
Tanya Lenz
5 min read
--
Observatories and telescopes, lasers and X-ray light sources, and other high-throughput instruments generate image data faster than CPU-bound pipelines can…
Bitwise determinism makes large-scale pretraining easier to debug, validate, and resume reproducibly. These benefits become especially valuable when training…
Ashwath Aithal
10 min read
Includes Code
--
Executable UDFs are now generally available on ClickHouse Cloud across AWS, GCP, and Azure, with native runtimes, network access, observability, and API and Terraform support.
Automatic speech recognition must handle how people actually speak, not only the languages and styles that dominate pretraining data.
Elizabeth Goodman
13 min read
Includes Code
--
Generative recommender (GR) systems are emerging as a powerful new approach for large-scale personalization. Instead of treating recommendation as a set of…
Tanya Lenz
10 min read
Includes Code
--
As language models grow, scaling dense architectures becomes increasingly expensive. In a dense transformer, every token passes through every layer…
Michelle Horton
6 min read
Includes Code
--
The MaxText team successfully reproduced AI2’s OLMo 3 7B language model from scratch on Google Cloud TPUs using JAX/XLA, precisely matching the original PyTorch-on-GPU reference across pre-training and mid-training stages on all held-out evaluations. The implementation achieved up to 57.4% Model Flops Utilization (MFU) and demonstrated robust infrastructure portability by surviving mid-run cluster resizes and cross-generation TPU shifts without requiring recipe alterations. Crucially, the exercise proved the necessity of comprehensive held-out validation by catching a silent data-loader memorization bug that artificially depressed training loss and would have otherwise faked a performance win.
Gagik Amirkhanyan, Ran Ran, Aireen Mei, Matt Davidow, TPU Inference Software Engineering Team, Google Cloud, AI2 Team
26 min read
Includes Code
--
As large language model (LLM) inference increasingly processes sensitive information and proprietary model context across personal, enterprise…
Tanya Lenz
6 min read
Includes Code
--
The compute and memory demands of generative AI increasingly exceed what a single GPU can provide. NVIDIA TensorRT multi-device inference is a new capability…
Weather-sensitive industries increasingly have access to observations that offer an earlier, more local view of changing conditions.
cuTile Rust () is a tile-based system for safe, idiomatic GPU kernel authoring in the Rust programming language. Extending the Rust ownership model to tile…
Tanya Lenz
18 min read
Includes Code
--
Biomolecular structure prediction is now often run at proteome scale, where the goal is to move an entire worklist through the pipeline efficiently.
Elizabeth Goodman
10 min read
Includes Code
--
Every NVIDIA CUDA Toolkit release adds functionality and performance improvements that help developers get more from NVIDIA GPUs and the broader NVIDIA software…
Deploy an Open Model from Checkpoint to Inference in Two Commands with NVIDIA TensorRT Model Connect
Open AI models are evolving faster than ever, but bringing them into native applications can still require model-specific conversion, preprocessing…
Tanya Lenz
6 min read
Includes Code
--
Alibaba released the model weights for Qwen3.8-Flash-Next as a preview of the upcoming Qwen4 architecture for developers to experiment with and evaluate.
Michelle Horton
3 min read
--
For years, a Python developer who needed a GPU had two realistic choices: Learn NVIDIA CUDA C++ well enough to write an extension, set up a build toolchain…
Elizabeth Goodman
12 min read
Includes Code
--
OpenAI Team
6 min read
Includes Code
--
MTIA 300 is the first of Meta’s family of in-house training and inference accelerators optimized for training ranking and recommendation models. We’re sharing how MTIA 300’s built-in NIC chiplets a…
Rajiv Krishnamurthy
6 min read
--
Use AdaptGrow, a GPU-accelerated matrix factorization algorithm, to turn rolling correlation and tail-dependence matrices into hard clusters…
Elizabeth Goodman
12 min read
Includes Code
--
Recommender systems (RecSys) are one of the most ubiquitous machine learning problems in the consumer internet industry yet notoriously difficult to train and…
Elizabeth Goodman
11 min read
Includes Code
--
NVIDIA Holoscan is a platform for building real-time AI applications at the edge, from medical imaging to robotics. HoloHub is its companion repository: a…
Elizabeth Goodman
10 min read
Includes Code
--
Modern vision-language models (VLMs) can support tasks such as visual question answering, captioning, and image-text reasoning. In practice, however…
Tanya Lenz
7 min read
Includes Code
--
Atomistic simulation requires three things: knowledge of the science, compute-efficient implementation of simulations, and accessible interfaces to the…
Elizabeth Goodman
11 min read
Includes Code
--
Teams customize their models to hit their targets for latency, speed, memory, and compute. With the open NVIDIA Nemotron family of models…
Tanya Lenz
18 min read
Includes Code
--
HeyGen ported their 18B+ parameter Avatar IV video generation model to Google Cloud's Trillium (v6e) TPUs via torchax and XLA, utilizing FSDP and Ulysses sequence parallelism across an eight-chip mesh. To achieve a 1.86x speedup for real-time streaming, the engineering team pipelined exposed all-to-all collectives, aligned sparse attention block sizes to eliminate mask padding, and bypassed softmax serial dependencies using a precomputed Cauchy-Schwarz upper bound. These custom Pallas kernel and compiler optimizations were deployed only after passing rigorous two-tier quality gates to guarantee byte-identical or mathematically equivalent pixel outputs.
13 min read
Includes Code
--
Alibaba released the open weights for Qwen3.8-2.4T-A95B (Qwen3.8-Max), its largest open-weight model, bringing near-frontier capabilities to the open ecosystem.
Michelle Horton
4 min read
--
Morteza Ramezani
17 min read
--
Meta’s Generative Ads Recommendation Model (GEM), the foundation model behind ads recommendations across Instagram and Facebook, now trains at LLM scale on several thousand of the latest-gene…
Darren Liu
24 min read
--