#

PyTorch Programming Tutorials & Engineering Articles

815 PyTorch tutorials, guides, and engineering insights from NVIDIA, Meta, Google, and more

PyTorch Articles & Tutorials

Filter:
NVIDIA logo
NVIDIA
Intermediate
As AI agents tackle increasingly complex workflows, reducing time-to-train is critical for rapidly iterating on both models and the agents they power.
Farshad Ghodsian
11 min read
--
NVIDIA logo
NVIDIA
Advanced
AI applications at the edge often need to run multiple neural networks within limited compute and power budgets. NVIDIA Deep Learning Accelerator (DLA)…
Nikshay Shrivastava
10 min read
Includes Code
--
NVIDIA logo
NVIDIA
Intermediate
Autonomous vehicle (AV) developers need reasoning models that can operate within the compute and latency constraints of an onboard system.
Marco Pavone
12 min read
Includes Code
--
Hitarth Trivedi, Jay Narale, Jianjian Xie, Aerial view of a busy multi-level highway interchange with cars and bright streetlights at night., Yunfei Chen, Sanah Imani
15 min read
--
Meta logo
Meta
Advanced
Triton Low-Level Extensions (TLX), is an extension to Triton that helps engineers optimize GPU performance while preserving Triton’s productive programming model.  TLX introduces explicit war…
Hongtao Yu
20 min read
Includes Code
--
Google logo
Google
Intermediate
Pathways is a single-controller distributed training system designed to coordinate large-scale model training across multiple accelerator groups by representing computations as a global dataflow graph. Unlike traditional SPMD models like DDP and FSDP that rely on synchronous collectives, Pathways uses asynchronous dispatch and a central scheduler to maximize hardware utilization and prevent deadlocks in complex workloads. By decoupling model definition from physical hardware via virtual devices, developers can efficiently scale complex training architectures and seamlessly handle dynamic workloads like reinforcement learning post-training.
Sujeeth Jinesh, Shaurya Gupta, Luke Baumann, Monica Song, Ivan Nardini, Akanksha Gupta, Anthony Owusu, Jing Yang, Sadi Kneipp, Yu Li
13 min read
--
NVIDIA logo
NVIDIA
Advanced
During reinforcement learning (RL) post-training, training trajectories are generated on the fly by running inference with the current policy model.
Tanya Lenz
11 min read
Includes Code
--
NVIDIA logo
NVIDIA
Intermediate
As distributed AI training scales, failures across hardware and software dependencies increasingly disrupt training progress, making fast…
Elizabeth Goodman
12 min read
Includes Code
--
NVIDIA logo
NVIDIA
Advanced
Physical AI developers working with world models face two distinct challenges: adapting models to their sensors, tasks, and data, and getting those models into…
Elizabeth Goodman
9 min read
Includes Code
--
NVIDIA logo
NVIDIA
Advanced
Atomistic simulations help researchers understand how molecules and materials behave, from chemical reactions and phase transitions to mechanical failure.
Anoushka Bhutani
11 min read
Includes Code
--
Google logo
Google
Advanced
Google Research has completed the 32-bit open-source Coral NPU RISC-V architecture with the introduction of its Matrix Execution Unit, which delivers up to 1 TOPS while targeting 10 mW average power for edge inference. Alongside this architecture, the release of the v1.0 open-source IREE/MLIR compiler allows developers to take models from JAX, PyTorch, and TensorFlow straight to Coral NPU hardware without writing custom assembly. Developers can now prototype and deploy these multimodal edge applications using new hardware options like the Synaptics Astra SR200 MCU series and globally available Coralboard devkits.12
Billy Rutledge
7 min read
--
Google logo
Google
Advanced
This post announces native PyTorch backends for vLLM and SGLang powered by TorchTPU, enabling developers to run and optimize serving workloads using familiar PyTorch tools. Built in collaboration with open-source project maintainers, these engines include optimized TPU kernel implementations, production-ready features like KV cache management with TPU Sync, and disaggregated serving. Public benchmarks demonstrate state-of-the-art performance and industry-leading cost efficiency on Ironwood TPU across long-context and agentic workloads.
Qi Zhou, Chris Chan
7 min read
Includes Code
--
Google logo
Google
Advanced
Google AI Edge Model Explorer now supports custom compiler intermediate representations and open-source adapters like TOSA, VGF, and PTE for on-device machine learning. Developers can leverage hierarchical graph visualization, split-pane views, and per-node trace overlays to catch export regressions and verify hardware delegation. These tools enable engineering teams to transparently debug execution graphs and pinpoint cycle-level latency bottlenecks locally.
Eric Yang, Alan Ma
4 min read
Includes Code
--
Google logo
Google
Advanced
TorchTPU integrates a native torch.device("tpu") backend directly into PyTorch, allowing developers to train large-scale models on Google Cloud TPUs without rewriting code in JAX. The TorchTitan TPU fork enhances this with production-grade scaling primitives like SimpleFSDP, Graph Trainer, and custom Pallas kernels that achieve high Model Flops Utilization (MFU) across dense and MoE architectures. Developers can leverage these battle-tested optimization playbooks, recipes, and the Hugging Face modeling backend to efficiently scale their own workloads on TPU infrastructure.
Aleksey Vlasenko, Rob Torres, Will Constable
22 min read
Includes Code
--
NVIDIA logo
NVIDIA
Advanced
NVIDIA CUDA Tile (cuTile) is a Python-based DSL and parallel programming model for NVIDIA GPUs that automatically leverages advanced hardware capabilities to…
Tanya Lenz
14 min read
Includes Code
--
Stripe logo
Stripe
Beginner
How Stripe scaled fraud detection and improved it with rapid agent-assisted experimentation for the Radar product.
Ryan Drapeau
10 min read
Includes Code
--
Google logo
Google
Intermediate
Meet the LiteRT Tensor API: giving developers native C++ control over the model graph to unify end-to-end ML computations on heterogeneous edge hardware.
Ping Yu, Quentin Khan, Chintan Parikh, Lu Wang
10 min read
Includes Code
--
NVIDIA logo
NVIDIA
Beginner
Bring coding agents, model customization, and CUDA-accelerated development to your everyday PC…
Elizabeth Goodman
7 min read
--
NVIDIA logo
NVIDIA
Advanced
Meta launches Muse Spark Open, a coding-optimized 816B mixture-of-experts (MoE) model with 42B active parameters and a 1M token context window.
Tanya Lenz
5 min read
--
NVIDIA logo
NVIDIA
Advanced
Observatories and telescopes, lasers and X-ray light sources, and other high-throughput instruments generate image data faster than CPU-bound pipelines can…
Quynh L. Nguyen
17 min read
Includes Code
--
NVIDIA logo
NVIDIA
Advanced
Bitwise determinism makes large-scale pretraining easier to debug, validate, and resume reproducibly. These benefits become especially valuable when training…
Ashwath Aithal
10 min read
Includes Code
--
ClickHouse logo
ClickHouse
Advanced
Executable UDFs are now generally available on ClickHouse Cloud across AWS, GCP, and Azure, with native runtimes, network access, observability, and API and Terraform support.
NVIDIA logo
NVIDIA
Advanced
Automatic speech recognition must handle how people actually speak, not only the languages and styles that dominate pretraining data.
Elizabeth Goodman
13 min read
Includes Code
--
NVIDIA logo
NVIDIA
Advanced
Generative recommender (GR) systems are emerging as a powerful new approach for large-scale personalization. Instead of treating recommendation as a set of…
Tanya Lenz
10 min read
Includes Code
--
NVIDIA logo
NVIDIA
Advanced
As language models grow, scaling dense architectures becomes increasingly expensive. In a dense transformer, every token passes through every layer…
Michelle Horton
6 min read
Includes Code
--
Google logo
Google
Advanced
The MaxText team successfully reproduced AI2’s OLMo 3 7B language model from scratch on Google Cloud TPUs using JAX/XLA, precisely matching the original PyTorch-on-GPU reference across pre-training and mid-training stages on all held-out evaluations. The implementation achieved up to 57.4% Model Flops Utilization (MFU) and demonstrated robust infrastructure portability by surviving mid-run cluster resizes and cross-generation TPU shifts without requiring recipe alterations. Crucially, the exercise proved the necessity of comprehensive held-out validation by catching a silent data-loader memorization bug that artificially depressed training loss and would have otherwise faked a performance win.
Gagik Amirkhanyan, Ran Ran, Aireen Mei, Matt Davidow, TPU Inference Software Engineering Team, Google Cloud, AI2 Team
26 min read
Includes Code
--
NVIDIA logo
NVIDIA
Advanced
As large language model (LLM) inference increasingly processes sensitive information and proprietary model context across personal, enterprise…
Tanya Lenz
6 min read
Includes Code
--
NVIDIA logo
NVIDIA
Advanced
The compute and memory demands of generative AI increasingly exceed what a single GPU can provide. NVIDIA TensorRT multi-device inference is a new capability…
Tanya Lenz
6 min read
Includes Code
--
NVIDIA logo
NVIDIA
Advanced
Weather-sensitive industries increasingly have access to observations that offer an earlier, more local view of changing conditions.
Elizabeth Goodman
13 min read
Includes Code
--
NVIDIA logo
NVIDIA
Advanced
cuTile Rust () is a tile-based system for safe, idiomatic GPU kernel authoring in the Rust programming language. Extending the Rust ownership model to tile…
Tanya Lenz
18 min read
Includes Code
--
NVIDIA logo
NVIDIA
Advanced
Biomolecular structure prediction is now often run at proteome scale, where the goal is to move an entire worklist through the pipeline efficiently.
Elizabeth Goodman
10 min read
Includes Code
--
NVIDIA logo
NVIDIA
Advanced
Every NVIDIA CUDA Toolkit release adds functionality and performance improvements that help developers get more from NVIDIA GPUs and the broader NVIDIA software…
Jonathan Bentz
14 min read
Includes Code
--
NVIDIA logo
NVIDIA
Intermediate
Open AI models are evolving faster than ever, but bringing them into native applications can still require model-specific conversion, preprocessing…
Tanya Lenz
6 min read
Includes Code
--
NVIDIA logo
NVIDIA
Intermediate
Alibaba released the model weights for Qwen3.8-Flash-Next as a preview of the upcoming Qwen4 architecture for developers to experiment with and evaluate.
Michelle Horton
3 min read
--
NVIDIA logo
NVIDIA
Advanced
For years, a Python developer who needed a GPU had two realistic choices: Learn NVIDIA CUDA C++ well enough to write an extension, set up a build toolchain…
Elizabeth Goodman
12 min read
Includes Code
--
Meta logo
Meta
Advanced
MTIA 300 is the first of Meta’s family of in-house training and inference accelerators optimized for training ranking and recommendation models. We’re sharing how MTIA 300’s built-in NIC chiplets a…
Rajiv Krishnamurthy
6 min read
--
NVIDIA logo
NVIDIA
Advanced
Use AdaptGrow, a GPU-accelerated matrix factorization algorithm, to turn rolling correlation and tail-dependence matrices into hard clusters…
Elizabeth Goodman
12 min read
Includes Code
--
NVIDIA logo
NVIDIA
Advanced
Recommender systems (RecSys) are one of the most ubiquitous machine learning problems in the consumer internet industry yet notoriously difficult to train and…
Elizabeth Goodman
11 min read
Includes Code
--
NVIDIA logo
NVIDIA
Intermediate
NVIDIA Holoscan is a platform for building real-time AI applications at the edge, from medical imaging to robotics. HoloHub is its companion repository: a…
Elizabeth Goodman
10 min read
Includes Code
--
NVIDIA logo
NVIDIA
Intermediate
Modern vision-language models (VLMs) can support tasks such as visual question answering, captioning, and image-text reasoning. In practice, however…
NVIDIA logo
NVIDIA
Intermediate
Atomistic simulation requires three things: knowledge of the science, compute-efficient implementation of simulations, and accessible interfaces to the…
Elizabeth Goodman
11 min read
Includes Code
--
NVIDIA logo
NVIDIA
Intermediate
Teams customize their models to hit their targets for latency, speed, memory, and compute. With the open NVIDIA Nemotron family of models…
Tanya Lenz
18 min read
Includes Code
--
Google logo
Google
Advanced
HeyGen ported their 18B+ parameter Avatar IV video generation model to Google Cloud's Trillium (v6e) TPUs via torchax and XLA, utilizing FSDP and Ulysses sequence parallelism across an eight-chip mesh. To achieve a 1.86x speedup for real-time streaming, the engineering team pipelined exposed all-to-all collectives, aligned sparse attention block sizes to eliminate mask padding, and bypassed softmax serial dependencies using a precomputed Cauchy-Schwarz upper bound. These custom Pallas kernel and compiler optimizations were deployed only after passing rigorous two-tier quality gates to guarantee byte-identical or mathematically equivalent pixel outputs.
13 min read
Includes Code
--
NVIDIA logo
NVIDIA
Advanced
Alibaba released the open weights for Qwen3.8-2.4T-A95B (Qwen3.8-Max), its largest open-weight model, bringing near-frontier capabilities to the open ecosystem.
Michelle Horton
4 min read
--
Meta logo
Meta
Advanced
Meta’s Generative Ads Recommendation Model (GEM), the foundation model behind ads recommendations across Instagram and Facebook, now trains at LLM scale on several thousand of the latest-gene…
Darren Liu
24 min read
--