Today, diffusion models have become one of the most influential families of generative AI systems. They power applications ranging from image synthesis and editing to video generation, scientific discovery, and multimodal content creation.
Despite their impressive capabilities, the fundamental idea behind these models is surprisingly simple. They begin with pure noise and gradually transform it into realistic data.
That simple description immediately raises a deeper question. If the model starts from nothing more than random noise, how does it know where to move at each step? What tells it whether a tiny change makes an image more realistic or pushes it farther away from the data distribution?
In 2019, Yang Song and Stefano Ermon proposed a new perspective that answered this question in an elegant and mathematically principled way. Rather than learning to represent the entire data distribution directly, their framework focused on learning local guidance that can steer noisy samples toward realistic ones.
This seemingly modest shift in viewpoint became one of the key conceptual foundations of modern score-based generative modeling and strongly influenced the evolution of diffusion models that followed.
The infographic below illustrates the conceptual shift that transformed diffusion models. Instead of viewing generation as the difficult task of reversing noise itself, it shows how Yang Song and Stefano Ermon reframed the problem as learning the score, a local direction that points toward more realistic data.
Following this intuition, the infographic walks through the motivation, the challenges of naïve score modeling, the introduction of Noise Conditional Score Networks (NCSNs), and the role of annealed Langevin dynamics, revealing how a sequence of small directional updates can gradually transform pure noise into realistic images.
Paper Overview
Generative Modeling by Estimating Gradients of the Data Distribution (2019) introduced score-based generative modeling, a new paradigm that learns the score of the data distribution rather than the distribution itself.
By reformulating generative modeling around score estimation, the paper provided a principled alternative to both likelihood-based models and GANs, combining flexible architectures with stable optimization and a tractable learning objective.
Its ideas laid the foundation for modern score-based generative models and played a central role in the emergence of today's diffusion models.
Here's a quick infographic of what we'll cover throughout this review, highlighting the paper's core ideas, methodology, and lasting impact.
Table of Contents:
Abstract
This paper introduces a new paradigm for generative modeling that shifts the learning objective away from modeling the data distribution itself. Instead, it learns the score, the gradient of the log data density, which indicates the local direction toward regions of higher probability. Once this score field is learned, new samples can be generated by starting from random noise and iteratively following these learned directions through Langevin dynamics.
The authors show, though, that this seemingly simple idea breaks down when applied directly to real-world data. Natural images are widely believed to lie on low-dimensional manifolds embedded in high-dimensional space, making the score ill-defined outside the data manifold.
At the same time, accurately estimating the score in low-density regions is particularly difficult because training data are scarce there, even though these are precisely the regions where the sampling process begins. Together, these challenges prevent naïve score-based generative modeling from producing reliable samples.
To overcome these limitations, the paper proposes perturbing the data with multiple levels of Gaussian noise, which spreads the data beyond the low-dimensional manifold into the surrounding ambient space, enriching the training distribution and providing the neural network with informative learning signals across regions that were previously sparsely populated. The model then learns the score of every resulting distribution using a single Noise Conditional Score Network (NCSN).
During sampling, the model employs annealed Langevin dynamics, beginning from heavily perturbed samples and progressively reducing the noise level. At each stage, the corresponding score estimate guides the samples toward increasingly realistic regions of the data distribution, eventually recovering high-quality data.
One of the strengths of this framework is its conceptual simplicity. It avoids adversarial training, doesn't require sampling during optimization, places no restrictive constraints on the network architecture, and provides a tractable training objective that enables meaningful quantitative comparisons between models.
Experiments on MNIST, CelebA, and CIFAR-10 demonstrate that the proposed method produces samples competitive with contemporary GANs and likelihood-based models, achieving a state-of-the-art Inception Score of 8.87 on CIFAR-10 at the time of publication.
Beyond image generation, the learned score representations also enable effective image inpainting, suggesting that the model captures rich structural information about the underlying data distribution.
Introduction
Generative models have become one of the central areas of modern machine learning, enabling systems that can synthesize realistic images, generate speech and music, improve semi-supervised learning, detect anomalies, imitate expert behavior, and support exploration in reinforcement learning.
Over the years, two major paradigms have dominated generative modeling: likelihood-based models and Generative Adversarial Networks (GANs). Both have achieved remarkable success, yet each comes with fundamental trade-offs.
Likelihood-based models often require restrictive architectures or expensive approximations, while GANs rely on unstable adversarial training. As a result, neither provides a unified framework that combines high-quality generation, stable optimization, architectural flexibility, and a tractable learning objective.
This gap ultimately motivated the development of score-based generative modeling by Yang Song and Stefano Ermon.
This paper begins by questioning whether those trade-offs are actually necessary. Instead of designing yet another variation of existing generative models, the authors introduce a fundamentally different perspective on the problem.
Their key insight is that, unlike likelihood-based models, high-quality generation doesn't require learning the data distribution directly. Instead, it's sufficient to learn the score, the gradient of the log data density, which tells the model the local direction to move toward regions where realistic data are more likely to exist.
Building on this idea, the paper develops a complete score-based generative modeling framework that combines a principled learning objective with an efficient sampling procedure. Along the way, the authors identify the theoretical and practical challenges that arise when applying this idea to real-world datasets and propose a series of solutions that make the framework both stable and scalable.
The result is a new generation paradigm that avoids adversarial optimization, doesn't require restrictive probabilistic models, and provides a tractable objective for training and evaluation.
More importantly, the ideas introduced here became the conceptual foundation for the score-based diffusion models that would rapidly reshape generative AI in the years that followed.
Before diving more into the paper, it's helpful to understand the broader landscape that motivated this work. The infographic below contrasts the two dominant paradigms that shaped generative modeling before this paper and highlights the gap that neither could fully address. It also introduces the central objective of the paper: finding a practical framework that combines expressive generation, stable optimization, and a meaningful training objective without forcing a compromise between them.
The left side summarizes the limitations of likelihood-based models, which rely on restrictive modeling assumptions or surrogate optimization objectives.
The right side highlights the strengths and weaknesses of GANs, whose adversarial training often produces realistic samples but can be unstable and difficult to evaluate quantitatively.
At the center, the infographic illustrates the conceptual gap between these two approaches and introduces the score-based perspective proposed in this paper as an alternative that aims to combine flexibility, stability, and tractable optimization within a single framework.
2. Score-Based Generative Modeling
At the heart of this paper is a simple but powerful change in perspective. Traditional generative models attempt to learn the data distribution itself, a task that is often mathematically intractable or computationally restrictive.
The authors instead propose learning its score, defined as the gradient of the log-density. Rather than estimating how likely every point is, the model learns the local direction that points toward regions where the data become more probable.
This viewpoint transforms generative modeling into a score estimation problem. A neural network is trained through score matching to approximate the score function directly from data. Once this vector field has been learned, new samples can be generated by starting from random noise and repeatedly following these learned directions using Langevin dynamics.
The framework therefore separates naturally into two complementary stages: learning the score field during training and using that learned field to guide sampling during inference.
The abstract definition of the score can initially seem unintuitive because it replaces probabilities with gradients.
The infographic below builds intuition by comparing the data distribution to a mountain landscape. Instead of measuring the height of every location, the model only needs to learn which direction points uphill.
This simple analogy captures the central insight behind score-based generative modeling and explains why learning gradients can be considerably more practical than modeling the entire probability distribution.
The left side of the infographic illustrates the traditional objective of estimating the log-density landscape, a task that becomes impractical because computing the normalization constant is generally intractable.
The center panel introduces the score function as a vector field whose arrows always point toward regions of higher probability, allowing the model to navigate the distribution without explicitly evaluating its density.
The right side connects this intuition to score matching, where a neural network is trained to predict these directions directly. Once the score field has been learned, Langevin dynamics follows the predicted vectors step by step, gradually moving random noise toward realistic data samples.
2.1 Score Matching for Score Estimation
Once the score function has been identified as the quantity of interest, the next challenge is learning it directly from data.
Score matching provides exactly this capability. Rather than estimating the probability density and differentiating it afterward, the method trains a neural network to approximate the score function itself. In doing so, it avoids explicit density estimation while still recovering the information required to generate new samples through Langevin dynamics.
A practical advantage of the formulation adopted in this paper is that the score is modeled directly instead of being constrained to the gradient of an energy-based model. This design eliminates the need for expensive higher-order derivatives during optimization, making the learning procedure considerably more efficient. Under mild regularity conditions, minimizing the score matching objective provably recovers the true score function.
Despite its elegant theoretical foundation, the original score matching objective doesn't scale well to modern deep neural networks. Its optimization requires computing the trace of the Jacobian of the score network, an operation whose computational cost grows rapidly with the dimensionality of the data. For high-resolution images and deep architectures, this quickly becomes impractical.
Addressing this computational bottleneck is one of the paper's next major steps and motivates the scalable score matching methods introduced in the following section.
2.2 Denoising and Sliced Score Matching
The original score matching objective provides an elegant way to learn the score function, but its computational cost makes it impractical for modern deep learning.
To overcome this limitation, the authors discuss two scalable alternatives that preserve the central idea of learning the score while avoiding the expensive Jacobian trace computation. Although both methods optimize different objectives, they ultimately seek the same goal: estimating the score function without explicitly modeling the underlying probability density.
Denoising Score Matching (DSM) perturbs each training sample with Gaussian noise and trains the network to predict the score of the resulting noisy distribution. This reformulation removes the need to compute the Jacobian trace, making optimization significantly simpler and more scalable. As the noise level becomes sufficiently small, the learned score approaches the score of the original data distribution, providing an efficient approximation that performs well in practice.
Sliced Score Matching (SSM) addresses the same computational challenge from a different perspective. Instead of adding noise, it estimates the Jacobian trace using random projections computed through forward-mode automatic differentiation.
This produces an unbiased estimate of the original score matching objective while avoiding its full computational cost. But it remains substantially more expensive than DSM, requiring roughly four times more computation, which makes DSM the preferred choice throughout the rest of the paper.
Both methods are designed to solve the same problem but take very different routes to reach it. The infographic below compares their training objectives, computational requirements, and practical trade-offs, illustrating why Denoising Score Matching ultimately became the primary training strategy adopted in this work.
The left side illustrates Sliced Score Matching, where random projection directions are used to approximate the expensive Jacobian trace, preserving the original objective at a higher computational cost.
The right side presents Denoising Score Matching, which instead perturbs data with Gaussian noise and trains the network to predict the corresponding score of the noisy distribution.
The comparison at the center highlights the key distinction between the two approaches. SSM provides an unbiased estimate of the original objective but requires considerably more computation, whereas DSM offers a much simpler and more scalable optimization procedure.
Despite these differences, both methods learn the same underlying score function and eliminate the need to compute the data density explicitly.
2.3 Sampling with Langevin Dynamics
Learning the score function is only one half of the framework. The remaining challenge is to use that learned information to generate new samples. Langevin dynamics provides this missing link by transforming the estimated score field into a practical sampling procedure.
Starting from a random initialization, Langevin dynamics repeatedly follows the estimated score while injecting a small amount of Gaussian noise at every iteration. The score guides the sample toward regions of higher probability, whereas the injected noise encourages exploration and prevents the trajectory from becoming trapped in poor local regions.
Together, these updates gradually reshape random noise into samples that resemble the underlying data distribution.
From a theoretical perspective, Langevin dynamics converges to the target distribution in the limit of infinitesimally small step sizes and infinitely many iterations, provided the score function is estimated accurately.
In practice, these ideal conditions can't be achieved, so the paper assumes that sufficiently small step sizes and enough iterations provide an adequate approximation for sampling.
Score matching and Langevin dynamics therefore play complementary roles within the framework. The first learns the vector field that describes how samples should move, while the second follows that learned field to synthesize new data. Together, they establish the core principle of score-based generative modeling on which the remainder of the paper is built.
3. Challenges of Score-Based Generative Modeling
Up to this point, the paper has established a compelling framework: learn the score function through score matching and use Langevin dynamics to generate new samples.
At first glance, this appears to provide a complete solution to generative modeling. But the authors show that applying this framework directly to real-world data leads to unexpected difficulties.
Before introducing their proposed solution, the paper examines the two fundamental challenges that prevent naïve score-based generative modeling from working reliably in practice. Understanding these limitations is essential because they directly motivate the design of Noise Conditional Score Networks and Annealed Langevin Dynamics, the two key innovations introduced in the remainder of the paper.
3.1 The Manifold Hypothesis
The first obstacle arises from a mismatch between the assumptions behind score matching and the structure of real-world data.
Classical score matching assumes that the data distribution has full support over the entire ambient space, ensuring that the score is well-defined everywhere.
Real images, however, don't satisfy this assumption. Instead, they're widely believed to lie on low-dimensional manifolds embedded within a much higher-dimensional space.
This creates a fundamental difficulty for score-based generative modeling. Since the score is defined as the gradient of the log-density in the ambient space, it becomes undefined outside the data manifold, where the probability density is effectively zero. As a result, the theoretical guarantees of score matching no longer hold, and directly learning the score from unperturbed data can produce unstable and inconsistent estimates.
To demonstrate this issue, the authors train a sliced score matching model directly on CIFAR-10 images. The optimization fails to converge, with the training loss fluctuating throughout learning.
They then repeat the experiment after perturbing the data with an almost imperceptible amount of Gaussian noise. This small perturbation spreads the data distribution across the ambient space, restoring full support and making the score well-defined everywhere. Under these conditions, training becomes stable and converges smoothly.
This experiment provides one of the paper's most important insights. Adding even a tiny amount of Gaussian noise isn't merely a numerical trick. It restores the mathematical assumptions required by score matching. This observation becomes the foundation for the noise-conditioned framework introduced in the following sections.
The manifold hypothesis is an abstract concept that can be difficult to visualize. The infographic below illustrates why score matching fails when data occupy only a thin surface within a high-dimensional space and shows how a small amount of Gaussian noise restores the conditions needed for stable learning.
The left side illustrates the manifold hypothesis, where real images occupy only a small, low-dimensional surface embedded in a much larger ambient space. Because the score is undefined away from this surface, directly applying score matching produces unstable optimization, as shown by the fluctuating training loss in the upper-right panel.
The lower half demonstrates the key observation of the paper: adding a tiny amount of Gaussian noise spreads the data distribution beyond the manifold, giving it full support throughout the ambient space. This restores the validity of score matching, leading to stable convergence and laying the mathematical foundation for Noise Conditional Score Networks.
3.2 Low-Density Regions
The lack of training data in low-density regions makes both score estimation through score matching and sampling via Langevin dynamics significantly more challenging.
3.2.1 Inaccurate score estimation with score matching
Even after resolving the manifold issue, score estimation remains difficult in another critical part of the data space: low-density regions. These areas contain few or no training samples, meaning the model receives little supervision where the probability density is extremely small. As a result, the learned score can become unreliable precisely where accurate guidance is most needed.
The authors illustrate this limitation using a simple mixture of Gaussians. The learned score closely matches the true score around the high-density modes, where training data are abundant.
Between these modes, though, the estimation quality deteriorates because the model has little information from which to infer the correct gradient. These poorly estimated regions become particularly problematic during sampling, since Langevin dynamics typically begins far from the data manifold and must traverse these low-density areas before reaching realistic samples.
This observation reveals that accurate score estimation near the data alone isn't sufficient. For score-based generative modeling to succeed, the model must learn reliable gradients throughout the entire sampling trajectory, including regions where little or no data are observed.
Addressing this challenge becomes one of the primary motivations for the noise-conditioned framework introduced later in the paper.
3.2.2 Slow Mixing of Langevin Dynamics
Even with an accurately estimated score, sampling remains difficult when the data distribution contains multiple well-separated modes. The reason is that the score provides only local information about the direction of increasing probability. It tells the sampler how to move within a mode, but it doesn't reveal the relative probability mass of distant modes separated by large low-density regions.
As a result, Langevin dynamics may struggle to move between modes and can produce samples with incorrect mixture proportions. When the modes are completely disconnected, the score inside one mode contains no information about the existence or weight of the others. Even when the modes are weakly connected, transitions across the intervening low-density regions become exceedingly rare, requiring very small step sizes and many iterations before the sampler approaches the correct stationary distribution.
The paper illustrates this behavior using a Gaussian mixture example. Even when Langevin dynamics is given the exact score function, it fails to recover the true proportion of samples assigned to each mode. This experiment demonstrates that the limitation isn't caused by inaccurate score estimation alone. Instead, it reflects an inherent slow-mixing problem that arises whenever sampling must traverse large low-density regions.
The previous section showed that score estimation becomes unreliable in regions with little training data. The infographic below takes the next step by explaining how this limitation affects sampling. Using the analogy of isolated islands separated by a vast ocean, it illustrates why local gradient information alone is insufficient to recover the correct balance between distant modes.
The infographic compares high-density modes to islands separated by wide low-density regions. Near each mode, the score field accurately points toward higher probability, but it provides no information about the relative importance of distant modes. Consequently, Langevin dynamics can become trapped within a single region and transition only rarely across the low-density "deserts."
The Gaussian mixture example demonstrates this effect: even with the true score function, the sampler fails to reproduce the correct mixture proportions. This observation motivates the need for a sampling strategy that can reliably explore the entire distribution rather than relying solely on local gradients.
4. The Proposed Solutions:
The two challenges discussed in the previous section point to the same conclusion: a score function learned only on the original data distribution is insufficient for reliable generative modeling.
The authors address both limitations through a unified strategy based on learning scores across multiple levels of Gaussian noise.
4.1 Noise Conditional Score Networks
Adding Gaussian noise fundamentally changes the geometry of the data distribution. Even a small amount of perturbation gives the distribution full support over the ambient space, restoring the mathematical assumptions required for score matching.
As the noise level increases, previously empty low-density regions become populated with training samples, enabling the model to learn meaningful score estimates throughout the entire space rather than only near the data manifold.
To leverage this idea efficiently, the authors introduce Noise Conditional Score Networks (NCSNs). Instead of training a separate model for every perturbed distribution, a single neural network is conditioned on the noise level and learns the corresponding score function across the entire noise spectrum, from heavily corrupted samples that are easy to model to lightly perturbed samples that closely resemble the original data.
Together, these learned score fields form a hierarchy that progressively bridges simple noisy distributions and the true data distribution.
For image generation, the paper adopts a U-Net-style architecture with dilated convolutions to combine dense prediction with a large receptive field. The network is conditioned on the current noise level through conditional instance normalization, allowing the same model to adapt its internal representations and predict the appropriate score for each level of Gaussian perturbation.
During sampling, generation begins from the score field associated with the highest noise level, where exploration is easier, and gradually transitions toward lower noise levels as the sample becomes increasingly structured.
Because consecutive noise levels define similar distributions, each stage naturally initializes the next, allowing the model to refine coarse structure into realistic images. This progressive sampling strategy is formalized as Annealed Langevin Dynamics.
The infographic below summarizes this complete framework. It illustrates how a sequence of progressively perturbed images defines multiple training distributions, how a single NCSN learns the corresponding score field for every noise level, and how the weighted denoising score matching objective, network architecture, and mini-batch training procedure work together to learn reliable gradients across the entire noise spectrum before gradually guiding random noise toward the true data distribution.
4.2 Learning NCSNs via Score Matching
The authors train Noise Conditional Score Networks (NCSNs) using denoising score matching, although they report that sliced score matching achieves comparable performance.
For each Gaussian noise level, the network learns the score of the corresponding perturbed distribution through a separate denoising objective. These objectives are then combined into a single weighted loss, enabling one network to estimate the score across the entire sequence of noise levels simultaneously.
Because the overall objective is simply the weighted sum of the individual losses, optimizing it recovers the correct score function for every perturbed distribution.
To balance the contributions of different noise levels during training, each objective is weighted by the square of its corresponding noise standard deviation, σ². This weighting prevents large-noise distributions from dominating the optimization while ensuring that lightly perturbed samples remain influential.
The resulting objective is straightforward to optimize, scales naturally to deep neural networks, and provides a tractable loss that can be used for quantitative model comparison.
4.3 NCSN Inference via Annealed Langevin Dynamics
Once the Noise Conditional Score Network has learned the score at every noise level, new samples are generated using annealed Langevin dynamics.
Rather than attempting to sample directly from the nearly noise-free data distribution, the algorithm begins with pure Gaussian noise and progressively moves through a sequence of decreasing noise levels. At each stage, Langevin dynamics uses the score corresponding to the current noise level to refine the sample before passing it to the next stage. As the noise gradually decreases, the sample evolves from a coarse random pattern into a realistic data point.
This progressive strategy makes sampling substantially more reliable than applying Langevin dynamics only at the final noise level. High-noise distributions are smoother and easier to explore, allowing the sampler to move freely across different modes before gradually focusing on finer details. Because neighboring noise levels define similar distributions, each stage provides a strong initialization for the next, enabling a smooth transition from global exploration to accurate reconstruction.
To maintain stable updates throughout the process, the step size is scaled by the square of the current noise level, keeping the signal-to-noise ratio approximately constant across the entire annealing schedule.
The paper demonstrates this advantage on a Gaussian mixture model. While standard Langevin dynamics struggles to recover the correct proportions of different modes, annealed Langevin dynamics successfully preserves the true distribution by allowing exploration at high noise before progressively refining the samples as the noise decreases.
The following table highlights the key differences between standard Langevin dynamics and the annealed version proposed in this paper, explaining why annealing is essential for reliable score-based generation.
And the following infographic illustrates how annealed Langevin dynamics transforms pure noise into realistic samples. It walks through the complete inference pipeline, showing how sampling progresses across multiple noise levels, why beginning with highly perturbed distributions improves exploration, and how gradual denoising allows the model to recover accurate mode proportions while refining image details at every stage.
The infographic begins with the high-level intuition, where a random noise image is progressively sharpened as the noise level decreases. It then presents the annealed Langevin dynamics algorithm and shows how each noise level performs several Langevin updates before passing the sample to the next, less noisy distribution.
The center panels explain why this gradual schedule avoids the poor mixing behavior of standard Langevin dynamics, while the lower panels demonstrate how coarse global structure emerges first and fine visual details appear only during the final denoising stages.
Together, these illustrations show why annealing converts a difficult sampling problem into a sequence of much easier ones, making score-based generation both stable and effective.
4.4 End-to-End Architecture Overview
By this point, we've discussed the individual pieces of the framework. We've examined how the model learns score functions across multiple noise levels and how those learned scores are later used to generate new samples. The next step is to view these components as a single, unified pipeline.
The infographic below summarizes the complete architecture proposed in the paper, following both the training and inference workflows from beginning to end.
It shows how a real image is perturbed with Gaussian noise, how the Noise Conditional Score Network learns the corresponding score function for each noise level, and how those learned scores are later reused by annealed Langevin dynamics to transform pure Gaussian noise into realistic images.
One of the most elegant aspects of the framework is the clear separation between learning and generation. During training, the network never attempts to synthesize images directly. Instead, it learns a family of score functions, each associated with a different level of noise. During inference, those learned score estimates become the only guidance required for sampling, allowing annealed Langevin dynamics to progressively remove noise until a realistic sample emerges.
The entire generation process therefore relies on the same score field learned during training, resulting in a simple and coherent end-to-end generative model.
The left side of the diagram illustrates the data flow during training. A real image is perturbed with a selected Gaussian noise level before being passed to the Noise Conditional Score Network, which predicts the corresponding score vector field. The predicted score is then compared with the denoising score-matching target, and the network parameters are updated through the weighted training objective.
The right side shows the inference procedure. Generation begins from pure Gaussian noise rather than a real image. Annealed Langevin dynamics repeatedly applies the learned score estimates while gradually decreasing the noise level, refining the sample over multiple stages until it reaches the final data distribution.
Together, these two workflows demonstrate how the same learned score function connects training and sampling into a single, unified generative framework.
5. Experiments
The experiments evaluate whether the proposed framework can translate its theoretical advantages into practical generative performance. Beyond measuring image quality, the authors investigate whether the combination of Noise Conditional Score Networks (NCSNs) and annealed Langevin dynamics successfully addresses the challenges identified earlier, producing stable training, reliable sampling, and competitive image generation across multiple datasets.
The evaluation is conducted on MNIST, CelebA, and CIFAR-10 using the multi-noise training strategy introduced in the paper. The authors assess both qualitative and quantitative performance, examining generated samples, intermediate denoising trajectories, image inpainting, nearest-neighbor retrieval, and comparisons against contemporary likelihood-based models and GANs. Additional ablation studies isolate the contribution of each component, allowing the proposed training objective and sampling strategy to be evaluated independently.
The results consistently support the proposed design. Samples evolve smoothly from pure noise into realistic images, while nearest-neighbor analyses indicate that the model learns meaningful data representations rather than memorizing the training set.
The ablation experiments further show that training with a single noise level or removing the annealing strategy substantially degrades sample quality, confirming that both multi-noise learning and annealed Langevin dynamics are essential parts of the framework.
Quantitatively, the model achieves a state-of-the-art Inception Score of 8.87 on CIFAR-10 at the time of publication and a competitive FID of 25.32, demonstrating that score-based generative modeling can compete with leading generative models without adversarial training.
The infographic below summarizes the experimental evaluation presented in the paper. It brings together the qualitative examples, quantitative benchmarks, and ablation studies to illustrate how the proposed framework performs in practice and why each component contributes to its overall success.
The figure begins by showing the complete generation process, where samples gradually evolve from pure Gaussian noise into realistic digits, faces, and natural images as the noise level decreases. It then summarizes the main experimental results across MNIST, CelebA, and CIFAR-10, highlighting competitive image quality and successful image inpainting.
The lower panels compare quantitative metrics with contemporary generative models and present ablation studies demonstrating that multi-noise training and annealed Langevin dynamics are both necessary for stable, high-quality generation.
Together, these results provide empirical evidence that the proposed framework is effective at both learning meaningful score representations and generating realistic samples.
Image Inpainting
Beyond unconditional image generation, the authors demonstrate that Noise Conditional Score Networks (NCSNs) can also perform image inpainting.
By slightly modifying annealed Langevin dynamics, the model reconstructs arbitrarily shaped missing regions while preserving the observed pixels throughout the sampling process.
Unlike autoregressive approaches such as PixelCNN, which generate images in a fixed raster-scan order, NCSNs naturally handle irregular masks without requiring a predefined generation sequence.
These results show that the learned score field captures sufficient structural information about the data distribution to support both realistic image synthesis and flexible image restoration.
From Raw Data to Final Results
By this point, the paper has introduced the complete score-based generative framework and demonstrated that it works in practice. Before moving to the concluding discussion, it's useful to step back and view the entire experimental pipeline as a single workflow, from data preparation to the final generated results.
The infographic below summarizes the implementation pipeline used throughout the paper. Rather than focusing on the internal operations of the network, it follows the flow of the data itself: benchmark datasets are prepared, multiple Gaussian noise levels are constructed, the model is trained with denoising score matching, and the learned score functions are finally used by annealed Langevin dynamics to generate and restore images. Viewing the process end to end helps connect the individual components into one coherent training and inference pipeline.
The workflow begins with the three benchmark datasets used throughout the paper: MNIST, CelebA, and CIFAR-10. After simple preprocessing, including pixel normalization and data augmentation where appropriate, a geometric sequence of Gaussian noise levels is constructed to create the perturbed training distributions. The model is then trained using denoising score matching with the weighted objective introduced earlier.
Once training is complete, the learned score functions are evaluated using quantitative metrics such as Inception Score and FID, alongside qualitative analyses including progressive denoising, nearest-neighbor retrieval, and image inpainting.
The final stage illustrates the outputs produced by the framework, demonstrating how the same learned score field supports both unconditional image generation and image restoration.
6. Related Work
The authors position Noise Conditional Score Networks (NCSNs) within the broader family of Markov chain-based generative models while highlighting the conceptual shift introduced by score-based learning.
Many existing approaches either optimize likelihood-based objectives or rely on expensive Markov chain simulation during training. In contrast, NCSNs learn the score function directly through score matching and postpone sampling entirely to inference, eliminating the need for iterative sampling during optimization.
This separation between learning and sampling provides greater flexibility. Different score estimation objectives can be paired with different gradient-based sampling algorithms without changing the underlying framework, allowing the training procedure and inference algorithm to evolve independently.
The authors also note that this formulation naturally extends to energy-based models by learning their score functions directly rather than requiring explicit likelihood estimation.
The paper further distinguishes NCSNs from earlier score matching, contrastive divergence, and transition-operator methods. Although these approaches also rely on gradients or Markov chains, many require computationally expensive sampling during training or were developed for different objectives.
Likewise, while previous annealing techniques had been explored for denoising autoencoders and representation learning, the proposed annealed Langevin dynamics is designed specifically for score-based generative modeling, where it plays a central role in producing high-quality samples.
7. Legacy: Why This Paper Matters
Although this paper introduced a new method for generative modeling, its greatest contribution became clear only in the years that followed. Rather than remaining an isolated research idea, it fundamentally changed how researchers approached generation from noise.
By demonstrating that learning score functions across multiple noise levels could replace direct density estimation, it established a new direction that would soon become one of the dominant paradigms in generative AI.
The infographic below places this work in its broader historical context. It shows how the paper connects two important research threads. One originated from nonequilibrium thermodynamics and reverse diffusion, while the other introduced score-based learning through Noise Conditional Score Networks.
These ideas converged into the score-based stochastic differential equation (Score-SDE) framework, which unified diffusion models and score matching under a common mathematical formulation. In parallel, the same principles inspired Denoising Diffusion Probabilistic Models (DDPMs), providing an alternative discrete-time formulation of the same underlying process.
The infographic also highlights the paper's major technical achievements. It introduced a practical framework that combined stable optimization, scalable training, and high-quality image generation without adversarial learning. By solving the manifold and slow-mixing challenges through multi-noise training and annealed Langevin dynamics, the paper transformed score-based generative modeling from an elegant theoretical concept into a practical learning framework.
Perhaps the most important message is that modern diffusion models are best viewed as different perspectives on the same underlying idea. While DDPMs describe generation as reversing a forward noising process, score-based models learn the gradient field that guides this reverse trajectory. These formulations differ in their mathematical presentation, but they ultimately describe the same generative mechanism and were later unified through stochastic differential equations.
Today, many influential generative models trace their conceptual foundations back to the ideas introduced in this paper. Techniques such as classifier guidance, classifier-free guidance, Score-SDE models, Imagen, Stable Diffusion, and many subsequent diffusion systems all build upon the score-based principles established here. For that reason, this work is widely regarded as one of the foundational papers that shaped the modern diffusion model ecosystem.
The left side of the infographic presents the historical evolution of diffusion research, illustrating how earlier work on nonequilibrium thermodynamics and this paper's score-based formulation led to the emergence of Score-SDE and DDPM before expanding into today's diffusion ecosystem.
The right side summarizes the paper's core contributions, compares the score-based and diffusion viewpoints, and emphasizes that both frameworks describe the same generative process through different mathematical formulations.
Together, the timeline and conceptual comparison explain why this paper became a cornerstone of modern generative AI.
8. Conclusion
This paper establishes score-based generative modeling as a practical alternative to both likelihood-based models and Generative Adversarial Networks by combining score matching for learning with Langevin dynamics for sampling.
To make this framework effective on real-world data, the authors introduce Noise Conditional Score Networks (NCSNs) and annealed Langevin dynamics, overcoming the limitations of naïve score-based methods through multi-noise training and progressive sampling.
The resulting framework eliminates the need for adversarial optimization and sampling during training while remaining flexible with respect to network architecture and providing a tractable learning objective. Experiments on MNIST, CelebA, and CIFAR-10 demonstrate that these ideas translate into competitive generative performance, culminating in a state-of-the-art Inception Score of 8.87 on CIFAR-10 at the time of publication.
More importantly, the significance of this work extends far beyond its experimental results. By showing that learning score functions across multiple noise levels can serve as the foundation of a scalable generative model, the paper introduced the core principles that would later evolve into modern score-based diffusion models and influence much of today's generative AI research.
9. Beyond This Paper: The Evolution of Diffusion Models
This review has focused on the 2019 paper by Song and Ermon, but its story doesn't end there. The framework introduced here became one of the defining turning points in generative modeling, influencing a rapid sequence of advances that reshaped the field over the following years. What began as a method for learning score functions across multiple noise levels ultimately evolved into the family of diffusion models that now powers many of today's most capable generative AI systems.
The timeline below places this paper within that broader historical progression. It begins with the physics-inspired work on nonequilibrium thermodynamics in 2015, continues through the introduction of Noise Conditional Score Networks in 2019, and follows the major milestones that established diffusion modeling as a practical and scalable paradigm. These include DDPM, DDIM, Score-SDE, Improved DDPM, classifier and classifier-free guidance, latent diffusion, and the emergence of large-scale text-to-image models such as Imagen and DALL·E 2.
Rather than representing isolated breakthroughs, these papers form a continuous research trajectory in which each generation addressed a different limitation of the previous one. Early work established the theoretical foundations, this paper demonstrated how score-based learning could be made practical, later research unified different formulations under a common mathematical framework, and subsequent advances focused on improving sampling speed, image quality, controllability, and scalability.
Viewed as a whole, this progression illustrates how a single conceptual shift, learning gradients instead of explicit probability densities, grew into one of the most influential paradigms in modern machine learning.
Many of the techniques used by contemporary diffusion systems can be traced directly back to the principles introduced in this paper, making it one of the pivotal milestones in the history of generative AI.
10. Resources:
Minimum Probability Flow Learning (Sohl-Dickstein, Battaglino & DeWeese, 2009)
Bayesian Learning via Stochastic Gradient Langevin Dynamics (Welling & Teh, 2011)
Generalized Denoising Auto-Encoders as Generative Models (Bengio et al., 2013)
Deep Unsupervised Learning using Nonequilibrium Thermodynamics (Sohl-Dickstein et al., 2015)
Learning to Generate Samples from Noise through Infusion Training (Bordes, Honari & Vincent, 2017)
A-NICE-MC: Adversarial Training for MCMC (Song, Zhao & Ermon, 2017)
Implicit Generation and Generalization in Energy-Based Models (Du & Mordatch, 2019)
Sliced Score Matching: A Scalable Approach to Density and Score Estimation (Song et al., 2019)
Contact Me