Evaluating Generated Images: FID, KID, and Practical Cautions

Evaluating Generated Images: FID, KID, and Practical Cautions#

A generative model should not be judged only by a few appealing samples. Human inspection matters, but it is subjective and unstable. A course on deep generative models therefore needs at least a basic language for quantitative evaluation. In image generation, two of the most widely discussed metrics are the Fréchet Inception Distance (FID) and the Kernel Inception Distance (KID). Both compare statistics of real and generated images after those images have been passed through a pretrained visual feature extractor, typically Inception v3.

This chapter is placed before the main model families on purpose. Once VAEs, GANs, diffusion models, and flow matching enter the picture, students should already know what it means to say that one model has better coverage, sharper samples, or a lower feature-space discrepancy. Without that vocabulary, later empirical claims remain much harder to interpret.

There is also an important methodological lesson here. In supervised learning, evaluation often feels straightforward: compare predictions with labels. In generative modeling there may be no single correct image corresponding to one latent draw, so pixelwise comparison is usually the wrong tool. We need metrics that ask whether the distribution of generated images resembles the distribution of real ones. FID and KID are attempts to operationalize exactly that question.

Why Pixelwise Error Is Usually Not Enough#

Suppose two images depict the same shoe, but one is shifted a few pixels to the left or drawn with slightly different laces. A pixelwise mean squared error can be large even though a human would judge the images as semantically very similar. This makes direct pixel distances a poor proxy for generative quality whenever multiple plausible outputs exist.

A more useful idea is to compare images in a feature space learned by a strong vision network. There, semantically similar images often lie closer together than they do in raw pixel space. FID and KID follow this philosophy. They first map both real and generated images into high-level features, and only then compare the resulting distributions.

A simple analogy helps. If one compares two paragraphs letter by letter, a synonym replacement may look like a large error even when the meaning barely changes. If one first maps each paragraph into a semantic embedding, the comparison becomes more meaningful. FID and KID do the image analogue of this move: compare representations rather than raw coordinates.

Fréchet Inception Distance#

Let \(\{\boldsymbol{f}_i^{(r)}\}_{i=1}^{n_r}\) be features extracted from real images and \(\{\boldsymbol{f}_j^{(g)}\}_{j=1}^{n_g}\) be features extracted from generated images. FID approximates each feature distribution by a Gaussian:

\[\mathcal{N}(\boldsymbol{\mu}_r, \boldsymbol{\Sigma}_r), \qquad \mathcal{N}(\boldsymbol{\mu}_g, \boldsymbol{\Sigma}_g).\]

It then computes the squared Fréchet distance between those two Gaussians:

\[\operatorname{FID} = \|\boldsymbol{\mu}_r - \boldsymbol{\mu}_g\|_2^2 + \operatorname{tr} \Big( \boldsymbol{\Sigma}_r + \boldsymbol{\Sigma}_g - 2(\boldsymbol{\Sigma}_r \boldsymbol{\Sigma}_g)^{1/2} \Big).\]

Lower values are better.

FID is attractive because it reacts to both mean mismatch and covariance mismatch in feature space. If generated images all look realistic but only cover one narrow subset of the data, the covariance term tends to worsen. If generated images are consistently shifted toward implausible or blurry features, the mean term can worsen. In this sense, FID tries to summarize both quality and diversity in a single scalar.

The price of this convenience is approximation. The true feature distribution is usually not Gaussian, and FID can be biased for small sample sizes. It is still very useful, but it should never be treated as a perfect oracle.

Numerical Example: Reading FID Intuitively

Imagine that real images produce two-dimensional features with mean \(\boldsymbol{\mu}_r = (0,0)\) and covariance \(\boldsymbol{\Sigma}_r = \boldsymbol{I}\). A generated model produces features with mean \(\boldsymbol{\mu}_g = (1,0)\) and the same covariance \(\boldsymbol{\Sigma}_g = \boldsymbol{I}\).

In this toy case, the covariance term cancels out and the FID becomes simply \(\|\boldsymbol{\mu}_r - \boldsymbol{\mu}_g\|_2^2 = 1\). So even if the generated distribution has the right spread, a systematic mean shift in feature space still creates a nonzero penalty. If the means matched but the generated covariance collapsed to a much narrower cloud, the covariance term would reveal that lack of diversity instead.

Kernel Inception Distance#

KID uses the same general idea of comparing Inception features, but it avoids the Gaussian approximation. Instead, it computes a maximum mean discrepancy style comparison between the real and generated feature distributions, usually with a polynomial kernel. In compact notation,

\[\operatorname{KID} = \operatorname{MMD}^2(\mathcal{F}_{real}, \mathcal{F}_{fake}),\]

where the underlying kernel defines how features are compared.

The practical interpretation is again simple: lower is better. The important conceptual difference is that KID compares distributions through kernel mean embeddings rather than by fitting Gaussians. In the literature, KID is often appreciated because it has an unbiased estimator under finite sampling, whereas FID can be noticeably biased on small datasets.

One should not overdramatize the difference. In real projects, researchers often report both metrics because each highlights slightly different aspects of distribution mismatch. FID is historically dominant and easy to compare against prior work. KID can be more trustworthy when sample counts are small. Reporting both is therefore a good habit in a teaching setting because it shows students that evaluation itself has uncertainty and design choices.

What These Metrics Actually Measure#

FID and KID do not directly measure truth, beauty, or usefulness. They measure closeness between feature distributions under a particular pretrained network. This has several consequences.

First, the metrics are only as meaningful as the feature extractor. If the feature representation is well aligned with the dataset semantics, the scores are often informative. If the dataset is very far from the domain on which the feature extractor was trained, interpretation becomes less reliable.

Second, a lower FID or KID does not guarantee that every single generated sample looks good. A model may improve its average feature statistics while still occasionally producing artifacts.

Third, the metrics can be manipulated by implementation details such as sample count, image resizing, preprocessing, and the exact feature layer chosen. For this reason, careful reporting matters.

For a PhD audience, the most mature position is neither blind trust nor blanket dismissal. These metrics are useful because they create a common empirical language. They are limited because they compress a complicated perceptual and distributional judgment into a scalar. The right habit is to use them together with visual inspection, task-specific analysis, and transparent reporting of how they were computed.

Computing FID and KID with torchmetrics#

In modern PyTorch workflows, a convenient implementation route is torchmetrics. The typical pattern is:

  1. Instantiate the metric object.

  2. Feed batches of real images with real=True.

  3. Feed batches of generated images with real=False.

  4. Call .compute().

The most important implementation detail is preprocessing. With the default Inception-based extractor, the metric expects three-channel RGB images. Grayscale datasets therefore need channel repetition, and images should be scaled consistently before being passed to the metric.

A second practical warning is that metric computation should also be batched. Even if one wants to evaluate on 10_000 or 50_000 images, those images should not be moved through the metric all at once. It is much safer to stream both the real and generated samples in small chunks and to expose progress with tqdm, especially in notebooks.

import torch
from torchmetrics.image.fid import FrechetInceptionDistance
from torchmetrics.image.kid import KernelInceptionDistance
from tqdm.auto import tqdm


def prepare_for_inception_metrics(images):
    # Expect images in [0, 1] with shape (N, C, H, W).
    if images.size(1) == 1:
        images = images.repeat(1, 3, 1, 1)
    return images.clamp(0.0, 1.0)


device = torch.device("cuda" if torch.cuda.is_available() else "cpu")

fid = FrechetInceptionDistance(
    feature=2048,
    normalize=True,
    reset_real_features=False,
).set_dtype(torch.float64).to(device)

kid = KernelInceptionDistance(
    feature=2048,
    subsets=10,
    subset_size=100,
    normalize=True,
    reset_real_features=False,
).to(device)

num_real = 10_000
num_fake = 10_000
metric_batch_size = 64

for start in tqdm(range(0, num_real, metric_batch_size), desc="real metrics"):
    batch_n = min(metric_batch_size, num_real - start)
    real_batch = prepare_for_inception_metrics(
        torch.rand(batch_n, 1, 28, 28, device=device)
    )
    fid.update(real_batch, real=True)
    kid.update(real_batch, real=True)

for start in tqdm(range(0, num_fake, metric_batch_size), desc="fake metrics"):
    batch_n = min(metric_batch_size, num_fake - start)
    fake_batch = prepare_for_inception_metrics(
        torch.rand(batch_n, 1, 28, 28, device=device)
    )
    fid.update(fake_batch, real=False)
    kid.update(fake_batch, real=False)

print("FID:", fid.compute().item())
kid_mean, kid_std = kid.compute()
print("KID mean:", kid_mean.item(), "KID std:", kid_std.item())

c:\Users\tivog\deep-generative-models\.venv\lib\site-packages\torchmetrics\utilities\prints.py:43: UserWarning: Metric `Kernel Inception Distance` will save all extracted features in buffer. For large datasets this may lead to large memory footprint.
  warnings.warn(*args, **kwargs)

FID: 0.3896703840758988
KID mean: -2.5343895686091855e-05 KID std: 0.00042420485988259315