2 Distinguishability as Geometry

The first answer starts from a statistical question: how well can you tell two distributions apart from data?

If and are very close, distinguishing them requires many samples; if they differ substantially, even a small sample may suffice. This notion of statistical distinguishability is the basis of Fisher’s approach [1].

The key object is the score function, which is the gradient of the log-likelihood with respect to the parameter. For a parameter family , the score at is:

The score measures, infinitesimally, how the distribution changes as you move in the parameter space. It is the velocity of the distribution in the direction of .

To quantify how large this velocity is—how much the distribution changes per unit movement in —Fisher took the expected squared magnitude of the score. In 1-D, this gives the Fisher information [1, 2]:

In higher dimensions, this becomes the Fisher Information Matrix (FIM):

This matrix is not merely a useful statistic. It is a Riemannian metric tensor on the parameter manifold : a smooth varying inner product on each tangent space , encoding the infinitesimal statistical distinguishability of nearby distributions. The systematic study of this structure is the subject of information geometry [3, 4].

3 The FIM as a Pullback Metric

The FIM has a clean geometric origin and is not an ad hoc construction. It is the metric that parameter space inherits from the space of distributions via the parametric family. This is the concept of pullback metric.

4 Chentsov’s Theorem: The Unique Invariant Metric

The pullback construction shows that FIM is natural. A deeper result, Chentsov’s theorem [5], shows it is unique.

Chentsov is not saying the FIM is a natural choice among many. He is saying it is the only choice consistent with the principle that geometry should not depend on how data is represented. If two experimenters measure the same phenomenon but record their observations differently—and neither loses information—their geometric picture of the statistical model must agree. The FIM is the unique metric with this property.

This is a strong and somewhat surprising result. It says the geometry of statistical inference is not a convention or a convenience: it is forced by the structure of the problem.

This uniqueness result is local. The Fisher metric is an infinitesimal object: it is a tensor at each point of parameter space, giving an inner product on tangent vectors. It measures how fast nearby distributions diverge statistically, but it says nothing directly about distributions that are far apart.

Recovering a global distance requires integrating along geodesics in the Fisher metric, which is generally difficult and depends on the path. For many parametric families (such as Gaussians, exponential families, etc.), geodesics can be computed explicitly. In general, though, they cannot. This locality is a strength (it gives a clean Riemannian theory that is well-defined and coordinate-free) and a genuine limitation (it is not a ready-made global distance between arbitrary distributions).

This is the first major contrast with Wasserstein geometry. Fisher gives a local ruler for statistical distinguishability; Wasserstein will give a global ruler for displacement through the sample space.

5 The Cramér-Rao Bound: Geometry in Action

The Fisher metric does more than organize the geometry of a statistical model. It places hard limits on what any learning or estimation procedure can achieve. A learner navigating parameter space with smaller Fisher metric in some direction cannot acquire information in that direction quickly, no matter how its update rule is designed. The Cramér-Rao bound makes this precise [1, 2].

For an unbiased estimator of a scalar parameter , the bound states:

Here denotes variance under .

In vector form, for an unbiased estimator of :

where denotes covariance under , and denotes the Loewner partial order on positive semidefinite matrices: means is positive semidefinite. No unbiased estimator can have covariance smaller than .

For deep learning, this statement should not be read too literally. We are usually not searching for an unbiased estimator of a true parameter vector; neural networks are overparameterized, non-identifiable, and trained for predictive performance rather than classical parameter recovery. The relevance is geometric: the inverse Fisher describes how sensitive the model distribution is to parameter movement. In optimization language, it acts like a curvature-aware preconditioner, shaping stable step sizes and update directions.

The geometric reading is direct: is the dual metric on the cotangent space, and the bound says that estimation uncertainty is bounded below by the inverse of the local metric scale of the statistical manifold. Where the Fisher metric is large in a direction—where distributions are highly distinguishable and change rapidly along that direction—estimation is easy. Conversely, where the local distinguishability is small, estimation is hard.

The Cramér-Rao bound is one of the oldest results in mathematical statistics but its geometric interpretation—as a statement about the inverse of a Riemannian metric—was not fully articulated until Rao’s formulation [2] and the subsequent development of information geometry [3].

6 What the Fisher Metric Cannot See

The Fisher metric is defined entirely within parameter space. It requires a parametric family and measures how the model changes as varies. It is sensitive to the statistical shape of the model—how distributions in the family differ from each other—and entirely indifferent to the geometry of the sample space itself.

If you relabel the points of arbitrarily—i.e., apply any measurable bijection to the data—the Fisher metric does not change. This is exactly Chentsov’s invariance property that we touched earlier, and it is both, a strength and a limitation.

It is a strength because it means the geometry is intrinsic to the statistical model, not an artifact of data representation. It is a limitation because it means the Fisher metric has no vocabulary for the spatial layout of the data. To visualize the distinction, imagine a density plot: the horizontal axis is the sample space, while the vertical axis records how much probability mass sits at each location. Fisher sees changes in the heights of that plot, but not the geometry along the horizontal axis. It cannot tell you that the image of a cat shifted two pixels to the right is closer to the original than a completely different image. It lives entirely in the vertical direction, how probability weights change, and is blind to the horizontal direction of where points live in space.

7 Preview: Fisher as Local KL Geometry

The Fisher metric does not stand alone. It is the local, symmetric residue of a richer asymmetric object: the KL divergence. This connection—which we will come back to in another blog post—is worth previewing here because it explains why the Fisher metric appears so naturally in learning.

This means two things. First, any algorithm that minimizes KL divergence inherits Fisher geometry in its local second-order behavior. Minimizing KL gives Fisher as the local geometry, but the actual update rule may still use Euclidean gradients—this is precisely the gap that natural gradient descent, which will be discussed in future posts, is designed to close. Second, the Fisher metric inherits its symmetry from the Hessian operation: even though KL itself is asymmetric (because in general), its second-order term is a symmetric bilinear form—the metric tensor .

The asymmetry of KL beyond second order encodes something important: the two directions of KL correspond to qualitatively different behaviors (mode-covering vs. mode-seeking). I will examine this asymmetry in detail and show how it points directly toward Schrödinger bridge.

8 The Fisher Metric in the Exponential Family

The previous section showed that KL locally contains Fisher geometry in its second-order term. Exponential families are the cleanest setting where this local picture becomes algebraic: the Fisher metric, KL divergence, and convex duality can all be written in terms of a single function, the log-partition function.

The Fisher metric takes a particularly clean form for exponential families, which include Gaussians, Bernoulli, Poisson and several other distributions used in practice.

An exponential family has density

where are the sufficient statistics, are the natural parameters, and is the log-partition function. A direct calculation shows that the FIM is the Hessian of :

9 A First Uniqueness Result

I close this post by returning to Chentsov’s theorem, because it gives Fisher geometry a special status.

The Fisher metric is not just a convenient way to measure local distinguishability. Given one natural requirement—that the geometry of a statistical model should not depend on how the data are represented, as long as no statistical information is lost—the Fisher-Rao metric is forced. Up to a constant scale factor, it is the only Riemannian metric with this invariance property.

That is the main lesson of this first ruler. Fisher information does not merely assign numbers to estimators or appear in asymptotic statistics. It defines the unique local geometry compatible with statistical distinguishability. If we care about how distributions change as models learn, this is the geometry that the model carries.

But this uniqueness also explains Fisher’s limitation. Because it is invariant under sufficient transformations of the sample space, Fisher geometry is blind to the spatial arrangement of the sample space itself. It knows how probability weights change; it does not know how far mass has moved.

This is exactly where the second ruler enters. The next post turns to Wasserstein geometry, where distance is not measured by distinguishability but by displacement. Brenier’s theorem [9] will give a second kind of uniqueness: under quadratic transport cost, the optimal way to move mass is not arbitrary, but generated by the gradient of a convex potential.

10 What Comes Next

This post introduced the first ruler: Fisher information, the local geometry of statistical distinguishability. The next post introduces the second ruler: Wasserstein distance, the global geometry of displacement.

Together, these two rulers set up the main tension of the series. Fisher asks how distributions differ statistically; Wasserstein asks how probability mass moves spatially. Learning needs both perspectives.

From there, the story becomes more interesting. KL divergence will appear as the asymmetric object underlying Fisher geometry. Schrödinger bridges will connect entropy and transport into a single stochastic path. Hodge decomposition will reveal the algebraic anatomy of optimal flows. Natural gradient will show what it means for optimization to respect statistical geometry. Langevin dynamics will then add calibrated noise, turning gradient descent from a mode-seeking optimizer into a sampler over distributions.

Finally, these ideas will reappear in modern machine learning, where diffusion models, flow matching, natural gradients, Langevin samplers, and Bayesian methods can all be understood as approximations to pieces of this geometry.

The goal of this series is not to argue that these subjects are secretly the same theory. They are not. Rather, the claim is that they discovered compatible pieces of the same geometric picture: information provides the local ruler, transport provides the spatial ruler, KL provides direction, entropy provides stochastic paths, Langevin connects optimization to sampling, and topology explains the obstructions.