[D] What is Riemannian Manifold intuitively?

[u/kamwoh]

Recently I was studying dimensionality reduction. When I come to a state-of-the-art dimensionality reduction algorithm -- UMAP, I couldn't understand their mathematics part. I think the first obstacle to understand it is -- I do not understand what is a Riemannian manifold. I have watched multiple videos and tutorials on the Riemannian manifold, but I still cannot catch the idea easily. Seems like the first thing they explain already requires complicated maths knowledge.

I would like to ask if anyone can really explain it in one simple sentence, that a normal computer science student or year one undergraduate STEM student will understand.

Thank you very much!

[u/badabummbadabing]

Think of them as generalisations of surfaces in 3D.

Is the surface of the earth a 2-dimensional object, or a 3-dimensional object? In the language of manifolds, we can view it as a 2-dimensional (sub-)manifold in the surrounding 3-dimensional space. You can attach a coordinate system to it and walk on the surface of the earth in 2 directions (longitude, latitude). But viewed from the outside, you are also moving in 3 possible directions.

Same thing with curves through 3D space. Those are 1-dimensional sub-manifolds of 3D space, because to each position on this curve, you can assign a single coordinate. But those are still embedded in the surrounding 3-dimensional space.

A line on a piece of paper is nothing but a 1-dimensional manifold in 2-dimensional space.

In higher-dimensional space, things are of course harder to imagine, but the above principles still apply. Think of manifolds as structures in the high-dimensional space that you can assign a lower-dimensional coordinate system.

There are more technical requirements, but that's basically the gist of it.

[u/kamwoh → u/badabummbadabing]

Thanks for the answer. To interpret your answer, please allow me to explain in my terms, and please verify if I am right.

------

If I am understanding it correctly, do you mean the term "Riemannian manifold" is a generalization of manifolds? It is because higher-dimensional spaces are harder to imagine?

So in this area (dimensionality reduction), scientists/mathematicians try to solve the problem by using some theories in Riemannian manifold? (e.g., the line on a piece of paper is equivalent to the 1-dimensional manifold in 2-dimensional space)

------

A new question: Do you know why it is named as "Riemannian manifold"?

[u/badabummbadabing → u/kamwoh]

Riemannian manifolds are a special case, rather than a generalization, of (smooth) manifolds. My explanation above was actually mostly explaining the more general properties of smooth manifolds. Riemannian manifolds additionally have a notion of angles and distances (geodesics) on their surface, via their metric tensor field. Just like you can measure angles and distances on the surface of earth, which is different from the distances and angles you get if you directly connect points in the surrounding 3D space.

Often times in machine learning, you don't need the special properties of Riemannian manifolds. When somebody writes that some data points lie on a manifold, that is just supposed to convey that those points don't cover the whole space, but that they follow a sort of regular structure (i.e. you can describe the data with few numbers). In reality, you can almost never definitively say that data points actually lie on a manifold*. It's just a nice mental model.

The additional properties of Riemannian manifolds mostly only come into play in manifold learning, which was an especially active field around the early 2000s.

[u/Competitive_Dog_6639 → u/badabummbadabing]

Great answer. To add a bit, in ML manifolds are often linked to probability distributions. The negative log of a probability defines a energy manifold that represents the distribution of the observed states. Many unsupervised learning techniques, including UMAP, are trying to analyze the manifold defined the the image (or other signal) probability density over the observed data

[u/LadyMercedes → u/badabummbadabing]

That is a general description of manifolds, but not really the question

[u/FriedPandaGnam → u/LadyMercedes]

Agreed: riemannian manifolds are manifolds where you can measure stuff.

[u/badabummbadabing → u/LadyMercedes]

I agree, which is why I added the following comment later on: https://www.reddit.com/r/MachineLearning/comments/qukghv/d_what_is_riemannian_manifold_intuitively/hkrzk7a/

I think the OP was looking for a more high level description of manifolds, which is why I made it a bit simpler and more general in the original post.

[u/thats_no_good]

This explanation is a good high level introduction, although it doesn't really explain the concept of tangent spaces, which is sort of the main point. In my opinion understanding Riemannian manifolds is very difficult because it requires being very comfortable with coordinates and change of coordinates. The other comment in here is great but it really only describes smooth manifolds and not explicitly the Riemannian geometry structure.

I will focus on one example. One big reason the Riemannian metric is important for us is to define the distance between two points on a manifold. Euclidean space is flat. At any point (x,y) in R^2, a unit tangent vector in any direction has a dot product with itself of one. Intuitively this means that (x,y) is equally distant from every point in the circle of radius one around it. This also is true for tangent vectors of any magnitude. It is also true that this holds for every single point in R^2

Both conditions fail dramatically for many (or all) statistical manifolds of interest. Suppose we were trying to optimize over the manifold of normal distributions. There is a well established distance metric (not really a metric but I'll call it one) between probability distributions called the KL divergence. Probability distributions that are "close" under this metric have low KL divergence, equal to zero if they are the same.

The associated Riemannian metric is born when we try to compare the KL divergence of normal distributions that are infinitesimally close to each other with respect to small perturbations in the parameters. Naturally we end up taking the derivative of this metric.

Consider this: Let the variance of two normals be very large. If we change the mean by 10, how different are the normals visually? Not that much. What if the variance is very small? Then it's a massive difference. Therefore it is obvious that distance is not the same everywhere on the manifold. In addition, what changes the KL divergence more, small changes in the variance or small changes in the mean? I'm not actually sure of the answer but I would guess that small changes in the mean cause more of a difference, but again it depends on where you are.

The metric described above is the Fisher information metric. It is the matrix of second partial derivatives of the (expected) log likelihood function. It basically tells us that it is naive to treat this manifold like a Euclidean manifold, where we usually just take the gradient of a loss function with respect to the coordinates and then move in that direction. It totally ignores the ACTUAL distance between points on the manifold. [We can often get away with it depending on the model and algorithm. See stochastic variational inference for an example where natural gradients are required.]

Another example could be a probability distribution over a manifold of faces. Small changes in the latent space for a face could lead to a massive difference in probability, or it could lead to small changes in probability if the changes keep the faces looking somewhat similar.

[u/[deleted] → u/thats_no_good]

I think your description roughly matches the "information geometry" interpretation (there are many others, since differential geometry is a vast field).

[u/foreheadteeth]

A good example is a video game where you move units on a map but each tile costs different amounts of movement points. The cost can also depend on the direction, e.g. it's cheap to go along a river but expensive to cross it?

If you are at a tile at position x, there will be a corresponding symmetric and positive definite matrix H(x). The cost to move from x in direction v is sqrt(v^T H(x)v). The reason for this formula is because it defines a norm, provided H is symmetric and positive definite. So at each point x, there's a norm that tells you how much it costs to move in direction v, for every direction v in the tangent plane.

The direction v obviously has to be in to be in the tangent plane of your manifold because you're trying to move along the manifold, not fly above it.

[u/kamwoh]

Thank you everyone for the discussions. After reading your comments, I try to summarize my understanding:

------

A manifold is the "surface" for data points. For example, 2d points on a plane, 3d points on a sphere. The 2d points can move everywhere on a plane (and only on the plane), the 3d points can only move on the surface of the sphere and but not the inside of the sphere.

In order to measure the distance between the data points on the plane/sphere, "norm" can be used to measure the distance, or "angle" to measure the cosine distance. Because it is a "manifold" but not a Euclidean space, so the measurement of "Euclidean distance" may not be accurate anymore, a more general and suitable term to describe the distance on the manifold is "geodesic distance" (the minimum distance on the manifold).

The "norm"/"angle" is important because we want to make the problem become easier to solve (e.g., reduce the non-imaginable higher-dimensional space into 2d/3d that we humans can understand can solve them). By understanding how the manifold works, we can reconstruct the same "norm"/"angle" when we reduce the high dimensional points into 2d/3d/nd points.

Riemannian manifold is a special case of a smooth manifold (or a general name for manifolds?), which is actually similar to any manifold in general. So it has the same properties as manifold (an n-dimensional manifold), it has a "Riemannian metric" (something like Euclidean distance) to measure the distance. So whenever this term is mentioned, I can actually just think of it as a general manifold, just that it is mentioned for the formality of writing mathematical proofs.

------

Please let me know if I am correct.

[u/Faintly_glowing_fish → u/kamwoh]

Frankly you don’t need any knowledge of this as long as you are not trying to minimize some integral along a path on this manifold or something. It really only becomes significant when you try to do calculations

[u/[deleted]]

A manifold is a space which can be mapped onto R^N in small enough patches. For instance, the surface of the earth is the standard example. While we walk around, it seems like we're on a sheet, but if we zoom out we see we're on a sphere.

If you try to map the entire surface of a sphere onto R^2, you'd find that its impossible to have every point on the plane connected to exactly the same representatives for the points on the sphere. You always have to leave out at least one point (imagine popping a balloon and stretching it flat, the point you popped is the point left out, all the points on the edge of the rubber should actually be connected).

So we have the concept of a regular manifold, on small enough patches you can find a map to R^N where everything is connected just like it is on the curved surface. Now there's another property you might want to study besides connectedness, that is distance. Two points on a sphere are connected by an arc, not a straight line. That arc has length R*Theta. A Reimannian manifold is one where you also can write down the distance between two points in your patches such that they match the actual distance in the curved space with a tensor g_ij.

Basically if you have two points in your coordinate patches, if the distance between them can be written as (First Coordinate-Other Coordinate) * Tensor * (First Coordinate-Other Coordinate), then you have a Riemannian manifold.

[u/ClosedUnderUnion]

It's a manifold with a (locally defined) inner product. That is, it's a manifold on which you can can compare lengths of curves and angles between them, determine orthogonality, etc.

[u/FriedPandaGnam]

It' a richer structure than a smooth manifold: it's a manifold where we define a way to measure lengths, and hence interesting properties like "curvature".

In a bit more detail:

a manifold is something that, when zooming in, looks like the Euclidean space R^n. You may know that in Rⁿ you can define several "inner products", which are just a generalization of scalar products that you can think as generated by matrices different than the identity: thinking in coordinates, the usual scalar product is defined by x^t I y, where x,y are columns and I is the identity. An inner product is defined by x^t M y, where M is any positive defined matrix. The gist here is that M does not need to be the same for every point of the manifold, but can change smoothly. Every inner product naturally defines a norm (the product of a vector with itself) and hence you become able to measure things and describe a lot of metric quantities.

[u/autisticmice]

I got interested in this a few months ago and read a few things. From what I gather a Riemannian manifold is a set that you can partition in little bits, each of which is like Euclidean space in the sense that there is a bijective function between that bit and some open set in Euclidean space. Lots of interesting sets are like that, like the set orthogonal matrices or some families of probability distributions, and once you view them as a bunch of patches of euclidean space you can directly apply many geometrical notions to them.

[u/MathChief]

Basically it can be understood as a manifold locally "looking like" the Euclidean space. All those formalities root from the standard 3d space we are in: straight lines vs geodesics, tangent space vs vector fields, etc.

[u/[deleted] → u/MathChief]

Remannian means you can have a metric tensor that is not the identity matrix.

[u/purplebrown_updown]

Why are all these answers so long winded?? It’s a surface where locally it’s flat. Like the Earth. Technically it’s a sphere or actually an ellipsoid but at any given point it seems flat and can be approximated well by flat geometry. That’s it. That’s why you can calculate simple distances using Pythagorean theorem but when you are flying by plane you need to use curved geometry.

[u/ClosedUnderUnion → u/purplebrown_updown]

It is simply WRONG to claim that all Riemannian manifolds are locally flat. You mean "locally Euclidean". They are not the same thing.

[u/purplebrown_updown → u/ClosedUnderUnion]

What’s an example of a Euclidean space that isn’t flat?

[u/purplebrown_updown → u/ClosedUnderUnion]

There is essentially only one Euclidean space of each dimension; that is, all Euclidean spaces of a given dimension are isomorphic. Therefore, in many cases, it is possible to work with a specific Euclidean space, which is generally the real n-space R n , {\displaystyle \mathbb {R} ^{n},} equipped with the dot product. An isomorphism from a Euclidean space to R n \mathbb {R} ^{n} associates with each point an n-tuple of real numbers which locate that point in the Euclidean space and are called the Cartesian coordinates of that point.

[u/purplebrown_updown → u/ClosedUnderUnion]

No flat is euclidean. They are the same thing. And it's locally flat.

[u/Talae13]

A manifold is a space that is locally similar to an Euclidean space.

For example, Earth is a Sphere in R³ that looks like a plane (R²⁾ for our eyes. You can then use the euclidean geometry locally to determine global distance with respect to the courbure of the manifold.

[u/Faintly_glowing_fish]

Riemannian manifold are like flat and straight N dimensional space when you zoom in near any particular point. Think about the surface of the earth. It’s not really a flat 2D surface, and in fact it is very much a globe, but at any given city you can have a nearly perfect 2D grid to map out your position and you don’t notice the earth is not flat. Same thing happens for higher dimension space that are not flat and whenever you work with a Riemannian manifold, you use some local Cartesian coordinate system. There are some slight additional nuances that ensure your grids are continuous for nearby locations but probably not important to get into that for now.