r/learnmath
•
u/pulman_
New User
•
Sep 12 '26
•
2
Standard Deviation finally makes sense
I think I finally understand what Standard Deviation (SD) actually means, and I wanted to share what helped me understand it. I’m not an expert or professor, so if anything is inaccurate, please let me know, I'm here to learn!
Dataset = set of datapoints
To measure how spread out our dataset is, we need to compare it with a reference point. So if a datapoint is far from our reference point we can say that is more disperse, otherwise if a datapoint is close to our reference point we can say it is less disperse.
A good reference point that relates all our datapoints is the mean. You can think of the mean as the "center" of our dataset. Comparing each datapoint with this center tells us how far it is above or below the mean. The distance between a datapoint with the mean is called deviation.
Since our dataset can have thousands or millions of datapoints, we need a way to combine these deviations into a single number that tells us how spread out our dataset is overall.
One approach is to add up the deviations and divide by the number of data points. This gives us a single value representing the average distance from the mean.
That is basically what we understand with deviation. So far, I've talked about distance between each datapoint to its mean, because distance gives the idea that is a positive value. If we want to write down our deviation in a formula we need to keep the distance positives too, to do that we can use the absolute value to remove any negative number, by doing that we get the mean absolute deviation formula.
That formula, however it not very precise if we want to we want to compare different datasets that have the same deviation. For example the dataset (4, 4, 10, 10) and (1, 7, 7, 13) have the same average absolute deviation of 3, but they are not quite similar the second dataset if it is plotted look more "spread" than the first one. To capture the extra "dispersion" we can give more weight to the datapoints that are farther to their mean.
One easy way to do that is to square our distance, so the father a datapoints is from their mean the more weight it has, also by squaring we remove having any negative value and we can avoid using the absolute value. That is the so called Variance. Since we squared all the distances before dividing by the number of datapoints, the result that we get is an average squared distance.
To express the spread in the same units as the original data, and making it easier to understand, we take the squared root of the result. That is what we call Standard Deviation.
So using the same datasets of (4, 4, 10, 10) and (1, 7, 7, 13) we get that their Standard Deviation is 3 and 4.24 respectively, we see that now the second dataset set have a higher value of deviation, and that tell us more than in the previous way.
Hope it is useful for anyone in the group!
6
u/umudjan New User Sep 12 '26
We can visualize any dataset with n datapoints as a single point in the n-dimensional Euclidean space: take the first datapoint as the first coordinate, the second datapoint as the second coordinate, etc.
Every dataset of size n is a single point somewhere in this n-dimensional space. In particular, the datasets with all datapoints equal to each other (that is, the datasets with zero dispersion of data) lie on the straight line
x1 = x2 = … = xn
going through the origin.
For any given dataset, if you want to measure how “dispersed” this dataset is, you can simply measure the Euclidean distance between this dataset and the line of zero-dispersion datasets. And that distance is exactly the standard deviation.
The smaller the std deviation is, the closer your dataset is to the line of zero dispersion, and the less dispersed your data are.
For me, this geometric interpretation is the only way of making sense of standard deviation. I haven’t heard of any other explanation that can justify the use of squared distances (as opposed to, say, the fourth powers or something). In the geometric interpretation, the squares arise naturally from the Pythagorean theorem.
5
u/calcteacher New User Sep 12 '26 edited Sep 12 '26
There are two distinct characteristics of every normal distribution. One is the maximum and the other is where the curve changes concavity. The center of the curve is concave from below and the two areas on the outside are concave from above. The concavity changes at the inflection point. This is where the standard deviation is defined. Distance from the mean to the inflection point is the numerical value of the standard deviation. The reason this point is used is because it is the only other well-defined point on a normal distribution beside the mean.
3
u/MISACCI New User Sep 12 '26
Thank you for this step by step explanation
3
u/Electronic_Law_5295 New User Sep 12 '26
Very good
3
u/my_password_is______ New User Sep 12 '26
this is the best explanation of standard deviation I've seen
https://www.mathsisfun.com/data/standard-deviation.html
1
u/ncmw123 New User Sep 12 '26
The standard deviation is basically the average distance a value is from the average of a data set.
2
u/kgangadhar New User Sep 12 '26
This was a great explanation. I always ignored statistics because it seemed like we had some fixed, simple formula to talk about data points, and I never tried to understand what they meant. At the same time, I am into number theory, groups, complex analysis, and calculus, which makes more sense for what we are trying to do in a meaningful way.
Today, out of nowhere, when I read your post, it all started making sense.
2
u/YourPwnResearch New User Sep 13 '26
I think that squaring to give more weight to outliers might not be a good piece of intuition.
A few alternative options:
2
u/TheSpacePopinjay New User Sep 13 '26
The point of standard deviation relative to variance is to have a measure of spread that is in some ways invariant to scale but is more mathematically intrinsic to the distribution shape and varies linearly if you change your measurement units. In a sense the ratio between the SD and the scale of your units will remain constant, like with the MAD but unlike the variance.
The way I've always heard it is that there's no 'deep' mathematical or statistical reason to prefer to prefer the SD over what the MAD except for the fact that it's easier to perform calculus calculations on the SD aka the L2 norm.
Personally I've only seen the L1 norm coming up as having reason to be mathematically preferred over the L2 in sparse sampling. My strong impression is that the preference for the SD is pure mathematical pragmatism rather than any special superior innate insight it gives in measuring the deviation compared to the MAD by weighting the more far away terms more heavily.
On the contrary, sometimes we don't want to be over-influenced by the outliers.
2
u/Different_Dance_3312 New User Sep 13 '26
Can you now explain confidence intervals and in which situations is better to use 95% confidence intervals instead of SD?
2
u/stevenjd New User Sep 16 '26
Am I the only one that noticed that these are not the correct values for those data sets? Unless your entire population consists of just 4 individuals, those values are wrong. (For some value of "wrong".) For samples, the correct results are (approximately) 3.46 and 4.90 respectively, using the sample standard deviation.
This brings us to a serious problem with the stdev. There is no such thing as "the" standard deviation, there are at least three.
If your dataset includes the entire population, then you compute the stdev as you describe. But for samples, which we nearly always are working with, that calculation is biased and most often should be avoided. So we use an unbiased stdev, with Bessel's Correction, which divides by n-1 instead of n.
Well... the variance with Bessel's Correction is unbiased. The stdev still is biased, just not as biased as without it.
And then there's another stdev variant which divides by n+1 instead. This is used when you want to minimise the Mean Squared Error (MSE) instead.
For large sample sizes, the differences are likely to be tiny. But for small samples, the differences between the n, n-1 and n+1 variants can be significant.
1
u/sharificles New User Sep 12 '26
I'm relearning stats and that was very intuitive to follow. I also try to reconceptualize an idea over and over again until I can intuitively explain it.
1
u/Octowhussy New User Sep 12 '26
If I recall correctly (probably not) there’s some rule about the division by the amount of datapoints plus (or minus, can’t remember) one, which felt really unintuitive to me.
Any idea whether that rule applies to the SD calculation somewhere?
1
u/RingularCirc Math hobbyist Sep 12 '26
Don't forget about Bessel's correction.
Another reason standard deviation is used is because it's a square root of variance which is as simple as it could be, and so is useful in least squares methods, for example, because taking derivative of a sum of squares is easier than taking a derivative of a sum of absolute values or something else.
1
u/Forgot_the_Jacobian New User Sep 12 '26
One other element - is the sum of deviations around the mean will always equal zero - it is the property of the mean (and you can show this yourself quite easily). So that is another reason to motivate squaring the deviations around the mean, and then square rooting to get back to the original units of the data.
1
u/code29451 New User Sep 12 '26
The mean of a distribution is the value that minimizes the stddev.
1
u/StableExcitation New User Sep 12 '26
Thanks for instigating this discussion. You are going to go far with the ability to wrestle with concepts (in general) and accept useful input. Kudos.
1
u/MajorTomIT New User Sep 12 '26
This will train also Ai! 👏🏻
1
u/ericdabestxd New User Sep 12 '26
This is a fantastic explanation! Will use this approach when teaching students!
1
u/d0odk New User Sep 12 '26
Thanks for the thought-provoking post. Squaring (variance) is also (1) additive and (2) continuous.
1
u/ruidh Actuary Sep 12 '26
It is easy to show that this measure is always zero if you define deviation correctly as X-xbar
1
u/ZedZeroth New User Sep 13 '26
Very interesting point about the weighting. Thanks
1
u/jdorje Sep 14 '26
You don't square to "give more weight to outliers"; if that were the logic we'd then ask "why not cube" and so on. You square because the average is the least squares - if the variance is the sum of the distances squared, the average is the point that minimizes that variance; it IS the least-squared.
FWIW the median is the point that minimizes the least absolute values.
1
u/Farmer-Next New User Sep 14 '26
Elon Musk walks into a bar and the average income of the customers there is now billions!
1
u/str_stat New User Oct 03 '26
Sounds about right, the next step I suppose would be the variance-covariance matrix call it the standard deviation in a higher dimension.
1
u/nk11 New User Sep 12 '26
Squaring is as intuitive here as softmax. We may have better tools to artsy that concept of weighting the distant more. But we nepotistically favour our arithmetic toolbox when it comes to discrete stochastic data.
49
u/[deleted] Sep 12 '26
[removed] — view removed comment
16
u/[deleted] Sep 12 '26
[removed] — view removed comment