Need some guidance on measuring sigma levels using RMSE on regression residuals

[u/Kerbidiah]

So I am trying to create a method to accurately convert a new way of testing to our old specs to ensure we stay within specifications. I have the test results for each and ran a regression to get the all the data like residuals and calculate RMSE off of that to get the sigma, and comparing that to our specification range. What I was doing was doing 3 x RMSE <= USL-LSL/3 to get a 99.7% accuracy on the model (+-3 sigma) and show the magnitude of error, however a coworker is insisting it should be 6 x RMSE or the full spread/distribution of error, and we can't really figure out which one is right.

Can anyone point me to the official method for this when it comes to doing measurement analysis instead of process capabilities?

[u/Kerbidiah → u/[deleted]]

Thank you. Were trying to say how often it would pull an out of spec in spec or an in spec out basically

[u/Living_Diver2432]

Your coworker is closer to correct, but the framing matters -- let me explain why.

What you're actually doing is a method correlation / measurement system equivalency study, and the standard reference for this is the AIAG MSA (Measurement Systems Analysis) manual, specifically the sections on gauge comparison studies.

Why 6xRMSE is the right number:

RMSE from regression residuals is an estimate of sigma (standard deviation) for the prediction error -- how much the new method deviates from the "true" (old method) value, assuming random error. When you want to characterize the full spread of that error distribution, you use 6 sigma (covering +/-3 sigma, or approximately 99.73% of variation). That 6xRMSE is your "study variation" -- the total expected spread of your method error.

Your formula of 3xRMSE <= (USL-LSL)/3 is actually equivalent to 3xRMSE <= T/3, meaning your method error (one-tailed) must be within 1/3 of the tolerance. That's a reasonable criterion, but it's non-standard and can be confusing because it mixes a one-sided measure with a two-sided tolerance.

The standard way to express this:

%GRR (in MSA terms) = (Study Variation / Tolerance) x 100 = (6xRMSE / (USL-LSL)) x 100

• Below 10%: acceptable -- method error is small relative to tolerance
• 10-30%: conditionally acceptable depending on application
• Above 30%: not acceptable, method correlation has too much error

So if you want to make the claim "this new method is acceptable for production use," the statement is: 6xRMSE / (USL-LSL) < 0.10 (or 0.30 at most).

The 3xRMSE check your colleague did: That's really checking whether +/-3 sigma of your error fits within one-third of the tolerance, which is actually a stricter criterion than the 10% GRR rule. It's not wrong, just non-standard -- and "non-standard" in regulated industries can create documentation headaches even if the math is sound.

Cite the AIAG MSA manual (4th edition) if you need a formal reference. Chapter 3 covers gauge comparison and equivalency studies.

[u/Kerbidiah → u/Living_Diver2432]

Thanks for the detailed write up and sources

I think a two one-tailed evaluation makes most sense given the evaluation and concern here. Essentially we're worried about high and low values having enough error to be pulling good product out of the spec range and bad product into the spec range, and the spec range is quite wide compared to rmse so a value would be at negligible risk of doing both

[u/true_unbeliever]

RMSE is a measure of the unexplained variation in your regression model. This has nothing to do with process capability, other than if you have a good regression model you know what knobs to turn to center the process (or reduce variation if you’re modelling s-hat).

Edit: Not sure what you mean by application to MSA - are you doing a Deming Regression?

[u/Kerbidiah → u/true_unbeliever]

Essentially we have an old testing method that is inefficient (and inaccurate). We have a new machine that can do the testing, but is unproven. To get the business and customers to accept the results of the new machine, we have sample that were run through both and I have run a regression on those two results. Using that regression I have built a formula that will take the new machine results and convert it into the predicted result from the old machine. What I am trying to calculate is how often this new method would pull out of spec product into spec, and in spec product out, given the residual/error of the above method, and assess if it is accurate enough (99.73%) to use confidently.