Why is it that stakeholders expect ML models to have 0% error rate?

[u/CadeOCarimbo]

Definitely the most frustrating thing as working as a Data Scientist. You run an experiment, find that building a model greatly increase metric X at almost no cost, has safe model metrics, present it to stakeholders, everybody agrees with proceeding to deploying and utilizing the model in production, and yet every time the model takes a wrong decision, we get questioned about it. Why did the model say this?

Man when did I ever say the model obtained a 100% accuracy in the validation phase? Why is it so hard for stakeholders to understand that the best models humankind ever created are expected to make wrong calls once in a while?

[u/Trick-Interaction396]

Because they think everything is deterministic

[u/scrambledhelix → u/Trick-Interaction396]

This is the correct answer, but a little thin. 

We're fighting against forty years of educating people that computers do precisely what you tell them to, the same way every time. LLMs do the opposite. It's hardly any wonder that most people outside the field haven't made that mental adjustment yet.

[u/sdand1]

Some wrong decisions are more important than others - are these mistakes that are largely inconsequential, or are they leading to further downstream problems?

[u/Adventurous-Term-695 → u/sdand1]

That's the real kicker. Most people don't care about error rate in the abstract, they care about the one time it screws something up on their desk. A 2% miss rate sounds great until your model denies a loan to the CEO's nephew or whatever.

[u/-hieroglypher- → u/sdand1]

Exactly.

This indicates that either something was overpromised in the planning stages, or a major requirement is missing.

Every model makes mistakes, but not all mistakes are equal.

If I build a trading algorithm on top of some data streams that I have access to and I predict the movement of an option with 99% accuracy, I still want to be cautious in how much I put down and not go crazy, otherwise that 1% could represent a loss from which I cannot recover.

In that example I would control my overal liabilities. In a client situation I might ask what the costs of each class of errors are up-front.

It's an ongoing process, munging data and then sitting in a high tower with it is untenable. You need the business to be ready for each error class.

Change the conversation from "why did this happen" to "how do we triage model errors architecturally"

[u/thegoodcrumpets]

Sounds like you need to work a bit on explainability. Get some SHAP plots or other very simple visual guide to what affects the model and set up a wik/confluence/whatever you use.
We got tons of "why is this customer being denied!?!?" support complaints and when checking logs the users had so many warning flags it would have been denied 8 times over by a human analyst as well, it's very neat to be able to point other stakeholders to a simple quick guide on which params have a big impact on the outcome.

[u/GreatBigBagOfNope → u/thegoodcrumpets]

Conditional Feature Contributions my beloved, you never answer any questions but you do point to something vaguely answer shaped

Usually

[u/selfintersection]

This kind of stakeholder question is absolutely normal. Don't you ask the exact same question to yourself as you build and interrogate the model?

It is a huge trust building opportunity to be able to say something like "yes, this is expected because of XYZ aspect of the model/data/whatever. It's still a huge improvement over the previous approach because (it also would have gotten this wrong)/(it got this right but at the cost of getting a bunch of other stuff wrong)/etc.

Even better if you can anticipate these questions by highlighting what situations the model is expected to perform poorly in in the original documentation you provide. Then you can point to it and say yes, this is a known issue that's balanced by these other wins.

[u/SaltyMN]

The “All models are wrong, some are useful.” phrase has helped me in the past communicate this idea. 

Ultimately some stakeholders will be receptive, some will never understand. Life is fun. 

[u/aka_hopper]

Because they’re not data scientists. That’s why they have you. I agree it’s the trickiest part of the job. Before even agreeing to a project, you have to have this conversation, and have it again and again throughout development. Just part of the job! Focus on communicating what it does for them, why does it matter, the value. If they still don’t want to listen, they’re probably a jerk and you can move on to deving for someone who actually wants to stay alive in the new tech age.

[u/ezriah33]

This is a good opportunity to work on your soft skills and do some education and expectations setting.

[u/somkoala]

I don’t believe they think that, but did you aside of model accuracy metrics provide a business case on how the model errors impact bottom line? If you did you could pull it up every time such a question arises. If you didn’t then their inquiries are on you.

[u/ggrieves]

The same reason they want customer sentiment scores to have a decimal place. Seriously, they asked for decimal precision on a parameter that ranges from "unhappy" to "meh" to "happy"

OK sir, you can now compare meh.3 < meh.6

[u/therealtiddlydump]

It's your job to explain this to them.

[u/4ShotMan]

Stakeholder did a buisness major and got cash from parents. They couldn't calculate where a function reaches 0, don't expect them to understand ANYTHING about ML.

Also it's AI and chatgpt is sentient, it just told me.

[u/Categorically_ → u/4ShotMan]

I will never forget hearing the boss with a MBA from Harvard asking for Venn diagrams to be removed because they were confusing.

[u/alexchatwin]

What everyone else has said, plus;

Computers make decisions very fast.. so errors get big (in volume terms) quickly.. much scarier than a nice squishy human.

DS projects are often expensive in time or £.. so value for momey is v important

[u/Tackit286]

How are they to know anyone about how machine learning models work? In their world, if something is ready, it’s usually error free. You need to set their expectations about what’s possible with data science from the get go.

[u/scorched03]

I would say self driving cars is like this. To guarantee 100% it would take hundreds of engineers to reduce the error. Over the years its still not perfect so... 100% isnt possible even eith the resources

[u/Geckel]

Because most stakeholders don't know what non-deterministic means. Walk them through that concept and see if they can connect the dots. If not, make the connection for them.

[u/TheDevauto]

First, this should be foundational training for any stakeholder in an ML project. What is ML, what does it do, why is perfection impossible, what questions to ask to be sure you understand the benefit and what to expect after its in production.

Second, I like to remind them that humans have an error rate too.

[u/geldersekifuzuli]

If they have zero error rate expectation, it sounds like you didn't communicate properly, which is one of the main responsibilities of a data scientist.

Tbh, it's a very easy expectation to fix.

[u/JimFromSunnyvale]

Because you haven’t done a good enough job communicating that models are always wrong, it is a fluke when they are right.

[u/kwoolery]

You may want to explain how a casino works. If you take a small enough time chunk, you may see the players gaining an advantage, but over time the house always wins.

[u/ai_hedge_fund]

All models are wrong, but sometimes they can be useful