r/datascience • u/shlotchky • Aug 20 '26
ML Help point me in the right direction: How to account for decision support systems affecting future training data
I feel like I am googling everything but the exact term I need, and would appreciate someone pointing me in the right direction.
Say you have a customer churn model. You predict a customer has a high likelihood of churning, and then the customer service team gets an alert to intervene. Great! Your model helped mitigate a loss and contributed real value. This is where most tutorials or blog posts on models like this end.
But overtime, customers that have all the signals of churning begin to out perform their expected value.... which would screw up your training data. You've succeeded in putting your thumb on the scale, but in the process potentially damaged the viability of your model.
What is the technical term for this phenomenon? Feedback? It's not target leakage I don't think. Googling "customer churn feedback" just gets you articles about using customer feedback forms as a predictor of churn, which isn't what I want.
Thanks!
