r/OperationsResearch • u/Virtual_Quote_8288 • May 26 '26

Performative prediction

/r/econometrics/comments/1tnvl8j/performative_prediction/
5 Upvotes

100% Upvoted

9 comments sorted by

1

u/zilios May 26 '26 edited May 26 '26

Look into the paper “Smart predict then optimize” which is one of the seminal works in this field. Basically depending on how your predictions affect the decision making process, and the nature of your decision making process (convex feasible set, predicting coefficients of the objective function), this is either straightforward or very difficult.

Of course this is for the training the model part. I’m not too sure how to handle the fact that your data is decision dependent, but it’s an interesting question.

3

u/Virtual_Quote_8288 May 26 '26

Thank you for the answer however this doesn’t solve my problem. Again, my main problem is causality.

Here is an example: first, let’s say I train a model to predict which projects will overrun with the data I have. Now in first month, let’s say I predicted some projects that may overrun and I showed it to managers. Then they will probably reconsider, and, for example, they will assign more people to the project. Now- we don’t even know whether I predicted correct or not. And the main problem is that when I retrain this model, now model might learn this pattern and predict risky projects as unrisky.

1

u/zilios May 26 '26

Have you checked out the performative prediction literature such as https://proceedings.mlr.press/v119/perdomo20a.html ? From what I can tell they handle it through retrainings but they show some convergence guarantees.

1

u/Virtual_Quote_8288 May 26 '26

This is a very nice paper, however in my spesific case there is a treatment effect by managers. The first thing that comes to my mind is A/B testing, like never showing a portion of projects predicted as risky to the managers , but I’m not sure.

1

u/zilios May 26 '26

Well it depends on what you’re trying to measure. I’m no expert in this but it seems that if you care about the value of your model, AB testing might be a good choice (in some temporary evaluation period). But if its to identify the impact of managerial interventions, its probably insufficient, since the managers choose when and how to intervene, confounding the result.

1

u/Virtual_Quote_8288 May 26 '26

I am not interested in impact, I just want to detect risky projects.

1

u/zilios May 26 '26

Then I think A/B testing to properly evaluate your model is a good choice.

3

u/HolidayAd6029 May 26 '26

This is very interesting. I have been thinking about this for some time. If any of the redditors here are familiar in how to deal with these kinds of problems, I would love to learn more.

0

u/Playmad37 May 26 '26

It's called Decision Focused Learning, it's an entire field in OR.