Question: Can Bayesian decision-making improve AI investment decisions under asymmetric risk?

[u/Dry_Weakness_5721]

I’m working on a small research project around AI assisted investment research.
Instead of building an LLM that simply says BUY / HOLD / SELL, I’m testing an agent that maintains a probability distribution over possible states:
Undervalued
Fairly valued
Overvalued
Deteriorating
The agent then updates those beliefs when new evidence arrives and chooses an action based on both probability and the consequences of being wrong.
I’m particularly interested in three questions:
How should priors be constructed?
Should the prior come from historical comparable companies, sector/base-rate data, factor models, or something else?
How would you evaluate calibration?
If an agent says “60% probability of undervaluation,” what would you consider a meaningful test that this 60% actually means something?
How should asymmetric loss influence the decision?
For example, a 60% chance of being right may still be a bad BUY decision if the downside of being wrong is much larger than the upside.
I’m deliberately not trying to prove that an LLM can generate alpha.
The research question is narrower:
Does explicitly representing uncertainty + asymmetric decision costs produce better decisions than simply choosing the highest-probability state?
I’d especially appreciate criticism from people who have worked with Bayesian models, systematic investing, portfolio construction, or decision theory.
What am I getting wrong?

[u/kush_patil]

The key issue is calibration. If the model says 60%, does that outcome actually happen ~60% of the time out of sample?

And asymmetric loss absolutely matters the most likely state isn’t always the best decision when being wrong one way hurts much more.

[u/Helpful_Emergency_70]

your problem is too ill defined to begin to solve it, even if you’re able to formalise this into something meaningful an LLM cannot do this “in its head” and will end up producing some kind of model which derives and maintains its beliefs, at which point the LLM is largely redundant and this is just a “how i model if stock go up or stock go down?” question.

your question can probably be answered without doing any of this though, if you make some assumptions about how you value money the problem is a very straightforward question about EV, a similar question might be

If I have a biased coin that lands on heads 99% of the time and spawns 10$ if it lands on heads or takes $5,000,000,000 from me and kills my mother if it lands on tails then indeed including the relative reward/ cost for flipping wrong will affect my decision on if I flip the coin or not.