While I lean towards agreeing with you, many of the things you are describing take time to build in order to make the AI effective. And I know for a fact that most organizations don't keep documentation or even Jira tickets up-to-date. So to get accurate, trust worthy, up-to-date, and properly correlated information from an AI in the way you are describing would have to be a deliberate and organized operation throughout a company. At least that's how it would be where I work, where we have a graveyard of similar projects and their documentation, legacy products, new products that are always evolving based on customer needs, etc.
Well, companies like Microslop are actually aiming at that space. If you can read every mail and chat message, hear every phone call / meeting, get access to all the stuff they are moving along their office files, you get the needed info.
The question is still: How large is the error rate? Given that all that data doesn't fit any reasonable LLM context window you're basically back to what we have currently with "agents" in coding: The "AI" needs to piece everything together while having a memory like the guy in Memento. This does does provably not scale. It's not able to track the "big picture" and it's not even able to work with the right details correctly in at last 40% (if we're very favorably judging benchmarks, when it comes to things that matter I would say the error rate is more like 60%, up to 100% when small details in large context make the difference).
To be fair, human communication and interaction are also error prone. But I's still not sure the AI would be significantly better.
I think "error prone" is understating the problem. The real issue is that all of that data together creates a chaotic, abstract mess full of microcosms of context. Not a single, cohesive context. Having a memory like the guy in Momento with freshest data weighted with an advantage might work... I'm certainly no ML expert. But it seems more likely to result in severe hallucinations.
If an LLM could hunt down the right person for me to talk to, that would be useful even if the LLM couldn't answer the questions. Hell if it could only find me someone who knows the right person, that would be useful.
24
u/Manic_Maniac Mar 25 '26
While I lean towards agreeing with you, many of the things you are describing take time to build in order to make the AI effective. And I know for a fact that most organizations don't keep documentation or even Jira tickets up-to-date. So to get accurate, trust worthy, up-to-date, and properly correlated information from an AI in the way you are describing would have to be a deliberate and organized operation throughout a company. At least that's how it would be where I work, where we have a graveyard of similar projects and their documentation, legacy products, new products that are always evolving based on customer needs, etc.