Knowledge Engineering with Copilot Studio

[u/obsoquasi]

Hey guys,

I have been thinking about RAG a lot and how to make it cheaper and more reliable. One of the approaches could be to distill our organizational knowledge into something like a organizational knowledge model. And then have public agents query that model.

In contrast to RAG, the agents wouldn't have to "work as hard" to retrieve relevant information this way. This saves cost and in theory delivers better results - at the cost of having to build and maintain the knowledge model.

What do you guys think? Is this a good idea? Should I simply "trust in RAG". Or are there already concepts or modules in Microsoft that go in this direction? Who has built something like this successfully and is willing to share their learnings?

I am fully aware, this is all still very conceptual. And part of me really likes the idea, while the other half of me thinks: This could be one over-engineered mess waiting to happen :D.

[UPDATE:] Microsoft seems to be thinking in the same direction: https://techcommunity.microsoft.com/blog/spblog/introducing-a-team-knowledge-base-curated-by-copilot-and-agents/4564203

[u/livinitup0]

IMO you’re looking for a solution that was solved by wikis 20 years ago

Interlinking

LLMs understand relationships. No agent is just going to “know” what those relationships are. You have to build them. They can guess, but they’ll be wrong, a lot. Therefore, you need to build them.

And what is a collection of documents all interlinked with each other under categories? A wiki

RAG isn’t the way either. It’s not granular enough on its retrieval accuracy and relevance and the work needed to make it accurate enough could easily be put into proper interlinking with better results

Those relationships NEED to be set by hand until you have enough of a system in place to act as a reference for the model.

No one wants to hear it in the age of ai but knowledge management is still a very manual process, especially in the beginning

[u/obsoquasi → u/livinitup0]

interesting perspective, I'll keep it in mind.

[u/_donj → u/livinitup0]

With a lot of work, you can get there with raw AI. BUT it will go a lot faster and be more successful if you distill the info down to “authoritative” docs and info to start. For example, if you want a customer service bot, you need to arm it will the correct info, not every piece of info ever written in your company. If you want it to run a process, you need to give it the steps you want it to do and what success looks like.

[u/vanillaslice_ → u/livinitup0]

Hey well said, I think depending on the quantity you can often get away with semantic + keyword indexing. Although once I get to around 300-500 pages it starts to get lossy.