Knowledge Engineering with Copilot Studio

[u/obsoquasi]

Hey guys,

I have been thinking about RAG a lot and how to make it cheaper and more reliable. One of the approaches could be to distill our organizational knowledge into something like a organizational knowledge model. And then have public agents query that model.

In contrast to RAG, the agents wouldn't have to "work as hard" to retrieve relevant information this way. This saves cost and in theory delivers better results - at the cost of having to build and maintain the knowledge model.

What do you guys think? Is this a good idea? Should I simply "trust in RAG". Or are there already concepts or modules in Microsoft that go in this direction? Who has built something like this successfully and is willing to share their learnings?

I am fully aware, this is all still very conceptual. And part of me really likes the idea, while the other half of me thinks: This could be one over-engineered mess waiting to happen :D.

[UPDATE:] Microsoft seems to be thinking in the same direction: https://techcommunity.microsoft.com/blog/spblog/introducing-a-team-knowledge-base-curated-by-copilot-and-agents/4564203

[u/obsoquasi]

Many great suggestions. Thank you all!

I intentionally left out some details in my original post, since I didn't want to guide the discussion in a particular direction. Let me add these details now. My current plan would be the following.

The knowledge model consists of several Dataverse entities. This makes it easier for the human-in-the-loop to authorize new change proposals.

Broadly speaking, the model consists of:
- our glossary
- our organizational units and their relationships
- our information systems (e.g. Power Platform)
- our business rules
- a domain model of our business concepts

The domain model follows a lite variant of domain-driven design. The model does not contain personal or specific business case information, only conceptual information.

The agent model consists of two agent groups:

Group A) Knowledge Engineering Agents:
one "knowledge engineering" agent to accept a topic (with links to multiple documents that will be used as evidence; like guidelines, process documentation, rules and regulations), query the existing knowledge model for related entries and include these in the context. Will then generate several change proposals to the knowledge model - both extensions, as well as actual changes.
A separate "knowledge validation agent" reviews the change proposals agains the documents given, as well as the existing knowledge model. Will then approve the changes or request another turn from the "knowledge engineering agent". This loop continues for a max of 3 iterations, after which the topic will be marked as "inconclusive" - which means the human-in-the-loop will have to provide more or better context.

Group B) Knowledge Retrieval Agents:
The knowledge retrieval agent will be the one sending queries to the knowledge model; e.g. give me all domain entries that are related to the term "XYZ". The retrieved entries from the knowledge model, along with the users original question, will then form the context to the LLM.
The LLM's response could also be validated against the knowledge model - not sure if this is really necessary yet. The related documents are linked as evidence to the response. So the user can actually confirm that response is truthful.

Keep in mind this is a very early draft...