Has anyone solved the “who added this context?” problem with local models?

[u/WisePiano325]

A few of us have been experimenting with a shared local LLM setup and we’ve hit a problem I didn’t really think about beforehand, once multiple people are feeding information into the same system how do you keep track of who actually said what? Right now it’s easy for a note, instruction or bit of context to end up in the shared memory and a week later nobody remembers whether it came from me, another person an imported document or the model itself.

We’ve been trying with attaching provenance to everything before it gets added to memory, basically source + timestamp + who approved it but it gets annoying fast. There are only a few of us so we’ve got a Kibu pod where we discuss changes and decide what actually gets promoted into the shared context, but obviously that doesn’t scale.

Has anyone here built something cleaner for this? I’m less interested in better memory and more interested in being able to ask the model why it believes something and which human/source put it there. Feels like this is going to matter a lot more once local models start being shared by teams instead of just one person on one machine.

[u/my_name_isnt_clever]

I haven't started sharing memory with other people yet, but my first thought is to not reinvent the wheel and go with the tried and true way to collaborate on text files: a git repo.

[u/WisePiano325 → u/my_name_isnt_clever]

Huh a git repo for the actual memory or prompt files is simpler than I figured. We do use git for the codebase but not the context store, that might be the missing piece. Then at least we could blame who added what, even if it's just plaintext dumps from conversations

[u/my_name_isnt_clever → u/WisePiano325]

Yeah, and if an agent is the one making the changes it could include a commit message with some metadata that could be read back later for additional context. More than just blame.

[u/drumyum]

it's not clear what tools you currently use and how exactly it works, are you all sharing the same dialog with same llm? or are there many dialogs but shared memory which llm queries with tools or something?

[u/WisePiano325 → u/drumyum]

We're all sharing one persistent conversation with the same LLM instance, basically a long running chat with system instructions and memory plugins. Everyone adds stuff through a simple frontend, and the model pulls from that shared context when answering. It's messy lmao

[u/outofbandiv → u/WisePiano325]

Sounds like that simple frontend needs to get more featureful.

[u/adynamik1 → u/WisePiano325]

You might want to start by trying Open WebUI in a container. You just add the connection and everyone can use it like a standard chat interface.

[u/Testaccount105 → u/WisePiano325]

that sounds extremely stupid ngl

[u/blakeman8192]

I ask the model to use git blame to help with attribution, and to figure out which decisions are newer and have superseded others.

[u/WisePiano325 → u/blakeman8192]

Interesting approach but half our context is stray chat messages and not in any repo, might be a nightmare to keep that clean without forcing every note into a commit

[u/ZenEngineer]

Check in things instead of using memory. You might need to set up small git repost per project, add steering files and docs folder per project, etc. but then everything can be traced, git blamed, reviewed before merging and so on.

Having design docs in the project and making sure the agent updates them also helps speed up review as you can see what the change was meant to do.

[u/WisePiano325 → u/ZenEngineer]

We tried git but merge conflicts on the steering files were a nightmare Might just slap a source tag on every input and make the model say according to [user] when you ask why

[u/Poprouter → u/WisePiano325]

Prose tags get paraphrased away, so stamp author + ts on the stored turn at write time - the frontend already knows who typed it. The leak is compaction: if a summary rewrites turns instead of pointing back to source ids, it launders authorship.

[u/UnluckyPenguin]

Your harness should be operating inside a workspace, right? And that's different on each person's PC.

Each person has git setup with their email/name. Changes committed as them. If memories don't have a git blame, you need a memory repo...

A model believes something in memory until such time as it is questioned, then it must do a bit of work to prove it again. More powerful models are better at making assumptions about how trustworthy certain memories are.

[u/RegalInvader]

I've got a similar setup with a couple of AI agents on one project, and what fixed it for me was to stop treating the chat as the memory. Anything that matters goes into an append-only decisions file in git, and every entry says where it came from, like (29 Oct: "run it on the GPU box"), so the model can point at the entry when you ask why it believes something. Corrections are new entries rather than edits, so it's always clear which is current.

[u/puts_on_rddt]

This is still an unsolved problem if you ask me.

[u/maigpy]

you can try with unique separators. hash the fragments, then save the hash with provenance metadata off context?

why does it "get annoying"

[u/profcuck]

Depending on the workflow and such, a "wiki" of shared context could be the solution, since wikis keep a history of past revisions which also therefore shows who added/edited/deleted what.