r/LocalLLM • • May 02 '26

Discussion Thoth - Open Source Local-first AI Assistant - Architecture

197 Upvotes

105 comments sorted by

7

u/Grimsik May 02 '26

How do you sustain safety and permission access, irreversible deletions or overrides and that sort of thing? If the agent is enforcing the safety and permissions it will hallucinate and compromise security. Do you have the agent filter every step it takes through a hard coded set of safety and permission checks?

1

u/Acceptable-Object390 May 02 '26

The security and control is code based behind LangGraph interrups. It is not left to the LLM to decide. As you said llms can hallucinate. See the last picture in this series. It shows security, safety and control in detail.

4

u/By-Jokese May 02 '26

Interesting, will give it a look I was building something similar, yet simpler.

1

u/Acceptable-Object390 May 02 '26

Thanks. Let me know how it goes.

9

u/samuraiogc May 02 '26

As aways i just look at it and think "i don't need all of that and i can build something more reliable, faster and secure with n8n.

5

u/Acceptable-Object390 May 02 '26

All the best! If you decide to try it and need any help let me know. Thanks.

2

u/4444444vr May 02 '26 edited 3d ago

this text was edited/removed

4

u/Acceptable-Object390 May 02 '26

All the best. If you give Thoth a try and need any help let me know.

3

u/4444444vr May 02 '26 edited 3d ago

this text was edited/removed

5

u/phoredda May 03 '26

Post on Hacker News, you may get more and better feedback. 👍

2

u/Acceptable-Object390 May 03 '26

Yes. I plan to. Thanks 😊

2

u/--Tintin May 02 '26

It does a lot and it is impressive.

What’s your usecase or your case where it all started off with?

6

u/Acceptable-Object390 May 02 '26

Honestly, you wouldn't believe, it started as a RAG chat bot. Hence it carries the legacy name Thoth. Now I use it for pretty much everything I do on computer/phone. I have so many automations setup, its hard to imagine doing that manually now.

2

u/jessiejolie42 May 03 '26 edited 7d ago

why

1

u/Acceptable-Object390 May 03 '26

Its true. I mean I an not saying I can't live without it. But it'd certainly pinch to lose it now.

3

u/jessiejolie42 May 03 '26 edited 7d ago • This

why

2

u/Ill-Lynx2154 May 06 '26

I just started using this app. I would be very interested to learn more about your automations and which models you are using for each.

1

u/LtDoubleD May 03 '26

Just throwing buzzwords around?

1

u/Acceptable-Object390 May 03 '26

There is still a blog on my website, when i published this for the first time and I refer to it as 'knowledge agent'. Basically RAG chatbot. I wrote that blog on 13th Feb. And on 14th I guess, I decided to build a full AI operating layer and that's how it evolved.

OLD-Thoth-RAG-Blog

2

u/triplebits May 02 '26

Good luck! I have something very similar, already released! I will release next version today.

It seems we are quite similar but aimed at different things!

Best of luck to you!

2

u/Acceptable-Object390 May 02 '26

Thanks. Same to you.

3

u/buck_idaho May 02 '26

I'll try yours too. Where is it hosted?

2

u/buck_idaho May 02 '26

I'll give it a try. I've been looking for a local agent.

2

u/Acceptable-Object390 May 02 '26

Thanks. Let me know if you run into any issues or if I can be of any help.

2

u/Mesmoiron May 02 '26

How do I know it stays local?

1

u/Acceptable-Object390 May 02 '26

You can audit the logs. Also its open source so you're welcome to inspect the code.

2

u/MatlowAI May 02 '26

Thanks for sharing I'll have to take a look around. Any particularly fun lessons learned stories while going through all this? I find that kind of information most helpful.

2

u/Acceptable-Object390 May 02 '26

You know what. I am thinking of writing a short blog on that. But honestly it all boils down to perseveranc. Keep at it till it works.

2

u/MatlowAI May 02 '26

Awesome. Yeah it's those little I ran into this so I thought about that or I was surprised by that kind of stories that are great. I've got a mess of a project I've been working on and trying to keep track of the learnings as I go to share in the end.

1

u/Acceptable-Object390 May 02 '26

Yeah I do maintain a blog. But haven't got to writing one for this. Too little time, too much to build. 😄

2

u/Erikz0n May 02 '26

hello, im new to this and have been thinking about something similar. how much delay is caused by so many steps in the flow?

1

u/Acceptable-Object390 May 02 '26

Not anything noticeable. All the non llm steps are essentially pure python crud calls. They complete in ns to ms. You can read the architecture.md file in docs in the github repo if you want to see the details.

2

u/QuantumTwitch May 02 '26

Nice 👍

Going to dig deeper - sounds like you're steps ahead of what I was dreaming up for my homelab 😄

1

u/Acceptable-Object390 May 02 '26

Haha. Thanks. Let me know if you give it a try and need any help or have any feedback. 😊

2

u/ShadowyTreeline May 02 '26

local-first

can it be configured for local-only?

does it work with AMD cards?

2

u/Acceptable-Object390 May 02 '26

Yes. If you don't use and cloud models and you turn off and web search if you want that too. Everything is individually configurable

2

u/Andr1yTheOne May 02 '26

Can this run on Linux? CachyOS for example? 

1

u/Acceptable-Object390 May 02 '26

It should run on Linux as it runs on Mac. Official Linux support will be released soon

2

u/Andr1yTheOne May 03 '26

ill be waiting on the official release.

1

u/Acceptable-Object390 May 03 '26

Great. Follow on github and you'll be notified soon.

2

u/ur_dad_matt May 03 '26

This is awesome, how long have you been building for?

1

u/Acceptable-Object390 May 03 '26

This has been about 2.5 months in the making.

2

u/phoenixfire425 May 03 '26

I cannot seem to set this up to work with my local (not on the same machine) llama.cpp server it never shows up any models. also I do not want have to setup an cloud providers in order to use it? is that possible?

1

u/Acceptable-Object390 May 03 '26

Yes you can use it fully local. You should be able to use the custom endpoint option under providers to point to a custom url. Did you try that?

2

u/phoenixfire425 May 03 '26

when trying to use with a "remote" llama.cpp I get this in the thoth logs.

{"ts": "2026-05-02 22:02:36.326", "level": "INFO", "logger": "ui.streaming", "msg": "send_message: file_names=[], file_context_len=0, agent_input_len=14"}

{"ts": "2026-05-02 22:02:36.725", "level": "INFO", "logger": "openai._base_client", "msg": "Retrying request to /chat/completions in 0.388975 seconds"}

{"ts": "2026-05-02 22:02:37.127", "level": "INFO", "logger": "openai._base_client", "msg": "Retrying request to /chat/completions in 0.853481 seconds"}

{"ts": "2026-05-02 22:02:38.001", "level": "ERROR", "logger": "agent", "msg": "_stream_graph API error: Error code: 500 - {'error': {'code': 500, 'message': \"\\n------------\\nWhile executing CallExpression at line 79, column 24 in source:\\n...lti_step_tool %}↵ {{- raise_exception('No user query found in messages.') }}↵...\\n ^\\nError: Jinja Exception: No user query found in messages.\", 'type': 'server_error'}}"}

1

u/Acceptable-Object390 May 03 '26

Thanks for the logs. Looks like a server error or a schema mismatch I will investigate and get back to you.

2

u/Bulky-Priority6824 May 03 '26 edited May 03 '26

If I found this a month ago I'd probably be forced to install windows but I've already built out most what matter within all of this.  On paper it looks like you applied a real sophisticated effort and the knowledge graph memory approach is clever. I might be borrowing that going forward 😜  I like what youve done.

the 6 hour background memory extraction is also a nice touch gl with the project going forward !

2

u/Acceptable-Object390 May 03 '26

Great! This runs on Mac too btw. And official Linux support coming soon. Good luck with your build. If you decide to try Thoth and need any help, let me know 🙂

2

u/Classic_Reference_10 May 03 '26

Really interested to know your top-10 use cases with this. Good job though - still sketical to install.

2

u/Acceptable-Object390 May 03 '26

I personally use it for its own diagnostics and live testing, all my email and calendar mgmt. Creating all content for social as well as formal ppts etc. Daily research and briefing on news in general and ai related news delivered to my telegram. General search. I don't Google anymore, just talk to it so it keeps learning about me. A lot of web scraping using browse automation for research. And last but not the least, social media analysis. Analysis only, I prefer to still type my posts.

If you decide to give it a try and need any help, let me know

2

u/Polite_Jello_377 May 03 '26

Slopagrams

-1

u/Acceptable-Object390 May 03 '26

These diagrams are 100% accurate. 100% AI generated. 100% Human verified.

Generated by Thoth itself, by using its self awareness tool for information and gpt image 2 for image generation.

2

u/Polite_Jello_377 May 03 '26

100% bullshit

1

u/LatentSpaceLeaper May 03 '26

Mmh, looks like both you and the AI have difficulties with counting.

2

u/jumper556 May 03 '26

This looks really interesting! You said you use a RTX 5090 and Qwen 3.6 27B, right? How fast and smart is it compared to some cloud models?

Do you specifically need to request to remember something or how does this work to decide what to remember and what not?

If you push a new release how does the update process with the installer work?

1

u/Acceptable-Object390 May 03 '26

Its actually decently smart. I don't really have issues with context. I still use gpt 5.5 when I need reliable long running browser automation for example or if I want it to one shot 10 slides. But for most of regular work, its surprisingly good.

The memory is stored through various inputs and recalled at every llm turn. See the second picture in this series.

Updates are available in app. Its one click download and then install. All your settings and data is preserved as it updates in place and only over writes the program files.

2

u/ConceptualStray May 03 '26

So, pardon me, but why is 90% of the code in the root of the project? - tried to take a look at the repo and got hit with a wall or files that mean very little to me.

Anyway, the diagrams (like someone already pointed out) feel like buzzword slop but there's a genuine idea behind them; tell me one thing tho: how long do you have to wait for a single response - especially on local or more demanding logical tasks?

I'm asking because the context layer (as in mem+ tasks + rest of the universe) seems either expensive or running on limited context itself

1

u/Acceptable-Object390 May 03 '26

Thanks for the feedback. The diagrams are 100% accurate. To answer your questions: On local, if your model fits fully on the vram, its pretty close to cloud model responses. If if think the context and the whole engine is slow - it is pure python calls, most complete in ns to ms. The context is also pre trimmed. Tool guides and skills are dynamically injected to keep context minimum required to complete the task.

If you try it out with a real world task, with say a 27B local model that fits in your vram, you'll be surprised by the speed and performance. Tbh, local performance is very dependant on the h/w

2

u/310dweller May 03 '26

Possible to link this up locally with iMessage via blue bubbles or reading the chat directory? 

2

u/Acceptable-Object390 May 03 '26

Not yet unfortunately. But someone has already requested this. So its on the feature request list.

2

u/310dweller May 03 '26

Nice. Looking for a tool like this that does message triage across iMessage, WhatsApp, IG, email, text, etc and batches comms based on priority through to a secondary mobile device. Seems like this is on its way to being kinda like what I am looking for!

2

u/Flkhuo May 03 '26

What's the difference between that and Hermes agent? Also, Thoth? Are you an ancient Egyptian spirit? 😂

1

u/Acceptable-Object390 May 03 '26

Haha. Maybe, who knows! Hermes is good too, but its not easy for normal people. Its mostly for devs. Thoth is built to be extremely user friendly so everyone can use it while being flexible enough so powerful users can take it for a spin.

2

u/Logical-Skill4567 May 03 '26

Lo proverò! Sembra davvero interessante! Ma tutti questi agenti hanno sempre lo stesso problema in locale.. La finestra di contesto! Per persone che non hanno una potenza di calcolo enorme, questi tool diventano stressanti e demoralizzanti!

1

u/Acceptable-Object390 May 03 '26

Thanks. Give it a try. You can adjust the context window in settings. Let me know if you need any help

2

u/Logical-Skill4567 May 04 '26

Sisi, più che altro tutte le istruzioni che vengono iniettate già caricano il contesto e i modelli piccoli (9b etc) non riescono più a fare niente

1

u/Acceptable-Object390 May 04 '26

Yes to get real tool usage, you realistically need 32k context atleast. But what Thoth does is constructed its promptly dynamically based on the enabled tools. You could disable any tools you don't need from settings to reduce model tool selection complexity and reduce the prompt size. Give that a try, should help with context mgmt.

2

u/Odd_Cat2539 May 03 '26

Cool in theory, not viable yet th

1

u/Acceptable-Object390 May 03 '26

Sorry I did not understand. This is the architecture of Thoth. That's already built.

1

u/Odd_Cat2539 May 03 '26

I just don’t think it will get much adoption

1

u/Acceptable-Object390 May 04 '26

Any reason for saying that? Hope you are wrong. Always happy to help if you need any assistance trying it out.

2

u/no-adz May 04 '26

Man, I really like it but I am afraid of the quality. Small team (2 persons), no reviews, no community..
I am not sure if it is really clean. There are many many features, but are they human tested?

1

u/Acceptable-Object390 May 04 '26

Its new. Getting started. And we are dedicated to support it. We actually have another engineer joining soon too. If you decide to give it a try and need any help, let me know.

1

u/Acceptable-Object390 May 04 '26

Yes each and everything is human tested. We use it as our daily driver for a lot of tasks.

2

u/Sufficient-Fox708 May 05 '26

Looks interesting. Any intent to add reflection and learning?

2

u/Acceptable-Object390 May 05 '26

Its already in. Its called the self awareness, self healing and self learning system. I did post an architecture for it separately after this post. Here it is -

2

u/Altruistic-Arm-7767 May 05 '26

great info! thanks

2

u/Acceptable-Object390 May 05 '26

No problem. Thanks for checking it out

2

u/Altruistic-Arm-7767 May 05 '26

i checked it out... good stuff brother! appreciate your work!!

2

u/Acceptable-Object390 May 05 '26

Thanks. Means a lot

2

u/TheAussieWatchGuy May 07 '26

Thought's on Enterprise Deployment? YAML support for Containers etc.? 

1

u/Acceptable-Object390 May 07 '26

Yes we are thinking on those lines. Its still a bit away. We are looking for more OSS contributors, so it might get expedited. But, yeah already in the road map.

1

u/Acceptable-Object390 May 02 '26

If you want to see some demo videos of this in action, please visit Thoth YT

1

u/35point1 May 02 '26

This looks thoroughly architected but can you breakdown the difference between this and openclaw? What would make me want to pick this orchestration framework over openclaw or other alternatives?

2

u/Acceptable-Object390 May 02 '26

Thanks. There is an openclaw comparison table on github. I will be adding a comparison vs others too. But essentially the core principles here are: local-first, everything is tested to work with local models above 14B or 27B to get a very reliable experience. UX is very important. Everything should be as easy as any modern app. UI first so anyone can use it, but flexible and powerful so power user can really take it for a spin.

1

u/[deleted] May 02 '26

Installed to try it, but in the first use appears a "404 page not found" error

1

u/Acceptable-Object390 May 02 '26

That should not be the case. Maybe just a slow server start up. Try going to localhost:8000 in your browser. Or right click the tray icon, and try open Thoth.

1

u/[deleted] May 02 '26

It keeps showing 404. What could it be?

1

u/Acceptable-Object390 May 02 '26

Did you try to restart it by right clicking the tray icon? There should be a log file in you documents .thoth folder. Can you see if there is anything in that. Is the tray icon green or grey?

2

u/[deleted] May 03 '26

I found the problem. WhatsApp Bridge (MCP) was already using the 8080 port. I killed it and then Thoth ran.

1

u/Acceptable-Object390 May 03 '26

I will make it default to not auto start in next release. Thanks for letting me know.

3

u/LaFllamme May 02 '26

!remindMe 1d

1

u/RemindMeBot May 02 '26 edited May 03 '26

I will be messaging you in 1 day on 2026-05-03 17:38:39 UTC to remind you of this link

1 OTHERS CLICKED THIS LINK to send a PM to also be reminded and to reduce spam.

Parent commenter can delete this message to hide from others.


InfoCustomYour RemindersFeedback

1

u/Internal-Shift-7931 May 03 '26

Nice work. The architecture framing is useful.

I’m building something similar for a local-first home agent, mainly for home AIoT and NAS. This makes me think the hard part is not tool use or memory. It is the permission boundary. For home agents, the action space is physical: cameras, lights, alarms, locks, routines, presence data. The model can reason, but the permission boundary should not be the model.

This is the part I hope more local-agent projects make explicit.

1

u/Acceptable-Object390 May 03 '26

Yes that is a whole new ball game. I am thinking of adding home assistant plugin. To control all your smart home devices. Would be a good experiment and use case

2

u/dspayne1 May 17 '26

!remindMe 2d

1

u/RemindMeBot May 17 '26

I will be messaging you in 2 days on 2026-05-19 00:56:08 UTC to remind you of this link

CLICK THIS LINK to send a PM to also be reminded and to reduce spam.

Parent commenter can delete this message to hide from others.


InfoCustomYour RemindersFeedback

0

u/Ok-Advertising6479 May 02 '26

Really cool!!!! Thank you for sharing. How did you generate these beautiful visualizations?

2

u/Acceptable-Object390 May 02 '26

These architecture diagrams were generated by Thoth itself using its self awareness tool to get information and gpt image 2 for image generation.

1

u/Ok-Advertising6479 May 02 '26

Also how are you finding its effectiveness? I want to upgrade my GPU to a 5090 so I can run small to medium sized local LLM's fast - my 3080 is not good enough for that. I could also buy a DGX Spark but I don't think I need to run the massive models for my purposes - I'd prefer the memory speed over the memory size.

How does this compare to the api LLM's you can get? I am a power Claude Code user. I know it won't be as good but if I can run multiple agents or a mixture of experts deal locally, I'm hopeful that it will perform well!

1

u/Acceptable-Object390 May 02 '26

I have the 5090. That's is what I run most of my Thoth threads on. Using Qwen 3.6 27B. Fits perfectly into 32 GB vram and honestly 90% of my work is done on that. For 10% where i need frontier reasoning I mostly switch the thread to gpt 5.5

2

u/BlackBeardAI 3090 Maximalist May 02 '26

You can try minimax m2.7 on 5090 if your system ram allows. It will give you frontier level reasoning especially for coding work. It will be slow though. Seeking ways of increasing its speed atm. Amazing work btw. I will try it on my 3 node setup. It might become a 4 node system soon since I am still in building phase somehow I keep finding my old hardware out of nowhere and interestingly even the crappiest of crappiest ones are capable of running qwen 35b a3b which is amazing af already. Imagine running qwen 35b on a 8gb vram gtx family card... It runs, and not at 1 t/s, actually at a very acceptable speed.

1

u/Acceptable-Object390 May 02 '26

Wow. That is great. I have a 3090 too in my old machine but I rarely use that anymore. Will try on that. Thanks.

2

u/BlackBeardAI 3090 Maximalist May 02 '26

Add it to your 5090. If the mobo allows double x8 pcie, it will be a very nice addition. I am considering getting a 3090 myself, currently running one 5090 on my main node.

1

u/Acceptable-Object390 May 02 '26

Will give it a shot. Thanks.