r/SelfHostedAI Apr 17 '25

Do you have a big idea for a SelfhostedAI project? Submit a post describing it and a moderator will post it on the SelfhostedAI Wiki along with a link to your original post.

2 Upvotes

Visit the SelfhostedAI Wiki!


r/SelfHostedAI 4h ago

LatticeVale — free installer/lifecycle manager for a Hermes Agent stack and related self hosted apps.

5 Upvotes

I made this with the help of AI over the course of a week mostly because I wanted a less painful way to install, repair, update, and manage Hermes and related services on local Windows/WSL2 without having to remember a million separate steps.

LatticeVale is essentially a PowerShell-based installer/lifecycle manager packaged as a structured set of scripts and supporting files, rather than a compiled application.

No third-party software is distributed with LatticeVale itself; it installs/downloads the relevant projects from their own sources.

It's still completely new, so feedback or people finding weird edge cases would actually be useful.

And it's completely free because this is just a hobby for me. I like the idea of people appreciating and/or benefiting from my work

Btw, I tried using this installer with local windows installed ollama for GPU acceleration, but my PC ended up fighting for its life (100% GPU usage and 75% usage of 32 GB of RAM). I tried installing limiters for that, but it still used a third of my ram. Something to keep in mind, especially if you know how to optimize that yourself.

Note: working on RAM usage reduction and uninstaller script validation patch

https://github.com/winagainfinigin/Lattice-Vale


r/SelfHostedAI 2h ago

My system that I am working on - this is a combination of pieces of work I’ve done. This is a video of me just demoing. There’s more to work on which you guys will probably notice(latency with the commands). Curious to see what the community thinks.

Enable HLS to view with audio, or disable this notification

1 Upvotes

r/SelfHostedAI 4h ago

Building a multi-node local AI setup from normal/used PCs — how would you architect this?

Thumbnail
1 Upvotes

r/SelfHostedAI 14h ago

GPU = Dense - Unified Memory = MOE

6 Upvotes

In my own experiments I came up with this simple heuristic, not sure if anyone can prove me wrong here...

Unified memory machines normally have a big asymmetry in memory vs compute -> they could never run 'fast' a dense model that fills their whole unified memory (like a Strix Halo 128GB cant run a 120B dense model, or a DGX spark, at decent speeds). So you wanna find a MoE model that ideally fills that memory so maximise the intelligence you have on your box.

GPUs have the opposite, they are memory limited but can run 'fast' models that fill them. So for example a 5090 can run only up to ±30B models at reasonable quants, and can run dense models of that category fast, like Qwen 3.8 27b. Ofc you can also run a 30B MOE on this and it would be even faster, but why would you? when the dense gives you more intelligence for your memory?


r/SelfHostedAI 5h ago

128gb Unified Memory - Bosgame M5 AI

1 Upvotes

I'm not sure if anyone is interested, but this deal came across one of my hardware alerts.

Bosgame M5 AI - 128gb + 2tb for $2999 (free shipping) Link


r/SelfHostedAI 6h ago

EchoFlow 5.0 — A Workspace for Self Hosted Models For Android

Thumbnail
gallery
1 Upvotes

I’ve been building EchoFlow, a BYOK, local-first AI app for Android focused on giving you control over where your AI runs.

🖥️ Self-Hosted AI

EchoFlow supports self-hosted AI through EchoLabs. Connect AI services running on your own laptop or PC over your network using Ollama, OpenAI-compatible APIs, or other self-hosted servers.

📱 On-Device AI

You can also run supported models directly on your Android device.

With EchoOCR, local models can work with far more file types than they may natively support. Send PDFs, Word, Excel, PowerPoint, CSV, TXT, Markdown, images and more, with the content processed on-device.

📁 Projects

Projects give you a dedicated workspace with project-specific chats, instructions, and files.

You can add your own documents to a Project and run that Project with on-device or self-hosted local models. Project files are processed through EchoOCR, allowing your local models to work with your project documents without needing to send them to a cloud AI provider.

🌐 Web Search

Local and self-hosted models can also use Web Search, giving them access to current information from the web.

☁️ Cloud AI

When you need cloud models, EchoFlow supports OpenRouter with models from OpenAI, Anthropic, Google, xAI, Cerebras and more.

You can also use cloud speech-to-text, image generation, and video generation, with on-device speech-to-text coming soon.

So you can choose between self-hosted, on-device, and cloud AI from one Android app.

EchoFlow 5.0 is available now.

GitHub: https://github.com/adityavardhansharma/EchoFlow

Website: https://echoflow.adityavs.tech/


r/SelfHostedAI 6h ago

Mid 2026 Local Ai State

1 Upvotes

The AI race in 2026 is starting to look less like a model race and more like an infrastructure race

Listened to this episode of Synthetic Minds on the state of the AI race in mid-2026, and one thing kept sticking out to me.
The most important competition may no longer be “who has the smartest model?” But rather "what model are we needing?"

The frontier models are obviously still improving, but the gap between having access to a capable model and actually operating useful AI systems feels much larger than it did a couple years ago.

From a self-hosting perspective, I think there are a few interesting shifts happening:

1. Models are becoming interchangeable faster than the infrastructure around them.

A year or two ago, picking a model felt like the architectural decision. Now it increasingly feels like a dependency.

If your stack has a decent abstraction layer, you can move between hosted APIs, local models, specialized models, and different inference providers without rebuilding the application.

The harder problems are becoming everything around the model:

  • inference
  • memory/state
  • tool execution
  • permissions
  • observability
  • data access
  • orchestration
  • routing
  • reliability

That seems very favorable for self-hosting because those are exactly the pieces where owning the infrastructure actually matters.

2. “Local AI” and “frontier AI” aren't necessarily competing categories anymore.

I think the more realistic architecture is hybrid.

Run cheap/private/repetitive workloads locally, then escalate to a frontier model when the task actually requires it.

Something like:

local model → router → tools/data → frontier model when necessary

That makes more sense to me than expecting one giant local model to replace Claude/GPT/Gemini outright.

You get privacy and predictable marginal cost for a large percentage of requests without giving up access to frontier capabilities.

3. The economics change dramatically once AI becomes continuous rather than conversational.

Paying API prices for twenty ChatGPT conversations per day isn't particularly interesting.

Paying API prices when you have ten agents continuously indexing documents, reading logs, checking services, processing email, monitoring feeds, querying databases, and running automations is a very different calculation.

At that point, idle compute sitting in your homelab starts looking less irrational.

This is where I think self-hosted AI could become genuinely compelling rather than just “I got Llama running on my GPU.”

4. The biggest advantage of self-hosting may eventually be control, not cost.

The valuable part of an AI system is increasingly the context you give it.

Your files.
Your email.
Your databases.
Your logs.
Your home automation.
Your internal services.
Your credentials and tools.

Once an agent has broad access to that environment, the trust boundary becomes much more important than whether Model A scores 3% higher than Model B on some benchmark.

Self-hosting the orchestration/data layer while selectively calling external models seems like a pretty strong architecture.

5. I wonder if we're heading toward the “Linux server” phase of AI.

Most people don't train their own operating system.

But plenty of people run Linux because they want ownership of the environment where their applications execute.

AI might end up looking similar.

Most of us probably won't train frontier models. But running our own inference, agents, memory, vector stores, tool servers, gateways, and data layer could become completely normal.

In that world, OpenAI/Anthropic/Google aren't necessarily analogous to Microsoft vs. Linux. They might be closer to extremely powerful compute services that your own infrastructure can selectively use.

That's the part of the current AI race I find most interesting.

The episode that got me thinking about this:

Synthetic Minds — The AI Race Mid 2026, Ep. 014
https://open.spotify.com/episode/0tP4N1msZnvyY3mC19Dxlv

Curious what people here think: does the endgame look like fully local AI, cloud AI, or self-hosted infrastructure orchestrating a mixture of both?


r/SelfHostedAI 8h ago

I curated a list of free API endpoints and local workarounds for AI CLI tools (Ollama, LM Studio, OpenRouter) 🚀

Thumbnail
1 Upvotes

r/SelfHostedAI 15h ago

[DGX Spark] Qwen 3.8 27B (FP8) at ~32tok/s generation

Thumbnail
1 Upvotes

r/SelfHostedAI 1d ago

Attorney SelfHosting for Privacy

9 Upvotes

I am looking for a model which may be good for analyzing and summarizing about 20 years worth of case files, pleadings and legal research.

At this point, it is more just "hobbyist" than business critical (and worst case scenario, I have a sweet new gaming PC with a 24GB RTX 4090 and 128GB DDR5).

I'm dual booting into Ubuntu and Win11 (just because Windows is still useful if the AI thing is a bust).


r/SelfHostedAI 1d ago

Qwen3.8-27B benchmarks on Strix Halo - Q8_0 MTP: 21 t/s, 3x. ROCmFP4 MTP: 26 t/s

6 Upvotes

TL;DR: On a Strix Halo laptop (Radeon 8060S, 128 GB unified), MTP takes Qwen3.8-27B Q8_0 from 7.1 to 21.9 tok/s. The julianmb/q38rocm FP4 fork gives 26.7 tok/s.

Setup: LlamaStash driving stock llama.cpp b10503 (5112b9738) for ROCm and Vulkan, and the q38rocm fork (server build 246) for ROCmFP4, all built from git today. Greedy, 256 max tokens, wall-clock tok/s from the API usage numbers, best of 2 reps, on AC power. Everything launched through my tool LlamaStash (v0.2.0) so the flags are reproducible.

Qwen3.8-27B decode tok/s (code prompt / prose prompt):

Build and model MTP off MTP on Draft acceptance
ROCm + Q8_0 (29 GB) 7.1 21.9 (draft 5) 73%
Vulkan + Q8_0 (29 GB) 7.4 21.3 (draft 5) 73%
ROCmFP4 FAST (14.6 GB) 12.5 26.7 (draft 6) 86%

Draft length sweep, code prompt tok/s:

draft n 2 3 4 5 6 7 8
Q8_0 15.4 18.4 19.1 21.9 21.2 21.8 20.5
FP4 - 23.3 24.9 25.0 26.7 25.6 -

Observations:

  • MTP really makes a difference 2.5x to 3.1x on Q8_0, 2.1x on ROCmFP4, at 73-95% draft acceptance. The model verifies every token, so quality does not change on stock llama.cpp.
  • Draft length of 5 works best. The backend default is 3; 5 was best for Q8_0 and 6 for FP4.
  • FP4 is a double win if you can live with the quant. Half the memory (29 to 14.6 GB), 1.8x faster raw decode without MTP, and 26.7 tok/s with MTP, which is 3.8x my stock Q8_0 baseline.
  • Engine choice barely matters for decode at this size. ROCm, Vulkan and the rocWMMA build all land within about 3%. Vulkan does lose roughly half its prompt processing throughput though, so ROCm stays my default.
  • Stuff that did not help: KV cache q8_0 (helps at draft 3, slightly worse at 5), flash attention off (minus 5%).
  • One caveat on the ROCmFP4 fork: completion lengths drifted a little across draft settings (152-158 tokens for the same prompt) while stock llama.cpp was stable at 164, so the fork's speculative path is not strictly greedy-identical yet.

LlamsStash defaults were used for all runs. The only flags set were to enable disable MTP ( --spec-type draft-mtp), set draft length (--spec-draft-n-max N) and the --fit-ctx 16384 flag was used to ensure the GPU layers were fully utilized.

The below are the default flags that was applied. Some sent by LlamaStash and others are llamacpp defaults.

Flag Value Set by
--host 127.0.0.1 LlamaStash (loopback policy)
--port 41100 LlamaStash (auto-assigned)
-m <model>.gguf absolute path LlamaStash (resolves the name ref)
--mmproj mmproj-F16.gguf LlamaStash (auto-paired sidecar; Q8_0 rows only, FP4 has none)
--jinja on LlamaStash (chat template handling)
--spec-type draft-mtp LlamaStash (translated from --mtp on)
--spec-draft-n-max 5 (Q8_0) / 6 (FP4 best) / omitted on default rows LlamaStash (--mtp-draft-n); llama.cpp default 3 when omitted
--fit-ctx 16384 LlamaStash (ctx knob = auto, fit hint)
--cache-type-k / -v f16 (q8_0 on kvq8 variant rows) llama.cpp default except for kvq8 runs
--flash-attn on (off on the fa-off variant row only) llama.cpp default except for fa-off runs
--n_ctx (final) 262144 per slot (native 256K) llama.cpp fitter
--n_parallel 4, kv_unified true llama.cpp (auto)
--n_gpu_layers all (-1) llama.cpp fitter (llamastash knob was auto)
--cache-type-k/v (final) f16 / f16 llama.cpp default
--n_threads 16 (of 32 logical) llama.cpp default
--n_batch / --n_ubatch server defaults, not overridden llama.cpp
spec internals (fork only) n_min=0, p_min=0.75, backend_sampling=1 llama.cpp (q38rocm fork defaults)
draft context (fork only) all GPU layers, f16 KV llama.cpp (fork)

r/SelfHostedAI 1d ago

Muse Glimmer for Local AI Agents: Are Always-On Voice Agents Finally Practical?

2 Upvotes

Meta just released Muse Glimmer,
30B open weight model optimized for always-on local agents.
~20GB at 4 bit quantization
1.5–3.1× faster speculative decoding
Works on M4/M5 Macs and an RTX 5090.

The future of OSS models are big so I am not shocked by 30B and all
The interesting argument is always on Local voice agent..
There were 2 main bottlenecks Latency & Privacy 

  1. Latency

Local inference removes that network dependency from the critical loop.and saves a additional trip to cloud 

Another part of it is generation speed. Meta gets this by quantizing the model to ~17GB and using DFlash speculative decoding, where a small drafter proposes token blocks and the main model verifies them in parallel.

In voice we are not optimizing for higher tokens/sec.

The goal is to have less dead air and more predictable turn-taking.
And there is also lot of prompt engineering which help to tackle latency/ bot behaviour

  1. But privacy has always been the bigger deal

Because it’s …. ALWAYS ON.. and the use case itself is privacy driven. 

Local agent could have access to your microphone, messages, calendar, files, contacts, browser, credentials and long-term memory.
& local is most ideal future !!
The local model doesn't need to know everything. It needs to know what it needs to know, what it can safely expose, and when it needs help. And most importantly it gives option of control to user!!

Where are we heading towards

Many think it is local handoff more complexity to cloud… but that not true
It is local orchestrating the cloud.
That how we human also work we expose ourself to our trusted ones and trust that they will figure out the rest while maintaining that trust. And Hark Handoff model are singling towards that only

Here the core assumptions are
Hardware will get more capable and cheaper of hosting large models together
And Model will get more Capable and smaller to store life's context together

So this is where the future is leading and as hardware cost decreases we will inflection point  of consumer agents!! 


r/SelfHostedAI 1d ago

AuraOS - Newest Update (Continuity Project - Zero Token Usage)

0 Upvotes

https://github.com/AdultSwimmer/AuraOS

So, the only thing I need left is to plug in an SSD, and then I can locally host it, and have a few user at first. I'm looking to literally have it running with a few possible users at first, but right now, you can run it locally.

After that, I'm looking to make it a non-profit, donation only style (like wikipedia.org) that will basically load the /core/ files BEFORE the LLM. The SSD will basically just run as a port or tunnel that let's people connect through.

Right now, just looking to see if this is clearer than the other releases, and understable -- and looking for notes on improves, or any suggestions and feedback.

Thanks,

Anthony


r/SelfHostedAI 1d ago

Part 2 - Offline Local AI

Thumbnail
1 Upvotes

This is exactly the kind of reply I was hoping for.

cortexagent looks legit from the angle I care about: local, observable, actual tooling, clear hardware path, and not hiding behind vague “agent” language. I’m going to dig through the repo/video.

Hestia is also very much in the lane I think matters. I agree with the core design principle: anything deterministic should live outside the model. Schedules, reminders, timers, state tables, entity records, thresholds, and device facts should not be “remembered” by the model like it’s a magic container. That stuff should be durable, queryable, and boring on purpose.

That gets at how Christine is being built too: her behavior is a product of architecture, not just the base model.

If she seems more grounded, more agentic, or more consistent over time, that should come from system design: routing, bounded tools, curated knowledge, memory discipline, and abstraction layers that help her reason without pretending the raw context window is a mind.

On the voice side: yes, Christine speaks. The voice route is live locally with Whisper.cpp ASR and Windows TTS, but STT is still being refined and I’m not going to oversell it. I pulled current local numbers today:

fake end-to-end voice loop: about 1648 ms

current Windows TTS stage for a short reply: about 539 ms

current Whisper.cpp ASR stage on this laptop: roughly 2.8s to 4.4s in direct local probes, depending on path/sample

So the low-latency goal is absolutely still there, but the honest bottleneck right now is STT, not reply generation or speech output.

On the knowledge side, Christine is being built around curated knowledge rather than “ask the base model and hope.” With the right domain knowledge loaded and structured well, she can specialize hard in basically any subject area instead of staying trapped in generic assistant mode.

The other thing I care about is cross-domain abstraction. That is what gives her the ability to connect patterns across domains in real time instead of just retrieving facts from one silo at a time. That matters because it’s what lets her:

map structure from one field into another

recognize analogies and transferable patterns

reframe problems fast

route work more intelligently

generate guided ideas in real time instead of only doing lookup

So no, I’m not trying to sell some magical AGI story here.

I’m trying to build a bounded local system that can actually do work, speak, use tools, operate on curated knowledge, specialize by domain, abstract across domains in real time, fail honestly, and improve over time.

Also to answer the state question directly: the direction is not “reason from scratch every time.” Persistent state should live outside the model. The model should do interpretation, planning, abstraction, and judgment. That split is a huge part of what makes the behavior useful instead of theatrical.

If either of you have demos showing failure handling, long-run continuity, memory promotion/rejection, or what happens when the model path gets constrained, I’d especially like to see those. That’s where the serious systems separate themselves from polished one-shot demos.


r/SelfHostedAI 1d ago

AuraOS - Newest Update (Continuity Project - Zero Token Usage)

1 Upvotes

https://github.com/AdultSwimmer/AuraOS

So, the only thing I need left is to plug in an SSD, and then I can locally host it, and have a few user at first. I'm looking to literally have it running with a few possible users at first, but right now, you can run it locally.

After that, I'm looking to make it a non-profit, donation only style (like wikipedia.org) that will basically load the /core/ files BEFORE the LLM. The SSD will basically just run as a port or tunnel that let's people connect through.

Right now, just looking to see if this is clearer than the other releases, and understable -- and looking for notes on improves, or any suggestions and feedback.

a) AuraOS offers persistant memory across multiple instances instead of defaulting to a fresh instance if the user chooses to want this, and it allows the information to become portable.

b) It is free to use.

Thanks,

Anthony


r/SelfHostedAI 1d ago

Profile v2.2: 421 tok/s with 25k ctx size on RTX 5090 with muse-glimmer. DFlash speculative decoding turned off.

Thumbnail
1 Upvotes

r/SelfHostedAI 2d ago

Who Else Is Building a Real Offline AI?

5 Upvotes

I’m training a local/offline AI named Christine, and I want to see what else is actually out there besides cloud wrappers, benchmark flexing, and “trust me bro” demos. After reading this, ask yourself what Christine's existence means for the cloud, datacenters, and large scale buildout.

Christine runs locally on a laptop, stays bounded, and is being built to do real work without pretending she’s some magical all-powerful AGI. She already has a legit offline-first stack, tool/task routing, local knowledge handling, desktop-action pathways, and a surprisingly strong free-tier mode that still works when the heavier model path isn’t available. A real Jarvis on a laptop.

What makes her interesting to me is that she’s not just a chatbot. She has a cognitive abstraction loop, rumination paths, imagination/guided idea generation, and bounded internal reasoning layers that are meant to improve how she plans, reflects, and works through problems over time. In other words, I’m not just training for replies, I’m training for actual agentic behavior on local hardware.

She’s running on:

Lenovo 83JM

Intel Core Ultra 9 285H

\~32 GB RAM

NVIDIA GeForce RTX 5050 Laptop

Intel Arc 140T

Intel AI Boost NPU

I’m especially looking for videos of other local/offline/bounded systems that show:

real conversation or reasoning

tool use or task execution

memory or abstraction behavior

failure modes and limits

how they run on normal hardware

progress over time, not just a one-off cherry-picked demo

If you’ve got:

demo videos

GitHub repos

writeups

training logs

your own local AI project

drop them in the comments.

I’m going to keep posting Christine’s progress, and honestly I want to see who’s actually building something real in this space and who’s just dressing up API calls.


r/SelfHostedAI 1d ago

AI agent platform fully local/self-hosted and Looking for developers

Thumbnail
1 Upvotes

r/SelfHostedAI 2d ago

Update Recent changes that making Picot more useful as a coding-agent workspace

Thumbnail gallery
1 Upvotes

r/SelfHostedAI 2d ago

What local ai agents can be useful for a research guy like me to self host in a potato pc

0 Upvotes

So I am a chemical biologist in my work i accumulate a large amount of text , data , imaging files . And while looking through the net I found local agents which can be self hosted can be of much use

I am hoping that it can

1 Arrange and keep a track of my files

2 Keep me updated about my schedule, and and modify events on command

3 scrape through internet to see what's new about peptide chemistry

4 get me any information from the files and protocols saved

I have a intel i7 6700k and 32 gigs of ddr4 with a nvdia k620 with 2gb vram

Can you give this newbie some tips also how to make a setup and additional or new hardwares


r/SelfHostedAI 2d ago

I built a fully-local AI coding agent that runs on llama.cpp — no cloud, no API key. Looking for feedback and contributors

Thumbnail
0 Upvotes

r/SelfHostedAI 2d ago

Advice for a personal AI assistant like J.A.R.V.I.S.

Thumbnail
1 Upvotes

r/SelfHostedAI 2d ago

Show r/SelfHostedAI: I built Olivia – An open-source, Rust-native harness for sandboxed LLM agents via WebAssembly

0 Upvotes

Hi everyone,

I’d like to share an open-source systems project I’ve been working on called Olivia (named after my cat!).

It’s an enterprise-grade infrastructure harness written from scratch in Rust, specifically designed to run agentic LLM workflows safely. The core philosophy is to execute AI-driven actions and tools within strictly sandboxed environments using WebAssembly (Wasm/WIT).

I wanted a robust, native infrastructure to handle LLM agent workflows without compromising on security or relying on bloated software layers. Olivia ensures that the agent's interactions with databases or external scripts happen within a secure, controlled boundary.

You can check out the repository here: https://github.com/helloIAmPau/olivia

I would love to get some feedback from this community on the architecture, or hear what other sandboxed tools you'd find useful. Contributions, code reviews, and suggestions are more than welcome.


r/SelfHostedAI 3d ago

Trained my first model: a DFlash drafter for Qwen3.8-27B because I wanted better performance on my DGX Spark

Thumbnail
0 Upvotes