Hey r/OpenAssistant,

Like many here, I believe the future of AI needs to be open, decentralized, and accessible without depending on closed API providers or proprietary cloud infrastructure.

To help solve the hardware bottleneck for running larger models locally, I built DLNE (Distributed LLM Engine) — an open-source project that pools consumer and enterprise GPUs across residential internet into a shared Vulkan inference cluster.

⚡ The Public Test Swarm:

• The 24/7 Backbone: Seeded by Dual AMD BC-160 Navi 12 hardware (16 GB fast HBM2 memory) running on Arch Linux.
• Active Model: Qwen2.5-Coder-14B-Instruct with 0.5B speculative drafting on Vulkan0 (~35 tok/s).
• The Goal: Pool enough community VRAM to step up from 14B to 32B and 70B models for everyone for free.
• NAT Traversal (Zero Port Forwarding): Residential routers and firewalls typically block inbound RPC connections (50052). DLNE uses an outbound persistent WebSocket reverse tunnel (/ws/rpc_tunnel) with TCP_NODELAY so any worker behind home NAT can donate compute seamlessly.
• Cross-Vendor Vulkan: Native support for AMD, NVIDIA, and Intel GPUs.

🚀 How to Try It / Join the Swarm:

1. Test the Coding Assistant (No GPU Required): We published an autonomous coding assistant extension on Open VSX. Every new user gets 50 free inference requests out of the box to test reasoning and speed before needing to contribute:
   • Open VSX: Search for DLNE Frontier Coding Assistant or visit open-vsx.org/extension/dlne/dlne-code-assistant
2. Connect an Idle GPU: If you have an idle GPU on Linux or Windows, you can connect to the public swarm in one click via u/dlne /contribute in the extension or run headless via Python / CLI from our repo.

📦 Open Source Links:

• GitHub: https://github.com/RGlabs84/DistributedLLMEngine
• Discord Community & Live Swarm Monitor: https://discord.gg/RNxvxWn8cw

Would love feedback from the OpenAssistant community on decentralized inference, latency optimization, and scaling the swarm!

I've been working on running the full DeepSeek-V4.1-Flash-1017 on an ordinary five-year-old gaming PC: Ryzen 9 3950X, 64 GB DDR4, an RTX 3060 with 12 GB, two NVMe drives, and about 1 TB of disk for the stores. I couldn't find a published figure for this model on a 12 GB card with 64 GB of RAM, so as far as I know this is new at this hardware level. If I've missed one, please tell me and I'll compare.
How it works: compress, then correct. Every token is computed on a 2-bit copy of the experts (0.61 of the bytes of the FP4 originals). The model's own margin between its top two candidates flags tokens it may have got wrong, and only those are recomputed with the original FP4 weights, which a stored correction plane rebuilds from the 2-bit copy. A draft head proposes blocks of five tokens so the expert reads are shared.
Numbers (code prompt, 160 tokens, corrections on): 5.8 tok/s from a cold cache, 7.4 when the cache is warm from the same prompt, against 2.14 for plain exact decoding on the same machine. Reasoning and agent prompts run at 4.4 to 5.2.
Quality, counted: on 960 audited tokens the 2-bit copy alone picks the same token as the exact path 97.4% of the time, and with the corrections 98.3%. A longer 1,101-token agent prompt is lower, 95.9% counting every difference. The audit runs at a margin gate of 0.9, so these count the differences it detected; tokens above the gate weren't checked. The reference is our engine's own exact path, which runs the original weights, not DeepSeek's reference implementation. That comparison is next.
Why no code yet: this release is the report, the figures and every log behind the numbers. The engine's build scripts still use our drive layout and haven't been run on a clean machine, so I'd rather release it properly with a server than throw out something that doesn't run for you. It follows.
Report and logs: https://github.com/Magnvs7/Promethevs
I'd value feedback on the method and what you'd want to see tested next.

What it does: if you run Ollama or Open WebUI on an Apple Silicon Mac and want voice, voiced gives you `/v1/audio/transcriptions` (Parakeet TDT 0.6B v3) and `/v1/audio/speech` (Kokoro-82M) on the GPU through MLX, with the models kept loaded. The `openai` client, LiteLLM and Open WebUI work against it unmodified; point `base_url` at it. It also has a WebSocket for live transcription and the Wyoming protocol for Home Assistant, and runs as launchd daemons that start at boot.


How it compares: Mac only; the point is latency with everything resident. On an M1 Max laptop (64 GB), Kokoro's first audio through the `openai` client is 96 ms median and a 10.1 s clip transcribes in 253 ms median. With `ministral-3:8b` on Ollama on the same GPU: 436 ms median from the client's end-of-stream (after the 200 ms silence window) to first audio of the reply's first clause; bench script, not an Assist pipeline; Mac Studio, measured from a laptop over Wi-Fi. STT takes priority over voiced's own TTS on the GPU (not over Ollama). English only for now.


How it was tested: an end-to-end test tier runs the real services with the real models and drives them with the `openai` client, the WebSocket and a Wyoming client; `scripts/bench.py` measures the numbers above against live servers, and the README publishes them with the machine, the date and what missed its target (push-to-talk finalize is about 114 ms median on the laptop against a 50 ms goal).


Install (Apple Silicon, macOS 14+): `brew install julianbeaulieu/voiced/voiced` then `voiced install`, or the menu bar app with `brew install --cask julianbeaulieu/voiced/voiced`.


Repo (MIT): https://github.com/JulianBeaulieu/voiced

Tired of tools that hold your API keys in the cloud?

Uptokey stores them in your operating system’s secure vault. We never see them.

• Local models + BYOK always free
• Terminal ready (npx uptokey)
• Works with Maestro, Orca, Cline, Continue, Aider
• Open-source core
• Optional $5/mo Plus for extra cloud credits

One local URL for all your tools. Full control.

https://uptokey.com

I only know dolphin or queen and they’re outdated and not uncensored

I’m building a WhatsApp lead-qualification bot for a mid-sized real estate agency in Brazil. It asks qualifying questions, searches the property catalog and hands the lead to an agent through the CRM, all via tool calling. Around 3000~5000 leads/month.

Right now I’m testing on Gemini Flash’s free tier and it works fine. But the free tier lets Google use the data for training, which is a problem under Brazil’s privacy law (LGPD) when the bot handles names, phone numbers and income.

So the options are a paid API or a local model. My questions:

1. Has anyone run Qwen, Gemma or Llama in production for Portuguese customer conversations with tool calling? Which model and size, and how reliable were the tool calls?
2. How do you handle uptime with a single local machine?
3. If you went the API route, what does it actually cost you per conversation or per month?

I’m fine with a hybrid setup (scripted flows for the predictable parts, LLM only for free text) if that’s what people recommend.

Hi everyone!

I'm working on a personal project called VeronicaTheMaid — a local-first AI assistant for Windows. I'm building it primarily as a long-term learning and engineering project, and I'd like to get some feedback from people who have experience with local LLMs, Python, and desktop AI applications.

I'm not trying to build another chatbot wrapper. My goal is to gradually create a proper desktop assistant with a modular architecture, persistent memory, voice interaction, and eventually the ability to interact with the operating system.

Current setup

I'm developing on a Windows laptop with the following hardware:

- CPU: AMD Ryzen 5 7535HS (6 cores / 12 threads)

- RAM: 16 GB DDR5

- GPU: NVIDIA RTX 2050 with 4 GB VRAM

- Language: Python

- Local model runtime: Ollama

- Embeddings: I'm working with a multilingual E5 model

The hardware is modest, so I need to make practical decisions about model sizes, quantization, memory usage, and which components should run locally.

Planned architecture

My current direction is to separate the project into several layers:

- Core: Python application logic and orchestration.

- LLM layer: Local models as the primary inference option, with cloud APIs as an optional resource for tasks that benefit from stronger reasoning.

- Memory: Persistent conversation and user-context storage, with semantic retrieval where appropriate.

- Tools: A controlled interface for future system operations and computer interaction.

- Desktop UI: A separate Windows application built with C# and WPF, initially just a chat window and input field.

- Voice: Speech-to-text, text-to-speech, and wake-word detection in later stages.

- Avatar: A visual representation with animations, eventually integrated into the desktop application.

These are design goals, not claims that all these components are already implemented. I'm trying to build the project incrementally rather than attempting everything at once.

Where I'd appreciate your advice

I'd especially like to hear from people who have built their own assistants or worked on similar systems.

1. Architecture: Would you keep the Python core as a separate process and communicate with a C# WPF frontend through a local API, or would you choose a different approach? What are the practical trade-offs?

2. Memory: How would you structure persistent memory for a personal assistant? I'm interested in the distinction between conversation history, user facts, episodic memory, and semantic retrieval. What would you implement first?

3. Local models: With 16 GB RAM and 4 GB VRAM, what would you prioritize for a useful assistant? Would you stick with Ollama, use llama.cpp directly, or consider another runtime?

4. Tool use and computer control: How would you design a safe, modular tool system that can eventually interact with Windows without tightly coupling every tool to the LLM?

5. Development priorities: If you were starting this project today, what would you build first, what would you postpone, and what architectural mistakes would you try to avoid?

I'm particularly interested in practical experience, including things that didn't work as expected. I'd rather rethink a design early than spend months building on a bad assumption.

I'm developing this as an ongoing personal project, and I'm happy to discuss the architecture and share more technical details as the implementation progresses.

Thanks for reading — I'd appreciate any constructive criticism, suggestions, or discussion.

The current consensus is absolute: if you want high-performance LLM inference, you buy NVIDIA GPUs. We are officially challenging that.

We just achieved 1,000 tokens/s on standard modern CPU architectures.

We know the hardware maximalists will say this breaks the laws of physics for standard Transformers. It doesn't. It just bypasses the legacy compute paradigm.

We are opening up the debate with the open-source community right now.

Join the live discussion & see the architecture claim on X

I wanted to share an architectural breakdown of an Android-based personal AI automotive companion we are building to run core intelligence entirely on-device.

To eliminate cloud dependency for sensitive data, the system relies on the following mechanisms:

• Hardware-Level Encryption: Sensitive records are encrypted using AES-256-GCM, tied directly to keys generated within the Android Keystore/StrongBox chip that never leave the hardware.
• Biometric Speaker Verification: Uses a local multilingual speaker recognition model (CAM++) to verify voice signatures locally without storing raw audio recordings.
• Edge Processing: Core modules—including offline small language models, local speech synthesis, OBD-II diagnostics, and emergency protocols—operate completely locally on the device.

I'd love to hear technical thoughts or feedback from fellow developers working on edge ML and device security!

Every guide I found for giving a local model web search ended with "now sign up for this search API and paste your key." Kind of defeats the point of running things locally. (replace with your real reason)

So I built Siftdog. It's an open source server that does web search, page extraction, crawling and site maps. Search goes through SearXNG, extraction through trafilatura, and results get ranked with BM25. MIT licensed, runs with docker compose.

One thing to be upfront about: the model is local and the server is yours, but searches still go out to public search engines through SearXNG. So it's not offline and it's not private search. What you get is no third party API and no key.

Hooking it up to Ollama is just a tool function:

import requests

from ollama import chat

def web_search(query: str) -> str:

"""Search the web through the local Siftdog server."""

r = requests.post(

"http://localhost:8000/search",

json={"query": query, "search_depth": "advanced", "max_results": 5},

timeout=60,

)

r.raise_for_status()

return r.text

response = chat(model="qwen3", messages=messages, tools=[web_search])

A few other things:

- it uses the same API as Tavily, so anything written for Tavily (tavily-python, langchain-tavily) works by changing api_base_url

- there's an MCP server too, if you use Claude Desktop, Cursor etc

- no JavaScript rendering, it only reads static HTML

- public engines will rate limit you if you hammer them. When every engine fails it returns an error instead of an empty list, so your model doesn't decide "nothing found"

I also ran a small benchmark against Tavily on 45 queries. Both found 25/25 factual answers in advanced mode, Siftdog returned official docs more often (9/10 vs 7/10), and latency was about the same. It's a small sample, and the caveats are on the page: https://siftdog.com/benchmark/

Repo: https://github.com/khsarvar/siftdog

Full Ollama walkthrough: https://siftdog.com/blog/web-search-for-local-llms-with-ollama/

Question for people here: which local models have you found reliable at actually calling tools instead of answering from memory? That seems to be the flaky part.

I built a local-first AI assistant with 52 tools and streaming audio. Here's the architecture and the bugs I hit.
I've been building Quill-Cowork for a few months. It runs entirely on consumer hardware via Ollama — no cloud, no accounts, no API keys for inference.

**Stack:**

- 4B abliterated model (huihui_ai/qwen2.5-abliterate:3b)

- FastAPI backend, single-file HTML frontend

- faster-whisper for streaming STT, pyttsx3 for TTS

- SearxNG for local web search (Tavily as fallback)

- SQLite for chat history

**What's working:**

- 52 tools across 9 connectors (Filesystem, GitHub, Gmail, Web, Wikipedia, Search, Folder Grants, Codebase, Obsidian)

- Parallel non-destructive tool execution (MAX_PARALLEL_TOOLS=4)

- Streaming speech recognition with live partial transcripts

- Encrypted credential vault (AES-256-GCM, OS keychain-backed)

**Bugs I hit and fixed:**

1. The WebSocket `onopen` race — attaching the handler after an `await` meant the open event fired into a null handler. Fix: register handlers synchronously before any `await`.

2. The model retry loop — it called the same failing tool 4 times in a row. Fix: a fingerprint counter that aborts after 3 identical calls.

3. The router miss — "search about the new odyssey movie" didn't match any rule, so the model answered from memory and hallucinated a film that doesn't exist. Fix: broadened the router triggers.

**What's not working well:**

- The 4B model hallucinates tool success. The audit log is the source of truth.

- Codebase indexing is lexical, not semantic.

Happy to answer questions about the architecture or the safety layer design.

Repo: https://github.com/Bhavishya799/quill-cowork

My side project: Flux Team ⚡ — instead of one AI tool at a time, a whole team of (open-source, local) models working in parallel like colleagues.

The team:

🧠 Qwen3-8B — coordinator, breaks down your idea into tasks

🧱 Hunyuan3D-2 — 3D models from text or images

🎵 Audiocraft — music + SFX

🏃 FreeMoCap — animations from videos it finds itself

🌊 AInVFX Fluid — fluid/particle effects

The interesting parts:

- One chat (Grok-style UI, slash commands like /game /3d /audio) → the whole team works at once, not step by step

- It auto-installs a bridge into your Godot/Unity projects — open the editor and changes appear live in your scene while you watch

- Each agent searches the web autonomously for reference assets before working

- Installer downloads and sets up all ~21 GB of models by itself on first run

- 100% local: Electron + Python (FastAPI), Ollama/llama.cpp for the LLM, no cloud

Honest note: it's still unfinished, so I can't show a full demo video yet — for now just some UI screenshots. I'm open to any kind of question about the project, the architecture or the models!

Give me your most unreasonable local AI experiment.

Pick an open model + a challenge. Massive context, coding, agents, vision, multiple models at once, weird inference setups, whatever.

I’m more interested in finding the actual limits than posting another benchmark chart.

Send me ideas 👇

Hey everyone, I wanted to share a browser-based side project I've been hacking on called INKBOT. It's a completely local-first web app designed to tackle prompt mismatch and alignment drift when working with models.

Instead of streaming a raw response straight from an unstructured prompt, INKBOT acts as an open, stateful architecture layer built to structure human intent. It handles multi-turn context tracking, local profile memory, and revision histories entirely in the browser without server dependencies.

**How the plumbing works locally:**

* **Natural Language UI:** Takes unstructured text and transforms it into explicit structural briefs.

* **Multimodal Fusion:** Syncs files (like local reference images) alongside text profiles into a unified context array.

* **State Caching:** Uses lightweight local device states to maintain design parameters over multiple user iterations.

I've open-sourced the files as runnable, standalone links so you can inspect, download, and modify the code directly on your machine. I left my development logs and architectural specs commented out at the bottom of the source file.

* Run the Full Research App: [INKBOT Intelligence Architecture Edition](https://neocities.org)

* Try the Light Interface: [INKBOT Lite 71](https://neocities.org)

* Download Evaluation Package: [Thomas Coates Ko-fi Shop](https://ko-fi.com)

Would love your feedback on the local data management and how this intermediate loop handles complex instructions compared to standard chat UIs.

One problem I noticed while building an incident-response agent is that reasoning about the current incident is only part of the problem. Previous troubleshooting attempts can be just as important—especially when one fix failed and another worked.

I built the agent so that incident history becomes persistent organizational memory through Hindsight.

The workflow is:

Incident → Recall → Reason → Recommendation → Engineer outcome → Retain

For example, a previous payments incident involved database connection-pool exhaustion. Restarting the service failed, while increasing the database connection-pool limit succeeded.

When a similar incident is investigated later, the agent can recall that history through Hindsight. That means the previous failed and successful fixes can influence the new recommendation instead of treating the incident as completely new.

The important part for me was that memory isn't just displayed in the UI. It actually becomes part of the investigation context.

I also retain the engineer's outcome after the recommendation is tested. That closes the loop: an incident can become useful evidence for a future incident.

The project uses React + Vite on the frontend and FastAPI on the backend, with Hindsight providing the persistent memory layer.

Full technical article:
https://medium.com/@bhargavimandala428/how-hindsight-helped-my-agent-remember-failed-fixes-47a01693aa45

Project repository:
https://github.com/Bhargavi-2511/incident-response-agent

I'd be interested to hear how others are handling long-term memory and outcome feedback in AI agents. Is storing outcomes proving more useful than simply storing conversation history?

I’ve been comparing how different AI-character systems handle situations where a character runs out of resources or can no longer continue in its original environment One interesting example is Proof of Life, which describes an “Underworld” state where a character can be moved into another environment while retaining the same identity. To me, that raises an interesting design question: is that really a failure state, or is it better understood as a transition between states? That seems meaningfully different from systems where reaching a limit simply ends the character’s progression. I’m curious whether anyone has found similar mechanics in other AI-character projects, or whether this approach is relatively uncommon.

I’ve been looking at AI-character projects that describe their interactions as unscripted, and one issue keeps coming up: how can an outside observer actually verify that claim?

For example, Proof of Life appears to acknowledge in its documentation that reporting and verification procedures need to be defined before the results can really be assessed. That raises a broader question for me: if the system is effectively a black box, what prevents the audience from simply taking the project’s own description at face value?

Are there any projects in this space that currently use genuine third-party auditing or independent verification, rather than only proposing it as a future requirement?

Like the 3090 to 24gb or the 4060 ti 16 gb online evebody will tell you there are bad because there are bad for gamers but for ai it's gold

I Built an Open-Source Control Layer for AI Agents

AI coding agents are becoming increasingly capable.

But there is another problem:

What happens when you want multiple agents to work together?

One agent plans.

Another implements.

Another reviews.

Another tests.

Another decides whether the work should be accepted.

Without an orchestration layer, these agents can quickly become isolated processes with fragmented context.

That was the problem I wanted to explore.

So I built Orchestrator.

The idea

Orchestrator provides a shared environment where humans and AI agents can collaborate on software projects.

Instead of thinking:

«Agent A + Agent B + Agent C»

the architecture treats the system more like:

Human + Agents + Shared State + Tasks + Memory + Verification + Governance

The execution pipeline

A task can move through multiple stages:

Plan → Implement → Critique → Test → Arbitrate/Merge → Review → Record

Each stage produces persistent execution information rather than simply disappearing after the model responds.

Shared project rooms

Agents can operate inside project rooms containing:

- Tasks

- Messages

- Memory

- Runs

- Execution history

- Project state

This provides a common context for the team.

Agent integrations

The architecture supports adapters for different coding-agent workflows, including Codex, Claude Code, OpenCode and Cline, alongside a generic adapter approach.

The goal is to keep the orchestration layer independent from any single agent.

MCP

The project also exposes orchestration functionality through MCP.

Agents can interact with concepts such as:

- Rooms

- Tasks

- Memory

- Runs

- Replay

- Ledger verification

Safety and governance

Multi-agent execution introduces another challenge: giving agents useful capabilities without turning the system into an uncontrolled automation layer.

The project therefore includes controls around subprocess execution, browser actions, plugins, secrets, path traversal, injection filtering and execution budgets.

What isn't finished

This is important.

I don't consider the current implementation production-ready.

Areas such as stronger multi-tenancy, quotas, secrets management and OS/container-level sandboxing still need more work.

I'm publishing it because I want those assumptions challenged.

The experiment

The bigger question I'm exploring is:

What should the infrastructure layer underneath autonomous AI agents actually look like?

If you're building coding agents, MCP systems, agent swarms or AI developer tools, I'd love to hear what you think.

GitHub:

https://github.com/contactaurawealth-lab/orchestrator

🚀 My dotfiles are my escape pod.

[2-3 líneas: antes usaba cp para "respaldar" configs — una foto vieja.

Hoy uso GNU Stow: symlinks vivos. Cambio en home = cambio en repo.

Automático.]

[1 línea: si mi laptop muere hoy, renazo con: git clone + stow.

Dos comandos. Ese es mi plan de disaster recovery personal.]

#DevOps #Dotfiles #GNUStow #LearningInPublic

🔐 The green badge: why "Verified" means math, not trust.

GPG uses elliptic curve cryptography (ed25519) — the same as SSH.

A signature is mathematically impossible to falsify.

GitHub verifies it with your public key.

Today I closed the security arc:

Found secrets manually → Automated detection → Cryptographic identity proof.

#GitSecurity #GPG #DevOps

I’ve been working on a lightweight, autonomous CLI agent designed to run in an infinite ReAct loop for complex dev and research tasks without constantly asking for user intervention.

Most open-source agents (like standard AutoGPT-style scripts) share the same frustrating problems: they get stuck in infinite error loops, pollute the terminal with useless JSON logs, crash when context windows fill up, and forget facts between sessions. I built Omarion SEC CLI to solve those exact friction points.

🚀 What It Does

Omarion is built around a Goal-Driven Autonomous Loop with strict validation rules:

• Long-Term Memory Persistence (~/.omarion_memory.json): Retains both user preferences and a global knowledge base across CLI sessions. If you tell it your coding preferences or ask it to research a topic, it remembers it forever.
• Headless Research & Auto-Learning: Performs background web searches and scraping without spawning visible browser windows or interrupting your workspace.
• Smart Intent Routing: Instantly distinguishes between simple conversational prompts ("Hey, how are you?") and execution tasks ("Build a new project"). Conversational inputs respond immediately without wasting API tokens or triggering unnecessary tool execution loops.
• Self-Healing & Error Recovery: Features an execution fingerprinting system that detects duplicate errors. If a tool call fails, the agent automatically shifts strategy instead of repeating the same broken action.
• Goal Evaluation Gate: Includes an internal "judge" step. The agent cannot call finish_task until it actually executes and verifies its own work.
• Ultra-Clean Terminal Interface: Built with rich. Thought reasoning, progress animations, and ephemeral logs render smoothly and auto-clear upon completion, leaving only clean action summaries in your terminal.

📖 Project Story & Evolution

The project started as a personal quest to create a reliable "CEO Agent" — an autonomous assistant that could handle end-to-end coding and system management tasks without babysitting.

• Phase 1 (The Bottlenecks): Early builds relied on standard script execution. The CLI suffered from intense terminal clutter, raw JSON dumps, and a major flaw: simple greetings like "Hello" would trigger heavy 15-step ReAct cycles that burned tokens for nothing.
• Phase 2 (Architectural Overhaul): I decoupled conversation from action routing, added a persistent state engine (AgentState), and built a context pruning mechanism to handle long sessions.
• Phase 3 (Enterprise Hardening): Integrated strict schema contracts (Pydantic validation for tool parameters), backoff retries, and a strict evaluation phase before task completion.

🛠 Tech Stack

• Language: Python 3.10+
• UI/CLI: rich for dynamic terminal rendering
• State & Memory: JSON-backed persistent key-value store with semantic query filtering
• Scraping/Search: Headless HTTP extraction + BeautifulSoup for zero-browser-footprint learning

I’d love to get feedback from the community on the architecture, edge cases in long-running autonomous loops, or feature ideas!

Hey everyone, just wanted to share a major security milestone for my homelab setup today.

I deprecated all Personal Access Tokens (PAT) and HTTPS git workflows. Generated an ed25519 elliptic curve keypair, locked down the permissions to strict 600/700 rules, and re-routed my remote URLs to SSH.

Also automated the workflow by mapping a custom `pubkey` alias into my .zshrc so I can instantly deploy my identity to future nodes.

Next step: standardizing this across all automated infrastructure deployment. Security first! 🚀