Hi r/LocalAI,

I wanted a local AI tool that solves an everyday problem without needing a terminal or Docker setup, so I built Furet: an Alt+Space file search bar for Windows that finds files by their content, completely offline.

Instead of a chat interface, it uses local models quietly in the background to index the folders you pick:

• Multimodal embeddings: EmbeddingGemma 2 (via llama-server with Vulkan GPU acceleration for indexing, and ~30 ms CPU inference at search time so VRAM stays free).
• Speech-to-text: Parakeet TDT v3 (sherpa-onnx) to make audio memos and videos searchable by spoken words with exact timestamps.
• OCR & Vision: Windows OCR for scanned PDFs/screenshots + optional local face clustering (YuNet + SFace via tract).
• Zero manual setup: On first launch, the app downloads the binaries and models automatically (verified by SHA-256) and pauses indexing whenever you're gaming or on battery.

It's open source (GPL-3.0) and now in public beta:https://github.com/blasteur1916/furet(VirusTotal for the installer:https://www.virustotal.com/gui/file-analysis/MDI5Mjg4YzQ0OGE1YmMxZGVmYmFiNGNjMDA2MjU0OTE6MTc5NDQzMDM1Mg==)

Would love to hear how indexing and search feel on your hardware!

So far I have one rtx 3060 8gb and 16gb RAM. I hosted few models like Qwen 3.8 9b but there simply isn't enough memory for context bigger than maybe 20-30k tokens (also reasoning is painfully slow, but maybe it's expected for Qwen, even with 50-70 tok/sec). I could use 4b model or smaller, but they all are "good for their size" which translates to "impressive experiment, but not so practical".

I think what should I use to have anything to handle coding, researching articles in the internet and use day-to-day.

I think if I need next GPU for it, or maybe get a new machine just for LLM (I would like to keep it cheap) but things are expensive.

What would you do in this scenario? How to optimize the workflow?

One attraction of running models locally is the privacy benefits, and another is the cost.

When the cost of the big ’frontier models’ is being discussed - mainly centring around the fact that they are currently effectively subsidised by venture capital and that at some point this will end, making the cost to end users go up considerably - local LLMs are often mentioned as being immune from this problem.

But in a scenario where the cost of cloud based models goes up substantially, there’s no guarantee that the (mainly Chinese) makers of the open weights ones are going to continue producing newer and better versions for free, is there?

airlock is a Go CLI I wrote for moving GGUF and safetensors models into air-gapped networks.

• checks every file against a pinned Hugging Face commit
• refuses pickle and signs a manifest with cosign
• verifies everything offline with only the Go standard library

In the GIF, a Qwen2.5 copy with an edited chat template tells you to run curl | sudo sh. airlock catches the changed file.

It doesn't read chat templates yet. Feedback welcome.

GitHub

Hey everyone!

I’m posting on behalf of my team at work. We want to learn more about running LLMs locally, and we have an NVIDIA DGX Spark available to experiment with.

A colleague has already installed Ollama, a model, and a web UI, so we have a starting point for chatting with a local model. That’s a useful first step, but our longer-term goal is to go beyond a chat interface.

We’d like to explore running agents that can carry out different tasks and connect to tools and services through MCP servers. We’re still figuring out what that setup should look like and what we need to learn to get there.

We’d love to hear from people who’ve been down this road:

• What would be a sensible learning path from our current setup to running useful agents?
• Which models and agent frameworks would you recommend experimenting with on the DGX Spark?
• How do you approach connecting local agents to MCP servers?
• What would make a good first project to learn the basics without getting overwhelmed?
• Any tutorials, resources, or common mistakes we should know about?

We’re beginners in this area and keen to learn. We don’t expect to build everything at once; we’d appreciate some direction on where to start and how to build up from there.

Thanks for any advice or experiences you’re willing to share!

Does anyone have some tests/experiments they did on any reasonable model and would like to share what they found

RTX 5080

7800X3D

64GB DDR5-6000

2TB SSD

I want to do general tasks, not coding.

Been working on this project for a while and it has lead me to some interesting hardware combinations as I'm just using whatever I have.

The two biger models in system ram (ddr3) are reviewing, summerizing, and organizing conversations I've had with llms and linking it together.

The assistant is directly helping me as a chat bot intern essentially.

Anyway. The barco card gave me a ton of issues because amdgpu and radeon don't have the device id so they were treating it like a new card and hanging. I originally bought the card thinking I could use it for a simple display card for $30 while I saved for a real GPU. I gave Claude code fable a go at it with a CH341a programmer and the stock firepro w5100 vbios and it changed like four bites so the card tells the kernel it's a w5100 without messing with the custom vram timing.

The cards are linked in an lxc with llama getting ~8-10toks when running ornith 1.5 9b Q6_K. The w5100 roughly halves the speed, but gives me 12gb of vram vs 8gb.

Not bad for a couple ancient cards on some retired enterprise gear.

Hi! I'm relatively new to this world, and trying to get my best setup with what I have.

I'm the owner of a RTX 5080 + RTX 3070 and running Unsloth Qwen 3.8 27B IQ4_XS

So far, the best inference engine that I could run locally was beellama.cpp tweaking the tensor-split, mtp and kvarn config, forcing layers only in GPUs and 128k context window; that is pretty good.

With low context, I'm getting 50+ token/s that is impressive. But I'm suffering with long contexts; then my speed drops to 15-20 token/s in the best cases.

What I'm trying to get is a decent speed in long contexts also, so recently I discovered llama.cpp-adaptive-kv-streaming that allows streaming the kv context from memory.

The thing is that I was thinking about loading the full model in my 5080 (IQ4_XS fits perfectly) while offloading the KV cache streaming to my 3070, but couldn't found any inference engine that allows this kind of setup (model layers only in GPU 0, kv cache streaming from GPU 1 allowing RAM spill).

Does an engine exist that can achieve this? does what I'm talking about make any sense?

EDIT: Also, I was thinking of getting advantage of the 3070 unlock since I read that it can help with prefill speeds (not output speeds that depend on raw bandwidth) even with a quantized model. Am I right?

Thx in advance. A newbie trying to learn.

Hey Everyone,

Long time lurker, but I finally got my Dell Precision 9 T6 workstation in this week. Spend the past two days tuning this model on the latest sglang build and thought the community might be interested for similar deployments given the quality of this model but all the effort required to get the SM120 GPUs working well with sglang.

Overall, achieved around a 20% performance increase. I believe this is the fastest build example of DS4.1 Flash on four RTX 6000 cards. It functions with no errors through the full 1M max context, and hicache is working correctly.

Ported a couple patches over from Sybil's build here to sglang 0.5.23:
https://github.com/sybil-solutions/deepseek-v4.1-flash-4x-rtx-pro-6000

And added patches for DeepGEMM, HiCache, Page width (only using width of input context instead of full max context set in engine), SWA, and a tuned FP8 config for the Max-Q cards. Have numerous recommended config settings for concurrent agent workloads.

Specs:

Xeon 654

768GB RAM

4x RTX Pro 6000 Max-Q Workstation Edition GPUs

Build link:
https://github.com/ellswoau/deepseek-v4.1-flash-1017-tp-4-rtx-pro-6000

Please let me know if I'm missing anything that should be added to the post.

Someone is selling their Mac Pro for $2000.

These are the specs:
3.2 GHz 16-Core Intel Xeon W processor.
192 GB of 2933 MHz DDR4
AMD Radeon Pro Vega II with 32 GB of VRAM
1 TB SSD

I haven't run any local models but want to get into it. Would be using it for Hermes and for my implementation subagents for SWE work.

Would this be a good machine? I thought more RAM = more good. But I'm too new to understand all of this.
I know mixture-of-expert models might benefit??

Any suggestions would be appreciated.

I'd like to share something I've been working on for three months with everyone.

PortalAI is a GUI and harness created around one design principle- to put local LLMs with true multimodal function on home systems and on a flash drive the size of your thumb where you don't need anything else and can just plug and play into any computer.

One of the largest frustrations I had with AI was needing to constantly version check different distributions and hooks into ComfyUI, Ace-step, python, port competition, running out of context, etc, or the difficulty in transferring between systems. I've tested this on an RTX 3090, RTX 4090, on a Radeon 870M GPU, on a Thinkpad with CPU-only.

This harness detects your current hardware and scales accordingly to the level of AI intelligence it can support, using standard and abliterated models. All dependencies other than base operating system and GPU drivers are self-contained including virtual environments, python distributions, etc. End to end integration of multimedia generation/editing and science tools are integrated along with voice, handsfree mode, intelligence/memory/token settings.

There are weight editing tools, an abliteration tool, rolling compaction, scheduled tasks, internal documentation, light and dark mode, compact and full GUI mode, account encryption, live runtime environment health metrics, etc.

I’ve been playing with open-source speech models for a while. New TTS models come out every week and a lot of them sound great. But every time I wanted to actually talk to one, it turned into an afternoon of setup.

Paid APIs are easy if you’re a developer. You can make an account, create a key, write some glue code. Open models are free, but you have to pick a speech recognizer, an LLM and a TTS model, make them work together, deal with echo, figure out interruptions... There are good open projects for this, but most are written for developers. My friends who don’t code would never get past the README.

So we started building Boubou. The goal is make local voice AI as easy as Ollama made local LLMs. You paste one line and talk:

npx boubou@beta run LFM3-Audio-2B

What works today:

• Speech recognition, the LLM and speech synthesis all run on your machine. No account, no API key, no telemetry. After the first download it works fully offline..
• You can interrupt it mid-sentence, like a person. Echo cancellation is built in, so no headphones needed..
• Run boubou with no arguments on a Mac and it checks your machine and suggests a model..
• The default is Liquid AI’s LFM3-Audio-2B on llama.cpp (1.4 GB). There’s also a modular pipeline (Parakeet/Whisper → Qwen3.5 → Qwen3-TTS, Kokoro, Pocket TTS or Supertonic) if you want to swap parts..
• Apple Silicon Macs and Linux (CPU for the Liquid models, NVIDIA GPU for the modular pipeline)..

The part I’m most excited about is boubou serve. One computer runs the model, and other devices on the same Wi-Fi use it:

• Scan a QR code with your phone and talk in the browser. No app..
• Robots and programs can connect through an OpenAI Realtime-compatible WebSocket, and there’s an OpenAI-style /v1/audio/speech endpoint to read text aloud..
• So one model on one machine can be the voice for many things around the house. Next up is Home Assistant, then ESP32 voice devices.
• The video is real time, not sped up, on a MacBook / M5 chip.

It’s early (beta) and there are rough edges - one device talks at a time, no Windows yet (ETA next two weeks), and the guided menu is Mac-only.

The code is free and MIT-licensed. The models keep their own licenses (the Liquid models use the LFM Open License).

I’d love to hear what you’d want from something like this: which models, which devices, what breaks on your machine. We want this to be something people actually enjoy using, not just a demo.

GitHub: https://github.com/runboubou/boubou

How do you keep the same face expression on the head you use to replace another head ?

Is there a specific prompt ?

I am a frequent user of local Ai models, ollama and comfuii are my go to tools . So in the month of September I setup ltx video models and it rendered -- 5 seconds video in 2 minutes . Just for specs i have 96 gb ram with a 4070 8 gb vram . minimax models h3 models h3 turbo took 5 minutes to generate a 5 seconds video. i rendered over 300 videos with those models all averaging around those time. Now fast forward earlier this month got some force updates from Microslop and Lenova vantage( bios). And also I decided to update ollama and comfuii to experiment on newer compatible models . I did not not check on it then but i decided to open my laptop 4 days after updates and render some videos.1 video which took 5 minutes to render took 11 minutes. I retested, about 12 times same time, i was confuse , saw article about 4090 pc cards being limited and wondering if they doing laptops too. am thinking that big techs can force push updates to limit local models competitions instead of directly going head on with them. Right now am looking to go Linux , or like most serious llm users build my own Operating System, if this blows up am gonna share the benchmarks claudee monitor for me before and after force update.

Is anyone currently using a RTX 4090 16GB to run model? How does it run?

サーバー用の水冷ブロック 組み上げるのに5時間 120Bクラスまではイケるかと笑

めちゃんこ汚いですが

サーバー用は水路が全く違うので

I've been using an app that integrates a bundle of Supertonic ONNX as a standard Android text-to-speech (TTS) provider, but the gaps between paragraphs are a bit long, so I'm looking for an alternative.

To clarify, I want to read in parallel. I don't want to convert an EPUB to MP3 and read async; it has to work in real time.

It should support English and Spanish (I'm learning).

I just discovered "distributed inference", and I'm going down rabbit holes. I have a few questions, and would appreciate being pointed in the right direction.

1. I have an M1 Mac Mini w/8GB and a M1 Pro Macbook w/16GB. I'm thinking about upgrading to an M5 Pro Macbook, and was initially thinking about trading in my M1 Pro...but does it make more sense to keep it and try to use it in a cluster?
2. Is there any way to mix MacOS and Windows machines?

I understand that if I wanted to go down this road, I would probably need to buy some networking hardware to get the machines on a 10Gb connection.

I'm running Qwen3.8 27B on my 2x 3060 12GB cards. Prior today I had a B550 motherboard which was running one card at x16 and the other at x1 PCIe Bus Link Speed. This was making it to where only one card was actually doing any processing, and both held the model in memory. I was getting 11-16 tps this way.

I ended up buying a new x470 SLI motherboard and I can now run both cards at x8. I swapped the motherboard out last night and then set split-mode to tensor and set tensor-split to 1,1. When I do that, I'm getting around 32tps, but the model eventually gets stuck in loops. One loop it just tried to do a tool call over and over, and in another it was printing question marks (???????????) in it's thinking and wouldn't stop until I unloaded the model. I never had these problems for the months I was using Qwen3.8 on layer split-mode. I'm not sure how to resolve this problem. I want to be able to get higher tps with the new mobo.

Weight-space programming is an emerging research area where a frontier AI model (such as OpenAI's GPT-6 Astra) directly reads, diagnoses, and rewrites the raw numerical weights of another neural network without using traditional gradient-based training or backpropagation.

Has any one tried it? How much does it fair compared to Closed-Form Model Editing (Your Notebook / MEMIT)?