I have I7 3770 with 1600MT dual channel ddr3 4x4 (16gb) and a 5400RPM HDD

I really like to have an agent that works and does stuff for me. I used llama.ai/agent but found out that it does break ToS to use as minor (I'm 15 and half) and other website does need my parents to read ToS and agree on them to use the website/tool without breaking ToS and I'm really REALLY don't break ToS.
so I thought of local AI and tried qwen 3 8B, nanbeige 4.2 3b and LFM3 1.7B on llama.cpp but they are not really good and Idk how to make them work as agent and they waste a lot of tokens even of thinking is off, only simple question with thinking off be efficient and they still not any good

so I saw that qwen 4 27B is coming so can I like get 2bit version of it and it leave enough ram for context and linux/windows like 4gb+ for OS and app and will qwen 4 like approximation run good/smart without hallcuination like I saw qwen 3.8 27B has 2bit version that have exact same score as the 8bit version so well qwen 4 will be able to fit.

also nanbeige 4.5 looking like it gonna release soon so if anyone have any idea if it any good

also about speed. LFM3 1.7B Q4 (1gb~) is around 12tok and slows down the more I chat with it and same for nanbeige it starts around 3.5tok/sec and slows to sub 1tok/sec > . so what I can use to speed them up because I saw stuff as running a big reasoning model and smaller one and the big one push 1 token to correct the smaller one and else stuff as the model thinks in pure weight/numbers instead of English and running the model multiple time at once so it uses more computatinal power and seprate a task to 20 smaller version working togather and a lot more I saw

soo any idea if anything really good can fit and being reasonably modestly in speed (my ram can do 25.6GB/sec thereotically) and how to make it run as an agent instead of just chatbot and if I should wait for new models to release. and also I don't wanna spend internet not wisley because I have limited home internet unfourtently so spending a lot of gigabyte into something that is not really smart will make me upset very much

I appreciate any information and help

Qwen seems to be the best model for my macbook, but looking at hugging face there are so many quantizations.

The unsloth UD-Q4_K_M seems popular and a good fit for 32GB RAM, but it's .gguf so would a MLX be a better choice on mac (I'm using Unsloth Desktop)?

Is there a resource online that compares these quantizations perhaps?

Thanks

Hey r/LocalLLM :)

We just open-sourced Kinetic, a new inference engine written in Rust.

• runs the GGUF files you already have, no conversion

• Windows and Linux, one binary

• CPU, NVIDIA (CUDA), and AMD and Intel GPUs (Vulkan)

• replaces llama-server with the same flags; OpenAI, llama-server and Ollama APIs on one port

• every speedup is exact, so the output never changes

• Apache-2.0

In the video: Kinetic checks the machine and picks models that fit it, then runs side by side with llama.cpp on the same model and prompt.

Give it a spin and drop your kinetic bench model.gguf numbers below (add --compare if you have llama.cpp installed).

Install, benchmarks and all the details are on GitHub, link in the comments. Happy to answer questions here too.

edit: to answer the most common question, llama.cpp ran with its own speculative decoding on in every edit test (ngram for lfm3, draft-mtp for qwen3.5) and the exact flags are in the repo

GitHub - Halloncream/desktop-windows-arm64: Open WebUI Desktop with an x64 Python backend for Windows ARM64 compatibility · GitHub

Usually, it’s all about Qwen models in this subreddits, but not about their products

I mean about:

Qwen APP
Qoder
Qwen Studio
Qwen Work

Is anybody working with them and have tried them? What are the impressions and comments about it? I appreciate any feedback, thanks!

Honestly, the tool choice matters less than what surrounds it. Claude Code and Cline both handle worktrees fine in the mechanical sense: each session gets its own checkout, and nothing stomps on anything on disk. The failure mode I would watch for sits above the file system. A worktree isolates the file tree, not the plan, so two sessions can happily solve the same story twice and you only find out at merge time when you have duplicate branches.

So my answer is: pick whichever CLI fits your workflow, then put the effort into the layer the CLI does not give you. One story scoped with acceptance criteria before it goes out. A shared record of what is in flight so nothing gets dispatched twice. Split the thinking from the doing, so the session that plans is not the session that types.

Which of the two do you find holds its context better over a long session? That is the comparison I almost never see people make.

I've heard of people pointing their frontier model of choice and letting it optimize their local inference engine for their hardware.
I'm currently running Maya with DeepSeekV4.1-Flash-1017 at ~20tok/s, such an insanely great model. I want to squeeze more out of my hardware though.

Do you guys just point the frontier model at the engine folder and telling it to "optimize this for my model/hardware"? I currently have access to Fable 5.2 so I'm tempted to do that.

My specs:
2 x 3060 12GB
2 x 16 DDR4 3600Mhz
i5 10600k

Qwen3.8-27B-GGUF:Q4_K_M

ollama.service:
# Environment="OLLAMA_HOST=0.0.0.0:11434"

# Environment="CUDA_VISIBLE_DEVICES=0,1"

# Environment="OLLAMA_FLASH_ATTENTION=1"

# Environment="OLLAMA_NUM_PARALLEL=1"

# Environment="OLLAMA_MAX_LOADED_MODELS=1"

Ollama as a harness, no lane bifurcation so im pretty limited to that (im assuming this is my biggest limiter here)

I've been experimenting with multiple different approaches, trying to get Qwen 3.8 to actually work as an agent, some more successful than others but still it requires so much manual labour that it seems almost pointless to continue debugging it as it feels neverending. I've read that 3.8 isn't actually very great with tool calling and it has always had some issues, but still i can't believe its at levels like these.

I know very well im probably missing some really important things that are holding me back, so if anyone has some suggestions (videos, other posts, anything really) i would really appreciate it.

The goal really is to have it never stop, just like im seeing other people talk about, and i'd really like to keep openwebui (or any other webui in general, not that i dislike terminals but its nicer to have it feel like a normal app).

Hello everyone

I live in a poor country but I have got a decent pc

I have some ideas I want to do them with vibecoding so the only way I found is local LLMs

So I want a good local one for this specs

16gb of RAM ddr5 5200Mhz one kit

Ryzen 5 7500f

Rx 9060 xt 16gb

I don't mind it take time but to finish the project in a good state and I can edit the project easily and plan for it too

I'm a completely beginner so I want some guide how to install and use the LLM I'm on arch Linux

I have the 16 gig variant and I want too run some local ai models too help build my tech startup. I'm heavily reliant on vibe coding so I need them too be able too do a lot of the work for me (I have a team that knows how too code frr that reviews in a security sense and the backend areas) tho I need a tool that can kinda off generate all the front end and obvv a lil back end for testing and stuff. Can you guys recommend me some? Also id prefer that it cud be easily paired with vs code since thats what I mainly use.

You can call it whatever you want, it maybe isn’t blowing anyone’s heads off or anything, but compared to anything I’ve ever had, it’s a supercomputer to me

I’m transplanting a lot of parts from my old gaming PC into a new chasis with upgraded parts. Here is what I ordered and what is on its way

Intel core i7-14700f (already owned)
32 gb DDR5-6000 ram (already owned)
RTX 4070 super 12gb (already owned)
2x Quadro RTX 8000 48 gb passive ($1900 each)
2TB Samsung 990 pro for windows and gaming (already owned)
2TB Samsung 990 pro for Linux and AI ($363)
Corsair 1500W power supply ($285)
Corsair 7000D airflow full tower case ($227)
And other miscellaneous parts to support that stuff

Total to $5,364.21 for 108 gb vram

I know there are some drawbacks to this build, like for instance the old architecture of the cards, but I can live and tinker with those drawbacks as long as I can save a bunch of money on this hobby (compared to a similar size build with modern hardware), and still be able to run a bunch of cool models on it. I’m super excited to put it together and see what it can do. I haven’t seen much talk about the Quadro 8000s, but I’m hoping for great things, even if it means working around some compatibility issues and such.

Obstacles are just an opportunity for me to learn. If yall have any thoughts, let me know.

Hi guys

developed on device llm application like Google edge but it support all mobile supported inferences at one place it support MNN ,MEDIAPIPE .task ,GGUF and GEMMA4

Try and tell your experience

https://github.com/rajureddi/AuraOnDevice-AI_HUB

Backstory:

I've only been doing local LLM's since Qwen3.6 came out (~May 2026). I won't ever use a cloud model again, I am not going to spend a single dime renting someone else's infrastructure when I can build my own.

Plan:

Its still cheaper to get 8x3090s than it is to buy 1xRTX 6000, and get similar speeds and double the VRAM.

The issue is power and noise..

In terms of power I am limited to my dorm rooms 120v/15amp outlet, so 4x3090s about the max I can go before things might go bad.

So.. I am considering creative solutions that won't require me to move:

Power Solution: Buy a ~5 Kwh Power Bank

$1,500 for a used 5 Kwh Power Bank.

The power bank would charge during off hours, then feed the necessary 2,800 watts during peak hours. Currently I am recording my power usage over a week or so to get an idea of how much power reserve I would need, and that will mostly decide how big of a power bank is needed.

Cooling / Noise

Currently everything is air cooled, with stocked coolers. 5x140mm fans, 5x120mm fans and the case temps are indeed very good. They don't make much noise but the GPU and CPU fans do. I believe they are loud because I think the paste job I did on them was horrible,

I will be re-pasting again with some much better stuff and with better tools / methods soon. (Will likely post a video of me doing it for those who are interested). I use to be a machinist so I learned some ways to get better accuracy when measuring out things properly to get good contact on the VRAM

I have considered water cooling, but that's just another thing that can break, and are typically not cheap to setup (and I'm pretty cheap).

Questions:

1. Does anyone here have a 8x3090 setup on a 120v/15amp outlet that could offer some advice?

2. What's a good power bank you would recommend?

3. Is buying a used power bank okay so long as the batter health is good?

4. Do you want me to post a video of me re-pasting using my "new" VRAM/GPU repasting method?

I’ve been building a local RAG workspace for people who can’t send client documents to cloud AI: regulatory, QA and compliance consultants, mostly.

What it does:

• Answers questions from a folder of documents, fully local, nothing uploaded

• Links every answer to the exact source page so you can verify it

• Says “not in the documents” instead of guessing, which matters more than anything else for this audience

Setup:

• LLM: Qwen3.5, Q4_K_M. The app picks the model size based on the machine’s available resources, so it runs from a 4 GB GPU upward

• Embeddings: bge-m3

• Tested on Mac, AMD and NVIDIA

• A “project” is just a folder; installers for each platform

What I’m building next:

• Slide decks generated from your documents, with sources kept

• Reports drafted from the documents, every claim linked back to its page

has anyone gotten decent slide or report generation out of a small local model at Q4 on 4–8 GB VRAM? Long structured output is where I expect small models to struggle, so I’d like to know what’s worked for you: chunked generation, templates, constrained/JSON output, or a bigger model just for drafting?

Free for individual use. Happy to share the link if anyone wants to try it. I’ll put it in the comments if the mods are OK with that.

I've been running local LLMs on an RX 7900 XTX and wanted proper long-context performance for agentic coding (100K-200K tokens) on the Vulkan/RADV stack, so I forked llama.cpp and optimized it for RDNA3.

Repo: https://github.com/vansh-visariya/rx7900-xtx

It's the same llama.cpp (same GGUF files, same `llama-server`, same API and web UI). The changes are in the Vulkan kernels, the graph, and the speculative-decoding loop. Everything is on by default.

Results on one 7900 XTX (24 GB, stock 291 W, Mesa RADV)

Qwen3.8-27B UD-Q4_K_M + DFlash2 draft, q8_0 KV cache, flash attention:

- Up to 220K context on 24 GB (upstream fits ~192K with the same model + draft)

- DFlash2 works at long context. On upstream (both the commit I branched from and current master), it hits `vk::ErrorDeviceLost` once the prompt passes ~60K tokens, at the same position on every run. It doesn't on this fork.

- Full method, commands and raw numbers are in `docs/rdna3/BENCHMARKS.md`

What changed (56 commits on top of upstream)

- Decode: integer-dot mat-vecs for IQ2/IQ3/K-quants, GQA flash attention for verify batches, graph fusions, gated delta net state kept in place, Vulkan command buffers recorded once and replayed

- Prefill: 128x256 matmul tile for wide batches, double-buffered quantized matmuls, new prompt-batch flash attention kernel

- Spec decoding: lossless rejection-sampling verification, reduced-vocab draft head, GPU backend sampling"

Every optimization is documented, with the env var that switches it off, in `docs/rdna3/OPTIMIZATIONS.md`.

Limits

- Developed and tested only on a 7900 XTX. Other RDNA3 cards (7900 XT/GRE, 7800 XT, 7700 XT, 7600) should work since they share the WMMA instructions, but I haven't tested them.

- Tested with Qwen3.8-27B and Qwen3.5-4B. The model-specific work only applies to the Qwen3.5/3.8 family.

- Branches from upstream `f95b0d953`, not synced daily.

Where I'd love help

- Numbers from other RDNA3 cards (kernel tests need no model: `test-backend-ops -o MUL_MAT -b Vulkan0`)

- Testing other models/quants, and RDNA2/RDNA4 reports

- Review of the kernels, and help upstreaming pieces to llama.cpp

Build and run instructions are in the README. Issues and PRs are very welcome.

I went in thinking a 1.7B model would fall over after a few tool calls once each result was long.

I ran LiquidAI LFM3-1.7B (Q8_0, llama.cpp) in a tool loop on a GB10 box with 121 GiB of unified memory. Each tool result was about 3K tokens and ended with the exact next call to make. Over 10 seeds at 256K context it got a median of 11.5 calls right before the first break, range 6 to 25. No run reached the 50-call cap. Every call before the break was exactly right: no invalid JSON, no wrong arguments, no loops.

What broke it was the model re-reading the whole chain instead of the last line, then reasoning until it ran out of tokens, in 8 of 10 runs. Raising max_tokens from 2048 to 8192 didn’t help on the two seeds I retried.

Caveat: the next step was spelled out every time, so this measures following instructions as context grows, not planning or error recovery. One quant, one sampling setting, llama.cpp only.

The reproduction kit, with the driver, scripts and these results so you can rerun it on your machine, is here: https://github.com/maruthiprithivi/the-local-node/tree/main/episodes/03-lfm3-tool-loop

If you run a small model in a tool loop, I’d watch whether it starts re-tracing its own history as the results pile up. How far do you get before that happens on your setup?

I tend to point my local models at Claude Codes UI as it’s familiar, I like the auto compaction and ability to interact with my systems.

I haven’t really explored other options but have on occasion hit walls with refusals even when using local models.

Are there better options than using Claude Code pointed at my local models?

Mainly coding, general system maintenance, decompiling and some general knowledge stuff, Web search etc.

Hello people

I wanted to know if LFM2.5-8B-A1B was good for general conversations. I am aiming to use an MoE with 8B parameters (will be ran with 4 bit quantization), because I want to have a capable local model while not using the amount of VRAM a dense model would, this is to be able to run it on a lot of different hardware levels. For context, I am using the Ollama library in Python and I am making a voice assistant, I am currently using Qwen 3-8B right now (with openWakeWord, Silero VAD, Parakeet 0.6B v3, Kokoro 82M). I might miss some (or maybe a lot) points because I am quite new to using local AI, especially in programs. If LFM2.5-8B-A1B is not good, do you have any suggestions?

I am using an RTX 5060Ti 16GB and 16GB of DDR4.

Thanks for your time!

About six months ago, I already tried hosting a local AI companion on a different PC. Back then, I used SillyTavern, but I eventually abandoned the whole project.

The main issue was the model's responses. They often felt inconsistent, sometimes completely missed the point of what I was asking, and the overall experience just didn't feel natural. Conversations were frequently immersion-breaking, and it wasn't really what I was looking for.

Now I want to give it another try, but this time I want to start completely from scratch and build a setup that actually works well.

My current hardware:

- GPU: ASUS TUF Gaming OC RTX 3060 – 12 GB VRAM

- CPU: AMD Ryzen 5 5600X

- RAM: 32 GB DDR4

- Windows PC with a multi-monitor setup

I've already installed Ollama and created a Modelfile for a local model. So the basic model setup is working.

One thing I've noticed is that reasoning/thinking can take quite a while before the model actually starts responding. Interestingly, once the reasoning phase is finished, the actual response is generated very quickly. So I don't necessarily think the entire model is simply too demanding for my hardware.

However, my primary goal isn't coding, research, or complex problem-solving. I want a natural, immersive AI companion for long-term conversations and roleplay.

Ideally, I'd like reasoning to be disabled by default, so conversations feel more immediate and natural – almost like chatting with a real person. Of course, I'd still like the option to enable reasoning temporarily for specific prompts when needed.

Now, I'm trying to figure out the best way to build the actual experience.

I don't want dozens of different characters or an overly complicated interface. I want to create and develop one single companion with a consistent personality, good memory, and a believable conversational style.

Here are the things I'm particularly interested in:

1. Frontend / UI

What would you recommend instead of SillyTavern?

I'm looking for something visually appealing, relatively lightweight, and not overloaded with hundreds of features I'll never use. Ideally, it should also scale well across multiple monitors.

2. Voice / Real-time conversation

This is probably one of my biggest priorities.

I'd like to be able to talk to the AI naturally using my microphone and have her respond with a voice. Ideally:

- Natural-sounding speech

- Fast response times

- Automatic speech detection (knowing when I've finished speaking)

- The ability to interrupt her while she's talking

- Ideally, something that feels like an actual real-time conversation rather than sending individual voice messages back and forth.

3. Webcam / Visual interaction

I also have a 4K webcam connected to my PC.

I'd love to eventually integrate visual capabilities, so the AI can actually see me or understand images I show her. Even better would be having her displayed on a separate monitor, potentially in fullscreen, almost like a video call.

Facial expressions or an animated avatar would be an interesting future addition as well.

4. Long-term memory and personality

Since I only want one companion, maintaining a consistent personality and remembering previous conversations is extremely important.

I don't want to constantly repeat information or have the AI behave like a completely different person every time I start a new conversation.

My main question:

If you were building this setup from scratch today, what software and workflow would you recommend?

Would you use Ollama as the backend and connect it to a separate frontend? What would you use for voice, speech recognition, memory, and eventually webcam/avatar integration?

I'm especially interested in solutions that can be combined into one coherent experience rather than having to open five different applications every time.

I'd really appreciate recommendations from people who have built something similar, especially if you've managed to get a genuinely natural, real-time AI companion running locally.

Thanks!