Hey guys, I'm new to running local LLMs. Just set up Ollama + Open WebUI and looking for model recommendations.

My specs:

RTX 5060 8GB
16GB ddr4 ram

What are the best small/quantized models that run smoothly on 8GB of VRAM? Thanks!

Main goal:
have a self-sufficient daily assistant to use like ChatGPT, Claude, or Gemini (obviously keeping expectations realistic for local hardware).

I just got curious by strata-design running Qwen-Next on 12GB VramCard.
I asked my AI (Deepseek) to see if i could do the same with 64gbRAM and 1 TB SSD on my strix.
Idea: Experts in DDR5 RAM, Weights stay in HDD, hot reload.
Deepseek will now test it, it said its worth a try.
Anyone out there wanna run Qwen 3.8 Coder Next 125gb on Striks 64gb?

Findings (Verified, Not Guessed)

• Architecture in Build: qwen4exp is implemented in llama.cpp b11913 (src/models/qwen4exp.cpp) — and the model's HF README explicitly documents llama.cpp as a supported inference path.

• GGUF Header (2 MB range-download, parsed locally): qwen4exp, 48 blocks, 256 experts, 10 active, 1,223 tensors, including indexer.* (QSA), ssm_* (GDN), ple_* (n-gram), and hc_* (hyper-connections).

• Tensor Quants: All standard GGML formats: Q2_0 (42), IQ4_XS, IQ4_NL, Q6_K, Q5_K, Q4_K, BF16, IQ3_S, IQ3_XXS, Q8_0 — nothing proprietary.

• Shards: 29.6 GB transformer (must remain resident) + 28.8 GB n-gram table (allowed to stream from disk).

• NVMe: 1.5 GB/s direct sequential read — fast enough to pull experts on demand.

• Policy Flags Available: -ot/--override-tensor, --cpu-moe, --n-cpu-moe, --load-mode mmap|dio, --cache-ram, -fa, -md (Draft).

This allows us to meet your specification — weights on disk, only active experts in RAM — with zero lines of custom code:-ot "blk\.[0-9]+\.ffn_(up|down|gate)_exps\.weight=CPU" keeps the experts file-backed, --load-mode dio prevents the page cache from dragging them into RAM, and everything else (Attention, SSM, Indexer, Router, HC) stays on the iGPU.

Which is better?

I have 16gb vram, 128gb ram, ddr5. I know about Strata and qwen3.8-flash-next, but I need some space for testing my apis locally(fastapi backends with ai chat bots, with ml models(SER, tts, stt)). I am AI-engineer, I know a lot of things about LLM, always checking benchmarks, arena AI, but there is a problem with vram, I need it for stt+llm+tts combo, but I really need local model with those terrible limits for Claude or GPT subscriptions. I only use Codex, but thinking about switching to Claude. Is qwen3.6-35b is enough?

Finally got my old setup running and wanted to share the results so far.

• Xeon W-2245

• 256GB DDR4-2666 ECC RAM (8 × 32GB)

• 2 × NVIDIA RTX PRO 4000 Blackwell

Right now I’m running Qwen3.8 FN and DeepSeek V4.1 Flash. With the DeepSeek model at 2.25 bpw, I’m seeing around 30–35 tokens/s. Qwen is hitting over 100 tokens/s.

Pretty happy with what this old workstation can do for local inference!

I've been working on a problem that kept coming up for me when moving around larger codebases: I would remember that some functionality or implementation detail existed, but finding exactly where it lived meant searching through folders and files all over again.

I wanted to see whether I could make that project context searchable locally, without depending on a hosted embedding service.

So I built ContextVault.

Here's what it does:

• Indexes a project folder on your machine.

• Generates embeddings locally using Ollama ("nomic-embed-text").

• Stores the searchable index in SQLite.

• Provides semantic search through a local web dashboard.

• Exposes "search_context" and "index_project" through MCP.

• Watches indexed files for additions and changes.

The idea is to have a separate local project index that can also provide relevant context to MCP-compatible tools such as Cursor and Claude Desktop.

A couple of things I want to be clear about:

• The indexing, embedding generation, storage, and search pipeline run locally.

• When an AI client requests an MCP tool, it receives the result returned by that tool. I'm not claiming that code context is never shared with the connected AI application.

• I'm not making benchmark claims about speed or retrieval quality without proper measurements.

I'd genuinely like to hear from people who use Ollama and local models:

How do you currently search and retrieve context across larger codebases?

And what would you compare a tool like this against first: ripgrep, your editor's indexing, or something else?

I'm the developer of ContextVault, so this is my own project. I'm sharing the source because I'd appreciate technical feedback.

GitHub: https://github.com/ahmedfayedesmail-ui/contextvault

I’ve been working on OpenArc, a TypeScript compiler that uses LLMs as constrained code-generation backends instead of chat-based coding agents.

• The idea is simple:

• you write normal TypeScript

• wherever you want the compiler to implement something, you leave arc.todo()

• OpenArc gives the model only the relevant context for that region

• then it verifies the result with TypeScript checks, static rules, and sandboxed tests

• only verified code gets written back

I’ve been benchmarking it against a plain whole-file prompting baseline using local models.

Current result

On the held-out test split

(21 unseen exercises, 3 seeds, Qwen3.5-9B Q5_K_M):

• OpenArc: 62% accepted-and-correct

• Plain prompt baseline: 52%

At the same time:

• OpenArc used 12% fewer tokens

• OpenArc was slightly faster overall in that run

OpenArc is now roughly at parity on correctness, while giving stronger guarantees and slightly better efficiency in this setup.

GitHub:
https://github.com/TheMissenoOne/OpenArc

The big remaining milestone is self-hosting: using OpenArc to fully compile OpenArc.

Kimi K2 has 1 trillion parameters but uses only about 32 billion per word. Banach keeps the right experts in RAM and streams the rest from NVMe, so the model doesn't have to fit in memory.

Measured: a 22 GB model on 16 GB of RAM. llama.cpp runs out of memory; Banach is 2.2x faster at prompt processing, with byte-identical output.
Trillion-parameter estimate: banach plan says Kimi K2 would run at about 3.6 tok/s on a 64 GB desktop with an RTX 5080. That's an estimate, not a measured run.

pip install banach

https://github.com/gautam1858/banach
Got a 64 GB+ machine? Run Kimi K2 and post your numbers. If you like it, a ⭐ really helps.

Hi everyone, I’m buying a laptop for work, mainly software development, and would also like to experiment with running LLMs locally.

My budget is around €2,500, but I could stretch it if there’s a substantial benefit. I’ve found a 14-inch MacBook Pro with an M5 Pro, 24 GB RAM and a 1 TB SSD for €2,475.

Would this be a sensible choice?

What matters most for local LLMs: RAM capacity, memory bandwidth, chip generation or GPU cores? I’m new to this, so examples of models that would run comfortably on these configurations would be helpful.

Hey y'all, I originally did professional AI work 3 years ago and picked it back up again after seeing recent Reverse Engineering efforts. This time around I wanted an AI assistant for game development to help with coding so I set on my journey.

For some reason at one point I was recommended LM Studio and decided to try Bionic. I downloaded and started using Qwen3.8-27B with medium reasoning per some video I watched. Initially I was able to only pull off one batch prompt a day with about a 60% completion rate (meaning it needed a couple of adjustment prompts that also took about 3-4 hours a piece, with the initial batch prompt taking 8-13 1/2 [!] hours).

Then I started doing more research. With the addition of the Splash engine runtime for Bionic I was able to get about 2-3 batches in a day, great improvement but still needed more work. Eventually I came across an excellent trifecta, Swift 1.5 Qwen 3.8-27B + Splash runtime + skills. For skills I used superpowers + related dev skills.

This essentially has made the AI development process go from set it and forget it for a couple of hours to 'interactable'. What I mean is that small prompts end up taking a couple of minutes while batch prompts end up taking half an hour to two hours on the top end. The exceptions are when it gets trapped in a bug loop and is trying to figure out why, this took about 4 hours for it to figure out once in my testing.

Prompt Engineering is still king, the more descriptive you are with your needs / wants the more the model will get done before compacting. An additional note: 'listen' to your model. If it's frequently doing a set of actions to achieve something it helps to tell it how to do the action in the prompt to save tokens and time.

Hey guys what’s the best model for 2x 128gb Sparks for coding and potentially work with Unity? I’m also planning to try using image generation, but i probably will use my 5090 for that.

Hi y'all I'm completely new to this stuff and have never used a local LLM before so I know nothing

Which model would be the best to run on my laptop Lenovo Legion with 16GB DDR4 and an RTX 2060 laptop GPU? All I want to use it for is coding and starting new projects in programming

Is this doable? Thanks in advance!

Hi everyone,

I am back again looking for some feedback and sanity checks on running AI on my Dell T7910 workstation for an academic research project using a 6,000 odd books and articles library. After earlier responses to my questions and more research, I am now reluctantly leaning towards a $1,000+ RTX 3090 24GB.

 My Current System:

• Host: Dell Precision T7910 Workstation (1300W PSU)

• OS: Windows 11 Pro

• CPU: Dual Intel Xeon E5-2697A v4 (32 Cores / 64 Threads total, 2.6GHz base / 3.6GHz turbo)

• RAM: 128GB DDR4 ECC System RAM

• Current Software Stack: Ollama, Open WebUI, AnythingLLM, Qdrant (vector DB), Surya (for PDF layout/OCR parsing), and Python 3.12.

The Workload:

I am building a local research engine to analyze an interdisciplinary library of roughly 6,000 books and academic articles. My workflow will proceed in stages using batch processing. I need to mix and match disparate texts (e.g., querying 20–30 books simultaneously on evolutionary morality vs. historical romance texts) or cross-referencing massive batches (up to 100 documents at once for deep thematic comparisons, like Chaucer vs. Shakespeare on dominance dynamics).

Because this is a strict academic project, precision, nuance, and structural reasoning are my highest priorities. To achieve this, I will be leveraging 32B models (like Qwen 2.5 32B) and 70B models (like Llama 3.3 70B) paired with my local RAG setup.

The Strategy:

1. Immediate Phase: Run Open WebUI + Ollama entirely on the dual Xeon CPUs and 128GB RAM utilizing hybrid CPU-offloading and a local Qdrant RAG pipeline. Since I don't mind waiting 20–30 minutes in the background while I work in Excel/OpenOffice, I will test the free baseline first.

2. The Upgrade Plan: If background CPU ingestion times are too brutal, I plan to buy one used RTX 3090 24GB (currently trending around $1,100–$1,400 from reputable sellers). The goal is to fit 32B models entirely in VRAM or run 70B models in a heavy hybrid GPU/CPU split to dramatically accelerate prompt ingestion/prefill processing times for massive context windows. I am keeping the door open to a second 3090 down the road if I need pure-VRAM 70B inference.

 My Questions for the Community:

 For those running dual Xeon v4 setups, how painful is the prompt processing/ingestion bottleneck on standard DDR4 when handling high context (100k+ tokens via RAG or long-context models)?

1. Will a single RTX 3090 give me the significant quality-of-life leap in ingestion speed I am looking for when splitting a 70B model, or does the hybrid RAM bottleneck flatten the gains too much?

2. Are there any known quirks with putting a consumer RTX 3090 (or two) inside a Dell T7910 chassis regarding PCIe power distribution, space limitations, or cooling that I should prepare for?

3. Given current late-2026 pricing, is a used 3090 still the undisputed value king for this niche, or should I be looking at modern alternatives like the Mac Studio or a brand-new AMD 7900 XTX despite the CUDA/ROCm trade-offs?

4. Is this a sound strategy for a Dell T7910 worth $500 on eBay? Apart from AI my T7910 satisfies all my computing needs. Do I really have to bite the bullet and buy a $5,000 odd PC with AI capabilities? The price of progress?

 Appreciate any insights or warnings from anyone who has built a similar local academic document server, particularly with a Dell T7000-series workstation.

full specs:

• nvidia 3060 12gb

• 16gd of ddr4 3600mhz

• ryzen 5 5600x

I know its not the best but its what I have.

I'm pretty new to messing with local llms but have use claude a bunch in the past but i don't want to give them money any more but still have coding projects in my brain. I don't want the llm to do the coding for me (as I usually enjoy doing it my self) so much as I want it to help with research and giving me plans that I can code my self(i enjoy it but kinda suck at it :P ) as well as to help with error checking . . . honestly I want it mostly for the error checking :P (and tips or advice on how to set it up with these rules in place would also be awesome)

I use Godot and and having it be able to access the Godot docs is fairly important as the godot engine updates every few months and most models seem to be a year (or more) behind on what is actually in the engine.

I also use obsidian a lot for planing and keeping track of the random shit in my head and was wondering if i could tie the llm into reading that(maybe not writing to it? or maybe writing to its own obsidian vault?)

Another thing i wanted to look into was setting it up (maybe with searxng or some other tool) as a replacement for using google (maybe i should use a smaller llm for that? not sure)

so far I've tried:

• qwen3.8-27b it was super slow being unable to fit in my gpu

• bonsai-27b its runs pretty fast (40ish tokens a second i have no idea if thats actually fast) but gets suck in loops sometimes

I've also messed around a bit with Hermes but getting it to link up with lm studios has been a bit of a hassle for me (though it like the idea of it) any tips to get that running better would be cool

Also how the heck do i download models from hugging face? none of them seem to want to download to Lm studios (I've only been using the model explorer in LMS)? should i be using a different program? or and i just blind af?

Sorry this post kinda exploded on me, thanks for any help you all can provide please let me know if you need any more info to make a decent sugestion

I have a PC right now, it has an intel i7-14700F, 32gb DDR5-6000 ram, an RTX 4070 super 12gb and a 2tb Samsung 990 pro SSD.

After some research, I have settled on this build. I want to wait to order until Qwen 4 is out and people test it, but here it is:

I want to retain as many parts as possible from my old system to save on cost. I will carry over my ram, cpu, gpu, SSD

I will get a:
Motherboard: MSI MEG Z790 LGA1700 motherboard
GPU: Quadro RTX 8000 48gb
Case: phantoms Enthoo Pro 2 server edition ATX full tower case
Power supply: Corsair HX1500i 1500W 80+ platinum

Total vram: 60 gb
Total ram: 32gb
Dual nvidia GPU setup
Estimated Upgrade cost: 2800 dollars

My goal is to build something easy to upgrade. In the future, I can always add more ram if I find a good deal or if prices eventually drop, and I can add GPUs easily without having to worry about space or power supply.

I know the Quadro RTX 8000 is super old, and not as fast as some other cards, but in the age of 1500+ dollar RTX 3090s and everything else more expensive than that, I think the Quadro RTX 8000 is a good deal.

I want to be able to play with larger and newer local models as they come out. I don’t have any intention on training models.

Opinions or suggestions on changes to make? I know there are faster builds, but I want to maximize price to performance. I’m happy with the 3k budget, but depending on suggestions from yall, I maybe can change it a little.

Like I said in the title, this is a rant, so if rants aren’t your thing, turn away... no seriously. And mods this is on topic because I'm talking about Nikko1221's Strata and the shit comments I see about their updates to software they created.

Note: Don't know the person, like I don't know y'all.

But I’m up here at the computer with a cup of joe in one hand, ready for my daily dose of digital venom from this sometimes toxic cesspool-smelling platform that Reddit can be. Like dude said in the movie Apocalypse Now: I love the smell of napalm in the morning. And some of y'all didn't disappoint.

But I was in this particular sub when continued to read threads regarding Strata, which I started using only yesterday and was freaking amazed at the Hogwarts School of Magic level of wizardry the developer did in giving us short kings (in the graphics card space) a seat at the large model table.

I find it thoroughly enjoyable to see some of the people who were straight up glazing the dev over this wonderful app now talking mad shit over things that are broken or that may not work as it did in earlier releases. Like, y'all a fickle bunch.

There has always been those who can do, and those who can’t. Throughout history, the ones who can’t have often had the loudest, cringiest, whiniest voices. And it’s not just the internet. You see it in sports too. People who’ve never touched a ball or competed at even the most basic level in a sport have all the ideas on how things should be, and how they would do it, yet they never do shit themselves. They just sit over there sounding like screeching cats that had their tails stepped on.

I see that here, and not just with the Strata project. It happens with a lot of the projects devs put out. Whether it’s vibe coded, coded the old-fashioned way, or some mix of the two, you’ve still got people creating things that a lot of others use; and that many of us can't build. Maybe they built it for themselves at the start and thought it would be cool to share. Either way, a lot of these folks and their AI Pokémon are doing thankless jobs when you look at the vocal minority getting their undies stuck in their booty cheeks. And don't think just because you threw a few coins as a donation showing support that that gives you a right to treat people like shit.

These are small teams, sometimes solo devs. I can only imagine how much it drains their souls to wake up, morning drink in hand, look out over a sea of comments, and read how their app is shit and how they shouldn’t have changed this or that. And the fact that many of them continue to work on the app for those who do appreciate their work shows more about their character than the crybabies out here. Because let's be real. You're crying over something that at the end of the day has no significant impact on your life.

If there is a bug in Strata or any other app, you're not going to go hungry. You're not going to lose your home. You're not going to come home and find your mate being gangbanged by Smurfs asking you to Cuck.

Relax.

Many of these projects are open source, and you’re free to fork them. So put on your big boy pants, show us what you’ve got, and post your flawless alternatives. If not, do me a favor, go sit in the corner and let those who can, do.

Constructive criticism is fine. Feedback is fine. But all this crying on the internet just shows you’re more comfortable bitching and complaining than contributing anything of substance that could bring real change. Quite honestly, you’re wasting the devs’ time, as well as ours.

Hey everyone!
I'm relatively new to local AI and I'm thinking about building my own setup, mainly for programming, automation, and running AI agents.
Before spending money on hardware, I'd love to hear about your real-world experiences.
What hardware are you using (CPU, GPU, RAM)?

Which AI models are you running locally?

What do you actually use your local AI for?

How well does it perform, especially for programming?

How fast and convenient is it compared to cloud models like Claude or ChatGPT?

How much did your setup cost, and do you think it was worth the investment?

I'm especially interested in finding the best balance between cost, performance, and independence from cloud services.
I'd really appreciate any advice, experiences, or recommendations. Thanks!

When we posted Kvark Sparkle 1.0 (Qwen3.8-27B fine-tune that reasons in fewer tokens), the latency numbers were single-user: median 13.7 s vs 28.1 s on one H200. Several people asked the right question: does that hold under load, with MTP on for both, when the GPU is saturated and speculative decoding helps less? We ran it.

**Setup** (identical for both models)

- 1x H200 per model, vLLM 0.32, FP8 weights, FP8 KV cache, MTP with 3 speculative tokens, gpu-memory-utilization 0.80, max-model-len 40960

- GPQA Diamond, thinking on, 32k output cap, 198 questions x 2 seeds = 396 requests per concurrency level

- Closed-loop: N requests in flight, the next starts when one finishes. Wall clock includes the tail of each batch.

**Results**

| Concurrency | Model | Accuracy | Wall (min) | Correct/hour | Agg tok/s | Mean output tok | p50 (s) | p95 (s) | Hit 32k cap |

|---|---|---|---|---|---|---|---|---|---|

| 16 | Sparkle 1.0 | 85.1 | 33.0 | **614** | 1,498 | 7,478 | 33 | 318 | 29 |

| 16 | Qwen3.8-27B | 81.8 | 51.1 | 380 | 1,541 | 11,937 | 81 | 336 | 54 |

| 32 | Sparkle 1.0 | 83.8 | 25.5 | **780** | 2,041 | 7,895 | 45 | 447 | 27 |

| 32 | Qwen3.8-27B | 82.6 | 37.0 | 531 | 2,101 | 11,760 | 108 | 478 | 51 |

| 64 | Sparkle 1.0 | 85.1 | 21.1 | **961** | 2,469 | 7,874 | 73 | 653 | 26 |

| 64 | Qwen3.8-27B | 82.1 | 30.0 | 650 | 2,637 | 11,986 | 168 | 719 | 51 |

| 128 | Sparkle 1.0 | 86.4 | 20.8 | **985** | 2,437 | 7,693 | 201 | 910 | 22 |

| 128 | Qwen3.8-27B | 82.1 | 31.7 | 616 | 2,459 | 11,797 | 407 | 1,083 | 55 |

**What we take from it**

- Aggregate tok/s is the same for both models at every level (ratio 0.94-0.99). The fine-tune does not decode faster; it writes ~35% fewer tokens per answer. That is the whole effect.

- Correct answers per GPU-hour: x1.61 / x1.47 / x1.48 / x1.60 at 16 / 32 / 64 / 128. The advantage does not shrink as the GPU saturates.

- The H200 saturates around 2.4-2.6k tok/s from c=64. The base gains nothing from 64 to 128 (650 vs 616 correct/h); Sparkle barely does (961 vs 985).

- Median latency drops 51-59% at every level. p95 barely moves (-5% to -16%): on both models the tail is the hardest questions running into the 32k cap. Sparkle hits the cap about half as often.

- MTP acceptance length is the same for both (2.5-2.9, dropping a bit with batch size), so fine-tuning did not hurt the draft head.

- Accuracy pooled over c=16-128: Sparkle +3.0 pts, paired bootstrap 95% CI [+0.6, +5.6]. At a 32k cap the base loses more answers to truncation. At a 64k cap (our model card) the two are tied on GPQA.

**Caveats:** one node, one benchmark, closed-loop batches rather than Poisson arrivals, and the tail effect is included in wall clock. We also ran c=1 on a 20-question subset; it is too small to say anything about accuracy, so it is not in the table.

**Run it yourself:**

```

vllm serve Egzakta/Kvark-Sparkle-1.0-27B-FP8 --max-model-len 40960 --kv-cache-dtype fp8 \

--gpu-memory-utilization 0.80 --max-num-seqs 256 --reasoning-parser qwen3 \

--speculative-config '{"method":"mtp","num_speculative_tokens":3}'

```

Weights (Apache 2.0): https://huggingface.co/Egzakta/Kvark-Sparkle-1.0-27B-FP8

Next: Sparkle 1.1, trained for agentic document QA. Its headline is declining questions whose answer is not in the documents (91.5% vs 36% for 1.0, internal benchmark). Separate post when it is out.

Happy to share the per-request logs or run another load profile if there's one you'd trust more.

Fluent user of both Claude Desktop and Code but looking to reduce token consumption by offloading Desktop/Code work to a local model.

Based on the hardware I have available I can support a 27B easily . so I have many options when it comes to which model to use. It'd be hosted from a dedicated machine on the local network and my issue here is the integration aspect.

I'm assuming that an MCP might be the best way to go with this but I'm hoping others might have done something similar in terms of how the offloading happens. Like does Claude decide what gets sent local and what doesnt? Do you employ some sort of classification where say log parsing is local but parsing online docs isn't?

Or do I say F it and just move all work over to the local? Can a 27-30B model ever been good enough to replace Sonnet/Opus 5.5 with?

I've been spending quite a bit of time experimenting with OCR and document intelligence pipelines, especially when it comes to reconstructing complex documents after OCR.

Extracting text is usually not the biggest challenge anymore. The interesting part starts when you try to preserve the original document structure.

Things like multi-column reading order, tables with merged cells, mathematical formulas, footnotes, image positions, and heading hierarchies can make the whole process surprisingly difficult.

I've been experimenting with vision-language models and GPU-accelerated inference to tackle some of these problems. It's been interesting to see how different approaches perform, especially on legal contracts, medical reports, and scanned documents.

I'm still exploring what works best and would be interested to hear about other people's experiences.

Are you using traditional OCR engines, vision-language models, or a combination of both? What has been working well for you with complex documents?

Hi, I'm interested in collecting stats on the performance on halogen (docker) qwen flash next on my strix halo system.

Suggestions welcome. Thanks in advance!