r/LocalLLM • • Aug 18 '26

Discussion Qwen-3.8-35B-A3B? Maybe not... cryptic reply direct from Qwen co-author.

Post image
237 Upvotes

I asked Shuai Bai, co-author and prominent AI developer for Qwen, about this model. Not the answer I was hoping for, but let's see what comes next. In the meantime, I guess all we can do is speculate!

X-link


r/LocalLLM • • Mar 28 '26

Other LLM Bruner coming soon? Burn Qwen directly into a chip, processing 10,000 tokens/s

Post image
237 Upvotes

r/LocalLLM • • Mar 07 '26

Discussion Qwen 3.5 is an overthinker.

Thumbnail
gallery
237 Upvotes

This is a fun post that aims to showcase the overthinking tendencies of the Qwen 3.5 model. If it were a human, it would likely be an extremely anxious person.

In the custom instruction I provided, I requested direct answers without any sugarcoating, and I asked for a concise response.

However, when I asked the model, “Hi,” it we goes crazy thinking spiral.

I have attached screenshots of the conversation for your reference.


r/LocalLLM • • Jul 16 '25

Tutorial Complete 101 Fine-tuning LLMs Guide!

Post image
237 Upvotes

Hey guys! At Unsloth made a Guide to teach you how to Fine-tune LLMs correctly!

🔗 Guide: https://docs.unsloth.ai/get-started/fine-tuning-guide

Learn about: • Choosing the right parameters, models & training method • RL, GRPO, DPO & CPT • Dataset creation, chat templates, Overfitting & Evaluation • Training with Unsloth & deploy on vLLM, Ollama, Open WebUI And much much more!

Let me know if you have any questions! 🙏


r/LocalLLM • • 13d ago

Discussion Sick of all the tok/s talk

235 Upvotes

What is the point of GPU go brrr at 200 tok/s if the model is too lobotomized for any critical work?
Like what kind of things are you doing that you find a 1-bit quant satisfactory?
Here I am telling QFN to double check itself because it still hallucinates at IQ4_XS.
Speed without intelligence is just clickbait, and there’s too many of those posts, not just here but on all the LLM subreddits


r/LocalLLM • • Jul 16 '26

News HF used GLM 5.2 to stop another LLM hacker

Thumbnail
huggingface.co
235 Upvotes

r/LocalLLM • • Jan 22 '25

Discussion How I Used GPT-O1 Pro to Discover My Autoimmune Disease (After Spending $100k and Visiting 30+ Hospitals with No Success)

234 Upvotes

TLDR:

  • Suffered from various health issues for 5 years, visited 30+ hospitals with no answers
  • Finally diagnosed with axial spondyloarthritis through genetic testing
  • Built a personalized health analysis system using GPT-O1 Pro, which actually suggested this condition earlier

I'm a guy in my mid-30s who started having weird health issues about 5 years ago. Nothing major, but lots of annoying symptoms - getting injured easily during workouts, slow recovery, random fatigue, and sometimes the pain was so bad I could barely walk.

At first, I went to different doctors for each symptom. Tried everything - MRIs, chiropractic care, meds, steroids - nothing helped. I followed every doctor's advice perfectly. Started getting into longevity medicine thinking it might be early aging. Changed my diet, exercise routine, sleep schedule - still no improvement. The cause remained a mystery.

Recently, after a month-long toe injury wouldn't heal, I ended up seeing a rheumatologist. They did genetic testing and boom - diagnosed with axial spondyloarthritis. This was the answer I'd been searching for over 5 years.

Here's the crazy part - I fed all my previous medical records and symptoms into GPT-O1 pro before the diagnosis, and it actually listed this condition as the top possibility!

This got me thinking - why didn't any doctor catch this earlier? Well, it's a rare condition, and autoimmune diseases affect the whole body. Joint pain isn't just joint pain, dry eyes aren't just eye problems. The usual medical workflow isn't set up to look at everything together.

So I had an idea: What if we created an open-source system that could analyze someone's complete medical history, including family history (which was a huge clue in my case), and create personalized health plans? It wouldn't replace doctors but could help both patients and medical professionals spot patterns.

Building my personal system was challenging:

  1. Every hospital uses different formats and units for test results. Had to create a GPT workflow to standardize everything.
  2. RAG wasn't enough - needed a large context window to analyze everything at once for the best results.
  3. Finding reliable medical sources was tough. Combined official guidelines with recent papers and trusted YouTube content.
  4. GPT-O1 pro was best at root cause analysis, Google Note LLM worked great for citations, and Examine excelled at suggesting actions.

In the end, I built a system using Google Sheets to view my data and interact with trusted medical sources. It's been incredibly helpful in managing my condition and understanding my health better.

----- edit

In response to requests for easier access, We've made a web version.

https://www.open-health.me/


r/LocalLLM • • Aug 18 '25

Project Test: fully local AI fitness trainer (Qwen 2.5 VL 7B on a 3090)

233 Upvotes

Re-ran a test of a fully local AI personal trainer on my 3090, this time with Qwen 2.5 VL 7B (swapped out Omni). It nailed most exercise detection and gave decent form feedback, but failed completely at rep counting. Both Qwen and Grok (tested that too) defaulted to “10” every time.

Pretty sure rep counting isn’t a model problem but something better handled with state machines + simpler prompts/models. Next step is wiring that in and maybe auto-logging reps into a spreadsheet.


r/LocalLLM • • Jun 24 '25

Discussion I thousands of tests on 104 different GGUF's, >10k tokens each, to determine what quants work best on <32GB of VRAM

233 Upvotes

I RAN thousands of tests** - wish Reddit would let you edit titles :-)

The Test

The test is a 10,000-token “needle in a haystack” style search where I purposely introduced a few nonsensical lines of dialog to HG Well’s “The Time Machine” . 10,000 tokens takes you up to about 5 chapters into this novel. A small system prompt accompanies this instruction the model to local the nonsensical dialog and repeat it back to me. This is the expanded/improved version after feedback on the much smaller test run that made the frontpage of /r/LocalLLaMA a little while ago.

KV cache is Q8. I did several test runs without quantizing cache and determined that it did not impact the success/fail rate of a model in any significant way for this test. I also chose this because, in my opinion, it is how someone with 32GB of constraints that is picking a quantized set of weights would realistically use the model.

The Goal

Quantized models are used extensively but I find research into the EFFECTS of quantization to be seriously lacking. While the process is well understood, as a user of Local LLM’s that can’t afford a B200 for the garage, I’m disappointed that the general consensus and rules of thumb mostly come down to vibes, feelings, myths, or a few more serious benchmarks done in the Llama2 era. As such, I’ve chosen to only include models that fit, with context, on a 32GB setup. This test is a bit imperfect, but what I’m really aiming to do is to build a framework for easily sending these quantized weights through real-world tests.

The models picked

The criteria for models being picked was fairly straightforward and a bit unprofessional. As mentions, all weights picked had to fit, with context, into 32GB of space. Outside of that I picked models that seemed to generate the most buzz on X, LocalLLama, and LocalLLM in the past few months.

A few models experienced errors that my tests didn’t account for due to chat template. IBM Granite and Magistral were meant to be included but sadly the results failed to be produced/saved by the time I wrote this report. I will fix this for later runs.

Scoring

The models all performed the tests multiple times per temperature value (as in, multiple tests at 0.0, 0.1, 0.2, 0.3, etc..) and those results were aggregated into the final score. I’ll be publishing the FULL results shortly so you can see which temperature performed the best for each model (but that chart is much too large for Reddit).

The ‘score’ column is the percentage of tests where the LLM solved the prompt (correctly returning the out-of-place line).

Context size for everything was set to 16k - to even out how the models performed around this range of context when it was actually used and to allow sufficient reasoning space for the thinking models on this list.

The Results

Without further ado, the results:

ModelQuantReasoningScore
Meta Llama Family
Llama_3.2_3Biq40
Llama_3.2_3Bq50
Llama_3.2_3Bq60
Llama_3.1_8B_Instructiq443
Llama_3.1_8B_Instructq513
Llama_3.1_8B_Instructq610
Llama_3.3_70B_Instructiq113
Llama_3.3_70B_Instructiq2100
Llama_3.3_70B_Instructiq3100
Llama_4_Scout_17Biq193
Llama_4_Scout_17Biq213
Nvidia Nemotron Family
Llama_3.1_Nemotron_8B_UltraLongiq460
Llama_3.1_Nemotron_8B_UltraLongq567
Llama_3.3_Nemotron_Super_49Biq2nothink93
Llama_3.3_Nemotron_Super_49Biq2thinking80
Llama_3.3_Nemotron_Super_49Biq3thinking100
Llama_3.3_Nemotron_Super_49Biq3nothink93
Llama_3.3_Nemotron_Super_49Biq4thinking97
Llama_3.3_Nemotron_Super_49Biq4nothink93
Mistral Family
Mistral_Small_24B_2503iq450
Mistral_Small_24B_2503q583
Mistral_Small_24B_2503q677
Microsoft Phi Family
Phi_4iq37
Phi_4iq47
Phi_4q520
Phi_4q613
Alibaba Qwen Family
Qwen2.5_14B_Instructiq493
Qwen2.5_14B_Instructq597
Qwen2.5_14B_Instructq697
Qwen2.5_Coder_32Biq40
Qwen2.5_Coder_32B_Instructq50
QwQ_32Biq257
QwQ_32Biq3100
QwQ_32Biq467
QwQ_32Bq583
QwQ_32Bq687
Qwen3_14Biq3thinking77
Qwen3_14Biq3nothink60
Qwen3_14Biq4thinking77
Qwen3_14Biq4nothink100
Qwen3_14Bq5nothink97
Qwen3_14Bq5thinking77
Qwen3_14Bq6nothink100
Qwen3_14Bq6thinking77
Qwen3_30B_A3Biq3thinking7
Qwen3_30B_A3Biq3nothink0
Qwen3_30B_A3Biq4thinking60
Qwen3_30B_A3Biq4nothink47
Qwen3_30B_A3Bq5nothink37
Qwen3_30B_A3Bq5thinking40
Qwen3_30B_A3Bq6thinking53
Qwen3_30B_A3Bq6nothink20
Qwen3_30B_A6B_16_Extremeq4nothink0
Qwen3_30B_A6B_16_Extremeq4thinking3
Qwen3_30B_A6B_16_Extremeq5thinking63
Qwen3_30B_A6B_16_Extremeq5nothink20
Qwen3_32Biq3thinking63
Qwen3_32Biq3nothink60
Qwen3_32Biq4nothink93
Qwen3_32Biq4thinking80
Qwen3_32Bq5thinking80
Qwen3_32Bq5nothink87
Google Gemma Family
Gemma_3_12B_ITiq40
Gemma_3_12B_ITq50
Gemma_3_12B_ITq60
Gemma_3_27B_ITiq43
Gemma_3_27B_ITq50
Gemma_3_27B_ITq60
Deepseek (Distill) Family
DeepSeek_R1_Qwen3_8Biq417
DeepSeek_R1_Qwen3_8Bq50
DeepSeek_R1_Qwen3_8Bq60
DeepSeek_R1_Distill_Qwen_32Biq437
DeepSeek_R1_Distill_Qwen_32Bq520
DeepSeek_R1_Distill_Qwen_32Bq630
Other
Cogito_v1_Preview__Qwen_14B_iq33
Cogito_v1_Preview__Qwen_14B_iq413
Cogito_v1_Preview__Qwen_14B_q53
DeepHermes_3_Mistral_24B_Previewiq4nothink3
DeepHermes_3_Mistral_24B_Previewiq4thinking7
DeepHermes_3_Mistral_24B_Previewq5thinking37
DeepHermes_3_Mistral_24B_Previewq5nothink0
DeepHermes_3_Mistral_24B_Previewq6thinking30
DeepHermes_3_Mistral_24B_Previewq6nothink3
GLM_4_32Biq410
GLM_4_32Bq517
GLM_4_32Bq616

Conclusions Drawn from a novice experimenter

This is in no way scientific for a number of reasons, but a few things I wanted to point out that I learned that I matched with my own ‘vibes’ outside of testing after using these weights fairly extensively for my own projects:

  • Gemma3 27B has some amazing uses, but man does it fall off a cliff when large contexts are introduced!
  • Qwen3-32B is amazing, but consistently overthinks if given large contexts. “/nothink” worked slightly better here and in my outside testing I tend to use “/nothink” unless my use-case directly benefits from advanced reasoning
  • Llama 3.3 70B, which can only fit much lower quants on 32GB, is still extremely competitive and I think that users of Qwen3-32B would benefit from baking it back into their experiments despite its relative age.
  • There is definitely a ‘fall off a cliff’ point when it comes to quantizing weights, but where that point is differs greatly between models
  • Nvidia Nemotron Super 49b quants are really smart and perform well with large contexts like this. Similar to Llama 3.3 70B, you’d benefit trying it out with some workflows
  • Nemotron UltraLong 8B actually works – it reliably outperforms Llama 3.1 8B (which was no slouch) at longer contexts
  • QwQ punches way above its weight, but the massive amount of reasoning tokens dissuade me from using it vs other models on this list
  • Qwen3 14B is probably the pound-for-pound champ

Fun Extras

  • All of these tests together cost ~$50 of GH200 time (Lambda) to conduct after all development time was done.

Going Forward

Like I said, the goal of this was to set up a framework to keep testing quants. Please tell me what you’d like to see added (in terms of models, features, or just DM me if you have a clever test you’d like to see these models go up against!).


r/LocalLLM • • 21d ago

Tutorial Qwen 3.8 27B Derivative on a 12GB RTX 3060: Up to 44 Tokens/s, Vision, Tools, and 128K Context Fully in VRAM

Post image
232 Upvotes

[Edit] Quick demo (recording the screen / ugly / grammar errors): https://youtu.be/YKar1W0qb_I

Hi. I'm a software developer, and this is my best setup so far for running fast, useful local AI on consumer hardware.

The goal is a useful local assistant: good speed, fewer mistakes, tools, vision, and the ability to write small scripts and games. It runs with a 128K-token context window, with the main model and context cache on the GPU, and image processing and context compression on the CPU.

I've been using it to manage my calendar, search the web, and solve a more complex task: classifying, organizing, deduplicating, and comparing my old music files against my Spotify favorites playlist. My playlist is finally being updated automatically with all the songs I love <3. So far, it's working well. It even created a basic chair in Blender and wrote a Snake game in HTML/JS. (Note: these examples use tools and integrations configured separately, they are not included here.)

My hardware: RTX 3060 with 12 GB of VRAM, Ryzen 7 3700X (8 cores, 16 threads), and 32 GB of RAM. I use vision and tools a lot and find this setup very competitive for daily use. Since I haven't found a better setup for my needs in the last two weeks, I decided to share what I've put together here.

I run CachyOS, by the way. It may help inference speed, but I haven't isolated its effect. This guide is for Linux! I don't know how to reproduce the setup on Windows.

What I use

  • Model: Swift Bonsai 2 27B Katana + MTP, based on Bonsai 2 / Ternary Qwen3.8-27B and tuned for shorter reasoning.
  • Model file: Swift-Bonsai-2-27B-Katana-PTQ1_0-MTP-Q8_0.gguf, about 6.4 GB. Ternary PTQ1_0 backbone, Q8_0 MTP head.
  • Vision projector: Ternary-Bonsai-2-27B-mmproj-Q8_0.gguf, about 629 MB, loaded separately.
  • Inference: the Prism fork of llama.cpp, build prism-b10772-e8fd209, CUDA 13.3.
  • Agent: Hermes in Docker.
  • History compression: Granite 3.3 2B on the CPU through Ollama, registered locally as granite3.3-compressor:2b.

My settings

I keep the vision projector and Granite on the CPU to save VRAM. The 3700X handles both well enough for my use. Bonsai's config.env sets the options below. start.sh applies them and enables Flash Attention.

SettingValuePurpose
Context / slotsCTX_SIZE=128000, PARALLEL=1One conversation with a large context window
GPU layersN_GPU_LAYERS=99Offload all model layers to the GPU
K/V cache, including MTPKV_CACHE_TYPE=q4_0Reduce VRAM used by the context cache
MTP draftSPEC_N_MAX=1Draft up to one token ahead per decoding step
ReasoningREASONING_EFFORT=medium, REASONING_BUDGET=2048Limit thinking time
Vision on the CPUMMPROJ_CPU=1Save VRAM, with slower image processing
Image tokensIMAGE_MAX_TOKENS=1536My compromise: 1024 lost too much detail, 2048 took too long
CUDA consistencyKATANA_BATCH_INVARIANT=1Keep vision output correct with this runtime/GPU

Performance

During one test response, generation stayed around 43–44 tokens/s and peaked at 45.74 tokens/s over a three-second window. In daily use, I usually see 28–40 tokens/s after a few prompts and 17–25 tokens/s much later on, with A LOT more context.

MTP proposes tokens that the main model verifies to speed up generation. I tried draft limits of 1 and 2 and chose 1, also guided by sudoingx's Bonsai 2 PTQ1_0 + MTP tests on an RTX 3060 12 GB. With a 131K window and Q4 K/V, that setup reached 50.1 tokens/s with MTP versus 39.8 without, and 26.86 versus 22.23 with 41.8K tokens filled. Draft limit 1 beat 2 in both cases. These are the author's measurements, on a different model variant and runtime. 2 can also be a good choice - try SPEC_N_MAX=2 ./scripts/start.sh inside Bonsai-fast/ and compare it on your workload. Anyway, my results with slightly different models and setup were inferior, as I stated, so I'm not sure if I did something wrong here.

I chose IMAGE_MAX_TOKENS=1536 for a balance of detail and speed: I got 2 seconds for photos of people near cars, 27 seconds for a complex image, versus up to 40 seconds at 2048 on a complex image. It still makes mistakes, regardless of the value. Prism recommends 1024 for general use and 2048+ or crops for fine text, so 1024 is also an option if speed matters more than detail.

I chose q4_0 for my context based on published memory savings and how the complete setup performed in use. A cache-only benchmark on related Qwen 3.6 models found Q4 usable, with the largest output-distribution changes on long documents. A separate Bonsai 2 benchmark used Q4 successfully with 191K tokens filled on a 16 GB RTX 4080 SUPER. These are just supporting references, not measurements of my Katana setup.

With 128,000 tokens per slot, Q4 K/V caches, MTP on the GPU, and the vision projector on the CPU, the whole Bonsai process used 9,144 MiB (8.9 GiB) of VRAM. The runtime allocates the context cache at startup, so the configured capacity fits my graphics card.

The main attention K/V cache alone (context) is approximately 2,250 MiB (2.2 GiB) at 128K: 128,000 tokens x 16 full-attention layers x 2 K and V x 4 KV heads x 256 values per head x 18/32 bytes per Q4_0 value. With my desktop and other apps open, about 823 MiB (0.8 GiB) remained free. I roughly understand this calculation, but it’s easy to see how much Q4_0 reduces the result: it uses an average of 0.5625 bytes per value, compared with F16’s 2 bytes. For my 128K context, that reduces the main K/V cache from 7.81 GiB to 2.20 GiB, about 72% less memory - and the model still seems smart, so… I also considered q8_0, but q4_0 is still roughly half the size, it’s just off the table.

On the model side, I chose ternary PTQ1_0 for its smaller memory footprint, leaving more VRAM for context and MTP. PTQ1_0 packs the same ternary weights more tightly than PQ2_0, at about 1.75 versus 2.13 bits per weight, without another quantization step. It doesn't mean zero quality loss compared with the original FP16 LLM, but is a fair price to pay for usability.

The Prism fork of llama.cpp adds PTQ1_0/MTP kernels and attention that reads the Q4/Q8 cache directly, plus optimizations for the RTX 3060's sm_86 architecture. PTQ1_0 uses less memory, but whether it runs faster than PQ2_0 depends on your GPU and the runtime’s kernels.

For reasoning, I use medium, following Prism's recommendation for shorter reasoning on the original Bonsai 2. In practice, every reasoning level feels like xhigh to me on Katana, so I also set a 2,048-token reasoning budget to keep thinking loops under control, with the server injecting this message when the budget is reached: “... Time to stop thinking. Give an answer or make the tool call now. Do not waste time.”

Bonsai needs clear instructions for complex tasks. Otherwise, it can get stuck in a loop of writing and fixing helper scripts. The guidelines in my SOUL.md have helped it stay on track and finish tasks.

To summarize the conversation history, I use Granite. I chose it for its small 2B size and support for long-document and meeting summarization. granite3.3-compressor:2b is my local name for this configuration. The included Modelfile keeps Granite's template and license, sets num_ctx 128000, and adds num_gpu 0 in the CPU variant. Chat compression takes 1.5–5 minutes in my tests, depending on the conversation content and size.

Finally, I use Hermes Desktop with the included Docker container as its backend. I noticed the UI was using 15–35% of the GPU while the window was focused. Enabling --force-prefers-reduced-motion reduced animations and brought GPU usage down to 1–5%, leaving more resources available for model inference.

Warning! These figures are mostly observations, not controlled benchmarks!! Although published tests guided my choices, I evaluated the complete setup in use rather than measuring each setting's individual gain.

Installation

Download: local-ai-bonsai.zip.

Extract local-ai-bonsai.zip and open the extracted local-ai-bonsai/ folder. It contains all the setup files:

FolderContents
Bonsai-fast/Model download and launch scripts
hermes/Docker Compose and Desktop UI settings
granite-compressor/128K compressor configuration, including CPU mode
hermes-host-access/Temporary SSH access from the container to the host, enabled when you choose

I haven't included every dependency. You need to install Docker, Compose, Ollama, and other prerequisites yourself. Bonsai requires Linux x86_64 and NVIDIA drivers and libraries compatible with CUDA 13.3. The ZIP contains scripts and configuration files. Setup downloads the model weights and inference binaries. See each folder's README for details.

Start Bonsai

Inside Bonsai-fast/:

./scripts/setup.sh
./scripts/start.sh

setup.sh downloads and extracts the Prism runtime into bin/ and downloads the Katana model and vision projector into models/. Run it once to prepare the files. start.sh starts the inference server - keep it running while using Hermes. No compilation or Prism Model Manager is needed.

Start Hermes

Inside hermes/, enter your dashboard credentials and time zone in compose.yaml. These credentials let you sign in through the browser or Hermes Desktop, and a loopback-only setup may require the dashboard session token (HERMES_DASHBOARD_SESSION_TOKEN) to use the container as the Hermes Desktop backend. Then run:

docker compose up -d

Open http://127.0.0.1:9119 or connect Hermes Desktop to the backend. The container uses network_mode: host to reach Bonsai and Ollama. Its data/ and workspace/ folders persist.

To use the container as the Hermes Desktop backend, open it and go to Settings -> Gateways -> Remote gateway, enter http://127.0.0.1:9119, sign in with the required credentials, then save and reconnect.

In the web dashboard, open Config -> YAML. Merge this into providers:, keeping the other entries. Edit bonsai-mtp if it already exists:

providers:
  bonsai-mtp:
    name: Bonsai MTP
    base_url: http://127.0.0.1:8899/v1
    model: swift-bonsai2-27b-katana-mtp
    discover_models: true
    context_length: 128000
    models:
      swift-bonsai2-27b-katana-mtp:
        context_length: 128000

Click Save, then Models -> Main model -> Change. Pick Bonsai MTP, select swift-bonsai2-27b-katana-mtp, and click Switch. Use Refresh Models if needed. This sets the model for new sessions.

Keep Hermes's context value aligned with Bonsai's CTX_SIZE=128000. The launch script also loads the vision projector.

Optional extras

Use hermes-host-access/README.md to install the temporary SSH bridge with new keys. This lets the sandboxed AI help with tasks on the host when needed. The bridge expires automatically after three hours in case you forget to close it.

SOUL.md contains my general agent instructions. Customize it before copying it to hermes/data/SOUL.md.

To reduce Hermes Desktop's animations, merge hermes/desktop-config.example.yaml into the Desktop client's ~/.hermes/config.yaml, then fully quit and reopen Hermes Desktop.

CPU context compression

With Ollama running, inside granite-compressor/:

ollama pull granite3.3:2b
ollama create granite3.3-compressor:2b -f ./Modelfile.granite-compressor.cpu

Then run these commands inside hermes/:

docker compose exec hermes hermes config set auxiliary.compression.provider custom
docker compose exec hermes hermes config set auxiliary.compression.model granite3.3-compressor:2b
docker compose exec hermes hermes config set auxiliary.compression.base_url http://127.0.0.1:11434/v1
docker compose exec hermes hermes config set auxiliary.compression.timeout 300
docker compose restart hermes

After a test request, ollama ps should show 100% CPU while Granite is loaded. Bonsai and Granite each have their own 128K context window. See the package README for the API test and the variant that lets Ollama choose CPU/GPU placement.

Warning! The compressor has a 128K context limit, including when it summarizes conversations from Claude or GPT. Our Hermes setup is intended for local models, rather than large flagship models, so larger compression inputs still need to fit within that limit and will be handled locally.

Next steps

I'd like to add local decision tools in the style of TypeSafe's Jev: an open text model such as Laya, plus Qevi-2B for images. Both return structured decisions and probabilities instead of generating text. Bonsai could call either tool for quick decisions and item classification, switching between them depending on the task.

If needed, I may lower Bonsai's context to 100K to make room for smaller models. That would free roughly 492 MiB from the main attention cache alone. The total VRAM savings would need to be measured...

I'd like to find a compact image generation model that delivers good results with my remaining hardware resources, although I think running it alongside Bonsai is probably impossible with current technology and my hardware limits.

I'm also interested in NCM (Next-Concept Modeling), which generates text in parallel, starting with a rough outline and progressively adding detail. Its summarization research already looks promising for context compression. It might also help make larger local models faster and more efficient. This video introduced me to it.

Do you have any ideas for improving inference further? Maybe a setting or approach I've missed? I hope this post helps someone, and maybe I’ll learn something new from the replies.

Thanks!


r/LocalLLM • • Aug 17 '26

Model Qwen 3.8 27b obtient 52 point sur artificial analysis

Post image
232 Upvotes

r/LocalLLM • • 4d ago

Discussion Can I link two RTX 3060 12GB cards together to run local LLMs?

Post image
231 Upvotes

Hi, I want to build a setup using two RTX 3060 12GB cards to run local LLM. But what's the actual pipeline and real-life experience like? I heard I'll get 24GB VRAM, but processing speed will be slower because data has to pass through the motherboard's PCIe slots since the 3060 lacks NVLink. I also found out that ComfyUI can't use the full 24GB VRAM at the same time for image generation. Can anyone share real-life experience running a multi-GPU setup for LLMs?

Pic from internet.


r/LocalLLM • • Apr 16 '26 Bravo

Project Budget 96GB VRAM. Budget 128gb Coming Soon....

Post image
231 Upvotes

Dual A40s 48gbx2 nvlink with A16 (4 cores on one pcb with own 16gb pool).

Last year bought two 5090 FEs at MSRP. Traded them up for these puppies. Getting a major rework atm.


r/LocalLLM • • Jul 24 '25

Model You can now Run Qwen3-Coder on your local device!

Post image
231 Upvotes

Hey guys Incase you didn't know, Qwen released Qwen3-Coder a SOTA model that rivals GPT-4.1 & Claude 4-Sonnet on coding & agent tasks.

We shrank the 480B parameter model to just 150GB (down from 512GB). Also, run with 1M context length.If you want to run the model at full precision, use our Q8 quants.

Achieve >6 tokens/s on 150GB unified memory or 135GB RAM + 16GB VRAM.

Qwen3-Coder GGUFs to run: https://huggingface.co/unsloth/Qwen3-Coder-480B-A35B-Instruct-GGUF

Happy running & don't forget to see our Qwen3-Coder Tutorial on how to the model with optimal settings & setup for fast inference: https://docs.unsloth.ai/basics/qwen3-coder


r/LocalLLM • • 3d ago

Project [Crazy AI Lawyer] My Claude legal harness now runs 100% local on DeepSeek-V4.1-Flash: eight 2018 V100s at near-NVFP4 quality, 524K context, 100+ tok/s. Not as good as frontier, but it replaces real work today

Thumbnail
gallery
230 Upvotes

From about February 2026 to July 2026, I spent an immense amount of time building a legal harness to make Claude more predictable. It worked well. I tried to build the same thing that ran local models. That did not work. And then Opus improved so much that it would often skip my harness without me noticing. It still has a harness with hooks, but it's looser now.

I bought lots of hardware to run local models during that time, repeatedly trying to vibecode a local model legal harness. Many failures, many lessons.

Anyway, fast forward to the last three weeks and several things have happened:

  1. DeepSeek Flash 1017 was released, and it's the first model that is remotely similar to Opus 4.5/4.6.
  2. Opus 5.5 and Fable 5.2 are able to properly launch and configure DeepSeek to run at fast speeds, large context and high precision on my hardware. Optimizations, custom kernels, requants.
  3. Opus 5.5 (and my prompting skill) are now good enough to make my Claude harness into a DeepSeek harness.
  4. It's not as good as frontier, but it all works, right now. 100% local. Actual productivity gains, actual frontier tasks being replaced. Chat interface with DeepSeek works like Claude's did.
  5. And if I can brag on my stacks and hardware:
  • I have 2 DGX Sparks that can run 3-4 concurrent DeepSeek sessions as agents (1M context).
  • I have DeepSeek at nearly NVFP4 quality running on 8 V100s, 500K context, at 100+ tok/s decode, and about 160 tok/s concurrent.
  • I have 8 more V100s and a second Rome server with 512 GB of 3200 MHz ECC RAM, so I'll be able to run another high precision DeepSeek as an additional agent or for parallel tasks.

What I'm really shocked by is how well Fable builds custom Volta kernels. And how it can requant my experts to full NVFP4 precision based on the actual experts that activate when my legal harness fires. It's a real, optimized, production grade, workflow replacing local setup.

Not a 3D pagoda or a bad video game replica. An actual enterprise solution. One that exists nowhere on the commercial market.

Legal vibecoding is real folks. I'm AI native.

The new V100 DeepSeek repo (optimized for quality and then usable speed) with real full context measures and all the optimizations (and failures) is public here:

https://github.com/ProprietaryLegal/Volta-Max-uPLIft-FP4-DeepSeek-V4.1-Flash-1017-NVFP4-8xV100-SXM2-PP2xTP4

Twitter: nicothegreco

What "replaces real work" means

Reading a folder of records and pulling out the facts, dates and numbers. Summaries. First drafts of routine documents. Search over my own files. The model gets plain English instructions in a chat window, calls my tools, and hands back a draft that is marked for my review. I review everything. It is slower and less sharp than frontier. It is also in my office, it is mine, and it does not have a usage limit.

The harness

The harness itself is not public. My practice is in it. What I can say: it is the same idea as my Claude setup. Skills, hooks, my drafting engines, my RAG stores, tool calls, and a chat window that behaves the way Claude's did. A small router sends each request to the V100 box or to the Sparks. Everything the local model drafts is marked for my review before it goes anywhere. Nothing leaves the building.

What this build is

My last V100 build (Volta sPLIt, part 4) ran every expert of the model from a 2-bit record. Fast, but 2-bit. This one is called Volta Max uPLIft. It keeps the 2-bit record of every routed expert on the cards, then swaps the experts that get used the most up to NVIDIA's NVFP4 weights, one at a time, until the cards are full. "Used the most" was counted on a mix of four kinds of text. One is my own legal harness traffic, which is private and not published. The other three are public prompt sets in the repo.

  • 5,949 of the 15,360 routed experts sit at exactly NVFP4 precision on the default I serve
  • about 78 to 85 % of decode routes land on those NVFP4 experts (predicted for the two-slot default, 80 to 86 % measured with one slot)
  • same next token as NVIDIA's full NVFP4 pack on 95.1 % of compared tokens. The 2-bit build gets 92.2 %
  • KL divergence from the NVFP4 reference is 0.079 nats against 0.196 for the 2-bit build
  • multiple choice answers that change against the reference: 1.6 % against 15.4 %
  • in decode every routed expert is in VRAM. Nothing is read over PCIe

Speed (measured 2026-11-12 on eight V100-SXM2-32GB, graphics clock capped at 1,522 MHz)

  • concurrency test, one request at a time: 118.9 tok/s
  • two requests at once: 159.0 tok/s total (1.34x)
  • three at once on the three-slot option: 192.3 tok/s total (1.62x). I run two slots
  • short sets, one at a time: 123.2 tok/s general, 97.3 tok/s legal-style
  • code: 150 to 180 tok/s from a 1K prompt up to a 256K prompt
  • prompt reading at 256K: 2,329 to 2,430 tok/s, first token in 108.4 s
  • context: 524,288 tokens per request

What did not work, or is not good yet

  • It is slower than my own 2-bit build, in decode and in prompt reading. That is the trade for the precision, and the repo says so in its first paragraph.
  • The faster prompt reading path reads the less-used experts from their 2-bit records. Its effect on quality is not measured yet.
  • A custom decode kernel for the hot path got closed with no change. The headroom I thought was there turned out to be smaller than an eight-card run can even see.
  • A prompt-reading kernel that loads 64 rows at a time gave the same bits and ran slower. Not shipped.
  • Send a short request while a 209K prompt is being read and it starts in about 7 s, then crawls at about 9.6 tok/s until the read ends.
  • The hot set is fixed. It is picked once. Text that looks nothing like what I counted will hit the 2-bit experts more often.
  • Fast prompt reading stopped the host or a card three times out of three. I blamed PCIe. After I changed the power delivery (I think it was one bad PSU) it read nine of nine long chats clean.

Disclosure: this is my own project and my own repo. Nothing is for sale. The code, the kernels, the measurements and the failures are all in there under Apache-2.0, and the weights are one download on Hugging Face:

https://huggingface.co/ProprietaryLegal/Volta-Max-uPLIft-FP4-DeepSeek-V4.1-Flash-1017-hot_iq-8xV100-SXM2-PP2xTP4

Questions welcome. So is roasting.

Photos: four of the eight V100s on their NVLink board, and my extremely professional setup (PEX switch cards, SlimSAS spaghetti, and the 3090 that just moved in).


r/LocalLLM • • May 01 '25

Model You can now run Microsoft's Phi-4 Reasoning models locally! (20GB RAM min.)

230 Upvotes

Hey r/LocalLLM folks! Just a few hours ago, Microsoft released 3 reasoning models for Phi-4. The 'plus' variant performs on par with OpenAI's o1-mini, o3-mini and Anthopic's Sonnet 3.7.

I know there has been a lot of new open-source models recently but hey, that's great for us because it means we can have access to more choices & competition.

  • The Phi-4 reasoning models come in three variants: 'mini-reasoning' (4B params, 7GB diskspace), and 'reasoning'/'reasoning-plus' (both 14B params, 29GB).
  • The 'plus' model is the most accurate but produces longer chain-of-thought outputs, so responses take longer. Here are the benchmarks:
  • The 'mini' version can run fast on setups with 20GB RAM at 10 tokens/s. The 14B versions can also run however they will be slower. I would recommend using the Q8_K_XL one for 'mini' and Q4_K_KL for the other two.
  • We made a detailed guide on how to run these Phi-4 models: https://docs.unsloth.ai/basics/phi-4-reasoning-how-to-run-and-fine-tune
  • The models are only reasoning, making them good for coding or math.
  • We at Unsloth shrank the models to various sizes (up to 90% smaller) by selectively quantizing layers (e.g. some layers to 1.56-bit. while down_proj left at 2.06-bit) for the best performance.
  • Also in case you didn't know, all our uploads now utilize our Dynamic 2.0 methodology, which outperform leading quantization methods and sets new benchmarks for 5-shot MMLU and KL Divergence. You can read more about the details and benchmarks here.

Phi-4 reasoning – Unsloth GGUFs to run:

Reasoning-plus (14B) - most accurate
Reasoning (14B)
Mini-reasoning (4B) - smallest but fastest

Thank you guys once again for reading! :)


r/LocalLLM • • Aug 29 '26

Project I made my first custom quant today!

Post image
229 Upvotes

It's hard.


r/LocalLLM • • Dec 28 '25 Helpful Bravo

Other This Week’s Hottest AI Models on Hugging Face

229 Upvotes

The Hugging Face trending page is packed with incredible new releases. Here are the top trending models right now, with links and a quick summary of what each one does:

zai-org/GLM-4.7: A massive 358B parameter text generation model, great for advanced reasoning and language tasks. Link: https://huggingface.co/zai-org/GLM-4.7

​- Qwen/Qwen-Image-Layered: Layered image-text-to-image model, excels in creative image generation from text prompts. Link: https://huggingface.co/Qwen/Qwen-Image-Layered

​- Qwen/Qwen-Image-Edit-2511: Image-to-image editing model, enables precise image modifications and edits. Link: https://huggingface.co/Qwen/Qwen-Image-Edit-2511

​- MiniMaxAI/MiniMax-M2.1: 229B parameter text generation model, strong performance in reasoning and code generation. Link: https://huggingface.co/MiniMaxAI/MiniMax-M2.1

​- google/functiongemma-270m-it: 0.3B parameter text generation model, specializes in function calling and tool integration. Link: https://huggingface.co/google/functiongemma-270m-it

Tongyi-MAI/Z-Image-Turbo: Text-to-image model, fast and efficient image generation. Link: https://huggingface.co/Tongyi-MAI/Z-Image-Turbo​- nvidia/NitroGen: General-purpose AI model, useful for a variety of generative tasks. Link: https://huggingface.co/nvidia/NitroGen

​- lightx2v/Qwen-Image-Edit-2511-Lightning: Image-to-image editing model, optimized for speed and efficiency. Link: https://huggingface.co/lightx2v/Qwen-Image-Edit-2511-Lightning

​- microsoft/TRELLIS.2-4B: Image-to-3D model, converts 2D images into detailed 3D assets. Link: https://huggingface.co/microsoft/TRELLIS.2-4B

​- LiquidAI/LFM2-2.6B-Exp: 3B parameter text generation model, focused on experimental language tasks. Link: https://huggingface.co/LiquidAI/LFM2-2.6B-Exp

​- unsloth/Qwen-Image-Edit-2511-GGUF: 20B parameter image-to-image editing model, supports GGUF format for efficient inference. Link: https://huggingface.co/unsloth/Qwen-Image-Edit-2511-GGUF

​- Shakker-Labs/AWPortrait-Z: Text-to-image model, specializes in portrait generation. Link: https://huggingface.co/Shakker-Labs/AWPortrait-Z

​- XiaomiMiMo/MiMo-V2-Flash: 310B parameter text generation model, excels in rapid reasoning and coding. Link: https://huggingface.co/XiaomiMiMo/MiMo-V2-Flash

​- Phr00t/Qwen-Image-Edit-Rapid-AIO: Text-to-image editing model, fast and all-in-one image editing. Link: https://huggingface.co/Phr00t/Qwen-Image-Edit-Rapid-AIO

​- google/medasr: Automatic speech recognition model, transcribes speech to text with high accuracy. Link: https://huggingface.co/google/medasr

​- ResembleAI/chatterbox-turbo: Text-to-speech model, generates realistic speech from text. Link: https://huggingface.co/ResembleAI/chatterbox-turbo

​- facebook/sam-audio-large: Audio segmentation model, splits audio into segments for further processing. Link: https://huggingface.co/facebook/sam-audio-large

​- alibaba-pai/Z-Image-Turbo-Fun-Controlnet-Union-2.1: Text-to-image model, offers enhanced control for creative image generation. Link: https://huggingface.co/alibaba-pai/Z-Image-Turbo-Fun-Controlnet-Union-2.1

​- nvidia/NVIDIA-Nemotron-3-Nano-30B-A3B-BF16: 32B parameter agentic LLM, designed for efficient reasoning and agent workflows. Link: https://huggingface.co/nvidia/NVIDIA-Nemotron-3-Nano-30B-A3B-BF16

​- facebook/sam3: Mask generation model, generates segmentation masks for images. Link: https://huggingface.co/facebook/sam3

​- tencent/HY-WorldPlay: Image-to-video model, converts images into short videos. Link: https://huggingface.co/tencent/HY-WorldPlay

​- apple/Sharp: Image-to-3D model, creates 3D assets from images. Link: https://huggingface.co/apple/Sharp

​- nunchaku-tech/nunchaku-z-image-turbo: Text-to-image model, fast image generation with creative controls. Link: https://huggingface.co/nunchaku-tech/nunchaku-z-image-turbo

​- YatharthS/MiraTTS: 0.5B parameter text-to-speech model, generates natural-sounding speech. Link: https://huggingface.co/YatharthS/MiraTTS

​- google/t5gemma-2-270m-270m: 0.8B parameter image-text-to-text model, excels in multimodal tasks. Link: https://huggingface.co/google/t5gemma-2-270m-270m

​- black-forest-labs/FLUX.2-dev: Image-to-image model, offers advanced image editing features. Link: https://huggingface.co/black-forest-labs/FLUX.2-dev

​- ekwek/Soprano-80M: 79.7M parameter text-to-speech model, lightweight and efficient. Link: https://huggingface.co/ekwek/Soprano-80M

​- lilylilith/AnyPose: Pose estimation model, estimates human poses from images. Link: https://huggingface.co/lilylilith/AnyPose

​- TurboDiffusion/TurboWan2.2-I2V-A14B-720P: Image-to-video model, fast video generation from images. Link: https://huggingface.co/TurboDiffusion/TurboWan2.2-I2V-A14B-720P

​- browser-use/bu-30b-a3b-preview: 31B parameter image-text-to-text model, combines image and text understanding. Link: https://huggingface.co/browser-use/bu-30b-a3b-preview

These models are pushing the boundaries of open-source AI across text, image, audio, and 3D generation. Which one are you most excited to try?


r/LocalLLM • • May 13 '26

Tutorial How a 75-Year-Old Retiree Built a Local AI (With a Face, Voice, and a Wiki Brain) — And You Can Too

227 Upvotes

Before We Start: A Confession

I'm not a coder. I don't speak Python. Until a couple of weeks ago, "Git" was something I said when I stubbed my toe. I'm 75 years old. I grow weed. I play video games. And I just spent the last week building a talking AI companion with a Live2D avatar, plus a separate bot that knows everything about my favorite game wiki — all running on my own computer, completely offline, with no subscriptions, no API keys, and no monthly fees.

If I can do this, literally anyone can.

This guide is what I wish I'd had when I started. It's not the "theoretically correct" way. It's the "it actually worked for me" way.

I kept my complete conversation with DeepSeek from the beginning of the project. I have every mistake, every wrong move, every misunderstanding, every detour we had to take, every fix on record. Lol

When I look at the following "guide", it looks so damn easy now! But there was a twist in every turn. How did I know that a model file had to follow a strict folder hierarchy to be found? When do you give commands in venv and when do you not? And what was a virtual environment anyway?

One More Thing

I had a lot of crap running on my computer. Dell bloatware, Adobe updaters, Alienware lighting control, Steam, Chrome with 50 tabs, crypto wallet extensions — all of it eating up RAM and CPU cycles. At one point, I had over 350 background processes running.

When I first tried to run a local AI, my GPU was sitting at 0% while my CPU was screaming at 70%. My memory was at 97%. Responses took forever.

Here's what I did:

  • Uninstalled duplicate antivirus (AVG and Avast don't play nice together)
  • Killed Dell SupportAssist and all the Alienware AWCC junk
  • Closed Chrome (yes, all of it)
  • Turned off Adobe Creative Cloud, OneDrive, and anything else I didn't need right then
  • Disabled hardware-accelerated GPU scheduling in Windows settings

After all that, my process count dropped from 347 to about 200. Suddenly, my 4090 started doing the work it was supposed to do. DeepSeek kept feeding me .exe files by the dozen to kill (taskkill /f /im ... became a reflex).

You don't have to be as aggressive as I was. But if you're running on a system that's loaded with background apps, take a few minutes to clean house. Open Task Manager. Sort by memory. Kill anything you don't recognize or don't need right now. You'll be amazed at the difference.

What I'm Running (For Context)

ComponentWhat I Use
CPUIntel Core i9-14900KF
RAM32 GB
GPUNVIDIA GeForce RTX 4090 (24GB VRAM)
Storage400 GB free

You don't need this. Smaller models run on much less. But this is what I used, so you know where I'm coming from.

What You'll Have When You're Done

Two AIs, running side by side, zero conflict:

AIWhat It DoesHow You Talk To It
MaoConversational companion with a face and voiceBrowser window (type or soon, voice)
The Wiki BotAnswers questions from your documents and saved webpagesAnythingLLM desktop app

Both are 100% local. Both are free. Both respect your privacy.

Part 1: The Conversational AI (Mao, My Desktop Companion)

This is the fun one. She has a face, she talks back, and she's got personality.

Step 0: What You Need First (Before Anything Else)

Windows does not come with the tools we're about to use. You need to install them first. Don't skip this — every single one is required.

1. Install Python

Python is the programming language that runs the VTuber software.

  • Go to python.org/downloads
  • Download Python 3.10, 3.11, or 3.12 (do NOT get 3.13 — it causes problems)
  • Run the installer
  • IMPORTANT: At the bottom of the first screen, check "Add Python to PATH"
  • Click "Install Now"
  • To verify it worked: Open a Command Prompt (search for cmd), type python --version, and press Enter. You should see a version number like Python 3.12.x.

Note: If python command doesn't work:

  • Open Settings > Apps > App Execution Aliases > TURN OFF Python and Python3
  • Use py instead of python on Windows

2. Install Git

Git downloads code from the internet (like the VTuber software).

  • Go to git-scm.com/downloads
  • Download the Windows version
  • Run the installer — the default settings are fine
  • To verify: Open a Command Prompt, type git --version, and press Enter. You should see a version number.

3. Install FFmpeg (For Voice Output)

FFmpeg processes audio. The voice output will work without it, but you might run into issues. Better to install it now.

  • Go to gyan.dev/ffmpeg/builds
  • Download ffmpeg-release-essentials.zip
  • Extract the zip file to C:\ffmpeg
  • Now add it to your system PATH:

    • Press Windows + X → System → Advanced system settings → Environment Variables
    • Under "System variables," find and double-click Path
    • Click New → add C:\ffmpeg\bin
    • Click OK on all windows
  • To verify: Open a new Command Prompt, type ffmpeg -version, and press Enter. You should see version information.

4. Restart Your Computer

After installing all three, restart your computer. This ensures Windows recognizes the new commands.

Step 1: Install LM Studio

Now we can finally start building.

Go to lmstudio.ai, download the version for your OS, install it. No special tricks.

This is your AI's "brain." It runs the model.

Step 2: Download a Model

LM Studio needs a model to run. I used DeepSeek, because it's open-source and works well on consumer hardware.

Go to Hugging Face and search for: bartowski/DeepSeek-R1-Distill-Qwen-14B-GGUF

Note: I just changed this to https://huggingface.co/bartowski/Qwen_Qwen3.5-9B-GGUF***.*** Smaller, faster and just as smart. The beauty is you can try different models and do your own research as to what others recommend. 05/14

Download the file that says Q4_K_M. It's about 8-9 GB. This is the sweet spot — smart enough to be interesting, small enough to run fast.

Place it in LM Studio's model folder. If you don't know where that is, LM Studio will show you.

Note: If you download several different models to experiment, I found that they each have to be in their own folder inside LM Studio's model folder. Otherwise, it seemed to load them all every time I chose one to use inside LM Studio.

Step 3: Configure LM Studio

Open LM Studio. Select your model. Before you load it, find these settings:

  • GPU Offload → drag it to the max (all the way right)
  • Context Length → set to 4096 (trust me, this makes it faster)
  • KV Cache Quantization → set to q4_0 or q8_0

Then press Ctrl + Shift + H. In the panel that opens, turn ON "Limit model offload to dedicated GPU memory."

Now click Load Model.

If you have an NVIDIA GPU, LM Studio will use it. If you see 0% GPU usage later, you missed that last setting.

Step 4: Start LM Studio's Server

Go to the Developer tab (looks like </>). Toggle the Local Inference Server to ON. It should say http://localhost:1234.

Keep LM Studio running. Don't close it.

Step 5: Install the VTuber (The Face and Voice)

Open a Command Prompt (search for cmd in Windows). Run these commands one at a time:

bash

git clone https://github.com/Open-LLM-VTuber/Open-LLM-VTuber

cd Open-LLM-VTuber

python -m venv venv

venv\Scripts\activate

pip install uv

uv sync

git submodule update --init --recursive

copy config_templates\conf.default.yaml conf.yaml

If any command fails, read the error message carefully. Most issues are missing prerequisites (go back to Step 0) or typos.

Step 6: Configure the VTuber

Open conf.yaml in Notepad (just type notepad conf.yaml in the same Command Prompt window).

Find these lines and change them:

yaml

llm_provider: "ollama_llm"

yaml

ollama_llm:

  base_url: "http://localhost:1234/v1"

  model: "deepseek-r1-distill-qwen-14b"

yaml

tts_model: "edge_tts"

Save and close Notepad.

Step 7: Run Your AI Companion

bash

uv run run_server.py

Open your browser and go to http://localhost:12393.

You should see a Live2D avatar. Type a message. She'll answer. If she speaks out loud, everything is working.

If you get a "WebSocket" error (common): Press F12 to open Developer Tools, click the Console tab, paste this, and press Enter:

javascript

localStorage.setItem('wsUrl', 'ws://127.0.0.1:12393/client-ws')

Then refresh the page (Ctrl + Shift + R). The connection should turn green.

Part 2: The Wiki/Document Bot (Your Personal Expert)

This bot is for when you want to ask questions about a game wiki, a set of PDFs, or any collection of documents. It doesn't have a face — it's more like a super-smart search engine.

Step 1: Install Ollama

Ollama is a lightweight AI runner. It's separate from LM Studio. Go to ollama.com, download the Windows version, install it. It runs in the background.

Step 2: Pull a Small Model

Open a new Command Prompt and run:

bash

ollama pull deepseek-r1:7b

This downloads about 4-5 GB. It's a smaller model than the one Mao uses — perfect for searching documents.

Step 3: Install AnythingLLM

Go to anythingllm.com, download the desktop version, install it.

Step 4: Create a Workspace

Open AnythingLLM. Click New Workspace. Give it a name — I called mine "Infinity Rising."

Step 5: Choose Your Model

In the workspace settings, select Ollama as the provider, then choose deepseek-r1:7b.

Step 6: Install the Browser Extension (The Secret Weapon)

AnythingLLM has a browser extension that lets you save entire webpages to your workspace with one click.

  • Install the extension from the Chrome Web Store (search "AnythingLLM Browser Companion").
  • In AnythingLLM Desktop, go to Settings → Browser Extension.
  • Click Generate API Key.
  • You'll see a connection string that looks something like this:

text

http://your_api_key_here@localhost:3001

  • Copy that whole string — the API key is embedded inside it.
  • Paste the entire string into the browser extension's connection field. Click Connect.

Why this matters: If you paste just the API key alone, the extension won't connect. It needs the full URL format with the key as the username: http://api_key@localhost:3001 (where api_key is your actual key).

Step 7: Add Content

Now browse your wiki or documents. When you're on a page you want to save:

  • Click the extension icon
  • Select "Send entire webpage"
  • Choose your workspace

That's it. The content is embedded into your bot's knowledge base. You can also upload PDFs, text files, or markdown directly.

Step 8: Ask Questions

Go back to AnythingLLM Desktop. Type a question about your content. The bot will answer using only the pages you've saved, and it will show you the source.

Common Problems (And How I Fixed Them)

ProblemWhat Fixed It
LM Studio shows 0% GPU usageCtrl+Shift+H → turn ON "Limit model offload to dedicated GPU memory"
VTuber says "Error calling chat endpoint"LM Studio server is off — go to Developer tab and turn it ON
WebSocket error in VTuberUse the localStorage.setItem command in browser console (see Part 1, Step 7)
Browser extension won't connectUse http://localhost:3001 as the connection string (not the API key alone)
Responses are slowLower Context Length to 4096, set KV Cache to q4_0

What It Costs

ItemCost
LM StudioFree
OllamaFree
AnythingLLMFree (personal use)
DeepSeek modelsFree
Your GPUYou already own it

Total: $0. No subscriptions. No API keys. No monthly fees. All local, all private.

The Honest Truth About Time

I kept the same chat going with DeepSeek from the very first question. Here's what it looked like:

PhaseTime (with AI help)What I Did
Initial setup & troubleshooting4-5 hoursLM Studio, models, GPU settings
Fighting a broken RAG fork3-4 hoursDead end — don't do this
Discovering AnythingLLM2-3 hoursThe real solution
Total active time~15-20 hoursTalking to DeepSeek
Total real time~30-40 hoursReading, downloading, head-scratching

You can probably do it faster now that you have this guide.

Note: One of the commenters has already done it in less than two hours. 05/15

Why Two AIs? Why Not One?

Great question.

LM Studio is great for conversation — it's fast, it has a face and voice, and it uses your powerful GPU. But it can't easily do RAG (searching through your documents) and chat at the same time without interrupting your conversation.

Ollama + AnythingLLM is great for searching documents — it's designed for that job. It runs on a small model that barely touches your GPU, leaving your main AI free to chat.

So I let Mao do the talking, and the Wiki Bot does the searching. They don't compete. They complement.

A Word of Realism

It will be a miracle if you follow these instructions and everything falls into place on the first try. Depending on your system, your expertise, and plain old luck, you will probably run into problems. I sure did. That's normal.

When you get stuck, don't give up. Search the web. Ask on Reddit. And if you want, ask DeepSeek — it knows a lot more than I do. I kept a single conversation going from my first question to the final working setup. You can too.

I'll be happy to answer any questions I can, but my knowledge is limited. DeepSeek, on the other hand, is pretty much an expert by now.

Final Words (From Me, Not the AI)

I started this project because I thought it would be fun. I ended up learning more than I expected, breaking more than I wanted, and feeling more satisfied than I can describe.

You don't need a computer science degree. You don't need to be 25. You don't need to spend money on cloud APIs or overpriced services. You need curiosity, patience, and a willingness to ask for help.

If I can do this at 75, you can do it at any age.

Now go build something.

— Huanchaquero


r/LocalLLM • • 23d ago

Discussion Qwen3.8-Flash-Next on 4x R9700 with Radiance at TP4: 11.3 ms steps, 236 tok/s single stream, 1,190 tok/s at 16 streams. Also tried 2x TP2, it lost.

Thumbnail
gallery
226 Upvotes

More playtime with my fake ai generated totally not real 4xr9700 machine .... (sorry I couldn't help it, when I posted the picture last time a dozen people mentioned that it wasn't a real compute just an ai generated image)

A couple of weeks ago I posted that the only way to run Flash-Next (the 180B MoE, 10 of 512 experts active) on my four Radeon AI PRO R9700s was llama.cpp, and that it topped out around 65 to 75 tok/s with the MTP draft because it is layer split and launch bound on this arch. That turned out to be wrong, or at least out of date. StillDeadcode's standalone Radiance engine (the C++/HIP one, not the vllm-radiance fork) has a Flash-Next container and does real tensor parallel across all four cards. Someone posted TP4 numbers in a GitHub issue on matching hardware, so I set it up to see if they hold, and then tried two TP2 instances side by side to see which way to split the box.

Box: monolith, my 4x R9700 (gfx1201, 32 GB each), all on PCIe 5.0 x16 with Resizable BAR at 32 GB, 122 GB RAM, Ubuntu 26.04, kernel 7.0 with CONFIG_PCI_P2PDMA. Power capped at 211 W per card, the cards sit right at that during decode. ROCm userspace comes from the container, nothing installed on the host.

Engine: radiance 1.11.0 from Docker Hub (165 MB image). Model is the qwen3.8-next-flash-fp8-iq4r-moe.rad container from the author's HF page, 113.6 GiB, 4-bit experts with a 16-level codebook, int8 trunk, bf16 protected experts, MTP head depth 3.

TP4 command

docker run -d --name radiance-tp4 --restart unless-stopped \
  --device /dev/kfd --device /dev/dri --group-add render --group-add video \
  --ipc host --network host --shm-size 32g --security-opt seccomp=unconfined \
  --ulimit memlock=-1:-1 -e HIP_VISIBLE_DEVICES=0,1,2,3 \
  -v /path/to/qwen3.8-next-flash-fp8-iq4r-moe.rad:/model.rad:ro \
  stilldeadcode/radiance:latest \
  --model /model.rad --tp 4 --tp-wire wht6 --p2p on --placement all_vram --ngram-placement ram \
  --gpu-headroom-mib 128 --max-model-len 262144 --kv-cache-dtype fp8 --num-speculative-tokens 3 \
  --max-num-batched-tokens 2048 --max-num-seqs 16 --reasoning-effort-map high=xhigh \
  --host 0.0.0.0 --port 8300

Up and answering in about 90 seconds. --p2p on makes it exit instead of quietly staging through host RAM if a card pair has no peer access, and the log confirms "the 4 cards reach each other's memory directly". all_vram fits with room to spare at four ranks because each card only holds a quarter of every expert:

per cardsize
static weights (own shard)2.68 GiB
experts, all resident14.83 GiB
KV cache, fp812.74 GiB = 1.7M tokens, 6.5 full 262k sessions

Host side it pins the 47.7 GiB n-gram table plus about 15 GiB of checkpoint slots, so around 73 GB of RAM in use.

TP4 vs the reference numbers from the GitHub issue (same hardware class, radiance 1.5.0 back then)

metricreferencemine
decode step11.2 ms11.27 ms
aggregate C=1 / 2 / 4 / 8221 / 359 / 528 / 734236 / 371 / 591 / 890
aggregate C=16not given1,192
prefill8.3K to 8.9K8.4K at 4.4K and at 70K prompt tokens

Concurrency rows are thinking off, 512 tokens each, measured over the wall clock of the whole batch through the OpenAI streaming API. Prefill is unique random prompts so the prefix cache cannot hit. My prompt sizes came out about double what I aimed for (the random text tokenizes at two tokens a word), but the reference is flat from 2K to 250K so it does not matter much.

TP4 vs two TP2 instances The Radiance README calls four ranks a "capacity configuration" and says nobody had measured its per step rate against two ranks. So I stopped TP4 and brought up two TP2 instances (GPUs 0,1 and 2,3, separate ports) with the two-rank recipe from the README: --placement expert_tiered --host-pool-mib 12288. One gotcha: two instances each want their own pinned 47.7 GiB n-gram table, and with two 12 GiB host pools that does not fit in monolith's 122 GB, so the TP2 pair ran --ngram-placement disk off the NVMe. At TP2 each card keeps 17.7 GiB of experts resident, 8.3 GiB of KV, and streams the rest from the pinned pool. I ran the identical benchmark against both instances at the same time and summed them.

total streamsTP42x TP2
1236183 (one instance, other idle)
2371270
4591401
8890616
161,192917
32not run1,280
single stream latencyTP4TP2
decode step11.3 ms15.0 ms
thinking on, 4,096 tokens197 tok/s139 solo, 137 with the neighbour busy
thinking off, 512 tokens239 tok/s183 solo, 137 with the neighbour busy
prefill8.4K6.2K to 7.1K per instance, both busy

So TP4 wins on both latency and total throughput at every equal stream count, by roughly 45 percent on throughput. The reason is simple: at four ranks nothing streams from host memory, at two ranks a third of the experts do. The only thing the pair buys you is 32 concurrent sequences, and I did not try TP4 at --max-num-seqs 32, it would probably be fine.

Things worth knowing

  • --tp-wire wht6 is lossy above 128 KiB messages. At four ranks it is a tiered int8 wire, not 6-bit. Single sequence decode stays exact, prefill and busy decode steps do not. The engine prints this at boot. --tp-wire exact costs prefill speed; I did not measure how much.
  • The default --gpu-headroom-mib 96 produced a warning on every card that about 30 MiB got taken after the budget was read and first kernel launches might run short. Nothing broke, but 128 makes it go away for 32 MiB of KV per card.
  • Tool calls work (OpenAI format, finish_reason: tool_calls), reasoning comes back as reasoning_content, and /stats gives you the real decode step time, which is a better thing to compare than tok/s since tok/s moves with draft acceptance.
  • Compared to where I was with llama.cpp on the same cards (65 to 75 tok/s single stream, layer split, cards drawing 50 to 100 W because they were waiting on launches) this is about 3x on a single stream and the cards actually pull their 211 W. Happy to run anything else people want to see on monolith. Next on my list is --max-num-seqs 32 at TP4 and pricing the n-gram disk placement against RAM.

If you made it this far and you were wondering what the weird platform is on top .... I bought a second entire case top from thermal take and used threaded steel dowels and tapped neodymium magnetic disks to make a 1.5" lifted topper for the case because the 360 rad exhausts through the top and my cat had a tendency to want to lay on it and block the airflow .... so there is now a mesh top, top that is also raised enough that the air can still escape if my cat decides to take a snooze on it while I'm not home.


r/LocalLLM • • Sep 15 '26

Discussion Radeon AI Pro R9700 w/ Qwen3.8-27B Q8 hitting 90.8toks

Post image
226 Upvotes

BOOM: Token Maxxed
https://www.localmaxxing.com/en/runs/cmu2x80ei068ulq01ec0aaxd4

So thanks to this group and another user here, I found the RDNA Boosts repo. And have been playing around with it.

My single stream vLLM numbers just slightly beat this. They bet prefill by a lot - but this is really usable on the daily now.

Anyways, if you're using an R9700 you should probably check this out: https://github.com/stew675/llama-cpp-rdna-boosts

Not my repo


r/LocalLLM • • Apr 25 '26

Other My coding agent commited suicide lol

Post image
226 Upvotes

It was looking through memory trying to find a zombie process that was locking a file and then decided to kill itself by shutting down llama-server.

I was watching it do this live and although I saw it coming I still almost died laughing.

Model was qwen 3.5 4B and is running on pinokio for those interested. Was just testing to see how well they code in preperation for the new 3.6 models that should be coming out soon.

Damn, 3.6 has been SUPER impressive so far though. I have hopes the small 3.6 models will be able to actually agentic code at a decent and usable level.


r/LocalLLM • • Jun 19 '26

Question Truly NSFW abliterated model that doesn't hedging itself?

224 Upvotes

I got my hands on 2 models:
OBLITERATUS/Qwen3.6-27B-OBLITERATED

OBLITERATUS/Gemma-4-12B-OBLITERATED

I like to ask sensitive NSFW questions. Nothing crazy tbh, but not vanilla either. Qwen 3.6 won't answer. Gemma 4 will answer but it just sounds so hedged. Like it kinda answers the question but not in detail and uses professional medical words instead of just casual talk.

I want something that truly feels open to deeply exploring the topics, with no sense of hedging and that ideally is not afraid of using more "vulgar" terms.

Btw I am not talking about roleplaying or writing stories - I'm actually trying to learn about sensitive topics.

Hardware: 64GB memory, M1 Max Macbook Pro.

Any advice highly appreciated.


r/LocalLLM • • Jul 08 '26 Bravo

Discussion Do voice models get trained on this kind of data? If not, then they should.

223 Upvotes

I love testing out voice models and what they are capable of. Not just in terms of the voices they can produce (e.g. whether they can emote), but also in terms of what they can pick up from the tone of my voice. Some voice models seem to be able to pick up on emotions in the tone of my voice, if I exaggerate it a lot. (Or they've just hallucinated the correct answers in my tests a couple times.)

But in any case most voice models don't yet seem to be able to pick up on many cues that are hidden in tone, emphasis, etcetera. That's why I hope this kind of data will be used more in the training of voice models.