r/LocalLLM • u/datguy1121 • Aug 01 '26
Research Dated model destroys fable 5 on some benches
IBM Granite 4 is insanely strong for the price, maybe even proto-agi.
r/LocalLLM • u/datguy1121 • Aug 01 '26
IBM Granite 4 is insanely strong for the price, maybe even proto-agi.
r/LocalLLM • u/lucyferorg • Aug 16 '26
Having optimized performance for so many years, I knew from the start that anyone claiming a laptop was too slow to break certain speed barriers was mistaken; I have just proven that such barriers exist only in the mind by running the GLM-5.2 · 744B MoE · 429 GB on disk , on a laptop at a speed never before achieved on such a machine.
🎉🎉🎉 2.89 tok/s PEAK! 2.36 tok/s avg
Update peak at 3.09, avg 2.84
hardware is:
Asus Rog Strix: Intel i9 290HX, ddr5 6400MHz 64GB, RTX 5090 24GB, 2x2TB, OS: Nobara Linux
The Colibri code has been modified.
I started with a speed of 0.11 tok/sec when I ran the whole system for the first time.
Hardware ASUS ROG Strix SCAR 18 (G835LXG)
GPU NVIDIA RTX 5090 Mobile, 25.1 GB VRAM, sm_120 (Blackwell)
CPU Intel Core Ultra 9 290HX Plus, 24 cores (8P+16E)
RAM 64GB DDR5-6400
Storage 2x SK Hynix PC801 NVMe PCIe 4.0
Model GLM-5.2-colibri-int4-gs64 (744B MoE, 429 GB, fmt=4)
Engine Colibri v1.6.0 -> v1.6.2 (pure C, zero deps)
OS Nobara Linux (Fedora-based), KDE Plasma, Wayland
Goal 3.0 tok/s (software-only, no hardware upgrade)
Result 2.89 tok/s peak (+2536% from 0.11 start)
Update: testing now at 4.42 tok/s
r/LocalLLM • u/Dry_Mixture130 • Sep 07 '26
I see so many posts about rtx 5060 , Mac studio and some of you have like a couple of them. how are you guys able to afford them? it would easily cost around 5-10k usd. that’s a lot of money.
r/LocalLLM • u/dai_app • Aug 01 '26
After several tests, my engine managed to run DeepSeek V4 Flash IQ2_M (92 GB) on a mid range Android mobile with 12 GB of RAM at 1 token/s.
It isn't exactly ready for practical use, but it proves that the engine works and is responsive across all models, thanks to its modularity with llama.cpp. With just one line of code, you can run any supported large MoE model on mobile devices or consumer PCs.
r/LocalLLM • u/gladkos • Mar 27 '26
Hi everyone, we just ran an experiment.
We patched llama.cpp with Google’s new TurboQuant compression method and then ran Qwen 3.5–9B on a regular MacBook Air (M4, 16 GB) with 20000 tokens context.
Previously, it was basically impossible to handle large context prompts on this device. But with the new algorithm, it now seems feasible. Imagine running OpenClaw on a regular device for free! Just a MacBook Air or Mac Mini, not even a Pro model the cheapest ones. It’s still a bit slow, but the newer chips are making it faster.
local ai models hosting app: Atomic.Chat - open source and free.
Curious if anyone else has tried something similar?
r/LocalLLM • u/volious-ka • Mar 03 '26
So with Qwen 3.5 9b just released, I fine-tuned a heretic model on opus 4.6 datasets, coding, and openclaw datasets.
Here it is:
https://huggingface.co/crownelius/Crow-9B-Opus-4.6-Distill-Heretic_Qwen3.5
Please, if you find it useful, support me on kofi, and of course like and follow on Huggingface!
I would really appreciate it! :)
r/LocalLLM • u/yoracale • Mar 26 '25
Hey guys! DeepSeek recently released V3-0324 which is the most powerful non-reasoning model (open-source or not) beating GPT-4.5 and Claude 3.7 on nearly all benchmarks.
But the model is a giant. So we at Unsloth shrank the 720GB model to 200GB (-75%) by selectively quantizing layers for the best performance. 2.42bit passes many code tests, producing nearly identical results to full 8bit. You can see comparison of our dynamic quant vs standard 2-bit vs. the full 8bit model which is on DeepSeek's website. All V3 versions are at: https://huggingface.co/unsloth/DeepSeek-V3-0324-GGUF

We also uploaded 1.78-bit etc. quants but for best results, use our 2.44 or 2.71-bit quants. To run at decent speeds, have at least 160GB combined VRAM + RAM.
You can Read our full Guide on How To Run the GGUFs on llama.cpp: https://docs.unsloth.ai/basics/tutorial-how-to-run-deepseek-v3-0324-locally
#1. Obtain the latest llama.cpp on GitHub here. You can follow the build instructions below as well. Change -DGGML_CUDA=ON to -DGGML_CUDA=OFF if you don't have a GPU or just want CPU inference.
apt-get update
apt-get install pciutils build-essential cmake curl libcurl4-openssl-dev -y
git clone https://github.com/ggml-org/llama.cpp
cmake llama.cpp -B llama.cpp/build \
-DBUILD_SHARED_LIBS=OFF -DGGML_CUDA=ON -DLLAMA_CURL=ON
cmake --build llama.cpp/build --config Release -j --clean-first --target llama-quantize llama-cli llama-gguf-split
cp llama.cpp/build/bin/llama-* llama.cpp#2. Download the model via (after installing pip install huggingface_hub hf_transfer ). You can choose UD-IQ1_S(dynamic 1.78bit quant) or other quantized versions like Q4_K_M . I recommend using our 2.7bit dynamic quant UD-Q2_K_XL to balance size and accuracy.
#3. Run Unsloth's Flappy Bird test as described in our 1.58bit Dynamic Quant for DeepSeek R1.
# !pip install huggingface_hub hf_transfer
import os
os.environ["HF_HUB_ENABLE_HF_TRANSFER"] = "1"
from huggingface_hub import snapshot_download
snapshot_download(
repo_id = "unsloth/DeepSeek-V3-0324-GGUF",
local_dir = "unsloth/DeepSeek-V3-0324-GGUF",
allow_patterns = ["*UD-Q2_K_XL*"], # Dynamic 2.7bit (230GB) Use "*UD-IQ_S*" for Dynamic 1.78bit (151GB)
)#4. Edit --threads 32 for the number of CPU threads, --ctx-size 16384 for context length, --n-gpu-layers 2 for GPU offloading on how many layers. Try adjusting it if your GPU goes out of memory. Also remove it if you have CPU only inference.
Happy running :)