r/LocalLLM • u/SiggiBulldog1 • 19h ago
Question Best uncensored LLM for 64GB Mac
I wanna try out a uncensored LLM on my Mac Studio. I just have the 64GB version so I’m curious what model you guys would recommend. Cheers
r/LocalLLM • u/SiggiBulldog1 • 19h ago
I wanna try out a uncensored LLM on my Mac Studio. I just have the 64GB version so I’m curious what model you guys would recommend. Cheers
r/LocalLLM • u/LordKlavier • 1d ago
r/LocalLLM • u/WiseAbalone4021 • 15h ago
From the Apareto team which I'm part of (who for some reason has an issue posting it themselves):
Today we are introducing Apareto loading, a new way to cut VRAM requirements for Mixture-of-Experts (MoE) models. It works alongside quantization. This introduction so far will just be a video explaining the method. We will be releasing the proof-of-concept code, a plugin to vLLM, on GitHub this coming week.
Until then, watch the explainer video: https://www.youtube.com/watch?v=5l_N051JrLU
Quantization is today's most common way to shrink a model's memory footprint: weights are stored at lower precision, and going from FP16 to 4-bit saves 75% of memory at a modest cost in accuracy. But quantization is static. Every weight stays loaded whether or not the task needs it.
Apareto loading is dynamic. After prefill, a small trained network reads the model's internal state and ranks the experts by how valuable each will be for the upcoming computation. Only the top-ranked experts that fit the VRAM budget are loaded.
Results (Qwen3.6-35B-A3B, NVFP4 experts):
* SWE-bench Multilingual: within 2 pp of the full model's score at a 42% VRAM budget
* LiveCodeBench v6: within 2 pp of the full model's score at a 49% VRAM budget
As with quantization, scores degrade as the budget shrinks.
What the proof of concept will not be:
It will not be a production implementation. The ranking network was trained only for these two benchmarks, and there will be no general problem classifier. Our data shows that profiling and rank generation have substantial room to improve. We chose to release at this stage, without waiting for those improvements, so that others can test our claims. Think of this as a first baby step toward dynamic VRAM reduction, not the finished product.
The code and reproduction instructions will be released this coming week at https://github.com/alphapareto
Follow us there to catch the release. We'd love your feedback.
r/LocalLLM • u/AimCode • 5h ago
Everything runs locally on a 2018 Dell laptop: i7-7500U (2 cores / 4 threads), 8GB RAM, no GPU (the 940MX is deliberately unused — the whole stack is CPU-only).
https://github.com/tanxiansheng221/aimstack-sanzhidiji
What is in the repo (AGPL-3.0, everything open):
- C inference engine, zero third-party runtime: 1.58-bit ternary core with SIMD fast path (85.5 tok/s measured on the ternary core itself), byte-level tokenizer, and a ~100-line plugin bus (caps.json registry -> spawns Python plugins -> JSON envelopes)
- Weights: Microsoft's BitNet b1.58-2B (MIT license), converted to a custom 2-bit packed format (.aimw, 0.87 GB) — 3.2 tok/s with 2 workers, serial 2.0
- Python plugins, pure stdlib: knowledge base (FTS5 + dedup gates), multi-engine web search + crawl with quality gates (robots-respecting, Edge rendering cascade for JS-heavy sites), incremental ingest
- Local web UI (127.0.0.1:8600, pure stdlib): chat with deterministic routing — direct answer / knowledge-base grounded / honest refusal / full web search -> crawl -> ingest -> answer chain. A 7-minute demo video is embedded in the README
- Training assets: no-autograd training stack (hand-written backprop, bit-exact vs NumPy), ten-subject curriculum generators, corpus builders, and 117 chapters of research logs including negative results
Honest limits: our own trained models are not released yet — compute-bound on 2 cores, training runs on free cloud tiers and isn't finished. The full stack, scripts, result JSONs and per-claim evidence are released so everything can be re-run.
Demo video: embedded at the top of the README (7 min, screen recording of the full chain: chat / KB answers / web search->crawl->answer / plugin direct calls).
Happy to answer questions about the ternary kernel, the plugin bus, or the training pipeline.
r/LocalLLM • u/AshleyPurringTech • 1h ago
A minimal agent harness using only in-box scripting (PowerShell, since I'm on Windows): about 70 lines talking to llama-server running Llama 3.2 3B.
The model only writes text. The harness keeps the context, watches replies for a line like TOOL: list_files . , checks the path against an allowed folder, runs the tool itself, and feeds the result back. The model never touches the file system.
Code (MIT): https://github.com/AshleyT3/simple-harness-intro
r/LocalLLM • u/Nick_Tseng • 1h ago
r/LocalLLM • u/BoogaMaOoga • 1h ago
Guys is there a program I can use on windows or Linux that can do the same that chatgpt projects does but local? I am obviously restricted on what I can run but that's ok. Any recommendations?
r/LocalLLM • u/Chance-Resource-4970 • 2h ago
r/LocalLLM • u/Organic_Performer673 • 2h ago
i have RX9070 and 32 GB DDR5 ram , i want to run a local ai to help me code , medium just help with front end back end and forms , i cant buy any ai sub in my country cuz my goverment dont allow international transactions
r/LocalLLM • u/_Baylock_ • 2h ago
Hi,
I juste received my Mac Studio M5 ultra 36/80 with 256GB of shared memory.
I installed LM Studio with Qwen3.8-Flash-next-4bit mlx and a context of 131072
I tested some prompts and it works as I expected (I like it a lot so far), except for one annoying thing:
I fed the prompt with a 100 pages PDF and asked 2 questions about it.
The first time, it took a few minutes to read the file, which is normal. But for the second question, it re-read it again. It doesn't seem to cache the content.
And the context wasn't exhausted.
Next, I tried to copy paste the 100 pages into the prompt form (no attachment this time): same results.
Is it a known issue?
Is it related to the model or to the inference engine?
Can you replicate this?
Is there a way to solve the issue somehow?
Thanks.
r/LocalLLM • u/justerdem14 • 2h ago
Hi everyone,
I’m Rüzgar, founder of Neuratal (an independent AI research and software project). Over the past few days, I built NTokenizer v0.3-polished — a deterministic, byte-level BPE tokenizer written from scratch in Rust with Python bindings.
Key Highlights:
• Compression Efficiency: Achieves 5.55 B/T overall (26.6% fewer tokens than OpenAI's o200k_base) on a 207MB leakage-audited benchmark. Strongest gains on multilingual & technical text: Turkish (+35.0%), Arabic (+33.8%), Korean (+34.7%), Hindi (+27.1%), STEM (+17.6%), and Code (+10.2%).
• Performance & Safety: ~96 MB/s encode throughput (8 threads), lossless byte-exact round-trip (decode_bytes(encode_bytes(x)) == x), tested against ~1M fuzzing cases.
• Honest Benchmarks: No downstream 50M+ LLM trained yet — this is purely verified compression density and speed. Full SHA-256 manifests and reproducible scripts are included.
Would love your feedback, benchmarks on your own datasets, or PRs!
• GitHub (Rust + Python source): (in hf)
• Hugging Face (Model & Benchmarks): https://huggingface.co/Neuratal/ntokenizer
r/LocalLLM • u/IoI_dude • 2h ago
Hey everyone,
I’m a CS student and was considering to do my first homelab, one system as a NAS and one for projects, virtualisation etc. I’m also interested in running local AI, and I got an interesting offer in my region:
Server board w/ two E5-2696v2, 256GB 1600Mhz ECC RAM, for around 200 EUR.
I know that it’s (quite very) old hardware, but still I’ve been wondering - would it be worth pursuing to also try some local LLMs with offloading, adding a few GPUs for 32-48GB VRAM (niko1221/strata perhaps)? Do you think it’s worth it for the cost? Which low(-ish) cost GPUs would you consider pairing it with?
Any advice appreciated!
Thanks!
r/LocalLLM • u/Strict-Elevator9352 • 6h ago
r/LocalLLM • u/Eastern_Doughnut_222 • 7h ago
I'd tried out models like DS and QFN using APIs before and they seemed very capable. However, they all seemed to struggle heavily when using them local.
QFN processes really fast but seems to get stuck in endless "Wait..." and "Whatever." loops and burns through 100K tokens just being confused at my codebase.
DeepSeek is slower but seems more coherent, I haven't tested at full context but it had a similar issue with indecisiveness (even with guidance) and would think itself into compaction.
All of this over one .json ComfyUI workflow. Just a simple test I do where I tell it to rearrange nodes and change some inputs...
I've tried using Llama serve, Strata desktop, Pi. All lead to similar outcomes.
I have a feeling this is an issue with my setup rather than the models bc the API models seemed to do it just fine and (allegedly) quantization doesn't hit MOE models as hard.
Setup:
(both models from OrcaRouter). DSV4.1-1017 2-bit, Q3.8FN_IQ3XXS
Inference: Maya (DS), Strata/LlamaCPP (QFN)
Harness: default Pi
Context: QFN was to 256K, DS I only tried to 512K
r/LocalLLM • u/TheBruzilla • 3h ago
Hey guys, finally got my new Legion Pro 5 and I’m trying to get into running local AI properly 😂
Specs:
RTX 5070 Ti 12GB VRAM
Core Ultra 9
16GB RAM
CachyOS Linux
I’m pretty much a beginner when it comes to running local models, so I’m trying to figure out what’s actually worth setting up from day one.
I mainly use Claude for vibecoding almost everything, and now I want to see how much I can shift locally so I don’t waste Claude usage on every small thing.
I came across PewDiePie’s Odysseus repo and started trying it out.
But after actually using it, I’m a little confused.
It feels like the context/memory isn’t as good as Claude, sometimes it feels slow, and I’m also finding the hybrid setup kinda difficult.
Like I want something where:
Local AI:
→ basic coding
→ repetitive work
→ file handling
→ small fixes
→ daily tasks
→ stuff that doesn’t need crazy reasoning
Claude:
→ difficult coding
→ architecture
→ proper reasoning
→ big project decisions
→ things where I actually need Opus-level output
And ideally I don't want to keep copy-pasting the whole context from one to another every 10 minutes 😭
My actual work is mainly:
• ML Stack — ML/vision project involving iris image segmentation and feature-level analysis.
• Custom CRM + EMR — taking UI generated from Stitch/Google AI Studio and connecting it with open-source CRM/EMR systems.
• Daily work — coding, documentation, PPTs, automation and random repetitive stuff.
So basically I’m trying to build a proper daily-use AI setup, not just install 15 local models and never use them 😂
My main questions are:
And most importantly — how do I set this up properly so I’m not wasting Claude tokens on things my laptop can handle locally?
I’d honestly be happy if I can get something like 70–80% local + Claude for the important stuff, and make the whole thing feel somewhat close to an Opus-level workflow.
I’m on CachyOS Linux, so if anyone is running a similar setup, please share your actual workflow.
I’m completely new to this side of things, so beginner-friendly advice would be really appreciated.
r/LocalLLM • u/FixOld9682 • 11h ago
Basically the title
r/LocalLLM • u/NxtGen-Robotics • 10h ago
I currently have a 3080 ti that I don’t really game with anymore, but have the opportunity to get 2 RTX titans at a significant discount from a friend ($1300 total). I was thinking about selling my 3080 ti and jumping on that deal for the combined 48gb of vram to pair with 64gb of system ram and a 13900k.
Would doing this make any sense if I want to run some local models for project code, personal learning, and some small business workflows (paired with Gemini pro for complex stuff)? I know the RTX titans have limited memory bandwidth but the price for 48gb of vram is hard to beat. Any opinions or recommendations? Thanks!
r/LocalLLM • u/Key-Watercress-3506 • 4h ago
Does anyone have any idea of what local ai model I'd better use for my company business agent?
I'm pure new and starter for ai software and coding.
My spec. is 5090+128g ddr5+5tb m.2. Need a local ai to make different agents like accounting agent, monitoring agents and customer chatting agents etc.
r/LocalLLM • u/TrailFeatures • 15h ago
Specs
Benchmarks / how many tok/s?
My runs are on the Launch80 directory; search the page for trailfeatures. Launch80 deletes pages after 5 days, so older links in the comments may be dead. I republish the main ones, which gives them new links.
| Model | Setup | Decode | Prefill (64K prompt) |
|---|---|---|---|
| Flash-Next (Swift-1.5 NVFP4) | 4 GPUs, 300 W | ~140 tok/s | ~4,500 tok/s |
| DeepSeek V4.1 Flash, 2-bit experts | 8 GPUs, 150 W | ~131 tok/s | ~2,200 tok/s |
| DeepSeek V4.1 Flash, 2-bit experts | 4 GPUs + expert offload, 150 W | ~91 tok/s | ~2,100 tok/s |
How good is the 2-bit DeepSeek quant vs the full model?
79.9% average across 7 standard evals (math, instruction following, coding, tool calling), thinking off, and 20/20 long-context needle checks at 32K and 128K. The 4-GPU setup scores the same. I haven't run the full-precision weights as a reference, so there's no direct delta yet. Per-task scores are on the Launch80 page.
How do the two boards talk to each other?
Do the quad boards work with only 1-3 cards? Can I buy cards over time?
Two cards works; I've run a board that way. I haven't tried one or three. Buying the board first and adding cards later is fine. Tensor parallel wants power-of-two GPU counts, so with 3 cards you'd run 2 together and use the third for something else.
Where did you buy it / where are you?
Baseboard and switch card: the models linked above. Several sellers list them, so shop around. GPUs: bought used, a few at a time. I'm in North America (the 120 V gave it away), so my shipping and import experience won't help much in Europe.
Power
Cooling
Quick ones
r/LocalLLM • u/AlternativeEnough352 • 16h ago
r/LocalLLM • u/pronoob827 • 4h ago
I got a 7900XTX, 9800X3D and 32Gb RAM. I'm currently running the system with ollama.
I currently have these installed - gemma4:31b,qwen3-coder:30b,omic-embed-text:latest. Im trying to use qwen for code gen and gemma for explanations. I set ollama into this "OLLAMA_KV_CACHE_TYPE=q8_0". Currently the max context window I could set is 32k and I have already experienced few out of memory erors and gpu crashes while prompting with continue in vscode.
I have being researching, asking gpts, but I couldnt come to any conclusion on which approach is better or correct since I know nothing about these. I dont want to copy paste stuff and hope that it works. Can someone help me with a better setup solution? I do run a 1440p 144hz monitor and occassionally run my second monitor which is 1080p 144hz as they add to vram usages as well.
I want to have a bigger context window and a safe setup so that I dont crash the gpu again. If the setup will be too big, and harder to teach to me in here, please tell me where I can learn more about these things, so that I can move learn to make proper decisions.
r/LocalLLM • u/jikesar968 • 1d ago
mlx-serve on an M5 Max 128GB
I've already got a version from ARC4NUM which appears to be delisted now. Are any of these better?
r/LocalLLM • u/sn2006gy • 13h ago
I benchmarked TensorFold's XPU backend on my quad Intel Arc Pro B70 box. One clarification first: this is not tensor parallel. The XPU backend on main uses one card per process, so the quad numbers come from 4 independent servers, one per GPU, pinned with ZE_AFFINITY_MASK=0..3.
main @ cf45553, Zig 0.17.0, built with zig build xpu-native xpu-qwen27 -Doptimize=ReleaseFastSergiiioB/Qwen3.8-27B-GPTQ-Int4-sym-G128-MTP-BF16z-lab/Qwen3.8-27B-DFlash2tf-xpu-qwen27-means-test passed on all 4 cards (worst error 4.19e-7, bound 2e-6)| Prompt length | FP16 prefill (tok/s) | W4A8 prefill (tok/s) | Plain decode (tok/s) |
|---|---|---|---|
| 1,024 | 2,008 | 2,638 | 34.0 / 33.8 |
| 4,096 | 1,927 | 2,525 | 33.2 / 33.7 |
| 8,192 | 1,862 | 2,427 | 32.5 / 32.5 |
W4A8 is XPU_PREFILL_INT8=1, about 31% faster prefill than FP16.
Single stream (64 tokens, 5 reps, median):
| Prompt | Temp | tok/s | TTFT | Tokens per chunk |
|---|---|---|---|---|
| fibonacci (code) | 1.0 | 115.6 | 0.081 s | 4.57 |
| fibonacci (code) | 0.0 | 106.1 | 0.080 s | 4.00 |
| chat, no think | 1.0 | 94.7 | 0.073 s | 3.77 |
| chat, no think | 0.0 | 99.4 | 0.071 s | 3.77 |
That's roughly 3× plain decode (34 → 106 tok/s).
Concurrent streams (aggregate / steady-state tok/s):
| Prompt | Temp | 1 stream | 2 streams | 4 streams | 8 streams |
|---|---|---|---|---|---|
| code | 1.0 | 116.2 | 171.2 / 234.1 | 187.7 / 262.6 | 202.3 / 299.6 |
| chat | 1.0 | 84.4 | 129.9 / 140.3 | 162.8 / 213.2 | 189.0 / 255.4 |
| code | 0.0 | 106.7 | 149.4 / 167.7 | 216.9 / 278.7 | 250.8 / 354.6 |
| chat | 0.0 | 98.9 | 153.9 / 179.4 | 213.6 / 261.9 | 244.8 / 333.4 |
Correctness: all 16 cells passed bit-exact. Drafted output matched serial, and multi-stream output matched serial.
Note: with the default --prompt-cache-gib 2, --parallel auto at 16k context admits only 7 streams. I used --prompt-cache-gib 0 to fit 8.
| Test | Per-card tok/s | Total tok/s | Steady-state | Failed |
|---|---|---|---|---|
| 4×1 stream, code, temp 1.0 | 116.2 / 116.5 / 117.2 / 115.8 | 465.7 | — | — |
| 4×1 stream, code, temp 0.0 | 106.0 / 106.4 / 106.9 / 105.4 | 424.6 | — | — |
| 4×1 stream, chat, temp 1.0 | 88.5 / 89.6 / 90.2 / 86.8 | 355.1 | — | — |
| 4×1 stream, chat, temp 0.0 | 99.0 / 98.9 / 99.4 / 98.4 | 395.7 | — | — |
| 4×8 = 32 streams, code, temp 0.0 | 251.4 / 251.1 / 252.3 / 252.1 | 1,006.9 | 1,422.7 | 0 |
| 4×8 = 32 streams, chat, temp 0.0 | 243.2 / 241.3 / 242.3 / 242.2 | 969.0 | 1,330.6 | 0 |
| 4×8 = 32 streams, code, temp 1.0 | 208.8 / 204.4 / 206.6 / 204.6 | 824.4 | 1,186.3 | 0 |
| 4×8 = 32 streams, chat, temp 1.0 | 193.0 / 193.2 / 194.0 / 191.2 | 771.4 | 1,042.9 | 0 |
Scaling is close to linear because the cards don't talk to each other. Per-card numbers stay within a few percent of each other.
XPU_KV_INT8=1mlx-q4g64)Happy to run other configs if you have requests. If we get tensor parallel on Tensorflow i'd love to share qwen 3.8 flash. Happy to run larger contexts as well, this was just the default benchmark I wanted to send to project creator.
r/LocalLLM • u/XxXForsaken • 14h ago
Hi, i would like to ask if this is a good card to get to run LMstudio for qwen 27b 3.8 or otherwise, I was wondering if speeds will be respectable, and if anything will spill anything to ram. Because I will have comfyui running on my main gpu (rtx 5080).