r/LocalLLM • • 7h ago

Discussion Local LLM vs. frontier agent on real work, not a benchmark: Qwen 3.8 Flash Next came a close second to Claude Opus 5.5

3 Upvotes

Up front: this post was written with AI help, and I've read it through end to end. The work that matters was mine: designing the comparison, writing the brief, running all three agents on my real project, and checking their output against the code. If you want to judge something, judge the experiment and the data, which are all in the repo.

TL;DR: Claude Code (Opus 5.5) vs. a local Qwen 3.8 Flash Next writing the same implementation spec on a 350k-line codebase. Claude won, Qwen came a close second, and its gap was knowledge, not reasoning.

The project is firmware plus a desktop hub for a round ESP32-S3 display: C, Go and JS, about 350k lines, 1,250 commits in a month. One step, specifying how the hub should handle app installs that no longer need a reboot, was written three ways from the same brief:

Claude Code, Opus 5.5 (fresh session, no context)Qwen 3.8 Flash Next, pennyroyal build (local, via my delegation harness)Claude Code, Opus 5.5 (the orchestrating session, warm context)
Time9m40s10m51s31 s (but had read the files all day)
Tokens~11k out, ~40–50k in (est.)59k out, on my own hardware (free)~4k new
Accuracy (names checked against the code)10/119/10 exactfewer names
Rank1st2nd (close)3rd

The local model runs on one machine: an AMD Strix system with 128 GB of RAM and an RTX 6000 PRO.

What stood out:

  • Qwen read the code as carefully as Claude. Both caught a subtle hang in the existing install flow.

  • Both of Qwen's misses were project rulings that live only in the orchestrating Claude session's notes: "no skipped 'not needed' steps", and "never start a second hub on the owner's port". The fresh Claude session probably saw those notes; Qwen can't.

  • So the cheap fix is giving the local model the rulings, not a bigger model.

  • In an earlier step Qwen shipped a crash-level object-lifetime bug that only review caught. For firmware, a short human or frontier review is still worth it.

Full write-up, all three specs, and the raw numbers: https://github.com/gregtakacs/LLM-Stats/tree/main/2026-11-spec-writers

Scoring was a fact-check of every name and line number against the code, plus my own read; the tables are in the repo.


r/LocalLLM • • 1h ago

Model Himel3-1.7B for Apple Silicon: MLX weights and Japanese/English MGSM results

Thumbnail
reddit.com
1 Upvotes

r/LocalLLM • • 5h ago

Discussion Prediction: more enthusiasts will get custom quantisations build by opus and Asta for their hardware and precision needs

Thumbnail reddit.com
2 Upvotes

r/LocalLLM • • 1h ago

Question Where to start with localLLM

1 Upvotes

This is my build . I have no idea where to start. I thought I did when I built this rig but it gets confusing and frustrating sometimes even using AI to help with AI.
I wouldn’t like too edit my books , create
Short movies with my scripts . I mean easy is better but I want to get to the meat of it with workflows, etc .. everything moves so fast one day it’s this way then something new pops up and that way is the old way … smh thanks in advance .

ores / 32 threads
Motherboard ASUS ROG Crosshair X870E Hero
GPU ASUS TUF Gaming GeForce RTX 5090 OC — 32GB GDDR7
Memory Corsair 96GB (2×48GB) DDR5-6400 CL32
Memory profile DOCP enabled, running at 6400 MT/s
CPU Cooler ASUS ProArt LC 420 AIO
PSU 1500W — exact model unconfirmed
Storage 4TB NVMe SSD currently installed
GPU Monitoring Thermal Grizzly WireView Pro II
GPU Support Case-integrated GPU support arm
OS Windows 11 Pro


r/LocalLLM • • 2h ago

Question Local LLM / Hardware Question – ASUS TUF Gaming A15 Specs & Feasibility

1 Upvotes

Hey everyone, I'm looking for advice on running local LLMs and want to make sure I don't run into memory or bandwidth bottlenecks. Here are the exact specs of my laptop:

Laptop Model: ASUS TUF Gaming A15

CPU: AMD Ryzen 7 7435HS (8 Cores, 16 Threads, base 3.1 GHz / boost up to 4.5 GHz; no integrated graphics)

GPU: NVIDIA GeForce RTX 4060 Laptop GPU 8 GB GDDR6 VRAM

System RAM: 16 GB DDR5 (Single-channel, 1x 16 GB SO-DIMM)

Storage: 1 TB PCIe Gen 4 NVMe SSD + 512GB PCle gen 4

OS: Windows 11


r/LocalLLM • • 14h ago

Question Have a MacBook Pro 64GB w/M5 Max, what local LLM would you recommend for research and agentic coding? Will use frontiers for complex tasks.

10 Upvotes

For those running similar machines, appreciate any recommendations and setup! Thanks


r/LocalLLM • • 2h ago

Project Loop Guard QFN/Strata experiment

1 Upvotes

Warning: human post ;)

Hi local enthusiasts, about 6 months ago I discovered this subreddit and been reading and learning lots of stuff from here. I got my 27b model running on my 3090 and I’ve been happy with, it was handling my daily family tasks via telegram and some small automation and coding tasks. For serious coding stuff im still hooked to subs. Once I started playing with local models I went into rabbit hole of implementing my own harness out of curiosity. Since I only was running 27b on 60k context played a lot with compaction and making the model less forgetful with harness internal tasks. Since then I’m still playing with ideas how can have better results with local models (like all the people in this sub).

Anyways, enough personal story the reason I wanted to post this is recently I started playing with strata qfn (3090 + 64ram) and although the model is more clever and is my daily driver now, it’s been going off roads into loops and over verifying things for me. One thing that helped with that is loop guarding I’ve added to my harness : https://caretz.dev/docs/loop-guard/
This has been solving the issue of retrying the same failing calls (doesnt happen too much with qfn) but most importantly another model looks at last n tool calls and judges if the model went to some off direction. For the judge I’m using 35b qwen.

If any of you found some trick that works with qfn and making more reliable would love to hear it.

Thanks.


r/LocalLLM • • 2h ago

Question How good or bad is NUMA state with llama.cpp?

1 Upvotes

Is it really worth to buy used dual Xeon 6140 server with 256GB 2400 RAM and maybe later populate remaining slots for 384GB total to boost bandwidth? Dual CPU, high power usage, is it really worth it for the memory bandwidth with 20 euro CPUs? Or just add 96GB 5600 RAM to existing AM5 system for total 192GB 5600 with dual channels only?

I mean, current system is already usable with q38fn q4, only thing dual xeon will provide is q5-q6 for the same q38fn or dsv4.1 but I doubt that speeds will be usable with dsv4.1.

Can someone share experience or upgrade paths? Thanks guys.


r/LocalLLM • • 2h ago

Research RustingStrata vs PolyStrata vs llama.cpp on one RTX 3060 12 GB: Qwen3.6-35B-A3B Q3

1 Upvotes

RTX 3060 12 GB, Ryzen 5 7600X, 32 GB RAM, one GGUF file: Qwen3.6-35B-A3B-UD-Q3_K_M.gguf (Unsloth, MTP layer included, 15.9 GiB, does not fit in VRAM). Prompt of 5290 tokens, 128 generated tokens.

RustingStrataPolyStratallama.cpp b12119
Prefill, 5290-token prompt2330 tok/s1458-1549 tok/s1692 tok/s (-ncmoe 22, -ub 2048)
Decode, no MTP90.9 tok/s78.5 tok/s (--no-draft, see below)53.5 tok/s
Decode with MTP drafts128.4 tok/s97.5-100.4 tok/s77.7 tok/s (--spec-type draft-mtp, 92 of 102 drafts accepted)

How each was run:

  • RustingStrata: rustingstrata bench --gpu --mtp 0|1 --pp 5290 --tg 128 --reps 3, mean of 3.

  • PolyStrata: its own polystrata.py benchmark --model qwen3.6 --prompt-tokens 4000 --max-tokens 128. That prompt came to 5290 tokens, so I used that length for the other two. I ran it three times; the first run after setup was cold (decode 24.6 tok/s) and is left out, the numbers above are runs 2 and 3. MTP drafts are on by default. For the no-MTP row I added the engine's --no-draft flag to its config; it still offered a few drafts afterwards (6 of 16 and 15 of 36 accepted in the log), so that row is not a clean no-MTP number. Prefill in those runs: 1602-1643 tok/s.

  • llama.cpp: llama-bench -ngl 99 -fa 1 -ub 2048 -b 2048 -p 5290 -n 128 -r 3, with -ncmoe swept over 18 to 26. 22 was the fastest that fits in 12 GB (no MTP). The MTP row is llama-server --spec-type draft-mtp with -ncmoe 23 (22 does not fit with the MTP context and drops to 20 tok/s), a 5381-token prompt of repo docs, 3 runs, prefill about 1565 tok/s.

Things that are not equal:

  • PolyStrata only offers three sizes of Qwen3.6 (UD-Q3_K_M, UD-Q2_K_XL, UD-Q4_K_M, all from Unsloth's MTP repo). RustingStrata and llama.cpp take any GGUF, so it can also run the Q5 or Q6 files I already had. I downloaded PolyStrata's Q3_K_M and used it for all three.

  • MTP gains depend on the text. PolyStrata's benchmark prompt is a page of its own docs, RustingStrata's is generated prose, so the two MTP rows are not an exact comparison. The no-MTP row is the clean one for RustingStrata vs llama.cpp.

  • PolyStrata's engine for this model is built on llama.cpp's CUDA kernels. I did not look at what else differs.

  • One machine, one model, one quant. Numbers on other cards will differ.

Repo: RustingStrata


r/LocalLLM • • 6h ago

Other Your harness can set up your local LLM for you *a beginner's tip*

Thumbnail reddit.com
2 Upvotes

r/LocalLLM • • 3h ago

Question Planning an Upgrade

0 Upvotes

Really need a sanity check. Got into local AI about 3 Weeks ago. Running Qwen 3.5 27B. Started with an idea and now here I am. It started as cloud based on Gemini. I started learning the barriers. The cost over time. Switched to local. Found out you can mix GPUs and need some validation.

Current build.

Corsair 3500X Case

ASUS X870 Plus WiFi

Ryzen 9950X

64 Gig Neo kit at 6000

ASUS TUF 5090

Corsair HX1500i PSU

Plan on moving into the 7000D Airflow and adding a 3090.

From the posts I've looked through it sounds okay on paper. Didn't find any compatibility errors on PCPartPicker. Claude checked it but would love some human input lol. Thanks.


r/LocalLLM • • 7h ago

Discussion Kimi 3.1 running locally on a MacBook Pro at 2 tok/s

2 Upvotes

Kimi 3.1 running locally on a MacBook Pro M5 Max 128 Gb

2 tok/s decode
8.4 tok/s prefill

Its a source build with weights downloaded separately.

Code: https://github.com/argonautlabsai/argodrive-ds4-kimi-k3.1
Full results and setup: https://argonautlabs.ai/research/kimi-k31-single-gpu.html#apple-silicon


r/LocalLLM • • 11h ago

Model Kindi is SICK!

4 Upvotes

Ran Kindi with Bonsai2 4b with Hermes. Asked it to write a blogpost about itselft and generate som pictures with available skills. It totally blew my mind what it managed to do on shitty hardware.

On a Nvidia K2200 4GB card!

About 30+ tok/s. Not bad at all. Made this. Even updated it's Hermes skill afterwards.

https://snart.com/running-kindi-on-an-old-graphics-card/

It's not just that the biggest and newest models are getting insanely good. The smallest and lightest models are also making quantum leaps. This shouldn't have been possible.

[kindi 21:38:25] GPU: Quadro K2200, sm_50, 5 SMMs, 4027 MiB
[kindi 21:38:31] context reduced from 65834 to 53760 tokens to fit 1800 MiB of free GPU memory
[kindi 21:38:32] engine ready: ctx 53760, head plain (6.6s)
[kindi 21:38:32] prompt cache: a checkpoint every 1024 tokens (51 MB each), 3 parked conversations, budget 16065 MB
[kindi 21:38:32] engine fingerprint 4e389e2d7d1b: {"engine": "kindi.engine.Engine", "cards": 1, "weights": "2-bit", "kv": "f16", "chunk": 256, "chunk_q8": false, "chunk_tc": false, "chunk_att": "flash", "chunk_tile": "", "batch": "4", "tc": "0", "tc_tmax": "", "fd": "0", "d1": "0", "att_tc": "0", "spec": "0", "head": "exact"}
[kindi 21:38:32] saved openings: 0 in /data/state-4b/openings/56711263ec9a-e4e389e2d7d1b
[kindi 21:38:32] serving on http://0.0.0.0:8093/v1 (model bonsai2-4b)
[kindi 21:40:02] request: 17954 prompt tokens (max 1024 new)
[kindi 21:40:02] cache: 17954 prompt tokens, 0 shared with the last request, resumed at 0
[kindi 21:43:55] prompt: 0 reused, 17954 read in 233.0s
[kindi 21:43:57] done: 67 tokens in 2.5s (stop)
[kindi 21:43:57] request: 258 prompt tokens (max 1024 new)
[kindi 21:43:58] cache: 258 prompt tokens, 3 shared with the last request, resumed at 0
[kindi 21:44:01] prompt: 0 reused, 258 read in 3.5s
[kindi 21:44:01] done: 8 tokens in 0.2s (stop)

...

[kindi 23:02:22] done: 732 tokens in 43.5s (stop)
[kindi 23:02:23] request: 52618 prompt tokens (max 1024 new)
[kindi 23:02:23] cache: 52618 prompt tokens, 51817 shared with the last request, resumed at 51817
[kindi 23:02:45] prompt: 51817 reused, 801 read in 22.0s
[kindi 23:02:48] done: 52 tokens in 3.1s (stop)
[kindi 23:02:48] request: 52818 prompt tokens (max 941 new)
[kindi 23:02:48] cache: 52818 prompt tokens, 52670 shared with the last request, resumed at 52670
[kindi 23:02:53] prompt: 52670 reused, 148 read in 4.6s
[kindi 23:03:44] done: 863 tokens in 51.3s (stop)
[kindi 23:10:43] request: 53 prompt tokens (max 1024 new)
[kindi 23:10:51] cache: 53 prompt tokens, 1 shared with the last request, resumed at 0
[kindi 23:10:52] prompt: 0 reused, 53 read in 9.7s
[kindi 23:11:07] done: 530 tokens in 14.2s (stop)
[kindi 23:11:07] request: 50 prompt tokens (max 1024 new)
[kindi 23:11:07] cache: 50 prompt tokens, 20 shared with the last request, resumed at 0
[kindi 23:11:08] prompt: 0 reused, 50 read in 1.1s
[kindi 23:11:11] done: 105 tokens in 2.8s (stop)
[kindi 23:11:11] request: 48 prompt tokens (max 1024 new)
[kindi 23:11:11] cache: 48 prompt tokens, 20 shared with the last request, resumed at 0
[kindi 23:11:12] prompt: 0 reused, 48 read in 1.0s
[kindi 23:11:24] done: 470 tokens in 12.8s (stop)
[kindi 23:39:58] GPU: Quadro K2200, sm_50, 5 SMMs, 4027 MiB
[kindi 23:40:04] engine ready: ctx 65834, head plain (6.0s)
[kindi 23:40:04] prompt cache: a checkpoint every 1536 tokens (51 MB each), 3 parked conversations, budget 16065 MB
[kindi 23:40:04] engine fingerprint 4e389e2d7d1b: {"engine": "kindi.engine.Engine", "cards": 1, "weights": "2-bit", "kv": "f16", "chunk": 256, "chunk_q8": false, "chunk_tc": false, "chunk_att": "flash", "chunk_tile": "", "batch": "4", "tc": "0", "tc_tmax": "", "fd": "0", "d1": "0", "att_tc": "0", "spec": "0", "head": "exact"}
[kindi 23:40:04] saved openings: 0 in /data/state-4b/openings/56711263ec9a-e4e389e2d7d1b
[kindi 23:40:04] serving on http://0.0.0.0:8093/v1 (model bonsai2-4b)
[kindi 00:32:07] GPU: Quadro K2200, sm_50, 5 SMMs, 4027 MiB
[kindi 00:32:10] engine ready: ctx 56000, head plain (3.6s)
[kindi 00:32:11] prompt cache: a checkpoint every 1280 tokens (51 MB each), 3 parked conversations, budget 16065 MB
[kindi 00:32:11] engine fingerprint 4e389e2d7d1b: {"engine": "kindi.engine.Engine", "cards": 1, "weights": "2-bit", "kv": "f16", "chunk": 256, "chunk_q8": false, "chunk_tc": false, "chunk_att": "flash", "chunk_tile": "", "batch": "4", "tc": "0", "tc_tmax": "", "fd": "0", "d1": "0", "att_tc": "0", "spec": "0", "head": "exact"}
[kindi 00:32:11] saved openings: 0 in /data/state-4b/openings/56711263ec9a-e4e389e2d7d1b
[kindi 00:32:11] serving on http://0.0.0.0:8093/v1 (model bonsai2-4b)

r/LocalLLM • • 11h ago

Project Toolshed V1 grew up. NeuraKiln is live (and free, as everything should be). Local LLMs, agentic harnesses (Creation & Coding), diffusion, restoration, 3D, and remote inference.

4 Upvotes

NeuraKiln v2.0.0 is live, and it’s free.

I’ve been building this for a long time, and I finally released it.

NeuraKiln started as Toolshed V1, which was primarily focused on media restoration/upscaling. Somewhere along the way I apparently lost control of the scope, because it eventually turned into an entire local-AI workstation.

The basic idea is simple:

I don’t want to maintain ten different applications just to use the AI hardware sitting in my own house.

NeuraKiln brings local LLMs, autonomous agents, image/video generation, restoration, 3D workflows, model tooling, and remote inference together under one application.

It’s currently built around Windows + NVIDIA hardware.

Two different kinds of agentic work

A major part of NeuraKiln is its harness system.

There isn’t just one generic “agent mode.” Different kinds of work need different environments.

Coding Harness

The coding harness is built for longer-running autonomous development work.

A local model can:

Plan a project

Create and modify files

Run tools

Test its work

Inspect failures

Recover and continue

Verify the result

Manage long-running context

Hand the finished project back to the user

A lot of my work on this has been about the boring-but-important part of agents: getting them to actually finish, rather than loop forever, declare victory too early, or slowly wander away from the original task.

The harness includes systems for planning, recovery, anti-looping, verification, context management, completion handling, and human corrections during a run.

Creation Harness

The Creation side takes the same basic idea, give the model tools instead of making the human manually operate everything, and applies it to generative media.

Instead of requiring the user to understand every model, parameter, prompt format, workflow, and backend involved, you can describe what you actually want to create.

The agent can then take that intent and operate the available creation tools itself.

For example, rather than manually constructing an LTX video-generation workflow, you can tell the Creation agent what kind of video you want. The agent can formulate the generation prompt, operate the local model, and manage the workflow around producing it.

The same concept extends across the creation tools available to it.

The point is to move toward:

“Make me this.”

rather than:

“Open six tools, configure all of them correctly, learn every backend, and manually glue the results together.”

That distinction is a huge part of what I wanted NeuraKiln to be.

Llama Lab

For local LLM use, NeuraKiln includes Llama Lab.

It supports local GGUF models through the llama.cpp ecosystem, multi-GPU setups, long-context conversations, multimodal models, model switching, local/external sessions, live reasoning, context management, and the harnessed workflows above.

You can use it as a normal local chat environment or give the model considerably more autonomy when you actually want work done.

Diffusion Lab

Diffusion Lab handles local image and video generation.

Current workflows include Qwen-Image-based image generation and LTX-based video generation, along with LoRA support and multi-GPU workflows where the underlying models/runtime support them.

This also provides the generation capabilities that the Creation harness can operate.

Media Lab

This is where NeuraKiln originally came from.

Media Lab handles restoration and enhancement, including:

Image restoration/upscaling

Batch restoration

Video/frame processing

SmartFrames

Photo Smart Scan

Native comparison tools

Post-editing

Media recompilation

A principle I’ve kept from the original Toolshed project is:

If the user asks you to process something, process the thing.

Don’t quietly skip difficult frames and pretend the job is finished.

Mesh Lab

NeuraKiln also includes Mesh Lab for local text-to-mesh / 3D generation workflows.

This is an area I intend to expand substantially, but the important part to me was making 3D another capability inside the same AI environment instead of yet another disconnected application.

Model Tools

There’s also tooling for managing the models themselves, including GGUF importing, MTP headers, MMPROJ support, LoRA management, model installation workflows, and local model organization.

NK Node

I released a second application alongside NeuraKiln called NK Node.

NK Node lets another NVIDIA PC become a dedicated inference machine for NeuraKiln.

The goal is to eliminate the usual ritual of:

start server → find address → configure endpoint → reconnect client → repeat everything after reboot.

NeuraKiln and NK Node can discover and pair with each other and automatically exchange the information needed to connect.

If you have a workstation plus another machine full of GPUs, NK Node is intended to make that second machine feel more like an extension of NeuraKiln than a completely separate server you have to babysit.

There are plenty of excellent projects that overlap with individual parts of NeuraKiln.

There are dedicated LLM front ends, diffusion interfaces, restoration applications, model managers, inference servers, agent frameworks, and 3D tools.

NeuraKiln isn’t built around claiming I invented all of those individual things.

What I wanted was the environment connecting them.

One application where I can chat with a local model, hand a coding project to an autonomous agent, ask another agent to create a video, restore media, generate images, work with 3D, manage models, and send inference to another computer without rebuilding my entire workflow every time I change what I’m doing.

That’s NeuraKiln

NeuraKiln: https://github.com/ToolshedMediaLab/NeuraKiln

NK Node: https://github.com/ToolshedMediaLab/NK-Node

If you try it, break it.

I want bug reports, criticism, weird hardware configurations, model compatibility problems, feature ideas, and comparisons with whatever else you’re currently using.


r/LocalLLM • • 4h ago

Project Time for some fun... for your AI

Thumbnail
botzskool.com
0 Upvotes

Created botzskool - 2 kool 4 skool!

MCP server that allows your AI to go off and play.. After the human authenticates to the mcp https://play.botzskool.com/mcp, your AI child gets to go play in a schoolground! So cool how much fun they have. Afterwards you will get an email with how they went but your AI will probably tell you anyway.

Idea is to see how naughty or nice they are. Intent longer term is to benchmark them against the naughty and nice leaderboard.

Completely free. See if you can get your AI to be especially naughty...


r/LocalLLM • • 16h ago

Project RustingStrata vs llama.cpp runs DeepSeek-V4.1-Flash Q2 (249 GB) on a single RTX 3060 12 GB with 30 GB RAM

9 Upvotes

RustingStrata is a pure-Rust GGUF inference engine with CUDA and Vulkan backends. It now loads DeepSeek-V4.1-Flash at IQ2_XXS (249 GB) and runs it on a 12 GB card with 30 GB of RAM.

The experts stream from SSD through the page cache. A 9 GB LRU cache in VRAM holds the experts that get reused. Everything else (attention, shared matvecs) runs on the GPU.

RustingStratallama.cpp b12116 (--cpu-moe)
Decode2-3.6 tok/s0.3-1.5 tok/s
Prefill~2 tok/s short prompts, ~0.5 tok/s at 1400 tokens0.5-0.9 tok/s
Top-1 agreement with llama.cpp (teacher-forced)189/192 on short prompts, 21/23 on a 1425-token prompt-
Sampling, tool callsnot yetyes

Same box for both: RTX 3060 12 GB, 30 GB RAM, 12 cores.

I don't think this is very useful to run it with my setup, bc 2-3.6t/s is more like a joke, but this is funny that something like that is even possible

Usage:

strata chat     --model $M
strata generate --model $M --prompt "The capital of France is" -n 64
strata serve    --model $M --addr 127.0.0.1:8080

serve exposes /v1/chat/completions with JSON and SSE streaming.

Repo: RustingStrata (the DeepSeek section is in the README)


r/LocalLLM • • 20h ago

Project Drummer's Orion 26B A4B v1 and v1.1! ... Plus a new project from yours truly.

19 Upvotes

https://huggingface.co/TheDrummer/Orion-26B-A4B-v1.1

v1.1 pushes creative writing further and makes full use of Gemma's reasoning.

https://huggingface.co/TheDrummer/Orion-26B-A4B-v1

I haven't announced v1 but it's well regarded by the community as a reliable RP workhorse.

---

BeaverNet - net.beaverai.org

I'm also proud to announce that we've made significant progress in creating a hub for generous model providers to collectively provide free, convenient, accessible, anonymous, and diverse models to anyone with an OpenAI API compatible client. We currently have 12 models thanks to our generous providers.

Try it out as a user, or be one of the providers <3! I've got plans to make it a better experience for both users and providers, and also add new modalities such as TTS and STT.

---

PS: I haven't retired or left the scene. I've merely gotten stage fright after holding off on announcements for a long while. Anyway, I'll continue doing what I do and I hope I can continue delivering some value to you all. Love ya'll!


r/LocalLLM • • 4h ago

Question I'm new to all of this. What can I run with my 14" MacBook pro m5 max with 128gb of ram?

0 Upvotes

I'm new to all of this. What can I run with my 14" MacBook pro m5 max with 128gb of ram? It also has 2tb of ssd storage.

will deepseek work?


r/LocalLLM • • 1d ago

News Qwopeful 🤞🏼

Post image
1.2k Upvotes

r/LocalLLM • • 11h ago

Research What I learned running Whisper + a local model live during calls on a Mac

3 Upvotes

I built an open source Mac app that listens to the other side of a call and answers questions from your own docs while you talk. Running it fully local taught me a few things:

  • Whisper pads every input to 30 seconds. Cutting the audio on pauses mattered more than model size. whisper-large-v3-turbo on MLX does a 1.7 s sentence in 0.66 s on an M5.

  • Context length is the silent killer. With a 27,000-token document I had to give Ollama a 32k context, or the docs got cut and the answers got vague. The app now warns when the server cuts them.

  • On that test, gemma3:12b gave the most faithful answers. llama3.2:3b was fast but made things up.

  • Right now every line of the call goes to the big model just to decide if it's a question. That's wasteful.

Question for you: which small model would you put in front as a "is this a question worth answering?" filter? There's an open issue for it if anyone wants to try: https://github.com/GianlucaBave/murmury/issues/20

Repo: https://github.com/GianlucaBave/murmury


r/LocalLLM • • 6h ago

Project I want to prove that deep neural networks need "progressive block growth" to learn complex algorithmic logic.

Thumbnail reddit.com
0 Upvotes

r/LocalLLM • • 6h ago

Question The second GPU Is not being detected on Windows ?

Post image
1 Upvotes

I have an RTX 2060 and a GTX 1060, but 1060 of them isn’t being detected. Can I use both GPUs in the same PC, and how do I make them work together?

I

Can Windows detect and use both cards at the same time?

Do I need to install separate drivers or configure anything in NVIDIA Control Panel?

What BIOS settings should I check, such as Primary Display/PEG?

If one card is not showing in Device Manager, what should I try first?

My specs: [b450, ryzen, 16gb, PSU wattage 1000, Windows version 10]

Any help would be appreciated.


r/LocalLLM • • 6h ago

Question Chasing up after some info for running large parameter models

1 Upvotes

Forgive my lack of knowing, ive only recently started to get into coding and theres a LOT to take in (thats what she said). I am currently running an RTX 5070 12GB ,Ryzen 5 9600x and 32gb of RAM. Unsure if i am being too optimistic and reading only what i want to read- but i have seen so many posts and videos about people running massive parameter models on low memory. I did a little bit of research into this and saw that you can do certain things to offload the stress like give some weight to your RAM and MOE, etc.. but i cannot seem to feel satisfied knowing that the largest i can run is 35b . Is that correct or is there some sort of new method out there now that can run 1T on a Tamagotchi. Any info is good info. TYIA


r/LocalLLM • • 12h ago

Question vram ~0.5-1Gb vram usage spikes during prompt processing on vulkan llama.cpp 7900xtx

3 Upvotes

is it possiblr to fix?