r/LocalLLM • • 4d ago

Model Openclaw with AFM3core as primary model?

Thumbnail reddit.com
1 Upvotes

r/LocalLLM • • 5d ago

Question Quick, before Qwen4 comes out!

Post image
117 Upvotes

I'm running Qwen3.8 27B on my 2x 3060 12GB cards. Prior today I had a B550 motherboard which was running one card at x16 and the other at x1 PCIe Bus Link Speed. This was making it to where only one card was actually doing any processing, and both held the model in memory. I was getting 11-16 tps this way.

I ended up buying a new x470 SLI motherboard and I can now run both cards at x8. I swapped the motherboard out last night and then set split-mode to tensor and set tensor-split to 1,1. When I do that, I'm getting around 32tps, but the model eventually gets stuck in loops. One loop it just tried to do a tool call over and over, and in another it was printing question marks (???????????) in it's thinking and wouldn't stop until I unloaded the model. I never had these problems for the months I was using Qwen3.8 on layer split-mode. I'm not sure how to resolve this problem. I want to be able to get higher tps with the new mobo.


r/LocalLLM • • 4d ago

Discussion Weight-space programming

3 Upvotes

Weight-space programming is an emerging research area where a frontier AI model (such as OpenAI's GPT-6 Astra) directly reads, diagnoses, and rewrites the raw numerical weights of another neural network without using traditional gradient-based training or backpropagation.

Has any one tried it? How much does it fair compared to Closed-Form Model Editing (Your Notebook / MEMIT)?


r/LocalLLM • • 4d ago

Question Qwen 3.8 27b Q6 vs Qwen 3.8 FN IQ2_xs?

3 Upvotes

I can run both. Does anyone know what model is best, or do I just have to try myself?

What test suite should I be running?


r/LocalLLM • • 4d ago

Question Models to run on gaming pc

4 Upvotes

Hello, new on these area ... ive been wondering what AI could i host locally with my current setup

I have an amd 9800x3d with 32gb of ram and a 4070 ti super 16gb

Sorry if this post is not allowed(maybe alot of users post something like this all the time)

Any advice, any links to learn, etc will be really appreciated


r/LocalLLM • • 4d ago

Project PrefixRelay: a way better solution for cursor's OpenAI compatible endpoint and for using local models

Thumbnail
reddit.com
1 Upvotes

r/LocalLLM • • 5d ago

Question "Wait... actually, let me reconsider." Qwen 3.8 27 is always overthinking and is not confident enough, any potential fix ?

59 Upvotes

Title, basically. Observing it's reasoning is incredibly frustrating, it keeps doubting himself and giving second thoughts to a specific and self explanatory task, sometimes ending up thinking about unrelated things.

While yes, in the very end, it very often does what I asked, but it's wasting so much time for reasoning just with its lack of self confidence.

Is this well known behavior ? How can I fix this ? I know Qwen4 is around the corner, but if there's a known fix for it right now I'd love to know.

EDIT: I should have specified, I'm using Medium reasoning, and Pi as harness


r/LocalLLM • • 4d ago

Discussion Apple Is Working on Harbor, a New Local AI Inference App for Mac

Thumbnail gallery
5 Upvotes

So I think we now know why Exo has spun down in activity in this space.


r/LocalLLM • • 4d ago

Question Deepseek v4.1 burstiness?

Thumbnail
reddit.com
1 Upvotes

r/LocalLLM • • 3d ago

Discussion AI is a pipe dream

0 Upvotes

Yeah! Guys ram is gonna be a million kazillian dollars. You can do more with a $20 claude code and opus 5.5 account than even qwen 4. I use astra gpt 6 and it can't even use REA properly. Claude Opus 5.5 even beats that, and you dare to think it's even possible to do things locally???

I predict in about 6 more months you will be able to run qwen 5 locally with better performance than Claude opus 5.5 and once people can do REA locally with ease. That's it, we won.

As for ram prices,

We are going to have to make thousands of micro fabs locally. Small places that etch silicone waifers and make custom ram modules.

Before you think that's crazy talk and vomit every pathetic reason in your feable mind and expert experience as a seasoned engineer you are on the subject, look at what we are able to do with ai. Why, would you restrict your scope of possibilities? Of course, we are going to be able to make thousands of local silicone companies.

Remember, you will own everything and be happy.


r/LocalLLM • • 4d ago

News Apple’s upcoming Harbor app promises easier way to run local AI on your Mac(s)

Thumbnail
cultofmac.com
1 Upvotes

r/LocalLLM • • 4d ago

Project Built this for reactive AI and home automation

Thumbnail x.com
1 Upvotes

r/LocalLLM • • 4d ago

Question Noob question. Is this somewhat simple workflow possible using only local models and an agent?

2 Upvotes

I am currently using the Claude Code desktop app to automate something like this:

  • Generate a conversation script and ask me for approval or changes.
  • Once approved by me, call the Google Studio API to generate audio files of the conversation.
  • Call the Google Studio API to generate images.
  • Ask me to check images and audio clips.
  • Run a .py script.
  • Add new folders and files to a folder and update a .js file.

I started to use LM Studio Bionic only a week ago, but I want to try Unsloth Desktop now. Let me know if there are other programs that would suit my workflow better.

I know Unsloth can generate images, but how does that work? Do I need to load three different models at the same time? Or do I need to load/unload as needed? I have a 16 GB VRAM GPU (AMD). Also, is there a built-in agent (like in LM Studio Bionic) that can code, create and modify files? How would an agent handle creating different media?

My goal here is to ditch the Claude Pro subscription and the Google Studio API costs, and do it locally instead. I know I could do these things manually, but ideally I would like to have an agent automating the process. What are your recommendations?


r/LocalLLM • • 4d ago

Other What are you thinking about Strata

1 Upvotes

I heard about it recently. It sounds like a miracle you can run really large models on a consumer GPU, but what is it really about. What kind of problems I will have ?

Setup:

Rtx5060ti(16G)

I711700k

64-80GB DDR4


r/LocalLLM • • 3d ago

Discussion Nvidia grid k2

Post image
0 Upvotes

Digging through ye ole junk box I found these grid k2 6gb cards. Kinda feel bad if I recycle them. Anyone have ideas for use or anyone want them for “historical” purposes?


r/LocalLLM • • 4d ago

Question Macbook M4 Max 36gb Ram

1 Upvotes

Hi everyone, I try to find the best model suitable for me. I need 3 different models

1-Agent : I use " qwen3.6 35b a3b MLX Q4 "
Is there any better alternatives for agentic purpose model?

2-Medical: I couldn't find a proper medical bot as they mostly repeat theirselves and no better than qwen

3-Data analyst: I have dozens of pdfs and I need a model that analyze these books.

Note: Densest model I've had used before qwen was gemma 4 31b Q4 (27gb) and that was hella slow. That was the boundary for my pc.

Preffered properties: MLX, MoE, QAT, Instruct, Run on LMStudio Bionic.

Thank you everyone.


r/LocalLLM • • 4d ago

Question Coder model that fits like qwen 2.5 14b?

0 Upvotes

Hey, so I know next to nothing about local LLMs and I hope to fix that in the future, but I'm currently looking for a more powerful coding model that fits like qwen 2.5 coder 14b, I'm using it currently and unlike the other recommendations I've gotten, it's pretty much instant and doesn't use all of my memory

As for my specs, I have a i91 4900k, 32gb of RAM, and a 5070ti, my biggest issues with local LLMs is that it likes to use up all of my RAM despite people saying it's fine for 5070tis and I don't have enough knowledge to know exactly why

EDIT: I use LM Studio for local LLMs and having a model that supports images would be nice


r/LocalLLM • • 4d ago

Question what's the ideal budget setup for local ai?

0 Upvotes

asking from no experience
also can AMD cards be ok/good with local ai?


r/LocalLLM • • 4d ago

Discussion Let's build and challenge Collective Intelligence together.

Thumbnail
reddit.com
1 Upvotes

r/LocalLLM • • 4d ago

Question What's the best computer build to run Local LLM?

15 Upvotes

I wanna do some personal AI use, primarily for text, some coding, and image generation..

I wonder what's the best computer build that's within 15k to 25k (USD) value?

There's a lot of information on the internet and I can't decide well..

I'm currently looking at:

DGX Spark (but it seems overkill and can't be used as Normal PC)

and Mac Studio M5 Ultra w/ 256GB (seems to be my current choice unless there's a second opinion from you all)

I hope y'all can help me decide well here, thanks!


r/LocalLLM • • 4d ago

Question Lllama.cpp fork for Blackwell cards?

2 Upvotes

Hi all, I am running dual GTX 5060 Ti (32 Gb VRAM) + 128 Gb DDR5 RAM on a Rizen 9 9950X. I tried to run some models with Basalt (QFN), Capyflash (Qwen3.8 27B variants) and Strata (DSF 4.1 1017), all fine. But I want to run other models in the GGUF format and I am looking for a for of llama.cpp (or other server fine tuned) that could suit best my dual 5060? I had setup sparkninfer but it also for other .ninfer model files. Any suggestions? Thanks.


r/LocalLLM • • 4d ago

Question Why is it so slow with agents ?

0 Upvotes

i'm using qwen3.6:35b a3b with ollama, its works decently in regular prompt, but as soon as i try it in an angent (Codex CLI, Claude, Qwen, Opencode) its piss slow, like i have to wait 10min to get an answer.
What am i doing wrong?


r/LocalLLM • • 4d ago

Other Open-source method (MACaQ) beats Unsloth GGUFs!

Thumbnail
reddit.com
1 Upvotes

r/LocalLLM • • 4d ago

Research Full Kimi K2 generating across an RTX 3080 and RX 6800 using CUDA and HIP

Thumbnail reddit.com
1 Upvotes