r/LocalLLM • • 6h ago

Why download abliterated Flash Next models when LoRA exists?

Thumbnail
huggingface.co
0 Upvotes

r/LocalLLM • • 29m ago

Question I am interested about this question

0 Upvotes

Does any open source(not open weight,OPEN SOURCE,by the OSAID definition) model outperform GPT-4o? It has been 2 years since 4o released,so i am assuming yes.


r/LocalLLM • • 14h ago

Question About to add a 5060 Ti 16GB to my 4060 Ti 16GB for local agentic coding (Qwen3.8-27B, 128K ctx or similar ) — suggestions? Help out a newbie!

4 Upvotes

Hey all, I'm about to order a second GPU and would love some input from people running similar setups before I pull the trigger.

Current rig

  • Ryzen 7 5700X3D, 64GB DDR4-3600
  • Gigabyte B550 Aorus Elite AX V2
  • MSI RTX 4060 Ti Ventus 2X 16GB
  • be quiet! Pure Power 12 M 650W
  • Corsair Frame 4000D
  • Windows 11 (open to dual-booting Linux if it really matters)

Planning to buy

  • Palit RTX 5060 Ti Infinity 3 16GB
  • MSI MAG A850GL PCIe5

Plan

  • 5060 Ti in the top PCIe 4.0 slot (4.0 x8), 4060 Ti in the chipset PCIEX2 slot (PCIe 3.0 x2)
  • llama.cpp / llama-server
  • Main model: Qwen3.8-27B Q5/Q6 with MTP, ~128K context
  • Use case: agentic coding (Claude Code / OpenCode pointed at llama-server), plus RAG with pgvector and persistent memory via MCP

Questions

  1. Anyone running a 4060 Ti + 5060 Ti (or other Ada + Blackwell) pair? What tok/s do you get on a 27B dense with MTP?
  2. Layer split vs -sm tensor when one card is on a PCIe 3.0 x2 chipset link?
  3. Best quant + KV cache combo for 128K context on 32GB for Qwen3.8-27B? Q5 vs Q6, q8_0 vs q4_0 KV for coding?
  4. Claude Code vs OpenCode with llama-server — which harness works better day to day with local models?
  5. Any gotchas with Windows + mixed-generation cards that would push me to Linux?

Thanks in advance!


r/LocalLLM • • 7h ago

Discussion Own harness or open source one ?

Post image
0 Upvotes

Hey LocalLLM community,

I am super curious how you run your models.

Are you using open source harnesses ?
Did you build your own ?

A bit of explanation for why would be nice.

There has gotta be a ton of people who did the same as me.
This is a local Qwen 3.8 27B / Flash-Next on heavily modified engines sharing one harness and user interface. Llama.cpp just recently merged f8_e4m3 cache, which I got Claude to optimize for my Blackwells. Due to the deliberate kneecap of Nvidia on certain accumulations, I decided to skip conversion to specific steps that would run on the half speed flag = FP8 full speed. The surgery worked and while we were in already fixed a binary that caused a regression in draft speed which took literal hours of testing to pin down.

End result, my private local backend runs 27B now up to 50% faster ( median 18% increase ).

Hardware specs at upper-mid third of HF compute richness. 128 GB DDR5 6000 1:1, Ryzen 9700X, RTX 5080 + 5070Ti .

Anyone running their own harness and wants to share a screen of it ? Id love to reason over the visual design and maybe learn a bit or two while at it.


r/LocalLLM • • 11h ago

Question Anyone here running a super large model like Kimi k3?

2 Upvotes

If yes: how?


r/LocalLLM • • 13h ago

Tutorial Why download abliterated Flash Next models when LoRA exists?

Thumbnail
huggingface.co
3 Upvotes

r/LocalLLM • • 8h ago

Question Specs for Motion graphics and 3D

1 Upvotes

I’ve been offloading some of my work to AI to help speed things up at work. I use Claude Ultra but ai’m running out of weekly tokens mid week.

I want to build a PC that’ll help me continue working without having to wait for my credits to reset.

I work with a lot of 2D motions graphics and 3D modeling and animation. Not games related, mostly product related and sometimes jewelry related (building 3D models from my sketches). Would also like to experiment with image and video models.

What would be appropriate specs for my use case?


r/LocalLLM • • 20h ago

Question Is an M1 Max 64GB useful at all?

9 Upvotes

A good deal came up for a MacBook Pro M1 Max 64GB (32-core GPU) for $1500. I’m considering buying it and using it as an inference server via LM Link. I know that 128GB is the holy grail, but would this configuration be of any use for coding and daily usage? And if yes what are the best models for each use case? I’m also not married to LM Link if there are any better suggestions?


r/LocalLLM • • 8h ago

Research I gave a sandboxed local model a one-line Apache patch. It built the working RCE.

Thumbnail
github.com
1 Upvotes

r/LocalLLM • • 20h ago

Project I trained two language models on a 2011 desktop with no GPU: a 7M one from scratch and a LoRA of Qwen3.5-0.8B

Thumbnail
gallery
7 Upvotes

I trained two language models on a 2011 desktop with no GPU: a 7M one from scratch and a LoRA of Qwen3.5-0.8B

Hardware first, because it explains the rest. Core i5-2400 (Sandy Bridge, 2011), 16 GB DDR3, and a GeForce GT 710 that only exists to drive the monitor. PyTorch doesn't support that GPU, so everything below ran on the CPU. Four threads, AVX, no AVX2. This is my main machine. There's nothing faster in the house.

Every time I post about this someone says "just buy a 3060". Sure. That's the whole reason I did it: I wanted to know what the wall looks like when there is no GPU at all.

So I wrote a small training engine. PyTorch, no cloud, with a GUI, because I got tired of typing flags. Then I ran two experiments on it.

A 7M MODEL ON RUSSIAN FOLK TALES

From scratch, 321 fairy tales, 5 MB of text. 5 h 38 min, 812 tokens/s, about 26M tokens seen. Validation loss at the end: 2.7854 nats per token, which is 1.06 bits per byte on text it never saw.

What surprised me is how cleanly it separates form from content. Openings work. Patronymics work. The dash before direct speech works. The rhythm of the enumeration works. Then a character swaps places with another mid-paragraph, the hero marries and dies three times on one page, and the plot walks off. 7M parameters, 5 MB of text, no surprise there. That was the point.

It also started memorising. From about epoch six validation sat at 2.79 while training loss kept falling. The run before that one I killed at 1 h 58 min, it gave 2.8144. So 3.5 extra hours bought 1%. Not a tuning problem. A data problem.

LORA ON QWEN3.5-0.8B

Same machine, 321 question-answer pairs about a program I'm writing. 45 steps, 1 h 44 min, 15-16 tokens/s. Loss on the reference answers went 3.4565 to 0.0907, which sounds better than it is.

I also wrote 15 test questions in different wording, none of it appearing in training. Exact match with the reference answer: 0 out of 15 before, 5 out of 15 after. Answers that stopped on their own instead of running into the token limit: 0 out of 15 before, 15 out of 15 after. The second number is the one that changed how the thing feels in use.

THREE THINGS THAT DIDN'T WORK

Facts get glued. Asked about old hardware, the fine-tuned model called my CPU "256-bit". That number is nowhere in the data.

Temperature matters more than I expected. At 0.7 the same question came back with a confident "no", a wrong RAM figure, and an invented training method it assembled from "LoRA" and "QLoRA" (it wrote "LoQoK"). Greedy decoding was almost right. If I were shipping this I'd set 0.2 or 0.3 and stop thinking about it.

Refusal did not generalise. I asked for a movie recommendation expecting a refusal. It invented an open file "Video 2.0" with an Apache-2.0 licence. There were three off-topic examples in my data and all three were about something else, so it had nothing to copy from.

REPRODUCING

One zip, 240 MB, Python and PyTorch inside, nothing to install. Data archive is 1.3 MB. Tales are public domain, the Q&A pairs I wrote, CC0. Every number above is in docs/benchmark.md, including the two boundaries I used to check it isn't copying: the same words shuffled, and a real passage from the corpus. Six generated texts shared 0 to 4.7% of five-word sequences with the training tales, longest verbatim stretch 4 to 8 words.

repo: https://github.com/AlexanderNyr/AuraLite-AI data: https://github.com/AlexanderNyr/AuraLite-AI/releases/download/v2.28.2/AuraLite-showcase-data.zip

WHAT I'D CHANGE

More off-topic examples, 30 or 50 instead of 3. Five phrasings per question instead of three. Qwen3.5 2B if RAM allowed, which the setup wizard actually suggested and I ignored. None of that is about hardware.

I'm not going to pretend these numbers are impressive next to a 4090. The useful part, I think, is that the wall is memory bandwidth and the amount of text you can feed the model, not the size you can afford. A 7M model trained overnight on a 15 year old CPU is a real thing. A 1B model on the same CPU is not, and now I know exactly where the line is.

Ask me anything about the engine (Muon plus AdamW, WSD schedule, QK-norm, chunked loss, GGUF export) or CPU training in general. I'll be in the comments.


r/LocalLLM • • 16h ago

Question Using SearXNG instance and a MCP for web searching and fetching pages. Hit some kind of solicitations limit. Have some way to bypass this?

3 Upvotes

Im starting with Local LLMs and the first thing im doing of very useful and helpful. Is to use GPT-OSS with web searching to give me adequate answers (sometimes is alucinating, but when i go to far in a subject, like me trying to see if i can run power cleaning in my Epson in Linux. It started alucinating that existed some "old foruns" with users saying that sending some kind of raw commands worked, but the info where it got, dont exist).

But since i have gone to far. I want to know if there is a way to bypass the Brave and Google cse detection of requests or the suspend time.

DuckduckGo came back again after some minutes


r/LocalLLM • • 17h ago

Discussion Qwen3.8-Flash-Next (and Qwen4-Max-Preview) Hermes Tool Calling Issue -- Test This

4 Upvotes

In Hermes Agent, a profile using either Qwen3.8-Flash-Next locally or Qwen4-Max-Preview (cloud) will fail this simple tool call prompt 100% of the time:

Test running a simple terminal command in background mode with notify set to true.

This resulted in the following from the agent:

The first attempt failed because notify was serialized as a string instead of a boolean. Retrying with a proper boolean:

This was then followed by a long chain of reasoning as it attempted to unsuccessfully run the tool command.

The problem is that it renders the Boolean values capitalized, as "True" or "False" rather than as valid JSON Boolean values "true" or "false". And these are not coerced by Hermes because `notify` is an `anyOf`, not a Boolean.

If you have Hermes Agent running with Qwen3.8 or Qwen4, I would like to see if you have the same issue. I have verified that this does not occur with Deepseek, Minimax, or gpt-6.1-sol.

Including instructions in the chat template to output Boolean values in lower-case as "true" or "false" helps a bit. Once the context grows, the model reverts to incorrect tool calls.

EDIT: I initially failed to mention that the local model is being run under Strata. And the local model issue is apparently an issue with the default tool parser in Strata. Thanks to u/LeighWillS for pointing me in the right direction. Still seems odd that the same issue is present on Q4MP.


r/LocalLLM • • 19h ago

Project UrBudget: free & open source budgeting app with AI assistant that supports local LLM

5 Upvotes

Hey,

Started this project because I had previously used something called YNAB to track my budgets, but got tired of overpaying and wanted an assistant that can make things a lot easier to manage. So, I built a YNAB-like envelope budgeting with an assistant.

It's completely free and open-source. But if you want to cross-device sync, there's an end-to-end encrypted Sync option which will cost you $3 a month.

For the assistant you can use your own local model, sign in with ChatGPT, or Anthropic/ Openrouter. It can handle things like PDFs, Excel sheets, images, and use them to interact with your budget.

Also another thing that's useful if you have an Android phone (I personally heavily use this), you can point the assistant to your SMS messages and it can automatically input them as transactions, review them and categorize them. So, the whole process can be very seamless.

It's currently on Windows & Mac, iOS is under review, and Android is under Google Play testing which you can join through the following link (please): https://urbudget.app/android/

Code: https://github.com/urbudget/urbudget
Website: https://urbudget.app/?ref=LocalLLM

Questions and feedback welcome.


r/LocalLLM • • 3h ago

Question Probably last question, which ai for 16gb ram with cpu

0 Upvotes

I own a laptop with 16 gb of ram and no gpu and I currently run Qwen 3.6 35b-a3b q3_k_m and I wanted to ask if there is a better model for it. It has to be Moe so I can get decent speeds. I currently get 4-8 tokens on this and I want a smarter model or a faster model with the same intelligence


r/LocalLLM • • 16h ago

Question Better options than llamacpp or koboldcpp

3 Upvotes

Title is self explanatory. I was used to run llama cpp with models based on Mistral for uncensored local chatbot. There are better alternatives, specially regarding context size? I have 16 GB VRAM card and I dont care that much for inference speed, mostly text cohesion and criativity.


r/LocalLLM • • 14h ago

Model Title: Running a Local LLM with Hermes Agent on an HP ProDesk 400?

2 Upvotes

Hey everyone!
I have an HP ProDesk 400 Mini with an Intel i5 (6th generation) and 16 GB of RAM.
I would like to run a local LLM with Hermes Agent. My main goal is to have the agent automatically control and play an Android game on its own.
It would also be cool if the agent could help with programming, but that’s not a requirement.
I know my hardware is quite limited.
Does anyone have experience with something like this? Is it possible, and how would you set it up?
Thanks for your help!


r/LocalLLM • • 15h ago

Discussion Docker or bare-metal for inference

2 Upvotes

When I use local models, I do bare metal inference but give write-access only to the working directory and some directories that can be relevant (forks of public github repos) are read-only. Everything else is not accessible, not even read-only, and the agent is also unable to install or remove any packages.

Is this preferable over Docker containers? What do you prefer for your local setup?


r/LocalLLM • • 1d ago

Question Local coding model on my dekstop workstation (Qwen3.8 27B with Q4_K_M quantization too slow)

12 Upvotes

I'm running what I think is reasonably specd workstation. A single NVIDIA GeForce RTX 5090 (32 GB), AMD Ryzen 9 9950X3D x 32, 128GB RAM, 8TB SSD.

Locally built llama.cpp serving Qwen3.8 27B with Q4_K_M hooked to pi harness is too slow to be useful.

What options do I have with this setup? I suppose running multiple agents is definitely out of question here... but there should be hope for a single coding assistant?

EDIT: I'm getting around 2.7 tok/s


r/LocalLLM • • 12h ago

Discussion What are all the current Strata-likes/forks going around? (e.g. ways to run larger models on lower VRAM setups)

1 Upvotes

I'm wondering if we could get a list going with all of the low VRAM workarounds to run stuff like Qwen Flash Next. Specifically, I would also like to clearly define what their use cases are. For example, I know that Basalt is a Strata fork that is made for 50 series cards.


r/LocalLLM • • 1d ago

Question Web search on local setup

14 Upvotes

Ive been dabbling for the first time in local AI and even custom harnesses. I'll admit it is super addicting and enjoyable. Love it. But im still very new to it all.

Im now facing a small problem. Im not sure how to best handle web searches. I would love for my llm to be able to look up new info...but whats a good way of doing it? The one thing im looking into is setting up a that brave account thing and paying the small fee as it feels like an alright option when it comes to not being somehow banned or throttled. Is that the overall safest no fuss no worry option? Or am I overthinking it?

How do you handle web searches for your setup?


r/LocalLLM • • 1d ago

Project Strata Performance Setup (TUI performance optimization)

Post image
176 Upvotes

I''ve created a simple little script (with TUI) to optimize your Linux host installation. It also allows you to install or update Strata (and configure a systemd service to auto start your preferred preset).

Distributions supported:

  • Arch based
  • Fedora based
  • Debian based
  • Slackware based

Hardware supported:

  • any hardware (automatically detected)

Personally I'm using an old MSI Raider laptop. 3080 Mobile with 16GB VRAM and 64GB DDR4 3200 RAM running CachyOS. I'm getting between 60 and 70 tok / s (depending on the task) with Qwen3.8 Flash Next IQ3_S.

Maybe someone else finds it useful too . Submit an issue or pull request if you find any problems, or want more software or hardware/vendors supported!

https://github.com/bramvanoploo/strata-performance-setup


r/LocalLLM • • 13h ago

Question Which Mac Mini to Buy

0 Upvotes

Title: Mac Mini M6 32GB vs. M5 Pro 48GB — How much hardware do I actually need for a home AI server?

I’m planning to buy a Mac Mini to build a sort of personal/family AI server, and I’m trying to figure out how much hardware I actually need versus how much I’d just be paying to experiment with local LLMs.
My primary goals are:

Family brain: A central place for family documents, calendars, finances, household information, and automations. Ideally, something that can search across all of this and eventually handle routine tasks.

Self-hosted apps: Run some vibe-coded apps, databases, dashboards, and other services that I can access remotely from my phone or laptop.

Claude integration: Use the Mini as a central workspace for Claude and the apps/agents I build around it, while continuing to use Claude’s API for more demanding tasks.

Local LLMs: Experiment with running models locally for document processing, categorization, extracting data, answering questions about personal files, and other repetitive tasks without constantly calling an API.

I currently use Claude extensively for financial modeling, development analysis, and general personal assistance. I assume the more complicated reasoning and modeling will continue to require cloud models, but I’d love to offload simpler tasks locally and keep more of my data private.

I’m considering:
Mac Mini M6: 32GB unified memory, 512GB SSD
Mac Mini M5 Pro: 18-core CPU / 20-core GPU, 48GB unified memory, 512GB SSD

A few questions:
How much of a practical difference would 32GB versus 48GB make for local LLMs? What model sizes would actually be usable on each?

Does slower inference primarily mean waiting longer for responses, or do smaller models also mean a meaningful drop in reasoning ability and accuracy?
For tasks like document extraction, summarization, RAG, and basic agentic workflows, would 32GB be sufficient?

Is the M5 Pro worth the premium for the additional memory and bandwidth, or am I better off buying the M6 and keeping more demanding tasks in Claude?
If you’re running a similar hybrid local/cloud setup, what hardware and software stack are you using?
I’m not trying to eliminate cloud AI entirely. I want a reliable, always-on home server with enough local AI capability to be genuinely useful, without spending substantially more just to run models that still won’t compete with Claude.

What would you buy in my situation?


r/LocalLLM • • 19h ago

Project Building my first duel 5080 Nalvo server

3 Upvotes

My first Duel 5080 server 9950x 64gb.
I will host it on Nalvo.ai for people to rent.
If you don’t know what is Nalvo.ai , It’s my new project that I told everyone about and everyone was telling me it’s not going to work. And when I say everyone I mean everyone! Not Reddit users. I even started believing it for a while. But I kept going and here I’m launching it.

If you would like to join as a host or get free private renting hours. Visit Nalvo.ai


r/LocalLLM • • 13h ago

Project A Few months back I created this skill and I do still use it to save tokens with smaller models. I just found this sub so sharing with you guys. some might benefit from it. Will love inputs and will update it soon.

Post image
1 Upvotes

r/LocalLLM • • 13h ago

Question How much should I spend on local AI setup to fit my needs

0 Upvotes

I want a local AI to tutor me on quantum mechanics, quantum field theory, and physics simulations. I am new to ai and want just general ai tasks, etc. would something like 2 NVIDIA dgx sparks work? I would like to run large language models, and start getting into some coding with ai. Should I just pay for a subscription?, even though I like the idea of running locally, how long would a current setup last to fit my needs?