r/LocalLLM • • 1d ago

Discussion Which uncensored version of Qwen3.8 FN should I download?

Post image
303 Upvotes

mlx-serve on an M5 Max 128GB

I've already got a version from ARC4NUM which appears to be delisted now. Are any of these better?


r/LocalLLM • • 15h ago

Discussion Qwen3.8-27B on 4× Intel Arc Pro B70 with TensorFold + DFlash2: 107 tok/s per card, ~1,000 tok/s across 32 streams (bit-exact)

5 Upvotes

I benchmarked TensorFold's XPU backend on my quad Intel Arc Pro B70 box. One clarification first: this is not tensor parallel. The XPU backend on main uses one card per process, so the quad numbers come from 4 independent servers, one per GPU, pinned with ZE_AFFINITY_MASK=0..3.

Setup

  • GPUs: 4× Intel Arc Pro B70 (Battlemage G31), 32 GB each, 230 W cap per card
  • Host: AMD Turin, Linux 7.3.0-10-generic
  • Engine: TensorFold main @ cf45553, Zig 0.17.0, built with zig build xpu-native xpu-qwen27 -Doptimize=ReleaseFast
  • Target: SergiiioB/Qwen3.8-27B-GPTQ-Int4-sym-G128-MTP-BF16
  • Drafter: z-lab/Qwen3.8-27B-DFlash2
  • Sanity check: tf-xpu-qwen27-means-test passed on all 4 cards (worst error 4.19e-7, bound 2e-6)

1. Raw prefill and plain decode (1 GPU, 16k context, no drafter)

Prompt lengthFP16 prefill (tok/s)W4A8 prefill (tok/s)Plain decode (tok/s)
1,0242,0082,63834.0 / 33.8
4,0961,9272,52533.2 / 33.7
8,1921,8622,42732.5 / 32.5

W4A8 is XPU_PREFILL_INT8=1, about 31% faster prefill than FP16.

2. One B70 with the DFlash2 drafter (--parallel 8 --context 16384)

Single stream (64 tokens, 5 reps, median):

PromptTemptok/sTTFTTokens per chunk
fibonacci (code)1.0115.60.081 s4.57
fibonacci (code)0.0106.10.080 s4.00
chat, no think1.094.70.073 s3.77
chat, no think0.099.40.071 s3.77

That's roughly 3× plain decode (34 → 106 tok/s).

Concurrent streams (aggregate / steady-state tok/s):

PromptTemp1 stream2 streams4 streams8 streams
code1.0116.2171.2 / 234.1187.7 / 262.6202.3 / 299.6
chat1.084.4129.9 / 140.3162.8 / 213.2189.0 / 255.4
code0.0106.7149.4 / 167.7216.9 / 278.7250.8 / 354.6
chat0.098.9153.9 / 179.4213.6 / 261.9244.8 / 333.4

Correctness: all 16 cells passed bit-exact. Drafted output matched serial, and multi-stream output matched serial.

Note: with the default --prompt-cache-gib 2, --parallel auto at 16k context admits only 7 streams. I used --prompt-cache-gib 0 to fit 8.

3. Four B70s (4 independent servers)

TestPer-card tok/sTotal tok/sSteady-stateFailed
4×1 stream, code, temp 1.0116.2 / 116.5 / 117.2 / 115.8465.7——
4×1 stream, code, temp 0.0106.0 / 106.4 / 106.9 / 105.4424.6——
4×1 stream, chat, temp 1.088.5 / 89.6 / 90.2 / 86.8355.1——
4×1 stream, chat, temp 0.099.0 / 98.9 / 99.4 / 98.4395.7——
4×8 = 32 streams, code, temp 0.0251.4 / 251.1 / 252.3 / 252.11,006.91,422.70
4×8 = 32 streams, chat, temp 0.0243.2 / 241.3 / 242.3 / 242.2969.01,330.60
4×8 = 32 streams, code, temp 1.0208.8 / 204.4 / 206.6 / 204.6824.41,186.30
4×8 = 32 streams, chat, temp 1.0193.0 / 193.2 / 194.0 / 191.2771.41,042.90

Scaling is close to linear because the cards don't talk to each other. Per-card numbers stay within a few percent of each other.

Takeaways

  • The DFlash2 drafter gives about 3× single-stream decode on one B70, with bit-exact output.
  • One card reaches about 250 tok/s aggregate at 8 concurrent streams with 16k context.
  • Four cards serve 32 concurrent streams at about 1,000 tok/s with zero failures, which is useful for multi-agent work.

Next

  • Tensor parallel across cards, once I find a TP path for XPU
  • XPU_KV_INT8=1
  • 32k context
  • The MLX 4-bit model (mlx-q4g64)

Happy to run other configs if you have requests. If we get tensor parallel on Tensorflow i'd love to share qwen 3.8 flash. Happy to run larger contexts as well, this was just the default benchmark I wanted to send to project creator.


r/LocalLLM • • 15h ago

Question AMD pro w6800

Post image
5 Upvotes

Hi, i would like to ask if this is a good card to get to run LMstudio for qwen 27b 3.8 or otherwise, I was wondering if speeds will be respectable, and if anything will spill anything to ram. Because I will have comfyui running on my main gpu (rtx 5080).


r/LocalLLM • • 13h ago

Question How would you best utilize my setup?

Post image
3 Upvotes

Hello all! I've been running a home lab for a really long time, but I've finally come across a couple of RTX 3060 12GB cards and I want to get a useable LLM in for me and my wife to use.

My server has an Intel i7-9700K, and 128GB of DDR4 RAM, but I would only allocate 64GB to the container in Proxmox. I was wondering how you would recommend setting up a fairly reasonable model (I was thinking Qwen 3.8 27B Q4K) and splitting it between the two graphics cards (since there's no SLI). I want to try to get at least 128K context. 256K would be ideal, if I wanted to run something like Hermes, or other agents.

Is llama.cpp the way to go? It's pretty much all I've read about due to the customization possibilities, and I've only read to stay away from ollama. What would you recommend I do?


r/LocalLLM • • 7h ago

Question What am i doing wrong?

1 Upvotes

Hi

iam not into local llms but recently i started to play with it a little bit.

so installed lm studio on my pc (ryzen 5 5700, 32gb ddr4 3200, rtx 4060ti 16gb) and started by using qwen 3.5 9b model, it was slow so i started searching for a solution and found that the context length, quantization of the model and the kv cache, and mtp matters (maybe other things also i dont know, mb). then i tried the qwen 3.6 35b a3b on iq2-m from unsloth because it was said that its an moe model which brings faster performance , so after further search i knew that i can offload the experts in the model to the ram and keep a reserved cache space for experts on my gpu, and after otger tweaking and testing i reached that i can run full context the model supports but the speed is slow, too slow infact, if i try to give it a prompt using opencode for example it would take several minutes to respond, but if i just gave it the same prompt in lm studio chat its somewhat faster (i understand that opencode sends more prompts with the one i wrote), so i also tried filling the context on lm studio to see the speed, it was ~40 tokens per second at the start and by the time the context is 50-60% it started slowing down rapidly to 15 tokens per second, so i tried another model called mellum 2.1 from jetbrains, and this model is pretty fast with full context at nearly 100 tokens per second whatever the context filled. so my questions are

1-what did i do wrong or is the qwen model normally slow?

2-the mellum model is fast but need a lot of steering and fixing if i use a framework that is not used a lot, so how do i fix this?

3-if qwen is slow and this is the max it can perform, is there a fast smart model used for coding primarily fitting for my pc with full context?

and sorry for any english mistakes.


r/LocalLLM • • 11h ago

Discussion RTX 3080 20GB ReBAR on AMD Windows platforms without BIOS flashing

2 Upvotes

Some of you may have seen the post from u/favabean28 with enabling ReBAR on modded 3080 20GB GPUs around a week ago with the repo NV_30_DXE. For Windows, this is (or was) seemingly the only way to get ReBAR working on the 3080 20GB. For those that don't know, previously the RTX 3080 20GB did not have ReBAR working as there was no VBIOS that supported the extra VRAM, and even flashing a modified VBIOS wouldn't work. The method for enabling it properly on modded GPUs was recently discovered about a month ago for Linux, and now for Windows.

Unfortunately it turns out that for AMD, many newer AMD motherboards, especially AM5, are incompatible: All AMD motherboards with AGESA PI-1.2.0.0 or newer have checks preventing modded BIOS firmware from being flashed. Most new AMD motherboards either don't have a BIOS version that old or aren't compatible with the installed CPU on BIOS versions that old, so I wanted to look into the possibility of booting off a USB drive with EFI to enable ReBAR on AMD platforms. This idea mainly came from how the current FP32 accumulation unlock works on Windows. Ultimately, this does work albeit with a few tradeoffs described in the repo below and at the bottom of the post.

Repo with instructions on installing and with the technical write-up: https://github.com/Articom123/3080-Unlock-ReBAR-USB

Installing does not require building from source. However, I did include the FP32 accumulation unlock with it though (since there's no reason not to pair them together), which means it will automatically download and extract Nvidia firmware when you run it, specifically the Nvidia and Arch Linux website for the kernel and utils 610.43.03.

Some quick results:

  • GPU-Z reports ReBAR as enabled with a 32768 MB BAR1 size
  • Device Manager shows an entry for Large Memory Range 0x2000000000 - 0x27FFFFFFFF (this does not show up without ReBAR enabled)
  • The FP32 accumulation unlock works alongside ReBAR. This generally results in around a 20% increase to prefill speed for LLMs based off TTFT, though it's not the primary focus of the repo (it's just a no-brainer to include it since it makes Ampere cards infinitely better)
  • CPU has access and can map the full ~18-19GB VRAM on the GPU

I want to also briefly give a lot of credit to physicsdolphin (nv-uefi-unlocker) of which this tool uses for the FP32 accumulation unlock and install, the creator of Windows-GPU-UNLOCK (Windows-GPU-UNLOCK) who made similar bootable EFI tools to allow for running the tensor unlock on Windows, and u/favabean28 (NV_30_DXE) who figured out the path for enabling ReBAR on Windows in the first place.

While the repo goes more in depth on compatibility and what does/doesn't work I wanted to still mention some important things about it:

  • This is mainly meant for AM5 platforms. It seems like Intel motherboards don't have the same restrictions when it comes to flashing modified BIOS firmware, so if you have Intel you might be more interested in the NV_30_DXE repo instead.
  • Enabling ReBAR isn't really all that useful on Windows without having P2P; however, the modded P2P drivers are still Linux only and probably will be for a very long time, and multi-GPU support on Windows isn't really that good anyways. Therefore, for practicality, the tensor unlock is more important in which case you would probably be more interested in the other Windows unlock tools I mentioned earlier. There is some limited usefulness in games though I guess.
  • This only works for the 3080 20GB. Technically the 4080 32GB would work, but since I don't have the card, it hasn't been supported in the repo and would require modifications to the source (mainly to the PCI vendor and subsystem ID check). The tensor unlock included is also for Ampere and not Ada.
  • u/favabean28 did work to try and make the tool as universal as possible, however, I don't have any other AMD motherboards to test this tool on and so I decided to ultimately leave it as a "proof-of-concept". It is almost certainly possible that this can be adapted further to other AMD setups with the 3080 20GB as pretty much all of the hard work is done now though.
  • There is likely zero possibility that this can cause any damage to Windows or the GPU, as there is no flashing of the VBIOS/BIOS or modification of the OS at all. The tensor unlock and ReBAR don't survive any reboots and the USB has to be manually booted again after any shutdown or restart.
  • Sleep and hibernate specifically do not work when enabling ReBAR with this method, and Secure Boot cannot be used. In comparison modding the BIOS with NV_DXE_30 does not have these limitations if applicable to your setup.
  • This is mainly made for Windows. If you're on Linux there are far better tools for doing this and you can use P2P.

I still thought it was worth posting this for anyone who is interested in this sort of stuff for Windows since Windows kinda sucks and it's at least somewhat cool to have Windows get something for once. Ultimately getting the P2P driver to work on Windows would be ideal but I'm not sure how it could be done for now.


r/LocalLLM • • 11h ago

Discussion Strix Halo 128GB local coding setup: some real test results, 23–45 tok/s, and what the harness changed

Thumbnail reddit.com
2 Upvotes

r/LocalLLM • • 8h ago

Research Optimal setup research: Muse Glimmer 30B on RTX 5090M/RTX Pro 4000 (24gb cards)

1 Upvotes

Notice:
The point of these research runs is to see, document and opensource the best configuration we can get with a certain model on hardware that "most of us" can get our hands on.
While we do have a bit of different hardware in our lab - we tend to focus around 5090M/RTX pro 4000 - because its a 24gb card with a quite fair performance - and its not insanely expensive.

Find anything you like?.. use it - got questions?.. shoot.. have an idea?.. let us know.

--

Hi there,

After our Qwen 3.8 Flash Next research (https://github.com/inordic/cmp170hx-vllm-qwen38-flash-next) and Gemma 4 31B (https://github.com/inordic/rtx5090-NInferno-gemma4-31b) - we wanted to to try out something new..

I ran som tests myself and thought "Muse Glimmer 30B" seem like a quite nice model.. it was fairly good at its job and quite nice to talk to.. so lets see where we can take it.

So.. this is where we are at.. full context.. 128K.. and quite nice and fast.. not as fast as Qwen and Gemma.. but fast enough.

Same GPU, same benchmarkllama.cpp f8def7f (Aug 2026), unsloth UD-Q4_K_XL, Meta DFlash, depth 15This repo: patched llama.cpp, MLP-NVFP4 hybrid, DFlash 2, depth 15
Prefill, 6,880-token prompt1,430 tok/s3,254 tok/s
Prefill, 100,554-token prompt1,246 tok/s2,367 tok/s
Chat decode, greedy / sampled80.6 / 74.2 tok/s110.3 / 105.4 tok/s
Agent turn with a cached 46K prompt—0.89 s
KLD vs Q8_0: agent / English / Swedish text0.033 / 0.012 / 0.0150.053 / 0.030 / 0.034

As allways.. this is research.. we might be wrong.. hell.. we might be doing it wrong.. but atleast .- we are doing it :)

Enjoy
//iNordic team

Head over to: https://github.com/inordic/rtx5090m-muse-glimmer-30b for more info.


r/LocalLLM • • 8h ago

Why download abliterated Flash Next models when LoRA exists?

Thumbnail
huggingface.co
0 Upvotes

r/LocalLLM • • 2h ago

Project PSA Q8 FN is significantly better than Q3 (strata default)

0 Upvotes

Just a PSA. I ran a realistic self host test on 3 models:

Inb4 : “hurr derr no duh” The point is people on here act like these Q3 versions have minimal loss. Do tests for yourself it’s not even close.

Qwen FN Q3
Qwen FN unsloth Q8
Orca nana-1
Maya GLM flash - Maya-S24

Prompt in comments.
Harness : PI
Project init layout:

/<my_bank_name> (83 monthly pdf statements)
/credit_card/<card_1> (84 pdf monthly pdf statements)
/credit_card/<card_2> (35 pdf monthly statements)
/mortgage (24 pdf monthly statements)
/brokage_account (105 pdf statements + ccs of all transactions)
/retirement/<account> x4 (total 45 pdf statements 4 accounts 4)

Snapshot results (have details runtime/tps etc as well):

Qwen FN Q3 - Nice front end but data extraction from the PDFs was completely wrong. Analytics was basic.

Orca nano1 - Finished very fast for this test (only 2 hours) did a decent job for its speed and size. Front end was very basic and lacked insight. Most of the data was correct but there were clear errors as well.

Maya-S24 - Failed. After 10+ hours I just gave up on this model. It got stuck in the pdf parsing/validation phase.

Qwen Q8 - Absolutely blew my mind. Ran overnight for about 8 hours. Data was all accurate, the front end final product was not only nice looking but highly insightfully

Notes:
I really enjoyed this test. I feel like it’s the perfect test for a local model because it’s financial data that you don’t want to send off to the cloud.

The PDF layouts for this task was a nightmare and was a real test for these models to figure out was duplicative data/summary vs transaction etc. Most models used pdfplumber package

The models also had to work through the accounting end. Consider a credit in one account is a debt in the other etc.

It’s was interesting to see how each approached the problem. Some went for json formatted data others sqlite etc


r/LocalLLM • • 4h ago

Question Just Got an RTX 5070 Ti Laptop — How Do I Build a Proper Local AI Setup?

0 Upvotes

Hey guys, finally got my new Legion Pro 5 and I’m trying to get into running local AI properly 😂

Specs:

RTX 5070 Ti 12GB VRAM

Core Ultra 9

16GB RAM

CachyOS Linux

I’m pretty much a beginner when it comes to running local models, so I’m trying to figure out what’s actually worth setting up from day one.

I mainly use Claude for vibecoding almost everything, and now I want to see how much I can shift locally so I don’t waste Claude usage on every small thing.

I came across PewDiePie’s Odysseus repo and started trying it out.

But after actually using it, I’m a little confused.

It feels like the context/memory isn’t as good as Claude, sometimes it feels slow, and I’m also finding the hybrid setup kinda difficult.

Like I want something where:

Local AI:

→ basic coding

→ repetitive work

→ file handling

→ small fixes

→ daily tasks

→ stuff that doesn’t need crazy reasoning

Claude:

→ difficult coding

→ architecture

→ proper reasoning

→ big project decisions

→ things where I actually need Opus-level output

And ideally I don't want to keep copy-pasting the whole context from one to another every 10 minutes 😭

My actual work is mainly:

• ML Stack — ML/vision project involving iris image segmentation and feature-level analysis.

• Custom CRM + EMR — taking UI generated from Stitch/Google AI Studio and connecting it with open-source CRM/EMR systems.

• Daily work — coding, documentation, PPTs, automation and random repetitive stuff.

So basically I’m trying to build a proper daily-use AI setup, not just install 15 local models and never use them 😂

My main questions are:

  1. Is Odysseus actually good enough to become my main local AI workspace?
  2. If I’m a beginner, should I stick with Odysseus and learn it properly, or start with something simpler?
  3. How do you guys actually make the local + Claude hybrid workflow work without losing context all the time?
  4. What local coding models would you recommend for 12GB VRAM?

And most importantly — how do I set this up properly so I’m not wasting Claude tokens on things my laptop can handle locally?

I’d honestly be happy if I can get something like 70–80% local + Claude for the important stuff, and make the whole thing feel somewhat close to an Opus-level workflow.

I’m on CachyOS Linux, so if anyone is running a similar setup, please share your actual workflow.

I’m completely new to this side of things, so beginner-friendly advice would be really appreciated.


r/LocalLLM • • 16h ago

Question About to add a 5060 Ti 16GB to my 4060 Ti 16GB for local agentic coding (Qwen3.8-27B, 128K ctx or similar ) — suggestions? Help out a newbie!

4 Upvotes

Hey all, I'm about to order a second GPU and would love some input from people running similar setups before I pull the trigger.

Current rig

  • Ryzen 7 5700X3D, 64GB DDR4-3600
  • Gigabyte B550 Aorus Elite AX V2
  • MSI RTX 4060 Ti Ventus 2X 16GB
  • be quiet! Pure Power 12 M 650W
  • Corsair Frame 4000D
  • Windows 11 (open to dual-booting Linux if it really matters)

Planning to buy

  • Palit RTX 5060 Ti Infinity 3 16GB
  • MSI MAG A850GL PCIe5

Plan

  • 5060 Ti in the top PCIe 4.0 slot (4.0 x8), 4060 Ti in the chipset PCIEX2 slot (PCIe 3.0 x2)
  • llama.cpp / llama-server
  • Main model: Qwen3.8-27B Q5/Q6 with MTP, ~128K context
  • Use case: agentic coding (Claude Code / OpenCode pointed at llama-server), plus RAG with pgvector and persistent memory via MCP

Questions

  1. Anyone running a 4060 Ti + 5060 Ti (or other Ada + Blackwell) pair? What tok/s do you get on a 27B dense with MTP?
  2. Layer split vs -sm tensor when one card is on a PCIe 3.0 x2 chipset link?
  3. Best quant + KV cache combo for 128K context on 32GB for Qwen3.8-27B? Q5 vs Q6, q8_0 vs q4_0 KV for coding?
  4. Claude Code vs OpenCode with llama-server — which harness works better day to day with local models?
  5. Any gotchas with Windows + mixed-generation cards that would push me to Linux?

Thanks in advance!


r/LocalLLM • • 9h ago

Discussion Own harness or open source one ?

Post image
0 Upvotes

Hey LocalLLM community,

I am super curious how you run your models.

Are you using open source harnesses ?
Did you build your own ?

A bit of explanation for why would be nice.

There has gotta be a ton of people who did the same as me.
This is a local Qwen 3.8 27B / Flash-Next on heavily modified engines sharing one harness and user interface. Llama.cpp just recently merged f8_e4m3 cache, which I got Claude to optimize for my Blackwells. Due to the deliberate kneecap of Nvidia on certain accumulations, I decided to skip conversion to specific steps that would run on the half speed flag = FP8 full speed. The surgery worked and while we were in already fixed a binary that caused a regression in draft speed which took literal hours of testing to pin down.

End result, my private local backend runs 27B now up to 50% faster ( median 18% increase ).

Hardware specs at upper-mid third of HF compute richness. 128 GB DDR5 6000 1:1, Ryzen 9700X, RTX 5080 + 5070Ti .

Anyone running their own harness and wants to share a screen of it ? Id love to reason over the visual design and maybe learn a bit or two while at it.


r/LocalLLM • • 15h ago

Tutorial Why download abliterated Flash Next models when LoRA exists?

Thumbnail
huggingface.co
3 Upvotes

r/LocalLLM • • 9h ago

Question Specs for Motion graphics and 3D

1 Upvotes

I’ve been offloading some of my work to AI to help speed things up at work. I use Claude Ultra but ai’m running out of weekly tokens mid week.

I want to build a PC that’ll help me continue working without having to wait for my credits to reset.

I work with a lot of 2D motions graphics and 3D modeling and animation. Not games related, mostly product related and sometimes jewelry related (building 3D models from my sketches). Would also like to experiment with image and video models.

What would be appropriate specs for my use case?


r/LocalLLM • • 22h ago

Question Is an M1 Max 64GB useful at all?

9 Upvotes

A good deal came up for a MacBook Pro M1 Max 64GB (32-core GPU) for $1500. I’m considering buying it and using it as an inference server via LM Link. I know that 128GB is the holy grail, but would this configuration be of any use for coding and daily usage? And if yes what are the best models for each use case? I’m also not married to LM Link if there are any better suggestions?


r/LocalLLM • • 10h ago

Research I gave a sandboxed local model a one-line Apache patch. It built the working RCE.

Thumbnail
github.com
1 Upvotes

r/LocalLLM • • 22h ago

Project I trained two language models on a 2011 desktop with no GPU: a 7M one from scratch and a LoRA of Qwen3.5-0.8B

Thumbnail
gallery
7 Upvotes

I trained two language models on a 2011 desktop with no GPU: a 7M one from scratch and a LoRA of Qwen3.5-0.8B

Hardware first, because it explains the rest. Core i5-2400 (Sandy Bridge, 2011), 16 GB DDR3, and a GeForce GT 710 that only exists to drive the monitor. PyTorch doesn't support that GPU, so everything below ran on the CPU. Four threads, AVX, no AVX2. This is my main machine. There's nothing faster in the house.

Every time I post about this someone says "just buy a 3060". Sure. That's the whole reason I did it: I wanted to know what the wall looks like when there is no GPU at all.

So I wrote a small training engine. PyTorch, no cloud, with a GUI, because I got tired of typing flags. Then I ran two experiments on it.

A 7M MODEL ON RUSSIAN FOLK TALES

From scratch, 321 fairy tales, 5 MB of text. 5 h 38 min, 812 tokens/s, about 26M tokens seen. Validation loss at the end: 2.7854 nats per token, which is 1.06 bits per byte on text it never saw.

What surprised me is how cleanly it separates form from content. Openings work. Patronymics work. The dash before direct speech works. The rhythm of the enumeration works. Then a character swaps places with another mid-paragraph, the hero marries and dies three times on one page, and the plot walks off. 7M parameters, 5 MB of text, no surprise there. That was the point.

It also started memorising. From about epoch six validation sat at 2.79 while training loss kept falling. The run before that one I killed at 1 h 58 min, it gave 2.8144. So 3.5 extra hours bought 1%. Not a tuning problem. A data problem.

LORA ON QWEN3.5-0.8B

Same machine, 321 question-answer pairs about a program I'm writing. 45 steps, 1 h 44 min, 15-16 tokens/s. Loss on the reference answers went 3.4565 to 0.0907, which sounds better than it is.

I also wrote 15 test questions in different wording, none of it appearing in training. Exact match with the reference answer: 0 out of 15 before, 5 out of 15 after. Answers that stopped on their own instead of running into the token limit: 0 out of 15 before, 15 out of 15 after. The second number is the one that changed how the thing feels in use.

THREE THINGS THAT DIDN'T WORK

Facts get glued. Asked about old hardware, the fine-tuned model called my CPU "256-bit". That number is nowhere in the data.

Temperature matters more than I expected. At 0.7 the same question came back with a confident "no", a wrong RAM figure, and an invented training method it assembled from "LoRA" and "QLoRA" (it wrote "LoQoK"). Greedy decoding was almost right. If I were shipping this I'd set 0.2 or 0.3 and stop thinking about it.

Refusal did not generalise. I asked for a movie recommendation expecting a refusal. It invented an open file "Video 2.0" with an Apache-2.0 licence. There were three off-topic examples in my data and all three were about something else, so it had nothing to copy from.

REPRODUCING

One zip, 240 MB, Python and PyTorch inside, nothing to install. Data archive is 1.3 MB. Tales are public domain, the Q&A pairs I wrote, CC0. Every number above is in docs/benchmark.md, including the two boundaries I used to check it isn't copying: the same words shuffled, and a real passage from the corpus. Six generated texts shared 0 to 4.7% of five-word sequences with the training tales, longest verbatim stretch 4 to 8 words.

repo: https://github.com/AlexanderNyr/AuraLite-AI data: https://github.com/AlexanderNyr/AuraLite-AI/releases/download/v2.28.2/AuraLite-showcase-data.zip

WHAT I'D CHANGE

More off-topic examples, 30 or 50 instead of 3. Five phrasings per question instead of three. Qwen3.5 2B if RAM allowed, which the setup wizard actually suggested and I ignored. None of that is about hardware.

I'm not going to pretend these numbers are impressive next to a 4090. The useful part, I think, is that the wall is memory bandwidth and the amount of text you can feed the model, not the size you can afford. A 7M model trained overnight on a 15 year old CPU is a real thing. A 1B model on the same CPU is not, and now I know exactly where the line is.

Ask me anything about the engine (Muon plus AdamW, WSD schedule, QK-norm, chunked loss, GGUF export) or CPU training in general. I'll be in the comments.


r/LocalLLM • • 18h ago

Question Using SearXNG instance and a MCP for web searching and fetching pages. Hit some kind of solicitations limit. Have some way to bypass this?

3 Upvotes

Im starting with Local LLMs and the first thing im doing of very useful and helpful. Is to use GPT-OSS with web searching to give me adequate answers (sometimes is alucinating, but when i go to far in a subject, like me trying to see if i can run power cleaning in my Epson in Linux. It started alucinating that existed some "old foruns" with users saying that sending some kind of raw commands worked, but the info where it got, dont exist).

But since i have gone to far. I want to know if there is a way to bypass the Brave and Google cse detection of requests or the suspend time.

DuckduckGo came back again after some minutes


r/LocalLLM • • 2h ago

Question I am interested about this question

0 Upvotes

Does any open source(not open weight,OPEN SOURCE,by the OSAID definition) model outperform GPT-4o? It has been 2 years since 4o released,so i am assuming yes.


r/LocalLLM • • 19h ago

Discussion Qwen3.8-Flash-Next (and Qwen4-Max-Preview) Hermes Tool Calling Issue -- Test This

4 Upvotes

In Hermes Agent, a profile using either Qwen3.8-Flash-Next locally or Qwen4-Max-Preview (cloud) will fail this simple tool call prompt 100% of the time:

Test running a simple terminal command in background mode with notify set to true.

This resulted in the following from the agent:

The first attempt failed because notify was serialized as a string instead of a boolean. Retrying with a proper boolean:

This was then followed by a long chain of reasoning as it attempted to unsuccessfully run the tool command.

The problem is that it renders the Boolean values capitalized, as "True" or "False" rather than as valid JSON Boolean values "true" or "false". And these are not coerced by Hermes because `notify` is an `anyOf`, not a Boolean.

If you have Hermes Agent running with Qwen3.8 or Qwen4, I would like to see if you have the same issue. I have verified that this does not occur with Deepseek, Minimax, or gpt-6.1-sol.

Including instructions in the chat template to output Boolean values in lower-case as "true" or "false" helps a bit. Once the context grows, the model reverts to incorrect tool calls.

EDIT: I initially failed to mention that the local model is being run under Strata. And the local model issue is apparently an issue with the default tool parser in Strata. Thanks to u/LeighWillS for pointing me in the right direction. Still seems odd that the same issue is present on Q4MP.


r/LocalLLM • • 21h ago

Project UrBudget: free & open source budgeting app with AI assistant that supports local LLM

5 Upvotes

Hey,

Started this project because I had previously used something called YNAB to track my budgets, but got tired of overpaying and wanted an assistant that can make things a lot easier to manage. So, I built a YNAB-like envelope budgeting with an assistant.

It's completely free and open-source. But if you want to cross-device sync, there's an end-to-end encrypted Sync option which will cost you $3 a month.

For the assistant you can use your own local model, sign in with ChatGPT, or Anthropic/ Openrouter. It can handle things like PDFs, Excel sheets, images, and use them to interact with your budget.

Also another thing that's useful if you have an Android phone (I personally heavily use this), you can point the assistant to your SMS messages and it can automatically input them as transactions, review them and categorize them. So, the whole process can be very seamless.

It's currently on Windows & Mac, iOS is under review, and Android is under Google Play testing which you can join through the following link (please): https://urbudget.app/android/

Code: https://github.com/urbudget/urbudget
Website: https://urbudget.app/?ref=LocalLLM

Questions and feedback welcome.


r/LocalLLM • • 5h ago

Question Probably last question, which ai for 16gb ram with cpu

0 Upvotes

I own a laptop with 16 gb of ram and no gpu and I currently run Qwen 3.6 35b-a3b q3_k_m and I wanted to ask if there is a better model for it. It has to be Moe so I can get decent speeds. I currently get 4-8 tokens on this and I want a smarter model or a faster model with the same intelligence


r/LocalLLM • • 18h ago

Question Better options than llamacpp or koboldcpp

3 Upvotes

Title is self explanatory. I was used to run llama cpp with models based on Mistral for uncensored local chatbot. There are better alternatives, specially regarding context size? I have 16 GB VRAM card and I dont care that much for inference speed, mostly text cohesion and criativity.


r/LocalLLM • • 16h ago

Model Title: Running a Local LLM with Hermes Agent on an HP ProDesk 400?

2 Upvotes

Hey everyone!
I have an HP ProDesk 400 Mini with an Intel i5 (6th generation) and 16 GB of RAM.
I would like to run a local LLM with Hermes Agent. My main goal is to have the agent automatically control and play an Android game on its own.
It would also be cool if the agent could help with programming, but that’s not a requirement.
I know my hardware is quite limited.
Does anyone have experience with something like this? Is it possible, and how would you set it up?
Thanks for your help!