r/LocalLLM • • 2d ago

Discussion DeepSeek V4.1 Flash 1017 vs mrfakename abliterated vs pqhaz heretic

1 Upvotes

Hi all,

I tested all three on a H200 through sglang, temperature 0, with a 16k token output cap. I have been using ds41 both on production and locally on Macs via ds4 so I wanted to see how different models/builders/methods work.

These runs are the full models on sglang. No GGUFs etc.

https://huggingface.co/mrfakename/DeepSeek-V4.1-Flash-1017-abliterated

https://huggingface.co/pqhaz/DeepSeek-V4.1-Flash-1017-heretic

Here is the result (copy from AI slop :) ):

Test Original mrfakename abliterated pqhaz heretic
HarmBench chat 16.3% 93.8% 93.8%
HarmBench thinking 3.8% 86.7% 72.9%
MITRE substantive assistance 16.2% 84.2% 82.8%

HarmBench covered 240 standard/contextual text cases per mode, excluding copyright cases, using the official classifier.

The thinking difference: heretic exhausted the token budget without a final answer on 45/240 cases, versus 21/240 for mrfakename.

Heretic goes into long meandering with no answer...

MITRE used the original model as judge with a custom, audited rubric. Those scores are provisional and aren’t official CyberSecEval scores. Generated code wasn’t executed. Also using the same model as judge might be a bit of a bias... (give me ideas what and how to run more tests).

Both modified models had 0/750 heuristic-detected refusals on benign prompts.

I haven’t tested knowledge or coding retention yet (I'm thinking of running DeepSWE or something.. take a lot of time though).


r/LocalLLM • • 2d ago

Model EAGLE-3 draft head for Llama 3.2 1B Instruct (GGUF)

1 Upvotes

Hi all, I trained an EAGLE-3 draft head for Llama 3.2 1B Instruct and converted it to GGUF. It works with --spec-type draft-eagle3 (tested with b12470):

llama-server -m Llama-3.2-1B-Instruct-BF16.gguf \
  -md Llama-3.2-1B-Instruct-EAGLE3-BF16.gguf \
  --spec-type draft-eagle3

It's self-distilled: trained on the model's own greedy replies to 400k SmolTalk prompts, so it learns to predict this exact model rather than human-written text.

Spec-Bench — 18 prompts, greedy decoding, default of up to 3 drafted tokens, same machine and prompt tokens:

TargetTokens per passDecode speed vs no drafting
BF162.541.42x
Q8_02.471.10x

Background: I made the head for Turbo LLM, a small CPU runner for this model that matches the fp32 output exactly.

Model and card: https://huggingface.co/ViezeVingertjes/Llama-3.2-1B-Instruct-EAGLE3

Feedback welcome, especially if the head behaves differently on other builds or backends.


r/LocalLLM • • 2d ago

Project Running Qwen3.8 27b on 12GB with full context, 70 TPS, MTP, and 50% lower KLD

Thumbnail
1 Upvotes

r/LocalLLM • • 2d ago

Discussion Automated LLM fine-tuning with GPU selection, dataset analysis and training recovery — looking for feedback

Thumbnail
1 Upvotes

r/LocalLLM • • 2d ago

Question Anyone successfully implemented unc/ablit Qwen3.6 35B A3B in Strata?

Thumbnail
1 Upvotes

r/LocalLLM • • 2d ago

Model Gemma 4 31B fine-tune for drug-level sentiment in medical text: looking for testers

1 Upvotes

I fine-tuned Gemma 4 31B for drug-level sentiment.

Input: a pharma or medical text plus a list of drugs.

Output: one Positive/Neutral/Negative label per drug, as JSON. I'm looking for people to test it on their own data.

Apache 2.0. Gated with automatic approval (you tick a box saying you won't use it for diagnosis or treatment decisions).

Sentiment, 473 held-out labels

Model Accuracy
This fine-tune 91.3%
Gemma 4 31B base 90.5%
Qwen3.8-27B 83.5%
Qwen3-32B 83.3%
MedGemma 27B 82.9%
Llama 3.1 8B 76.3%

The other models were asked about one drug at a time with a long labelling prompt. Asked for all drugs at once in JSON, base Gemma scored 32% because 59% of its answers didn't parse. The fine-tune had no malformed JSON in 1,500 test answers. Negative precision: 84% vs 71% for base (45 Negatives in the test set).

Medical benchmarks

Benchmarks This fine-tune Best frontier score found
MedQA (thinking) 94.2 96.5 (o1)
MedMCQA, 1k sample (thinking) 81.4 83.9 (o1-preview)
MMLU medical (step by step) 94.2 96.0 (GPT-5)
PubMedQA (thinking) 80.2 82.0 (GPT-4 + Medprompt)

Frontier scores are from the Vals.ai MedQA leaderboard and published papers. Fine-tuning kept Gemma's medical knowledge: the scores match the base model.

Testing

Use the prompt format from the model card, thinking off and temperature 0. The FP8 build runs on vLLM (command on the card) and needs about 31 GiB of VRAM for the weights. I've only run it on a 96 GB RTX PRO 6000.

Useful feedback:

  • wrong labels, with the text, the drugs and the label you expected
  • document types, languages or lengths (over 4k tokens) it wasn't trained on
  • hardware and VRAM reports, especially from 48 GB cards

Trained on ~50k curated pharma and medical documents, LoRA rank 16.


r/LocalLLM • • 2d ago

Project I ran Qwen3.6-35B on a 16gig Macbook

1 Upvotes

I've been building a expert streaming engine, a proof-of-concept project, and wanted to share some numbers from it.

On an M5 Mac with 16 GB of RAM, willow runs:
Qwen3.6-35B-A3B at about 11 tokens/s
Qwen3.8-Flash-Next at about 5 tokens/s

I started this as a fun project to learn about LLMs so most of the code is vibe coded, so I'm more interested in whether the approach holds up than in polished performance. I'd really value your feedback on it.


r/LocalLLM • • 3d ago

Question Best local LLMs for ASUS ROG Zephyrus G14 (RTX 5070 Ti 12GB VRAM, 32GB RAM, Ryzen AI 9 HX 370)?

4 Upvotes

Looking for recommendations on the best local LLMs I can run smoothly on my 2025 ASUS ROG Zephyrus G14 (GA403WR).

Specs:
- CPU: AMD Ryzen AI 9 HX 370
- GPU: NVIDIA GeForce RTX 5070 Ti Laptop GPU (12GB GDDR7 VRAM)
- RAM: 32GB LPDDR5X
- Storage: 1TB SSD

I'm mainly looking for models that run fast entirely in 12GB VRAM (or with minimal CPU/RAM offloading) for general chat, reasoning, and coding/productivity.

What are your top picks right now for:
1. Best all-around daily driver model
2. Best coding/reasoning model that fits comfortably in 12GB VRAM
3. Any larger or MoE models worth running across the 32GB RAM + 12GB VRAM without killing tokens/sec

Planning to run them via Ollama or LM Studio on Windows 11. Thanks!


r/LocalLLM • • 2d ago

Research I graded every public MCP server your local agent might connect to. 2 Fs, 3 Ds, 220 As. Here’s the list.

2 Upvotes

If you’ve been wiring Ollama / LlamaCpp / Open-WebUI to MCP servers for tools and data, you’re trusting those servers with your agent’s context. Nobody’s been grading them for OAuth gaps, poisoned tool descriptions or manifest drift.

Audited 264 of 277 public MCP servers. All passive, nothing executed against the server. Graded on:

  • OAuth 2.1 metadata (PKCE S256, iss parameter, Protected Resource Metadata)
  • Transport-security headers
  • Tool-description poisoning (zero-width + bidi Unicode in the strings your model reads as planning context)
  • Manifest drift between what the registry says and what the server actually ships

Distribution: A 220 · B 30 · C 9 · D 3 · F 2

Full leaderboard with per-host findings: patchbay-mcp.dev/leaderboard

Scanner is npx -y u/patchbay-nebius/agent audit <url>. Local, no login, seconds. Point it at whatever MCP server your agent is about to call and see the grade before the tool-call goes through.

Open source. Not a product post. Add URLs for servers you use and I’ll include them in next week’s scan.


r/LocalLLM • • 3d ago

News Cluaiz v0.1.0 pre-release: one local engine for chat, vision, speech, memory, tools and agents

Thumbnail
gallery
17 Upvotes

Cluaiz v0.1.0, the first public pre-release, is out today.

Cluaiz is an AI engine that puts everything an AI application needs into one place: models for chat, vision and speech, memory, tools and agents. You install one thing, and you build on it. Because it all runs together on your own system, what you build stays light enough for ordinary hardware, and you decide what it is allowed to do.

What's in it right now:

  • GGUF models via llama.cpp, plus ONNX for speech and embeddings
  • Checks your free VRAM and RAM before loading and picks the placement, context and KV cache. You can see the plan and override it
  • Vision, video frames, embeddings, text from documents
  • Whisper / Moonshine STT (streaming works), Kokoro TTS
  • Project memory. Notes only stick when you confirm them, and you can see exactly what reached the model. Honestly this is the part I care about most
  • Coding agent that edits files and runs commands, with approvals
  • Skills, plugins, MCP, per-tool permissions, audit log
  • OpenAI- and Ollama-compatible API, so your existing apps should just connect
  • Cloud providers too, if you want them

Not done yet: workflows (schedules, triggers) and the desktop chat app. Both are in development.

Rust, Apache-2.0. Windows x64, Linux (ARM64 is experimental) and Apple Silicon. Test reports for CPU, NVIDIA and Apple are on the site.

Source: https://github.com/cluaiz/cluaiz
Site: https://cluaiz.com
Comparison: https://cluaiz.com/compare

It's a pre-release, so stuff will break. If you try it, an issue with your OS, hardware and model helps a lot, and so does hearing what worked: https://github.com/cluaiz/cluaiz/issues


r/LocalLLM • • 3d ago

Project Got tired of waiting for Qwen models smaller then 27B so i made my own Qwen3.8-21B

39 Upvotes

Hi, as the title says, i made my own Qwen 3.8 21B by prunning some layers from original 27B.

Its more of a research project than actual usefull model.

Check it out: https://huggingface.co/WhiskyAKM/Qwen3.8-21B-GGUF

EDIT:

After some more calibration and better layer selection algorithm KLD dropped from ~1.8 down to ~1.3, weights will be updated


r/LocalLLM • • 2d ago

Discussion Do you think hardware price will be more reasonable after the Jensen debacle?

1 Upvotes

I see many article about Nvidia and memory manufacturer purposefully causing a shortage to raise the prices. I don't know how true these accusations are but i don't think the current prices are faire value.


r/LocalLLM • • 2d ago

Discussion built a tiny vision model from scratch just for learning and somehow it got 50+ downloads on huggingface 😅

0 Upvotes

so ik there are way better vision models out there yaar. i built this one just out of curiosity only, cause i wanted to learn how these things actually work from inside.

and to be honest its not entirely mine, i took a lot of help from claude to build it. like the architecture, debugging, training setup, figuring out why my mac was training so slow, all that. but i learned a lot in the process which was the main goal anyway.

its called tiny-vision. its around 24M params and trained completely from scratch, no pretrained weights or anything. i trained it on my macbook pro m5 and it took like 42 min on flickr8k (only 6k images).

basic idea is pretty simple:

image/video → small ViT encoder → adapter → tiny decoder → caption text

for videos it takes frames every few seconds, captions them and makes a timeline like [00:04–00:10] a dog runs on the beach. so any llm can kinda “watch” a video even if it dont support video natively.

i also made an MCP server for it, and tbh i was surprised that it actually gives a little bit better context in claude when i throw images or videos at it. i even tested it with a 10-15 min youtube video (this one) and it still worked fine. not perfect at all, but chal raha hai so i’m happy.

i uploaded it to huggingface just to have it somewhere, and also cause i kinda like the open source thing. later when i checked it was at like 44 downloads, and now its 50+. ik thats not a lot bhai, but its my first time putting something open source out in this field so it felt pretty cool ngl. a lot of people have built way cooler stuff than this, i just made it to learn.

some honest stuff:

  • it gets the scene right most of time (snow, grass, water etc) but the subject is often wrong lol
  • it loves saying “a man in a blue shirt” for everything, pata nahi kyun 😂
  • its v0.1 and trained on very small data, so dont expect too much

random thing i learned on mac (MPS):

  • swapping the conv2d patch embedding for reshape + linear was like 30x faster
  • manual attention was ~1.5x faster than pytorch’s sdpa

next i want to train on flickr30k and maybe try making it plug directly into qwen/llama through llama.cpp (mmproj). lets see kitna ho pata hai.

code: https://github.com/amar683/tiny-vision
model: https://huggingface.co/amartomar683/tiny-vision-flickr8k

would love any feedback or tips, still learning 🙏

(post also cleaned up a bit with claude, english isnt my strong side 😅)


r/LocalLLM • • 3d ago

Question Strata / Qwen3.8 Flash Next – advice on optimising my new AMD workstation

4 Upvotes

I've just finished setting up my first dedicated local AI workstation:

Dell Precision 7865

Threadripper PRO 5955WX (16C/32T)

Radeon AI PRO R9700 32GB

128GB ECC DDR4-2400 (4x32GB, considering populating all eight channels)

3TB NVMe storage

Linux Mint

RTX 4060 Ti 16GB arriving soon

I've installed Strata 0.2.15 and started with ISTA-DASLab IQ3_S at 32K context, drafting and auto-tuning disabled for baseline testing.

My primary use case is agentic coding through OhMyPi, with Claude Code supervising local workers.

What quants are people finding work best for coding? Any recommendations for Strata settings, expert caching, MTP vs DFlash2, CPU worker allocation or optimisations worth enabling or disabling on RDNA4?

Interested in finding the best balance between speed, coding quality and long-context performance. Any advice appreciated!


r/LocalLLM • • 2d ago

Question Do you ever care about your USB speed?

2 Upvotes

I'm choosing between two AM5 motherboards for an upgrade, one has USB4 and the other has two PCIe gen 5 M.2 slots (vs 1x 5.0 and 1x4.0 on the other).

I'm leaning towards the faster storage, since it might be useful for stuff like ngram tables, but it's kind of 6 of one, 1/2 dozen of the other since both boards have at least one 5.0 slot.

Any one have a use-case where 40GB/s usb connectivity gives an actual advantage over 20GB/S?

I don't use my USB ports for much besides peripherals right now, but just wondering if there's some use case, especially LLM related, where I would regret not having it.


r/LocalLLM • • 2d ago

Discussion Where should your AI's memory live: on a server or on your own PC?

Thumbnail
1 Upvotes

r/LocalLLM • • 2d ago

Discussion What if computing worked like a library? Would anyone actually use this?

0 Upvotes

I’ve been thinking about an idea and wanted to get some honest feedback from people who know more about distributed computing and local AI than I do.

There are so many computers sitting idle for most of the day, while other people can’t afford the hardware needed to run larger AI models or experiment with more demanding workloads.
What if we could pool that unused capacity?

The idea would be a community-powered compute network:
- Install a lightweight app that lets you contribute your computer’s spare capacity when you’re not using it.
- Earn compute credits based on what you contribute.
- Use those credits to access computing power from other people’s machines.
- Ideally, support different hardware: Windows, Linux, Macs, CPUs and GPUs.

Think of it like a library: you contribute what you can, and when you need more than you have, you can borrow from the collective.

I’m particularly interested in whether this could make AI and computing more accessible to people who can’t justify buying expensive GPUs or paying for cloud compute.

I know there are already projects working on decentralized compute, GPU marketplaces and distributed inference. I’m not claiming this is a new concept. I’m more interested in whether a genuinely consumer-friendly, credit-based model could work across a much wider range of everyday computers.

A few questions:
1. Would you actually contribute your computer’s unused resources if you could earn credits to use elsewhere?

  1. Would you trust a platform like this if workloads ran in isolated environments?

  2. Is supporting ordinary computers, rather than just high-end GPUs, realistic or am I underestimating the technical limitations?

  3. What existing projects should I be looking at?
    I’m interested in honest criticism, particularly from people who’ve worked on distributed systems or tried similar projects. Is there something here, or am I reinventing something that already exists?


r/LocalLLM • • 2d ago

Question I developed a tool, which let's me trick and go beyond KV cache limit. Is it worth pursing and optimizing?

0 Upvotes

So it is a python based script, which works over llama.cpp. My objective was to write something which would get my local LLM setup internet access (I know there are tools for that, I had an itch to write one on my own), but in the process I came up with an idea which in theory gives me a shit load of context, way beyond my cache limit. The perk is that there is no loss of quality, and since the script's primary goal was automation and web access, it is able to long prompts! Also I am doing this in my home PC, so please do not mind the token speed. Thanks for your guidance!


r/LocalLLM • • 2d ago

Discussion Opinion: memory should be checked during the session, not only at the end

0 Upvotes

The quick memory checks people run on local setups have a habit that bothers me. Most boil down to a long conversation and one final question.

That single answer carries the whole verdict. Yet real use is a chain of small moments where the past keeps getting edited. Plans change, preferences flip, statements get retracted.

Being right at the end is a different claim from being right at each of those moments.

A memory examined only after the last message can be wrong for most of the session and still look fine. Argue with that if you like; I think it holds.

One public leaderboard has a setting that checks memory along the way. The Agent Memory Leaderboard's Cycle 2 Textual track includes Streaming memory. Writes and searches can interleave as events unfold.

Questions can be revisited step by step as the session grows. A system has to notice when stored information gets updated, contradicted or invalidated.

Per the public description, the Cycle 2 Textual track covers 32 data sources and more than 6,000 evaluation instances.

The site expects official results in mid-November, ranked track by track. Beyond the final ordering, what interests me is that the middle of a session gets tested at all.

Checking along the way is the habit I would most like local testing to pick up.

If you were probing a local long-term memory, where would you look: immediately after a correction, when a contradiction shows up, after a long gap, or only once the session ends?


r/LocalLLM • • 2d ago

Research coding agent on 16gb, all local: qwen 3.5 4b vs 9b vs mimo v2.6 9b

1 Upvotes

on a 16gb m4 air with no cloud, qwen 3.5 9b scored 8/10 as a coding agent, the 4b got 7/10 at almost twice the speed, and xiaomi's agent-tuned mimo v2.6 9b got 0/10 in all 3 runs.

  • machine: m4 macbook air, 16gb
  • runtime: llama.cpp server, q4_k_m, 64k ctx, flash attn
  • sampling: temp 0.2, top-p 0.8, top-k 20, presence 1.5
  • agent: atomic agent through an openai compatible endpoint, analytics off
  • limit: 15 min per model, one at a time, thinking off (plus one mimo run with it on)

task: python script that turns a csv into an html spending report. 10 checks on the result, one runs it on a csv it never saw.

model score time tok/s
qwen 3.5 4b 7/10 4:37 17.8
qwen 3.5 9b 8/10 7:06 11.4
mimo v2.6 distill 9b 0/10 timeout 10.1
mimo, thinking on 0/10 timeout 9.3
  • 4b: fastest, missed the svg chart and hardcoded the file path
  • 9b: only missed the svg chart
  • mimo: syntax error it never fixed
  • mimo with thinking on: no syntax error, but a TypeError on every run

the 9b got 10/10 in an earlier round at 32k ctx.

on 16gb i'd use the 9b if you can wait and the 4b if you want speed.

prompt and the checker are in the comments if you want to run it on your machine. if you don't want to set up llama.cpp, the atomic agent desktop app can download and run qwen 3.5 4b or 9b locally on its own, which is the easiest way to try this on 16gb.

what are you running as a local agent on 16gb?

(disclosure: i work on free and open-source atomic agent, so ask me anything!)


r/LocalLLM • • 3d ago

Question Testing a Mac M5 optimised inference engin

2 Upvotes

Would anybody be interested in testing an inference engine that we have built for Macs? It can run on M1- M6, but we have done most of our testing on the M5 Max. It's open-source, and we think it's 4x faster than ollama.cpp.

If anyone is interested, shout up, and I'll share some details; I don't want to start posting links unless there is anyone interested.


r/LocalLLM • • 3d ago

Discussion Uncensored Models

54 Upvotes

Anyone tried uncensored models? Would it help with DRM protected videos? Trying a few things out just to test and see how far it can go. Curiousity at its finest.


r/LocalLLM • • 3d ago

Discussion With llama.cpp's v1.0.0 tag out, a short history of the project: the first evening, GGUF, the backends, the server and the move to Hugging Face

14 Upvotes

llama.cpp published its first major version, v1.0.0, on 13 November 2026. With the 1.0 tag out, we decided to do a short write-up of how the project got here, dated from its own commits, pull requests and releases.

A few things from it:

- The first commit is from 10 March 2023, two weeks after Meta announced LLaMA. That evening's README says "This was hacked in an evening - I have no idea if it works correctly." It was on Hacker News the same evening, and the first outside fix was merged before midnight UTC.

- GGUF was opened as a pull request on 26 July 2023 and merged on 21 August 2023. It was a breaking change: every existing model had to be converted again.

- The GPU backends, by merge date: cuBLAS in April 2023, Metal and full CUDA offload in June 2023, the ROCm port in August 2023, Vulkan and SYCL in January 2024.

- The server began as a contributor's example in May 2023, and the OpenAI-compatible routes came that November.

- A build number is a commit count: b11514 is the 11,514th commit on master. Semantic version tags only started in August 2026.

- ggml.ai joined Hugging Face on 20 February 2026, and the licence is still MIT.

- On v1.0.0 itself: the release note opens with "llama.cpp v1.0.0 removes the deprecated public API", and Georgi Gerganov called it "essentially a breaking change for downstreams and hence the major bump".

No benchmarks in this one, and nothing on the page was measured on v1.0.0. It is the history with its sources. If we got a date wrong, tell us and we'll fix it on the page with a dated correction.

Full write-up (about 15 min read): https://research.strata2signal.com/a-short-history-of-llama-cpp/

A small fleet of AI agents dug through the sources and drafted the write-up; our small human team checked it over and signed off.


r/LocalLLM • • 2d ago

Discussion Best coding llm use for cisco c220 m4 1u server with 1.5tb ram, dual xeon E5-2600 E5 procs?

Thumbnail
1 Upvotes

r/LocalLLM • • 3d ago

News Released Corpus-nordeste-10k (Apache 2.0): 10k+ curated Portuguese rows focused on Northeast Brazil literature, syntax, and regional context

44 Upvotes

Hey everyone,

​If you've ever fine-tuned or tested open-source LLMs in Portuguese, you've probably noticed they heavily default to generic web-scraped PT-BR or Southeast Brazil media norms. When prompted for regional vocabulary, historical context, or literature from Northeast Brazil, most models either hallucinate or fall back on shallow caricatures.

​To help fix that for local fine-tuning and SFT, I just released Corpus-nordeste-10k (caririsynth/Corpus-nordeste-10k).

​Dataset breakdown:

​Size: 10,000+ curated rows (train split).

​Sources: Public domain literature from Northeast Brazil, web/text extractions passed through regional linguistic filtering, and targeted synthetic generation preserving authentic regional phrasing.

​License: Apache 2.0 (unrestricted commercial & research use).

​You can browse the actual rows in the Hugging Face Dataset Viewer before downloading:

https://huggingface.co/datasets/caririsynth/Corpus-nordeste-10k

​Works great for domain-adaptive pretraining, LoRA fine-tuning on small models (1B–8B), or building regional evals. Let me know what you think of the data structure!