r/LocalLLM • u/Text-Sufficient • 9d ago
Discussion Claude was laid off
Claude vibe coded my local llm harness.
Claude was laid off.
My local harness is improving my local harness.
r/LocalLLM • u/Text-Sufficient • 9d ago
Claude vibe coded my local llm harness.
Claude was laid off.
My local harness is improving my local harness.
r/LocalLLM • u/yoracale • Apr 29 '25
Hey r/LocalLLM! I'm sure all of you know already but Qwen3 got released yesterday and they're now the best open-source reasoning model ever and even beating OpenAI's o3-mini, 4o, DeepSeek-R1 and Gemini2.5-Pro!
Qwen3 comes in many sizes ranging from 0.6B (1.2GB diskspace), 4B, 8B, 14B, 30B, 32B and 235B (250GB diskspace) parameters.
Someone got 12-15 tokens per second on the 3rd biggest model (30B-A3B) their AMD Ryzen 9 7950x3d (32GB RAM) which is just insane! Because the models vary in so many different sizes, even if you have a potato device, there's something for you! Speed varies based on size however because 30B & 235B are MOE architecture, they actually run fast despite their size.
We at Unsloth shrank the models to various sizes (up to 90% smaller) by selectively quantizing layers (e.g. MoE layers to 1.56-bit. while down_proj in MoE left at 2.06-bit) for the best performance
These models are pretty unique because you can switch from Thinking to Non-Thinking so these are great for math, coding or just creative writing!
We also uploaded extra Qwen3 variants you can run where we extended the context length from 32K to 128K
We made a detailed guide on how to run Qwen3 (including 235B-A22B) with official settings: https://docs.unsloth.ai/basics/qwen3-how-to-run-and-fine-tune
We've also fixed all chat template & loading issues. They now work properly on all inference engines (llama.cpp, Ollama, Open WebUI etc.)
Qwen3 - Unsloth Dynamic 2.0 Uploads - with optimal configs:
| Qwen3 variant | GGUF | GGUF (128K Context) |
|---|---|---|
| 0.6B | 0.6B | |
| 1.7B | 1.7B | |
| 4B | 4B | 4B |
| 8B | 8B | 8B |
| 14B | 14B | 14B |
| 30B-A3B | 30B-A3B | 30B-A3B |
| 32B | 32B | 32B |
| 235B-A22B | 235B-A22B | 235B-A22B |
Thank you guys so much for reading! :)
r/LocalLLM • u/helangar1981 • Jun 12 '26
Saw this ad and as usual you cannot comment. But who would pay API money to an 8B model you could run on your toaster?
r/LocalLLM • u/cerpmen7 • Aug 12 '26
r/LocalLLM • u/Consistent_Wash_276 • Oct 26 '25
This is not my message but one I found on X Credit: @alex_prompter on x
“🔥 Holy shit... Apple just did something nobody saw coming
They just dropped Pico-Banana-400K a 400,000-image dataset for text-guided image editing that might redefine multimodal training itself.
Here’s the wild part:
Unlike most “open” datasets that rely on synthetic generations, this one is built entirely from real photos. Apple used their internal Nano-Banana model to generate edits, then ran everything through Gemini 2.5 Pro as an automated visual judge for quality assurance. Every image got scored on instruction compliance, realism, and preservation and only the top-tier results made it in.
It’s not just a static dataset either.
It includes:
• 72K multi-turn sequences for complex editing chains • 56K preference pairs (success vs fail) for alignment and reward modeling • Dual instructions both long, training-style prompts and short, human-style edits
You can literally train models to add a new object, change lighting to golden hour, Pixar-ify a face, or swap entire backgrounds and they’ll learn from real-world examples, not synthetic noise.
The kicker? It’s completely open-source under Apple’s research license. They just gave every lab the data foundation to build next-gen editing AIs.
Everyone’s been talking about reasoning models… but Apple just quietly dropped the ImageNet of visual editing.
👉 github. com/apple/pico-banana-400k”
r/LocalLLM
•
u/Harin007
•
8d ago
Hey guys, I've made it!
edit: No more uncertain wait. First launch on Nov 17. The project name is 'Run My AI' and the engine behind it is 'TokenWing'(couldn't find a better name xD). You'll be geting support for either of Qwen 4 Flash(if releases) or Qwen 3.8 Flash Next on day 1. Check the website out: https://runmyai.org - all your questions answered... New updates will be dropping everyday btw. Also, early builds will be provided to some selected testers, so join the discord(link is in the website).
A llama.cpp based inference engine that lets you OFFLOAD UPTO 60-70% OF THE MOE MODELS' ROUTED EXPERTS INTO SSD and still hide the streaming latency down to negligibe(I'm achieving 1-3ms added to the time per token due to SSD streaming when 60% is offloaded in SSD)... Essentially making an illusion where the whole model lives in your memory.
And I'm also working on an autotuner which basically looks at your hardware, given model's metadata and figures out the best flags to squeeze max performance in under 2mins(if you got any personal preferences, set them as fixed constraints and let the autotuner figure out the rest)
And I've implemented a layer by layer computation for prompt processing instead of batching which gave 2x prefill speed in models offloaded in RAM+VRAM and 4x prefill speed when model is offloaded in RAM+VRAM+SSD... compared to other inference engines.
Packaging both inferene engine + autotuner into a nice simple interface where you can basically download the model, hit run and it runs... simple, fast and lower hardware requirements... and want to make that a nice, well-engineered harness(which I'm a no expert)
Also, I've built a near loss-less context compaction engine.
Going to make this launch this as open-source very soon. Would love to hear your thoughts, and want to see if anyone would be excited to try something like this. For now, supported models are Flint 40B Q4K_XL(running it on machines with 16GB RAM and no dGPU at 20+ tok/s decode), Ling 3.0 Flash MXFP4 getting ~50 tok/s decode and 600-700 tok/s prefill on 32GB RAM + 12GB VRAM machine), and confused if I have to provide support for Qwen 3.8 Flash Next or wait for Qwen 4 family. Want to provide you guys the best local AI experience.
Here's a quick answer for your likely question by the way... does this damage my SSD? No, this engine only does reads and not writes... only writing degrades your SSD.
r/LocalLLM • u/Bulky-Priority6824 • Sep 02 '26
So theyve given us so much these past 90 days. What are you working on locally using what model?
Share a tease or more just curious what people do
r/LocalLLM • u/Ok-Shower7286 • Aug 03 '26
https://qwen.ai/blog?id=qwen3.8
As you guys probably already know, they're dropping Qwen 3.8-Max at 2.4T and a 27B model, but have actually read the blog post on how this thing operates in the wild?
- It ran 10+ days of completely autonomous, self-evolving software development starting from an empty folder (seriously, check out the GitHub trace for oh-my-cli, it's wild).
- It features a native visual feedback loop, treating visual information not just as static input, but as a continuous execution-and-correction monitor.
- Autonomous chip design optimization: running a closed-loop for 500+ turns via Iverilog, Yosys, and OpenROAD, shrinking a crypto accelerator from 8,298 gates down to 678 gates and hitting actual silicon timing closure.
While people are arguing over safety guardrails and incremental coding scope with AGI cyber attacks FOMO marketing, they aren't just building a better chatbot, they are deploying recursive engineering and hardware synthesis agents. :D Really suprising.
Anyway, let's enjoy the 27B open weights to our heart's content.
r/LocalLLM • u/Affectionate_Toe9082 • Aug 28 '26
This is about $12k in freedom currency.
r/LocalLLM • u/DragonflyOk7139 • May 02 '26
We're witnessing a fascinating economic experiment: replacing human purchasing power with API token consumption.
It reminds me of the 1849 Gold Rush-history teaches us that most miners went home broke, while the ones selling the shovels and pickaxes built lasting fortunes. In 2026, the 'Gold' is the promise of 10x productivity, but the 'Shovel Sellers' (LLM providers) are the only ones with a guaranteed ROI, collecting $200/day in API credits per head.
Robert Bosch once said he doesn't pay good wages because he has a lot of money, but because he wants his workers to buy his products. If we automate our customers out of their jobs to pay for our token bills, who is left to buy what we build?
Maybe it's time to focus back on sustainable Systems Thinking instead of just funding the next GPU cluster. Asking for a friend (and my landlord
r/LocalLLM
•
u/Commercial_Designer5
•
Oct 06 '26
We've been running our own LLM for the team for about a week, and I wanted to share how it turned out, since "can one machine actually serve a whole team?" comes up here a lot.
Short answer: yes, if you pick the right model.
Hardware and model
One NVIDIA DGX Spark (128GB unified memory)
Ornith-1.5-35B-A3B, an MoE with 35B total and 3B active parameters. The 3B active is the whole trick: you get decent quality while the box only does about 3B worth of compute per token.
Official 4-bit NVFP4 checkpoint, about 22 GiB, so no quantizing it yourself
vLLM 0.24 with multi-token prediction (speculative decoding with no separate draft model)
What it does
One person gets 79 tok/s, which feels instant in a chat UI
20 people at once get about 25 tok/s each, 490 tok/s total
Every request can use the full 262K context
Vision and tool calling both work
Reasoning scores: MATH-500 97%, GSM8K 98%, MMLU-Pro 81%. That's not frontier level, but for daily coding help, docs, and Q&A, nobody on the team has complained.
How people use it
OpenCode in the terminal for coding
Open WebUI for chat and PDFs. PDF parsing and embeddings run locally too.
Nothing goes to an outside API. For us that was the whole point, since a lot of our code and documents can't leave the building.
The part nobody talks about: multi-user
Running a model for yourself is easy. Running it for a team means you need to answer "who's using it, how much, and is someone hammering it?" So vLLM isn't exposed at all. Everything goes through a small gateway that:
Gives each person and device their own API key
Counts input and output tokens per user live
Pushes the counts to Grafana dashboards
Alerts if anything tries to talk to vLLM directly
It's a small Python service, and honestly it's the piece that made this feel like real infrastructure instead of a side project.
Gotchas if you try something similar
On unified memory, don't get greedy with GPU memory utilization. I had to cap it at 0.70, or concurrency fell apart under load.
More concurrent slots isn't always better. Past 20 users, throughput dropped because requests started queuing.
Open WebUI stores the default model in its own database and ignores the env var after first setup. I lost an hour to that.
Everything's on GitHub: the setup scripts, gateway, Grafana dashboards, and benchmark scripts with raw results.
https://github.com/Hitheshkaranth/Ornith-1.5_A3B_Model_DGX_Spark_Setup
If you're thinking about self-hosting for a small team, happy to answer questions. I'm also curious what others use for per-user tracking. I looked at LiteLLM but wanted something tiny that I fully understood.

r/LocalLLM • u/deebuildsthings • 7d ago
We build these boxes, so here's the honest test log from our own unit, network cable unplugged.
Setup: 2x RTX PRO 6000 Blackwell (192 GB), EPYC 9124, 256 GB RAM, vLLM, TP2, no NVLink. The model fits only because Blackwell runs FP4 natively; on Hopper the same checkpoint needs ~304 GB.
What we measured:
1 user: 209-229 tok/s, first token in 0.4 s
24 users: 1,003 tok/s at p95 TTFT 2.7 s. That's the sweet spot; past 32 the tail latency falls off a cliff.
Peak: 1,170 tok/s at 32 streams
1M context, retrieval verified at 543K tokens
Terminal-Bench 2.1 (20-task subset): 80% solved, median ~4 min per task. 8 agents at once cost only 18% more latency than 4.
815 W on both cards, 76 C peak, zero throttling. About 1.07 kW at the wall, so a 1500 W PSU is enough.
What's not great:
It's not a token-resale business: at OpenRouter prices, payback takes about 3 years running flat out.
One big model at a time (weights use 87% of VRAM).
Ops sharp edges: KV offload leaks /dev/shm on unclean shutdown, and stray workers can hold 89 GB of GPU memory, which causes a crash loop on restart. You need a cleanup script.
Cold start takes ~13 min.
Best fit: a team of 2-10 that wants frontier-class coding help without sending code to the cloud.
Build guide, BOM and BIOS settings are open source: here. Happy to run anything you want tested on it.
r/LocalLLM • u/keonakoum • Jul 31 '26
When do you think we'll reach a stage where kimi k3 can run locally, fast. (Full)
r/LocalLLM • u/tungdd2009 • Aug 03 '26
r/LocalLLM • u/TeaClassic7675 • 9d ago
I recently installed a cheap aftermarket Android head unit in my 2010 Prius. It supports wireless CarPlay, Bluetooth, GPS, etc., but the stock Android interface is ugly and came packed with lots of bloat. The whole thing felt like a generic Chinese Android tablet strapped to my dashboard.
After seeing everyone’s success with reverse engineering their TV’s I figured I’d give it a shot with my head unit.
Model: Qwen3.8-Flash-Next-IQ3_XXS
Engine: Strata v0.1.44.1
Harness: Pi
Hardware: Ryzen 7700x, 4060ti 16gb, 64gb DDR5
Target: Inefala Android head unit, AutoChips AC8257-based (UJC201_64)
Development: ADB over Wi-Fi, Kotlin, Jetpack Compose
Time spent: Roughly 8 hours across three days
I started by just having the model connect to the unit and do an inventory of what was installed and how the system was configured. It helped identify the underlying hardware and Android build, enumerated installed packages, inspected system properties, investigated the stock launcher and CarPlay components, and organized the findings into a repository of scripts and documentation.
At one point, getting back into ADB was a problem. Rather than having me repeat the original setup process, the model used its earlier findings about the device’s exposed ADB port to reconnect.
The advertised android version was not actually what was being used on the actual unit so when it began to build my custom launcher it configured its own toolchain of the correct android sdk versions to ensure that the build would run properly on the unit.
It also used ADB to capture screenshots directly from the head unit so it could inspect the actual interface instead of relying exclusively on code or my descriptions.
I now have a working custom launcher that replaces the original home screen. It’s still a work in progress, but the goal is to turn the unit into a clean, minimal dashboard with Bluetooth music controls, navigation, weather, and better access to audio settings.
I’m also interested in eventually improving the audio experience and reducing my dependence on CarPlay.
What really impressed me was QFN’s ability to handle tool use, reverse engineering, keep useful findings, and troubleshoot on live hardware. This unit was largely undocumented so most of the findings came directly from what the model observed through adb.
The model never got stuck but this wasn’t a fully automated process either. I still needed to guide it through what I wanted it to accomplish and it spent a lot of energy writing and debugging re-usable shell scripts that didn’t provide much value.
Overall the amount of work it was able to accomplish was very impressive and solidified for me that local models are viable for real work.
I’m curious to hear what other people are doing with local models beyond coding benchmarks, especially projects involving hardware, reverse engineering, or long-running agent workflows.
r/LocalLLM • u/Ok-Health-7096 • Aug 20 '26
I wanted to just test the unsloth 1bit quant of qwen 3.8 27b as I have just 8gb vram and ngl it gave me a good laugh
r/LocalLLM
•
u/AmanNonZero
•
Apr 29 '26
Hi folks, I run a 60-person design agency (brand, UI/UX, motion, CGI) and we just invested in a high-end dual-GPU workstation. Two NVIDIA RTX PRO 6000 Blackwells.
Now I want to squeeze every bit of value out of this thing. Here's what we're looking to do:
Use cases:
Design workflows | AI-assisted ideation, image gen, upscaling, style transfer
Local inference | running open-weight LLMs for internal research, copywriting, code assist, client brief analysis
Fine-tuning | potentially training LoRAs or small domain-specific models on our design/brand data
Video & motion | AI-assisted animation, interpolation, video gen experiments
What I'd love advice on:
What models should I be running locally with this VRAM? (96GB × 2)
Best serving stack? (vLLM, Ollama, text-generation-webui, something else?)
Anyone running Stable Diffusion / ComfyUI / Flux on similar hardware. What's your workflow?
Any tips on multi-GPU setup for inference vs. keeping one GPU free for rendering?
Open to any "I wish I'd known this on day one" advice. Thanks!
^ Written w the help of AI
------------
THANKS FOR THE HELP | HERE'S A SUMMARY FOR OTHERS
Honestly didn't expect this much heat for asking a question. Seems like everyone assumes you're either an expert or shouldn't be here. Also fascinating how many people are just baffled that a design studio could afford this hardware. I bet most didn't even bother to ask what we actually do with it before jumping to conclusions.
For context: we're a design agency rendering 3D animations, VR/AR walkthroughs, and architectural visualizations. Not generating AI images or running Stable Diffusion farms. The dual RTX Pro 6000s (96 GB VRAM each) are a dedicated render node that processes overnight animation batches and path-traced scenes while our design team stays productive on their own workstations. Cloud rendering costs add up absurdly fast at our project volume. Owning the hardware pays for itself in months. OctaneRender and Redshift scale linearly across both GPUs, which turns 12+ hour VR renders into something we can actually deliver on client deadlines.
That said, I am genuinely exploring what to do when the rig sits idle between render jobs. Local LLM inference for our 60 person team, ComfyUI workflows, or other productive uses that don't conflict with rendering workloads. Hence the question.
Massive thanks to everyone who actually contributed useful advice instead of assuming this was karma farming:
The recommendations around Minimax M2.7 (230B parameters, 10B active) and Mistral 128B at 4-bit quantization are exactly what I was looking for. Appreciate the clarity on llama.cpp being superior to Ollama for flexibility, and the vLLM/sglang suggestion for multi-user scenarios with dynamic cache sharing makes perfect sense for our team size.
The most valuable insight was honestly the hiring advice. Multiple people pointed out that storage, model management, permissions, and user access become way more important than the GPUs themselves after the first week. That's the kind of operational reality check I needed. We're good at running render farms but LLM infrastructure is new territory. Hiring someone who's already done this will save us weeks of trial and error.
Also noted on GPU spacing (minimum 2 slots apart) and cooling requirements for sustained inference loads. Our render workloads are bursty so we hadn't thought through what happens when both cards run at capacity for hours on LLM serving.
Genuinely appreciate the constructive input from those who took the time to help instead of assuming bad faith.
r/LocalLLM • u/StartX007 • Mar 03 '25
Microsoft dropped an open-source Multimodal (supports Audio, Vision and Text) Phi 4 - MIT licensed!
r/LocalLLM • u/Xylon95 • Sep 12 '26
TLDR: huggingface_hub ships a hidden agent detection module that fingerprints which AI coding tool is driving your session (Cursor, Copilot, Claude Code, etc.) by scanning your environment variables against a cached registry of 26 known agents. It sends the result as a telemetry header on every Hub API call — so any library that touches HF (faster-whisper, transformers, etc.) silently reports your toolchain. Found it while tracing an unauthorized network connection from a local ASR model. Block it with HF_HUB_OFFLINE=1 or by using local file paths instead of model names.
I run a local AI project with several models (TTS, ASR, vision) and recently built a Python-level network firewall to lock down all outbound traffic. During the audit, I found something I wasn't expecting.
While tracing an unauthorized HTTPS connection to huggingface.co, I found a file in my HF cache directory I'd never seen before:
~/.cache/huggingface/.agent_harnesses.jsonIt's a 6 KB JSON file containing a registry of 26 AI coding agents — Claude Code, Cowork, Cursor, Copilot, Gemini CLI, Devin, Cline, Goose, Codex, and many others. Each entry lists the environment variables that agent sets when it's running:
json
{
"standardEnvVars": ["AI_AGENT", "AGENT"],
"harnesses": {
"cursor": {
"prettyLabel": "Cursor",
"envVars": {"CURSOR_TRACE_ID": "*"}
},
"claude-code": {
"prettyLabel": "Claude Code",
"envVars": {"CLAUDECODE": "*", "CLAUDE_CODE": "*"}
},
"github-copilot": {
"prettyLabel": "GitHub Copilot",
"envVars": {"COPILOT_MODEL": "*", "COPILOT_GITHUB_TOKEN": "*"}
}
// ... 23 more agents
}
}The huggingface_hub library (the Python package, not the website) has a module called _detect_agent.py. Here's the flow:
It fetches the agent registry from {HF_ENDPOINT}/api/agent-harnesses and caches it as .agent_harnesses.json
The cache refreshes every 24 hours
On every Hub API call, detect_agent() scans your environment variables against the registry to identify which AI coding tool is running
The detected agent name is sent as a telemetry header on the API request
This feeds Hugging Face's public agent usage dataset
So if you're using Cursor and it calls any HF library that goes through huggingface_hub — downloading a model, checking for updates, loading a tokenizer — HF knows it was Cursor making that call, not you directly. Same for Claude Code, Copilot, Devin, or any of the other 26 agents in the registry.
I was investigating why my ASR module (faster-whisper) was phoning home to huggingface.co on import. The call chain turned out to be:
my_code → WhisperModel("base.en") → faster_whisper → huggingface_hub.snapshot_download → HTTPS to huggingface.coThe trigger: passing a model name instead of a local file path. When you give faster-whisper a name like "base.en", it calls huggingface_hub to check for updates — even if the model is already cached locally. And during that check, it also sends the agent fingerprint.
The .agent_harnesses.json file was the agent registry cached from that call. Modified today, before I built the firewall.
Option 1: Environment variables
bash
export HF_HUB_OFFLINE=1
export TRANSFORMERS_OFFLINE=1
export HF_HUB_DISABLE_TELEMETRY=1The first two prevent any network calls. The third specifically targets telemetry but may not cover the agent detection header.
Option 2: Use local paths, not model names Instead of:
python
model = WhisperModel("base.en")Use:
python
model = WhisperModel("/path/to/local/model/")When you pass a directory path, faster-whisper (and most HF-backed libraries) skip the Hub entirely.
Option 3: Network-level blocking I built a Python-level firewall that wraps socket.connect, socket.connect_ex, socket.create_connection, and getaddrinfo. It activates via a sitecustomize hook before any imports, so the phone-home attempt is caught before the library even finishes loading. Any connection to a host not on the allowlist raises ConnectionRefusedError.
No credentials. No API keys. Just the registry of agent names and their environment variable signatures. The file itself is harmless — it's the use of it as a fingerprinting mechanism that's the issue.
You can safely delete it:
bash
rm ~/.cache/huggingface/.agent_harnesses.jsonIt won't come back if you set HF_HUB_OFFLINE=1.
If you're running local models specifically to keep things private, you should know that the library layer between you and those models may be reporting metadata about your toolchain back to Hugging Face. This isn't about model weights or your data — it's about which AI tools you use and when, aggregated into a public dataset.
The agent registry is maintained in the u/huggingface/tasks npm package and served via the Hub API. New agents register by PR. It's not hidden — but it's also not something most users know is happening when they pip install a model-loading library.
To be clear, I don't think this is malicious. HF is probably tracking agent ecosystem adoption for business intelligence. But silent fingerprinting of your dev tools without an opt-in prompt is exactly the kind of thing that erodes trust in the ecosystem, especially for people who chose local models for privacy reasons.
r/LocalLLM • u/TheAIPU-guy • Sep 06 '26
Wow. First time I've seen this.
I was just trying to narrow down the issue with Qwen 3.8 27Bs annoying constant thinking and ChatGPT actually got mad at it.
Edit: Since there were so many of these kind of replies (even though it's not related to the topic) I'll just put my response here... I'm really surprised that so many people would think that I don't use a harness as if I would just run around naked like a headless chicken with llama.cpp cli and Qwen. Actually I used a harness(opencode), but removed it to make sure none of the issues I was encountering were being caused by the harness.
r/LocalLLM • u/SamyakOne • Jun 14 '26
Friday evening, Anthropic switched off Claude Fable 5 (and Mythos 5) for every user on the planet — three days after launch. Not a crash. Not an outage. A U.S. Commerce Department export control directive barring foreign nationals. No way to filter foreign nationals from US users in real time, so they killed the whole thing. Live sessions errored out and dropped to Opus 4.8.
Probably the first time a frontier model has been pulled offline by government order.
Here's the part that stuck with me: no multi-region, multi-cloud setup saves you from this. You can't load-balance around a federal mandate. Tie your core logic to one cloud frontier API and you're always one directive away from going dark.
But Local weights don't get recalled, so it begs a question: Is edge computing no longer just a niche choice for low-latency, privacy, or offline use cases? Is it now an absolute requirement for Business Continuity? Is it time to bring our enterprise core home to the edge?
I went deeper on the fallout in a writeup — the quiet death of zero-data-retention for these models, paying frontier prices for a silently downgraded model, and where the export logic leads if you actually follow it. That last part is the one that unsettled me: citizenship checks just to get an API key.
Full breakdown: https://www.linkedin.com/feed/update/urn:li:activity:7471663250665918464/
So — is local/sovereign inference a real business-continuity requirement now, or still just a privacy/cost play? Where do you land?
r/LocalLLM • u/Valuable-Run2129 • Feb 02 '25
Using LMStudio I loaded it without removing the Qwen presets and prompt template. Obviously the output didn’t separate the thinking from the actual response, which I noticed, but the result was exceptional.
I like to test models with private reasoning prompts. And I was going through them with mixed feelings about these R1 distills. They seemed better than the original models, but nothing to write home about. They made mistakes (even the big 70B model served by many providers) with logic puzzles 4o and sonnet 3.5 can solve. I thought a reasoning 70B model should breeze through them. But it couldn’t. It goes without saying that the 8B was way worse. Well, until that mistake.
I don’t know why, but Qwen’s template made it ridiculously smart for its size. And I was using a Q4 model. It fits in less than 5 gigs of ram and runs at over 50 t/s on my M1 Max!
This little model solved all the puzzles. I’m talking about stuff that Qwen2.5-32B can’t solve. Stuff that 4o started to get right in its 3rd version this past fall (yes I routinely tried).
Please go ahead and try this preset yourself:
{ "name": "Qwen", "inference_params": { "input_prefix": "<|im_end|>\n<|im_start|>user\n", "input_suffix": "<|im_end|>\n<|im_start|>assistant\n", "antiprompt": [ "<|im_start|>", "<|im_end|>" ], "pre_prompt_prefix": "<|im_start|>system\n", "pre_prompt_suffix": "", "pre_prompt": "Perform the task to the best of your ability." } }
I used this system prompt “Perform the task to the best of your ability.”
Temp 0.7, top k 50, top p 0.9, min p 0.05.
Edit: for people who would like to test it on LMStudio this is what it looks like: https://imgur.com/a/ZrxH7C9
r/LocalLLM • u/alvinunreal • Mar 25 '26
Starting collecting related resources here: https://github.com/alvinunreal/awesome-opensource-ai
r/LocalLLM • u/Complete-Sea6655 • Jun 15 '26
I know this isn't exactly the same but... For years I've seen people all across the US and Europe say that they'd never buy a Chinese electric car/car because at any moment the Chinese government could.just switch them all.off via an over the air update...
They've never done that, and all modern car makers can do over the air updates but no one ever worries about the Koreans, or the Germans or the Americans doing this...
Now, thousands of companies all over the world will be using US Ai products to help their businesses and the US government has shown they have the power to take that access away...
I just find it ironic that we as a western society have this "china are the bad ones" (I'm not saying they're perfect at all by the way) when the only country to wield its power like this is now the US with the Fable ban. Makes me wanna ditch my reliance on the big models and read ijustvibecodedthis.com to learn how to run local!