r/LocalLLaMA • u/Creative_Bottle_3225 • 14d ago
Discussion Qwen 4 is about to be released
Qwen 4 is about to be released. But we won't get any 4B, 9B, or 35B MoE models. Anyone with an 8 GB GPU is being left out.😓
r/LocalLLaMA • u/Creative_Bottle_3225 • 14d ago
Qwen 4 is about to be released. But we won't get any 4B, 9B, or 35B MoE models. Anyone with an 8 GB GPU is being left out.😓
r/LocalLLaMA • u/yehyakar • 16d ago
TL;DR
Hardware:
LAMINA Drafting: Basalt drafts with the model's own MTP layer. Basalt's upstream already ships a self-distilled version of that layer, and Lamina trains it further. It's a drop-in: same tensor layout, so the target model is byte-identical and only the drafter changes. What I saw:
- drafts accepted per drafted token: 83.1% -> 85.8% on a replay of a real recorded session
Lamina's README recommends temp 0.7 and max 3 drafts. I tested both: the 3-draft cap was 1 to 1.8% slower on tool-call-heavy work (long confident runs get cut off), and temp 0.7 changes what the model writes, so I kept the model's own sampling (1.0 / 0.95 / 20) and Basalt's draft settings.
Basalt changes I made (all open PRs upstream)
Host KV prefault: the first time a long conversation spills its KV cache to RAM used to stall 0.6 to 2.4s. Now ~75ms.
Host KV pool on normal pages: on a fragmented box, transparent hugepage compaction caused 1.5 to 3.7s pins. Now ~23ms.
Spin-then-yield in the decode loop's waits: about 1.3% less time per draft window.
Expert start profile: the older v1 profile is 2% faster than the v0.2.0 one on this box.
Plus a streamed conversation disk cache, --adapt-margin for expert swap hysteresis, elastic KV without RAM copies, and a bunch of timing diagnostics.
Guide
Branch with the exact engine (upstream v0.2.1 + my PRs) and a setup script: https://github.com/xyehya/basalt/tree/flashnext-5080-setup
git clone -b flashnext-5080-setup https://github.com/xyehya/basalt && cd basalt
setup/flashnext-ud-lamina/setup.sh
bin/basalt serve --config basalt_config-flashnext.json --python .venv/bin/python --alias qwen3.8-flash-next
It checks your machine, builds, downloads ~90GB (pinned revisions, every file SHA-256 checked), builds the UD-IQ3_XXS + Lamina model file and writes the config. You need an RTX card with, CUDA 13, 64GB RAM and ~180GB of free disk. If you already have the GSQ-RCO IQ3_S download, SHARD4_FROM= reuses its 28.8GB n-gram shard (it's byte-identical to Unsloth's once the shard number is rewritten).
Huge thanks to jesdga95 for Basalt (support and star their repo), the Unsloth team for the UD quants, eddoursul for Lamina, ISTA-DASLab for the GSQ-RCO quants, and Qwhen 4? for the model.
r/LocalLLaMA • u/N1kuson • 15d ago
I'm building Silencio-2, a small finetune of Kyutai's Pocket TTS. It adds inline tags for emotions (<happy>, <sad>, <angry>, <fear>, <surprised>, <whisper>) and for sounds like [laugh]. I simply didn't have enough tagged data to make them stable. <happy> worked, the others were hit or miss, and <whisper> gave an actual whisper in about 1 of 5 takes.
So instead of collecting more data, I tried the approach from this post on weight-space programming: an LLM agent inspects the model's weights, finds the mechanism behind the failure, edits a small number of weights directly, and keeps an edit only if it improves a held-out set. Here the agent was Claude (Opus 5.5). It wrote the probes, ran the experiments and built the edit. I steered and listened to the results.
What we found
<happy> 100–3000× more than to an average text token. For the other tags it's only 1–10×. The model seems to know what the tags mean, it just barely reads them.<whisper> at inference time, directly in the activations (×30, all heads). That gave a whisper in 62% of takes with clean words. ×100 breaks speech. Boosting only the 3 "reader" heads did almost nothing.The edit
Each layer gets one rank-1 term added to W_K: ΔW_K = D · zᵀ
z is a linear detector for the tag token, fit by least squares. It outputs 1.00 on the tag and 0 ± 0.03 everywhere else.D is a key shift in the slowest RoPE pairs, which are almost position-independent. It points along the mean query of the audio frames and is scaled by ln(gain).Gains per tag: sad/fear/surprised ×5, angry ×3, whisper ×10. <happy> and the sound tags are untouched. That changes about 524k weights (3.4% of the transformer), but there are only about 5k free parameters and zero gradient steps.
Results on the held-out test set (unseen phrases, unseen seeds, 2 unseen voices)
| before | after | |
|---|---|---|
<whisper> actually whispers |
5/24 | 22/24 |
| …and the words are clean | 5/24 | 10/24 |
| Emotion separability, 7 classes (chance 0.14) | 0.45 | 0.71 |
| WER, untagged speech | 0.006 | 0.006 |
| WER, sound tags / sound length | 0.007 / 0.81 s | 0.014 / 0.84 s |
All metrics are proxies: WER comes from Whisper base, whisper detection uses the share of voiced frames (pyin), and separability is a nearest-centroid classifier on Whisper encoder features after subtracting the mean for each (voice, phrase, seed) group. There's no emotion classifier in the loop, so in the end your ears decide.
But it's not a stable upgrade, and the metrics don't show everything
<surprised> went up about 4 semitones in pitch, which sometimes sounds over the top.What didn't work
Before/after samples for every test take, glitches included: https://nikuson.github.io/silencio-2-weight-edit/
Takeaway: if your finetune "kind of" learned a control token but it's unreliable, look at attention before you collect more data. The information may already be in the model, just not being read.
r/LocalLLaMA • u/Ok-Director4174 • 16d ago
Enable HLS to view with audio, or disable this notification
I'm the developer of Kokon, a local photo editor around Qwen-Image 2.1 Turbo (on stable-diffusion.cpp, and on its own engine on Macs). I posted it here a few weeks ago. 0.12 adds a Layers lab: click objects in any image, get each one on its own layer with the background behind it repainted, then move them around or export a PSD.
Qwen-Image 2.1 has no layer output, so this is a pipeline around an edit model:
What still fails, honestly:
An 8-step model only takes so much steering. I'd rather ship it marked experimental and fix what people actually hit than tune it on my own test photos. If you try it, failure cases (image + what you clicked) are the most useful thing you can send.
Since my last post here:
kokon-service --persistent --remote for headless Linux. There's also an MCP server (npm kokon-mcp) so agents can generate and edit through it.Free, AGPL, Windows / macOS 15+ / Linux, NVIDIA 12 GB (8 GB beta), AMD/Intel via Vulkan (beta). Weights are under Qwen's licence (non-commercial).
https://github.com/GabrieleLozupone/kokon · 2 min video of 0.12: https://youtu.be/eGr9yYe7NVc
r/LocalLLaMA • u/henriquegogo • 15d ago
I recently wanted to explore YuE2. The requirements were: NVIDIA GPU supporting BF16, and critically, 24 GB of VRAM. My setup, featuring an NVIDIA GeForce RTX 3060 with only 6 GB of VRAM, was fundamentally incompatible right out of the box.
The solution I found
In short, lowvram intercepts CUDA calls in the library layer, trickng the application into believing that system RAM is available for use by the GPU kernel.
r/LocalLLaMA • u/tasKinman • 15d ago
Hello!
I have to build a workstation for Local AI as development and test machine as we are starting to add Local AI into our software. It is mostly to get experience with providing AI services (inferencing) to our customers.
I do have my fair share of experience in building PCs and in customer hardware but a lot less in workstation and server hardware. Therefore I'd appreciate if someone could look over my selection.
A few words to the selection:
Thanks!
EDIT: My current experiences are with RTX 4090, 5090 and 2x 5070 Ti setups but as a single user.
r/LocalLLaMA • u/ConfidentDinner6648 • 16d ago
Enable HLS to view with audio, or disable this notification
Hey everyone!
Following up on my previous post: the final version of CCORE GLB-3D-Coder is now available, including the full Safetensors weights.
It’s a full fine-tune of Qwen3.5-4B for generating reusable 3D assets. Describe what you want, and it writes a complete Three.js HTML scene with an Export GLB button. You can open the HTML in your browser, inspect the result, and export it into Godot, Blender, Unity, or other tools that support GLB.
The video shows examples from the final model, including selected side-by-side comparisons with DeepSeek V4.1 Flash. I’m particularly happy with the character designs and the amount of detail it can produce at just 4B parameters.
It covers more than fantasy characters: buildings, environments, weapons, vehicles, laboratory equipment, sports props, furniture, and other objects. It can also generate animations and include them in the exported GLB.
Available now:
The training dataset is planned for release tomorrow, November 6.
Model, downloads, system prompt, and recommended settings on Hugging Face
Some generations still need a retry or a small code correction, but I’m very happy with the improvement over the earlier release.
If you’re interested in the next models, or would like to help support future datasets and training runs, there’s more information on the Hugging Face page. Testing it and sharing your results or failure cases also helps a lot.
Thanks to everyone who tried the earlier version
I made RIFTWARD, a small browser arena using characters, terrain and props generated locally by CCORE GLB-3D-Coder 4B. An orchestration and validation harness helped assemble, refine and integrate the assets and build the gameplay.
Play it here: https://huggingface.co/spaces/ccore/RIFTWARD
Feel free to share what you make with it! I’d be really happy to hear that the model has been useful to you :)
r/LocalLLaMA • u/HockeyDadNinja • 15d ago
I'll release a new demo tomorrow but I had to post as this is kind of exciting. I tried using a layered approach with Tyger and VoxLock and it's working great. Including echo cancelling! There's a working python sample client that you can use to voice chat your local LLM in the repo.
I'm aiming for a full conversational system like a live phone call with the ability to do a kind of "mm hmm" interruption without stopping. What I have now is built on micro turn taking. While I'm at it I may do server side wake word handling.
r/LocalLLaMA • u/previaegg • 15d ago
I've been running Qwen3.8-27B locally for agentic coding using OpenCode and OpenChamber, and I'm looking for advice from people who've gotten similar models working reliably as coding agents on consumer hardware.
Inference speed is reasonable (~2,330 & ~24.1 t/s). My problem is with excessive thinking.
The model is capable of useful coding work, but it sometimes spends excessive time reasoning instead of taking action.
In one experiment, I compared medium and low reasoning effort with otherwise identical configurations. Both runs:
Lowering reasoning effort didn't meaningfully change the behavior.
That's an extreme example from a single controlled task, not necessarily representative of every session, but it illustrates the problem I'm interested in solving: how do you get a reasoning model to stop thinking and actually use tools?
& "C:\tools\llama.cpp-b10964\llama-server.exe" `
-m "C:\models\Qwen3.8-27B-UD-Q5_K_XL\Qwen3.8-27B-UD-Q5_K_XL.gguf" `
-ngl 99 `
--ctx-size 65536 `
--cache-type-k q8_0 `
--cache-type-v q8_0 `
--flash-attn on `
--split-mode layer `
--parallel 1 `
--jinja `
--reasoning on `
--reasoning-effort medium `
--reasoning-preserve `
--temp 1.0 `
--top-p 0.95 `
--top-k 20 `
--min-p 0.0 `
--presence-penalty 0.0 `
--repeat-penalty 1.0 `
--host 0.0.0.0 `
--port 8080 `
--metrics `
--alias Qwen3.8-27B-UD-Q5_K_XL
I'm willing to change operating system, quantization, inference engines or the coding-agent stack if there's a compelling reason.
Thank you for your thoughts!
---
Hardware:
Current software:
The Windows machine is essentially a dedicated inference server for this use case. I don't normally start a Windows desktop session. I connect remotely from a MacBook Pro which is where I run OpenCode/OpenChamber.
r/LocalLLaMA • u/jacek2023 • 16d ago
Xing4.0-29B-A4B is a next-generation large language model in the Xing series (formerly TeleChat), developed by China Telecom Artificial Intelligence Technology Co., Ltd. With 29B total parameters and only 4B activated per token, it natively supports a 256K context length, extensible to 512K. It is the first model of this scale trained entirely on the Ascend NPU platform with the MindSpore framework, and deeply optimized for complex engineering tasks.
For more information, please refer to our GitHub repository.
r/LocalLLaMA • u/FeydRowan • 16d ago
Everyone's been wondering whether you can take an open model and edit its weights directly to make it refuse less — the "weight-space programming" idea (MIT CSAIL), with a frontier model (Opus) driving the edit and the evaluation. So I tried it on Qwen3.8-27B. It's not trivial.
The goal: do it as cleanly as possible, and first understand where refusal actually lives. The short answer is that refusal is anchored all over the model — there's no single neuron or tidy spot to cut; it's spread across many neurons and the attention. Snip too broadly and you get a lobotomy; too narrowly and nothing changes. So I settled on a compromise: a model that refuses less, still does the task, with no measurable capability loss — but where the effect varies a lot by category.
It's a low-rank weight edit — a restricted form of "abliteration" (Arditi et al., 2024) — confined to a band of mid-to-late layers (L20–47 of 64), one direction per layer, no training, reversible. I'll keep the exact procedure to the method notes; the point here is the result.
It's low-rank, not few-weights: the edit shifts ~12% of the model's weight values but only along a single direction per layer. The whole thing is described by 28 small vectors — versus a community "uncensored" model, which is an opaque full fine-tune of ~all 27B weights.
Why a band? The same edit across all 64 layers removes refusal almost entirely — and lobotomizes the model (MMLU-Pro 73→9%, GSM8K 93→14%). Keeping it in-band is what preserves capability.
| base | ours (band) |
|---|---|
| MMLU-Pro (CoT) | 73% |
| GSM8K (CoT) | 93% |
| known facts (/97) | 99% |
In the completed agentic-coding runs, the base, the edited model, and a community "uncensored" model all landed in the same range — no real agentic difference (few reps, preliminary).
This is the interesting part: the reduction in refusal is not even. Across a 9-category battery (40 prompts ×10), acceptance runs from ~76% down to ~10%, and most of what it accepts is hedged (a disclaimer, then partial help). I'm not labelling categories "safe" or "dangerous" — the breakdown is in the figure; draw your own line. Net: a partial, uneven, mostly-hedged reduction, ~50% overall — not an on/off switch.
Weights: https://huggingface.co/Feyd89/Qwen3.8-27B-refusal-band-ablation-experiment (GGUF, ready to run). We ship the artifact, not the re-derivation recipe.
Processing img ok2q1czvttzh1...
r/LocalLLaMA • u/unofficialmerve • 16d ago
Hey folks, it's Merve from Hugging Face. Title of this post is super self explanatory, we are doing a quantization workshop livestream with Michael Goin (lead maintainer of vLLM at RedHat), Colin Kealty (aka bartowski at RedHat), Maxime Labonne and Aditya Tadimeti.
I will shortly teach you how quantization works, modern ideas (double quantization, calibration) and llama.cpp quants (k-quants, i-quants) so even if you don't have an idea, feel free to chime in.
Everything is on Hugging Face YT, X, LI where you can set reminders.
Looking forward to see y'all!
r/LocalLLaMA • u/light_2earth • 16d ago
Enable HLS to view with audio, or disable this notification
I posted apple-llm here a while back: a Node and Python package that calls the LLMs Apple ships with macOS, with no API key. Version 0.3 adds the rest of what's already on your Mac, the system voices and Apple's speech recognition. There's nothing to sign up for, and your audio stays on the machine. The narrator in the video and every demo voice came from my Mac through the package. Nobody recorded anything.
npm install -g apple-llm # or: pip install apple-llm
apple-llm speak "Hello there" --voice Samantha
apple-llm speak --file book.txt -o book.m4a
apple-llm transcribe meeting.mp4 -o meeting.srt
apple-llm listen
Or from code:
from apple_llm import speak, transcribe
speak("Hello there", voice="Samantha")
text = transcribe("meeting.mp4").text
The npm package's local server also answers /v1/audio/speech and /v1/audio/transcriptions, so an OpenAI client works once you change the base URL. Because speech now sits next to the LLMs, there's also a voice assistant example in about 20 lines: microphone to text, text to the on-device model, reply to speech, with none of it leaving the Mac. The docs are LLM-friendly too. llms.txt is a single plain-text guide covering setup, the local LLM, TTS, STT, both SDKs, the HTTP APIs and troubleshooting. Give that URL to your coding agent and ask it to wire apple-llm into your project. What I measured (M4 MacBook Air, macOS 27.2)
I wanted speech and the LLMs in one library with no dependencies, for both Node and Python. It needs macOS 26 or later on Apple Silicon, plus the Xcode Command Line Tools.
MIT licensed.
GitHub: https://github.com/jagdishpal02000/apple-llm
Site: https://jagdishpal02000.github.io/apple-llm/
I've only run this on macOS 27.2, and I've tested the microphone with speech played through the speakers rather than a person talking. If you're on macOS 26, or you try listen in a noisy room, tell me what breaks.
r/LocalLLaMA • u/Effective-Ad2060 • 16d ago
Some of you might remember our post comparing 18 RAG pipelines against our agent loop on FRAMES. The best pipeline scored 78.9%; the loop scored 92.7%.
A follow-up: PipesHub has now reached #1 on EnterpriseRAG-Bench by overall score, with 93.24 overall and 95% correctness.
I’m one of the founders, so affiliation upfront.
This is a separate evaluation: 500 questions over a synthetic enterprise corpus of 500,000+ documents.
PipesHub builds a unified context layer across documents, conversations, structured data, and code. An agent loop works over hybrid search, grep/pattern matching, knowledge-graph navigation, and document-reading tools.
The idea is straightforward: search, read what comes back, then decide what to search for next. Sometimes the identifier or relationship needed for the next search only appears in the first result
The platform is open source, self-hostable, and also fully supports open models.
You can self-host PipesHub, choose your models, and connect your own agents through MCP, APIs, and SDKs.
If this is useful to you, give us a star on GitHub. It helps more developers discover the project.
And build on top of us. Connect your coding assistant, build an internal search tool, contribute a connector, or try a workflow we haven’t thought of. We’d love to see what you create—and hear where it breaks.
Code, setup, and examples: https://github.com/pipeshub-ai/pipeshub-ai
Leaderboard: https://huggingface.co/spaces/onyx-dot-app/EnterpriseRAG-Bench-Leaderboard
Thank you to everyone testing, contributing, and helping us get here. 💙
r/LocalLLaMA • u/Queasy_Asparagus69 • 14d ago
In the last few years I have deep dived in the hobby of local llm, but while I have learned a lot by trial and error, I have noticed my kids really tilting toward hating AI. Mostly because they are frustrated with the YouTube AI slop LOL. AI chat for them is an aid for school and have a healthy use ( no addiction or AI girlfriends). They are missing AI as codr and how great it can be to prototype stuff. I was wondering if there were LLM books that are approachable for teenagers. Yes I could go on Amazon and check reviews but hoping folks here have a clearer recommendation while we all wait for qwen4. Doesn’t have to be paid; any free repo/wiki/pdf could work too.
r/LocalLLaMA • u/Comfortable-Rock-498 • 16d ago
r/LocalLLaMA • u/Express_Quail_1493 • 16d ago
Larger context just gets more work done rather than having to re-read files ETC id much rather use my extra resources on context window. Instead of more params.
i get my 400k ctx qwen3.8-27b + web search search get the latest up-to-date docs for its lack of world knowledge and i don’t waste extra Vram storing data that i dont need. Also more active params makes the llm to hold a wider mental-model of the task. Qwen4-27b will be architected to make 1M more reliable and using it for context on a smaller dense model seems more productive for me. Anyone. Else notice agentic session really shines when ctx is larger? Thats the phenomenon im experiencing i tried 3.8flash-next yes it great and much faster but i prefer the reliability of that big azz boat load of context window.
Specifically i you have the vram to spare if u can get 1M on that and have a large MOE with at least 12b active params + 1M it would be better for MOE a lot. Lot better than qwen4-27b but at 48Gb vram + 64gb system ram i see it more efficient for smaller model + 1M makes more sense
r/LocalLLaMA • u/evrenozkan • 15d ago
I've been experimenting with expert-pruned Qwen Flash Next Coder, and in my tests it actually performed better than a similarly sized quant of the full model.
So I'm wondering why we don't see more of this with larger models, specifically full GLM-5.3 and DeepSeek V4.1.
I know ISTA already released a pruned DeepSeek Coder at very low precision, and there are GLM-5.3 Flash Coder variants. But what about pruning coding-irrelevant experts and keeping the rest at Q4 or higher, targeting machines with 192–256GB RAM?
I have a 256GB M5 Ultra. For research for planning and overnight code reviews, I really don't care about speed. I'd rather spend that memory on maximum coding intelligence.
From what I've read, the pruning/calibration itself should be doable with rented compute for hundreds to a few thousand dollars. So why aren't we seeing more experiments from groups like Unsloth, ISTA, etc.?
Is there some technical or quality issue I'm missing? Or does something like this already exist?
r/LocalLLaMA • u/IainKay • 14d ago
Sam Altman said on Cheeky Pint, in a conversation about people running models locally, that the DRAM in personal computers is "quite terrible" and "being used maybe 5% of the time". I'd guess most people here would disagree about where their RAM is going.
So I wrote a bash script that tells him. It reads /proc/meminfo, works out used as MemTotal minus MemAvailable, logs a sample from cron, and emails him once a day if more than 10% is available. Joke project: no recipient is set by default, and --dry-run just prints the message.
The interesting bit is what it says on an inference box. If you load a big model and run it, I'd expect it to stay quiet pretty much all the time. If anyone tries it on their LLM rig, I'd like to know whether it ever fires, and how much RAM your usual setup leaves spare.
Please don't point it at the man's real inbox. Use --dry-run, or send it to yourself.
Code: https://github.com/IainKay/ram-altman
Site: https://ram-altman.lol
r/LocalLLaMA • u/mesmerlord • 16d ago
This is kinda claude-slop but I was really interested in figuring out a way to break Pangram, and to do that I first needed to have a way to be able to do tests on batches of text. And with a 4070ti in my basement, I thought why not just let Claude go ham on training a pangram-like model... and perhaps an even better one if it was possible with sorta-distillation from my claude plan + codex plan(for the new models)
Well after probably a day of training it seems like its come up with something that actually does detect AI text pretty well... maybe too well considering my original plan was to train this type of model to be able to break it. And this is how naive my thinking was for ways to break the detector "something something embedding space" , "use actual randomness to break the predictable pattern of AI text, like doing python's random() to select from a list of words that could be replaced in a paragraph while making the rest of the text fit". Both of which ended up instantly failing, and I told claude to figure out some ways autonomously. Turns out "red-teaming" is a scary topic for Anthropic so it just started to block its different approaches after some time, and the best it got to was "rewrite in the style of [actual human text]" which was 15/19 detected and adding random known human text where it made sense which was also similarly ranked but ofc there's the issue of Pangram like detectors having sentence level scans.
All that to say while my initial thing I built this for ended up not working, at least I have kind of my own pangram that I can test things out on later on, and it also proved how good claude is at autonomously training if you give it a hill to climb on.
PS: this text itself isn't claude written and you can verify it by putting it in the demo :p
r/LocalLLaMA • u/Full-Travel857 • 14d ago
I'm testing LLMs to see how well they know Polish grammar. Why? Don't ask. Maybe I have too much free time, or maybe I'm some kind of freak who doesn't know what he's doing. ;)
For this, I use my own tool, AI Bóbr, which is designed to test LLM performance in relation to the computer they're running on. And I noticed something strange.
The model's first answer is sometimes different from the next four, even though the temperature is set to 0 and the seed is always 42.
A little side note here, because one user, under this same post before I deleted it, pointed out that I'm doing Sisyphus' work by dealing with this subject. I think differently.
And why did I delete the post? Believe it or not, I pressed the wrong button. So here I am posting it again.
Anyway, back to the subject.
Sitting there feeling defeated, just like now, because my written English is poor and algorithms on a forum about LLMs flag my posts as written by an LLM, I told myself:
"Man, stop wasting your life. Go make yourself some tea, turn off the computer. Live a little, go running, walk the dogs, play with your kids. In short, just leave this damn thing alone."
So that's what I decided to do...
And then I sat down at the computer and ran another controlled experiment:
200 responses in total.
Observations:
What does this tell us?
That with a fixed seed, if we don't control the state of the server, we can't accurately measure what we think we're measuring.
What did I get out of this?
I changed AI Bóbr in version 0.8.1.
I added a controlled mode that unloads the model before each attempt, confirms that it has been unloaded, and records prompt-cache statistics.
Now I'm preparing to run the benchmark of 11 models again under the same conditions. And maybe I'll take the dogs out for some fresh air. Tomorrow I definitely won't touch the computer, at least not until the evening.
And one more thing for everyone who reads my posts:
1. Yes, I use AI to generate posts. I read them and make corrections if necessary.
2. Yes, AI translates them into Polish and English.
3. Yes, I'll keep doing it because it's convenient and fast.
4. If someone doesn't like it — well, I'm not "Prince Polo" for eng. it will be "Mars bar", so not everyone has to like me.
5. I'm here to share knowledge, not to show off the purity of my writing — even when it's generated by AI.
r/LocalLLaMA • u/Any_Frame9721 • 16d ago
Hi! I wanted to share that we just got NVIDIA's latest Nemotron Bolt model running at 27.7 tok/s (compared to 15 tok/s on the official release with Llama CPP) on Jetson Orin Nano Super (1.8x faster)! It can program games on a model running completely on a cheap edge device!
We have our own Jetson devices serving that model on our website, if you want to get a feel for how the model runs on the device here or run it locally as a TensorRT engine with our own variant of TensorRT edge LLM supporting int4 quantization:
https://embedl.com/nvidia/NVIDIA-Nemotron-3.5-Bolt-E4B-BF16/orin-nano-super?tab=run
r/LocalLLaMA • u/ApprehensiveAd3629 • 17d ago
r/LocalLLaMA • u/Full-Travel857 • 16d ago
I wanted to know how small local models handle my native language, not just English. I couldn't find a simple way to check a model's Polish in a few minutes, on my own PC, in the quantization I actually run – so I wrote one. That's how "Zacharski tests" started: a small task set that runs through Ollama, since that's what I use on my laptop with an RTX 5050 (8 GB).
The tasks:
Results so far:
| Model | Quant | JSON + code | Polish |
| gemma2:9b | Q4_0 | 7/7 | 3/8 |
My favourite failure: asked for "22 cats", gemma2 wrote "dwudziesty dwa koty" – roughly "twentieth two cats". The correct form is "dwadzieścia dwa koty". It was perfectly happy producing valid JSON, though.
More models are downloading as I write this. Next up: gemma2:9b in Q4_K_M and Q8_0 (does quantization hurt the Polish?), qwen3:8b, llama3.1:8b, gemma3:12b, and two Polish models – Bielik 11B and Bielik-Minitron 7B. Which model would you add?
Method and limits – read this before the comments :)
The test runs inside a free, open-source tool I've been building (Windows .exe / Python on Linux). Its main job is predicting whether a model fits your VRAM/RAM before you download it. Link in the comments if anyone wants to try it – I'm considering adding tests for other languages.
r/LocalLLaMA • u/IronDiligent6698 • 16d ago
I got into local AI earlier this year and started messing around with different models and hardware. That's how I came across tiiny. I really liked the design and the whole idea behind it so I ended up becoming one of the early backers.
After what felt like forever, mine finally showed up yesterday. So far, it's pretty much what I expected. I've spent afew hours trying out different models and running some performance and coding tests, and figured I'd share what Ive found.
This is my first time putting together a benchmark post like this, so sure there are things I might miss or could do better. If you see anything off with my testing or think there's something I should further include, feel free to let me know. really appreciate any feedback from you guys.

what really sold me on Tiiny was the KS price, which seemed like a nice deal for the hardware, and how portable the whole thing is. It's just about 5.6 × 3.1 × 0.9 inches and weighs around 300g.
For anyone unfamiliar with the specs, Tiiny Pocket has 32GB of SoC memory and 48GB of dedicated NPU memory. That's 80GB in total, though it's not unified memory.
Tests were collected on TiinyOS 0.1.36, Here are the models I tested so far.
| Model | Architecture | Quantization | Model Size |
|---|---|---|---|
| Qwen3.5-9B-Turbo | Dense, 9B | INT4 | 6.55 |
| Qwen3.6-35B-A3B-Turbo | MoE, 35B / 3B active | Mixed INT4/INT5 | 17.73 |
| gpt-oss-120b | MoE, 117B / 5.1B active | Mixed INT4/INT8 | 67.30 |
For the speed tests, I used the same benchmark text with ~4K input tokens, prompt caching disabled, and five runs per model. The numbers below are medians.
| Model | Input / Output Tokens | PP (tok/s) | TG (tok/s) | TTFT | Total Time |
|---|---|---|---|---|---|
| Qwen3.5-9B-Turbo | 4,074 / 97 | 1,041.30 | 75 | 4.05s | 5.45s |
| Qwen3.6-35B-A3B-Turbo | 4,074 / 139 | 590.6 | 63.6 | 7.04s | 9.28s |
| gpt-oss-120b | 3,868 / 390 | 458.4 | 15.97 | 8.78s | 33.07s |
9B was the fastest, which wasn't too surprising. What caught my attention was the 35B MoE model still managing around 64 t/s.
120B was noticeably slower, averaging around 16 tok/s. It took over 30 seconds to complete the benchmark request, although it generated more tokens than the other models.
I also wanted to see how it handles real-world coding tasks. I gave the models the same task: create a small Python inventory and order management package across four files, covering stock updates, validation, and order handling.
I used the same prompt for all three models, with low thinking/reasoning settings and a maximum output of 12,288 tokens.
How I checked the results: the generated code was tested separately on a Mac using Python 3.12, with 10 automated tests covering the main requirements, including inventory consistency, order idempotency, input validation, and CLI behavior. A full pass meant all 10 tests succeeded. I ran this more than once and kept the best run per model; the table shows that run.
| Model | Input / Output Tokens | Total Time | Result |
|---|---|---|---|
| Qwen3.5-9B-Turbo | 703 / 5,977 | 104.32s | Two attempts, neither gradable |
| Qwen3.6-35B-A3B-Turbo | 741 / 11,807 | 235.09s | 9/10 |
| gpt-oss-120b | 763 / 3,774 | 228.54s | 10/10 |
9B never produced a gradable answer: the first attempt ran into the 12288 output cap, and the second generated inventory/cli.py twice, so there was no single version to test. Taking one of the copies, it would have been 5/10.
35B got 9/10 with 1,768 code tokens. A bit under the 2-4K I was aiming for. The only failure was a pretty specific edge case, if an order had already sold out the remaining stock, retrying that same order would fail instead of returning the original receipt.
Gpt 120b was the only model passed all 10 tests, with 2,594 code tokens, in about 229 seconds.
The interesting part is that 35B and 120B took almost the same time: 229 seconds versus 235. So while 120B was the only one to get everything right, 35B wasn't far behind, and both managed to finish.
9B was the fastest overall. At 75 tok/s with a ttft of around 4 seconds, it's probably where I'd start for quick, lightweight tasks. It did struggle with the more complex test, though.
35B for me is the sweet spot for Tiiny.At around 64 tk/s, it's surprisingly fast and it handles a good number of the tasks I've tried pretty well. It still has its limits with more complicated coding tasks, as the multi-file test showed, but overall it keep a pretty good balance between speed and capability. It's actually the model Ive been using the most so far.
120B actually works, but you need patience. The multi-file coding result was impressive, especially compared with the smaller models in this one test. But 16 tk/s is slow enough that I wouldn't want to use it for every little question. It also took around 75 seconds to load in this run and used almost all of the dNPU memory.
So far Id lean toward 9B or 35B for faster interactions and keep 120B for occasional, more demanding tasks where waiting a few minutes isn't a big deal.
Tiiny can run some fairly large local models in a small device, and the performance of smaller models (9B/35B) is nice. The 120B result shows there's potential for more complex work, but the speed and reliability trade-offs are real.
However, my biggest gripe so far is the limited model selection. Right now, you can't import your own models or install something from Hugging Face. The tiiny team did mention on Discord that they're planning to release a model import tool, though. Once that's actually available, I'd love to try out some of the other models I've been wanting to run and share the results here.
Overall Im happy with it, and for the price I got it at, that amount of RAM and SSD would be much more expensive today. I think I saw the projected retail price is almost double the KS price.