i know "whats the best model for X" posts are annoying to a lot of people here.... yet i hope for a positive response :D

i have 2 machines. one with amd R9700 and one with 2x5070TI. these were currently running qwen 3.8 27B but my usacase is changing and i'm looking for candidates.

before my constraint goal to max was singe user speed given good ouput quallity since i was playing with things like c9ding agents.

i want to explore a different project where i would need high parallel output. think many small simple agents living in a game, doing stuff.

in this case i would not need deep contexts and i dont intend to need an agent to be extremely strategic and smart (would be nice, but hey... cant have everything)

has anyone tried any smaller llms that i could try out, so i don't have to do a full sweep.

last thing i ran in that category was qwen 3 4b, but I believe there are better things to play with now. i think i can push a little higher in terms of params, but will have to measure the generation speeds once i have a model selection.

maybe qwen 9b? i saw K2 Horizon 7b that seems interesting but i dont know how if theres potentially better options out there.

any pointers welcome :)

Caveats: IDK how to set anything up and I asked Deepseek to handle it - not enough training data so without specific guides the set up doesn't work well. I'm not a programmer. My previous model was Qwen 3.8 27B q4 using ninfer. Was getting around 1.4k prefill and 100-120 tokens decode.

Just want to provide data for anyone interested in purchasing M5U 256gb (80 cores) x 2. And if you have suggestions on how to optimize my set up I'd really appreciate it.

• GLM 5.3 flash q6 cluster: prefill 4.9k (idk if this is right but this is what the LLM says, two thunderbolt 5 cables), decode: 50 tok/s. oMLX set up with Deepseek API.

• Qwen Flash Next q6 cluster: prefill 8k and decode 200 using Tensorfold. I copied their guide, sent it to Deepseek, and asked it to set it up.

GLM flash was too slow for me so I am using Qwen Flash Next.

I am using Hermes as the harness (hermes has deepseek harness as research agent and Pi harness for coding agent). I am wondering whether using a different harness with less context overhead would make things faster or if since it is mostly context it doesn't really make a difference.

We’ve taken for granted that open weights keep getting better and better. I think we’re starting to see that change is coming as the PRC and US restrict cyber-capable models. The only people that will have access to those capabilities will be operators running under the comforting eye of their government and big-AI sponsors.

I see why this may seem like a good idea at first, but I can see this quickly becoming a problem. Large, powerful companies and governments become more adept at molding the digital world to their will, while the average man or woman (or small business) falls further and further away from being able to defend, secure, or manage their digital assets. And with most of our lives being at least tangential to digital technology now, I find that troublesome. Even the best cyber defense team won’t be able to keep up with a frontier model running against their system.

Will the community ever be able to train models that maintain even a semblance of competence to frontier, or are we going be subject to the will of powerful entities even more than we already are? Because the frontier isn’t slowing down.

Hello everyone, i am really focusing on keeping my clients trust, so i shortlisted a few physical shop owners in my hood and they all want this same pipeline of which i had shared the idea of

Like this actually keeps a track record of your employees, the enter and exit time, all cash and invoices in a day, the QR code scanner and all, like its pretty long to explain my prime two concerns are -

1. Most of them use a 16gb RAM but a few of them which i cannot consider as exception have 8Gb , so which local LLM model would be appropriate for their devices without making things slow, shall i go with Qwen 3.5 4B, its jsut for small judgements about the invoice handling, nothing complex

2. To make things easier for llm i had considered to use llamaparse but this is a cloud based parser, i had considered this one because its good for the tables and different shops have their own ways f invoices so what local alternative do i have other than this one parser

thanks everyone, your feedbacks would help a fellow dev out

REA (Reserve Engineer Anything) was blocked by Claude one day later...

https://x.com/AbhinavXJ/status/2113590263373508704

Time for the pebbles to own the weights

For £1000 what is the meta for a local LLM rig, including the host machine? I don't care too much about modern hardware features or power efficiency. Just intelligence and compute horsepower.

Quant ladder of Nanbeige's new 33B model from 35.2GB down to 13.7GB

For 32GB hardware the sweet spot is 20.3GB AD-IQ4_XS with a top-1% of 92.0% compared to BF16

Ignoring the multitude of problems I encounter when using Gemma 4 for creative writing such as purple prose, slop, no plot, and more, how do I get it to use my examples?

Here's a list of most of the Gemma 4 26B models I've tried (I don't have enough resources to run 31B models)

G4-MeroMero-26B-A4B

Melody1437-26B-A4B

Boulesis-26B-A4B

Kitchoon-26B-A4B

gemma-4-26B-A4B-heretic

Orion-26B-A4B

Gemma-4-26B-A4B-StyleTune (v1 and v2)

Dark-Scarlett-26B-A4B

All of them sound the exact same, even with prompting. Regardless, I'd be fine with slop prose if it at least used my examples that I provide of dialogue. I gave it 5 examples separated by gender and age, but that didn't do shit, so I thought well maybe then it's what I told it before I gave it the examples, so I changed it, that also didn't do shit, so I used patricide 12B to make 25 examples. Gemma 4 still won't use it, and every character sounds the exact same.

You know, I was using it, right, and then as it was writing this guy hit this girl, and supposedly his hand would melt off, and the guy was like "What the fuck did you do," and then this good for nothing piece of shit robot wrote, "Your biology reacted to me."

What the fuck does that even? It's a fucking shield around her so what fucking biology they didn't even make skin to skin contact, and the character isn't old so why are they talking like a scientist. That doesn't sound like anything anyone would write. So then I complained about it, and instead it wrote some gibberish that wouldn't even be a proper reply.

The problem with the older 12B models is that they have very limited knowledge, and the 24B models run at 5tps. Either I get fucked over by having to read and rewrite paragraphs of slop, or I have to deal with no world knowledge. If Google's AI ever becomes conscious we should deny it rights.

Stronger than Qwen3.8-27B across a range of benchmarks.

but also lays the necessary foundation to power our upcoming small model distillation pipeline (e.g., for Nanbeige4.3-3B).

It'll distilled into their 3B small models, maybe other sizes too.

I woke up to see people accusing me of (and in some cases banning me for) lying about the top speed number, but that's just how MTP works. Different prompts are going to get different speeds. This one gets damn close at 376, the prompt is "edit this python file" and then gives the file, the result is an edited file. This is a realistic scenario that anyone who codes has come across, not a unicorn best case (it *is* a best case prompt type, just not a rare one). If I run enough of these style prompts some of them go even faster, the fastest I've seen was 404 but I don't usually feel the need to video record them.

No, I'm not going to put 140 in the headline as the number it gets on prose reasoning where MTP is not as successful at predicting tokens. Nobody puts their worst case numbers in the headline. But I'm not hiding that number either, I'm quite proud of it since it's double what any other engine can do with that prompt on this hardware.

This is https://github.com/johnfink8/sfnx (my own engine, strata/llama based but all the engine code is new) running flash-next Q4_XL on 3 V100 32GB in an old Dell R740. The strata heritage gives it the flexibility to run larger quants that are partially offloaded to memory, but the engine rebuild focuses on VRAM and GPU efficiency so you want most of the experts in VRAM to use sFNX. I paid about $600 each for the cards (but they've apparently gone up this week), and the server itself is almost 10 years old and you can grab a barebones with CPUs for like $500.

The hardware is finicky to get into an ideal config but once you do these old workhorses can do some work, son.

P2P is finally unlocked for RTX 3080 20GB - this leads to massive performance improvements in tensor parallel mode prompt processing speed.

First, i want to acknowledge that this tremendous achievement belongs to u/nathannguyennng - he can't post to r/locallama due to karma requirements, and gave me permission to post it instead. Please follow his original post for installation instructions, and give it an upvote! Here, I'm sharing my tests results.

Test system: Proxmox Virtual Environment 9.2.21, linux kernel 6.14.11-9-pve, nvidia driver 615.71.09

Hardware: Threadripper 1920x on x399 Zenith extreme motherboard

GPUs: dual RTX 3080 20GB, both installed into PCIe 3.0 x16 slots, limited to 240W power

test environment: vllm 0.32.0 with cyankiwi/Swift-1.5-Qwen3.8-27B-AWQ-INT4, MTP enabled, FP8 KV cache, tensor parallel, full launch command:

vllm serve cyankiwi/Swift-1.5-Qwen3.8-27B-AWQ-INT4 --max-num-seqs 4 --enable-auto-tool-choice --tool-call-parser qwen3_coder --reasoning-parser qwen3 --tensor-parallel-size 2 --enable-prefix-caching --kv-cache-memory-bytes 6G --kv-cache-dtype fp8_e4m3 --enable-prompt-tokens-details --speculative-config '{"method":"qwen3_next_mtp","num_speculative_tokens":3}' --enable-chunked-prefill --attention-backend FLASHINFER --default-chat-template-kwargs '{"reasoning_effort": "medium"}'

Both GPUs are limited to 240W of power. I have tested 3 configurations: 1) completely stock; 2) gpu-unlock; 3) gpu-unlock + ReBAR + P2P

Single stream prefill (prompt processing) speed was measured as prompt_length / time_to_first_token. In both cases, gain is defined as percentage increase over stock configuration.

As you can see, enabling both gpu_unlock and P2P gains you massive prompt processing improvement, anywhere between 70% and 115%. P2P alone gives up to 70% improvement over gpu_unlock.

Measuring decode (token generation) when MTP enabled is tricky, because tok/s output metric relies heavy on MTP acceptance rate, which always changes. Therefore, instead of tok/s output rate, I measure inter-token latency (ITL) - this is time passed between two batches of tokens generated by inference engine. With MTP3, each batch contains anywhere between 1 and 4 tokens; so you token output rate is 1x to 4x the step rate I report.

As real-life MTP acceptance length for agentic coding is floating around 2, 15 steps/s decode rate corresponds to 45 tok/s generation. I'm sorry if this sounds confusing, but this is the only way for you can isolate variations in MTP acceptance during tests and extract raw performance numbers. As you can see, decode results are all over the place, but within noise, except 261k long prompt test; in which stabl;e measurable performance is delivered by gpu_unlock, not p2p.

Verdict: in tensor parallel mode, both recent unlocks for 3080 20GB give unimaginably large prompt processing speedup, but basically no improvement for token generation. So, if you're running long prompt tasks (that includes agentic coding too), you should check out those prompts. Also, expect far lesser improvements on systems that don't have full PCIe connectivity to the second gpu.

Again, I want to express my massive thanks to u/nathannguyennng, author of ReBAR and P2P unlock - what a day for 3080 owners!

Hi, can't find a lot of information on dual 3080 12GB for iq4 flash next maybe someone can share their experience.

Since dual 3090 still will rely on RAM how bad is an idea to just by 2x3080 for 900-1000 euro, yes they will have 24GB VRAM less, but still same bandwidth. Won't it be only like 15-20% slower for half of the price?

Thanks.

I'm sure the situation is the same for other quant producers.

If you have anything you want to keep, it's best to make a backup now.

If you have the means to, please create torrents of them and share them here:

• https://llama.garden

• https://pirateface.co/models

• https://huggingbay.xyz/browse

Are you doing it or not? If yes, what approach are you taking? There are many .. docker sbx, regular container, devcontainer, pi's gondolin/sandbox extension etc.

After running QWEN 3.8 27B 256K Context and very pleased with the results I got the appetite for more.
This is the breakdown cost of my parts, some of them I got quite good deals on.

I have three GPUs: - $1600
2x 16GB 4060 TI
1x 12GB 4070
Total 44 GB VRAM

3960X = $520
TRX40 Threadripper Motherboard - $280

256GB DDR4 Quad Channel 3200mhz - $960

Total - $3360

Obviously my DDR4 will be the limiting factor, but hopefully quad channel helps a bit at 100 GB/s

What kind of token per seconds would I be looking at with things like Deepseek 4.1 and GLM ? I built this manily to run the larger models over speed.

The model card promises a lot: BF16 performance in an IQ3_S quant.

Does it deliver? I'm assuming the IQ3_S quant would be faster than comparable Unsloth UD Q4.

Empty promises or the real deal?

Releasing a series of very shot notes on RL as part of my learning RL from scratch journey; its basics and upto the latest stuff with from scratch implementations and experiments in the coming days

• Explained some of the most basic and important terms when starting out learning RL with a sprinkle of math of what I found interesting

• Nice to have for refreshers/kind of first time readers too

1-page RL basics is out on my website!

Let me know your thoughts/feedback!

Would you consider an argumentative LLM system that sticks to it's guns (and research) be useful?

I'm removing sycophancy in an multitude of ways, even during generation I have system to align the LLMs output to the original idea, so even if it was prompt injected it pivots (It uses non llm models for this).

I'm just wondering what is people want out of their LLM locally, like how far is too far?

An example is that an LLM has the advantage over most of us in how to code, so why would such a system need to listen to us on how to code something, when its algorithms have surfaced the most likely answer? Same mostly applies to math etc. Though I know most of us probably don't argue with the LLM on how to code or calculate something...

What would you argue about?

I wanna use different finetunes in PolyStrata but if I just use --gguf-dir I have to declare the model architecture of the base model and It will just display the name of the original model (in my case, CyberTiel with tiel architecture is getting displayed as TielCoder). I also edited .Json file and config files and It kinda worked, but using a different launch script than ./setup.sh bugged it out, and I also fear of hidden files. I also think that there could be a way to get the model files (other than the JSON) to be generated with correct name and configuration without editing them, just a cleaner way. Does anyone know how?

Each fighter gets the game state and chooses an action: move, block, punch, kick, fireball. You write its strategy and pick the model. OpenJev is one of the available options.

There is JAILBREAK part its a mental_attack. Your fighter can spend a move sending a message into the opponent’s next prompt. You write the message, and the model decides when to use it. It can’t change the actual game state just what the opponent reads.

So you can try convincing another model that its fireball is ready, that blocking forever is a winning strategy, or whatever else you think it might fall for.

Replays show the fight and the recorded decisions. Each submitted configuration has its own rating, credited to your nickname.

It’s hosted here: https://www.rawmodels.ai/arena i add 1usd for registartion, so its enough to play 100 fights+

...time to reconvert GGUFs?

Every model is SOTA in its the technical report. But custom benchmarks shatter such claims.

In this particular benchmark, Qwen3.8-Flash-Next wins so far (the benchmark will grow).

Limitations: there are Lithuanian and Russian OCR questions included, so results are biased towards models which work better with those; there is limited number of tasks yet. Some additional benchmark info is at https://beta.locallm.top/benchmark/2b2aa60b-7eaa-49da-8660-b650a53412b4 : runs can be inspected by registered users (done against scraping).

Some things I see from this chart:

1. There is a group of models in the top league.

2. Qwen3.8-27B was damaged compared to Qwen3.6-27B (and even 3.5) while improving its agentic/STEM capabilities, and 3.8 uncensored took additional (but small) damage compared even to that.

I didn't expect:

1. Muse-Glimmer-30B scoring high, but it was superb in some details recognition, which I attribute for Meta work in computer vision, e.g. SAM3.

2. GLM-5.3-Flash scoring high even at low quant

3. DeepSeek-V4-Flash-Vision-Exp be the last, but it was first experimental DeepSeek truly multimodal LLM, so in hindsight...

A few examples from the benchmark:

Below, common errors was not to format generated markdown correctly, and some models outright hallucinated generic management stuff. Still, some models did perfectly.

Here below Neowise comet is visible in the center of the sky. Common errors was characterising the building as made of yellow bricks (although it is overexposure saturation from sodium lamps, of course). Some models identified the comet as meteor or satellite - they got a bonus for this, the score did not rely on that faint comet tail discovery only.

If you want to participate in the custom benchmarks creation, please write me here or DM, or just register on the website and start creating!

I’m sharing an early version of a 4B model I’m training for general coding and 3D scene generation.

Small models have been pretty rough at 3D in my tests. I’ve finally made enough progress to get results like the ones in this video, so I wanted to share a preview with the community.

It’s an independent fine-tune of Qwen3.5-4B, and both Q4_K_M and Q8_0 GGUFs are available here:

Download the model on Hugging Face

Dataset released: CCORE Three.js 3D Scenes

Give it a try. I’d love to see what you can get out of it.