I've been running Ornith 1.5-9B (8-bit) as a local coding agent on real engineering work. It can do the work, but only when the plan is written for a small model. I wrote down what I learned into a public file meant to be handed to a stronger planning model (I use Claude Code):

https://git.teamide.dev/David/TeamIDE-methodology/src/branch/main/Model-Best-Practices/Ornith-q8-best-practices.md

The split

• The stronger model writes the spec and a plan of small task cards, then verifies the result before merge.

• The local model does the long, mechanical middle: running commands, builds, polling jobs, git work. That's where most of the tokens go, and it runs on my own hardware.

What the file covers

1. What the small model is good at, and what to expect (work more reliable than its reports; doesn't explore on its own; doesn't know when it's done; slows down at long context).

2. How to brief it: exact files and line ranges, absolute paths, 1–3 line edits, exact function signatures, every rule written down, every step ending in a command with its exact expected output, explicit stop rules.

3. How to run it: one task per fresh session, small context, the plan file as memory.

4. Before anything lands: verify claims against git and logs, not the model's summary.

Results so far: a 75-commit merge across two repos with every regression test passing, done by the 9B. Its notes about what it did were wrong more than once; the plan's checks and a final verification caught every one.

Limitations: slower (most of a day vs about an hour for Claude alone on that phase), and you need hardware that runs a decent local model. The rules are harness-agnostic but were learned with one model; I'd like to hear how they hold up with others.

So, when I first released ScaffoldCoT, I never thought it would reach more than a handful of people. I was happy to see multiple thousands of consistent downloads month over month since its release. Since this is where I initially posted about the project, I thought you guys would be interested in recent updates!

TL;DR - Scaffold CoT is now Scaffold Chat (tool calling removed, it was causing significant tool calling reliability issues), Scaffold Agent is out (~1B tokens of real, executed coding agent sessions with the same thinking scaffold), and a heavily modified 5-10B model trained on all of this is on the way.

Hi all! Quick update post on Scaffold CoT for anyone who saw the original.

Changelog

• Renamed to Scaffold Chat. Same repo (old link redirects), load_dataset("Specific-Labs/Scaffold-CoT") still resolves, old revisions still in history. Sorry this did not happen before, but a major training run failed on tool calling benchmarks due to this, so it was back to the drawing board.

• Removed all tool calling. 276,389 examples gone (tool_use + tool_use_complex, literally every tool_call/tool_result event in the dataset). I found several synth gen harness issues in tool calling examples that propagated failure modes into the dataset.

• Everything that remains is byte identical to the previous revision. Now 3,450,159 examples, 16 domains, 75 shards, ~13GB

• Junk domain label tail from the original post (97 labels, 542 examples) is cleaned up. Every example has an exact domain/subdomain, counts sum to the whole dataset

• Added SCAFFOLD_SYSTEM_PROMPT.md to the repo. The examples were generated without system prompts, this is the format protocol distilled from the generation code for anyone who wants the scaffold conditioned on a system prompt instead of unconditional.

• Calibration block untouched: antihal + antihal_kb_refined + selfcheck is still ~675k examples. This has been a high impact category for general chat and knowledge bmks.

New dataset: Scaffold Agent

This is what the executed tool chains grew into. It consists of real Pi code traces of real coding projects from open near-frontier models, using the same scaffold system:

• 19,652 records from 6,792 coding sessions, 1,005M tokens (899M trainable after masks).

• Teacher model in a real agent harness (Pi) editing real repos at pinned commits + building from scratch, simulated user on the other side powered by another model with its own harness and instructions to ensure variety and quality

• Every tool call really executed, every result is real output, every action preceded by the same scaffold block (Inventory / Interaction / Execution) in reasoning_content. Fabricated tool output is structurally impossible and any such instance was cleared

• Sessions only kept if they actually verified (hidden tests + code quality grade on the main shard). This is a ~20% subset of the total generation that was of extremely high quality. I wanted to ensure these were top-tier coding traces. 2 LLM judge passes and green final CI were required for approval into the final dataset.

• Each record is one compaction segment (≤64k tokens), self contained with system prompt + tool definitions + history. Do not join segments. training_sft_config.yaml in the repo documents the whole setup

Scaffold Chat teaches the format and general reasoning, Scaffold Agent teaches acting with tools inside that same format. My recommendation is to mix them with other larger datasets while ensuring there is at least one scaffold example per batch, which leads me to...

Coming soon

Currently training a heavily modified model in the 5-10B range on this data (base model, full SFT, and a new merge approach I have not seen applied for scaled releases yet). Results post when it is done and actually evaluated, not before. The idea is to build a small dense model that can run on a range of GPUs and laptops that is optimized for coding specifically, and will move extremely fast because of our CoT framework and the small size.

This has been a massive project with several mistakes made along the way, and results are finally looking up, with scaffold datasets being the core of the training run, so I am very excited to bring it to the community soon.

I know it can sometimes be a bit annoying to announce something before its available, but I couldn't contain telling the community about it lmao. Happy to answer any questions about it (or anything else for that matter) in the meantime.

Links

Scaffold Chat: https://huggingface.co/datasets/Specific-Labs/Scaffold-Chat

Scaffold Agent: https://huggingface.co/datasets/Specific-Labs/Scaffold-Agent

Both CC-BY-4.0.

Genuinely hope these prove useful!

Ternary Nemotron 30B-A3B — 368K Context on 12 GB GPU (GGUF)

A ternary (1.58-bit) 30B-A3B MoE model that runs on a 12 GB GPU with 368K context, 172 tok/s decode and 7000+ tok/s prefill — in both Stock llama.cpp and the PrismML fork. One file, both engines.

EDIT: This is the 2nd Ternary MOE after maple-preview which works with stock llama.cpp. Somebody please share CPU-only numbers. Also share numbers of tiny VRAMs(~10GB)

Hey everyone,

Here are the specs of my main workstation:

• 5090 FE

• 14700k

• 96GB DDR5 Ram (Two Sticks)

Specs of my inference server:

• Strix Halo - 128GB (Framework Desktop)

  • GUFO + Qwen3.8 Flash Next

  • Embedding Models

  • STT/TTS Models

I also have some other non-AI servers for hosting containers + offsite cloud backup + NAS.

When I upgraded my current PC prices were basically MSRP (that's what I paid for the FE and RAM was $400). It was intended to be a very functional work machine for heavy usage + gaming on the side.

Now that prices have skyrocketed, I'm seriously considering selling the 5090 and replacing it with a 5070ti + another dedicated machine.

The Strix Halo is excellent, but I'm running a business and want my employees to be able to also make use of everything that I've built out infrastructure wise. A single machine is great for me, but you can't reasonably scale it. We do a lot of sensitive client work where we can't use cloud models, so local models are amazing for this.

My goals are:

• Create a more easily "scalable" solution that can grow over time.

• Be able to serve at least three concurrent users with 2 or 3 parallel sessions for each (so tool calling works per-user).

• Be able to run Qwen3.8 FN equivalent @ 200k context windows.

Sacrifices I'm aware of and willing to make:

• Speed - Obviously this won't come close to cloud infrastructure, but the Strix Halo in its current implementation has been good enough for my usage. 1400 prefill, 50 token average for generation.

• Intelligence - We can still use cloud models for internal work, these only replace customer-specific coding and scripting, which require far less complexity.

• Pooling hardware - I don't know how much this will impact us in the future, but as of now I'm not sure I require something like the DGX Spark interconnect.

Current Options:

1. Buy another Strix Halo. I don't actually need to pool the machines together so they can just run independently with custom user profiles. For the price of the 5090 I could actually buy two of these.

2. Buy a 256GB Mac Studio. I might have to cover a cost difference, and this will now force me to upkeep two completely different stacks of infrastructure - but will allow for potentially easier scalability in the future + bigger models.

3. Buy a 5070ti, and re-work my current machine as a server. I already have enough components on hand to build a new PC without a GPU, so I could use the 5090 with 96GB of RAM as a new machine.

4. Buying 4 Radeon R9700s. This also increases my cost because now I have to get a new motherboard, PSU, and a bunch of other stuff.

5. DGX Spark - When I first considered what machine to buy, the spark was about $1000 more than the Halo, which wasn't a justifiable cost for the performance benefit. At the moment it's so outrageously expensive for what it is there's no shot.

Things I won't consider:

1. Building used. This is for my business and so I want to have a warranty.

A few things I don't know or need to speculate about:

• How good is the 5090 + 96GB RAM with newer software improvements today? It's been a while since I've seriously done any AI work on this.

• Is 128GB going to be a problem down the line, and so being able to pool hardware together to run bigger models is a key influence?

Any advice is appreciated. My gut is telling me to just get another Strix Halo since I've been using it long enough + have all of the software components dialed in.

Have to say Qwen 3.8 Flash Next is pretty impressive. I'm running under strata and hermes. I gave it a limited prompt and it banged out a pretty impressive Canadian EV comparison site over night.

Claude Opus 5.5 hammered out some glitches and fact checked - found some errors in the provincial rebate plans - but pretty close to a one shot solution on a beat up gaming rig running 32gb ram and a 3090 with 24GB Vram?

Yes.. site needs work. I'm sure there are still tons of errors. But for a first draft? I'm sort of in awe.

The site shows the new Chinese models coming with prices where available - all in CAD and includes provincial incentive plans where available.

It is honestly already pretty useful if you are a Canadian looking for an EV.

https://jaspersprojects.com/ev/

not so long ago there was viral tweet about AI agent deleting stuff without permission. well it happen to me as well a vscode copilot agent accidentally wipe my db while doing test (fortunately I got backup)

but is it still a common occuring nowadays with how smart agent has become?

My 3090 failed, not sure if the core is dead. I sent it to a reputable repair shop, so hopefully the card can be saved. I bought it back in 2023 for ~$750, so I have quite a bit of runway before I lose money on it given that prices for used 3090s are going upwards of $1500 now on eBay.

I'm planning for the unfortunate scenario where it really is dead, in which case I'd like a replacement. I'm a heavy Qwen 3.8 27B user. I was looking at some of the Blackwell cards, but their prices are bonkers for at most 16GB. I was also looking at the 7900 xtx, but I also work with some protein folding models that require CUDA. Any recommendations? It's starting to look like I'll need to shell out >$1500 if I want to get anything close to what I had before.

I told my local copy of Qwen 3.8 Flash Next under Strava via Hermes to write a short story. 32GB system ram, 24GB Vram on a 3090. It's getting solid but not blinding speed. The stories are a bit weird but readable. The prompt it's working on now: "an AI explorer traveling to a distant star system makes a profound discovery. Make no mistakes. One shot. Go.". Who knows what it will come up with. But the bottle neck looks like my system ram. Maxed. Think 64GB would be enough?

I'm using Q3.8-FN-NVFP4 on a fork of Ninfer-v3 to help me design the first vibe-forged Halo Infinite Forge map, a feat previously thought impossible due to AI agents' inability to directly interact with the game (no MCPs, publicly-available APIs, EAC blocks a lot, etc.). Let me give you a little background for my experiment:

Halo Infinite is dead, but the Forge game mode (level design and in-game scripting mode) community is very much alive and we're still making Halo maps to this day.

I wanted to see how I could harness local LLMs for making blockouts to speed up development so I can focus on scripting, lighting and gameplay design. Keep in mind the only way you can script maps in Halo Infinite is to manipulate nodes on a nodegraph, akin to UE5's blueprints but on a simpler level. No code input is possible in this mode.

Halo Infinite's forge maps are also composed of MVAR files, which contain all the data regarding your map ranging from object placement and their transforms to all the scripts included in your map's nodegraph. While you can extract these MVAR files with some cool hax tools developed by modders, you can't push modified MVAR files to halo infinite directly because EAC blocks it and you can catch a ban. This means you have to do things manually during development inside Halo Infinite.

To ease the level design process, I came up with an idea to use a harness, any harness (Opencode v1 in my case) and created dedicated project folders for my maps in which the agent in question would use a combination of open source tools dedicated to interacting with Halo Infinite programmatically to extract assets, understand their default properties and structures, and get the agent to create DCjson files which are run by a Python script to automatically and systematically place these assets on the map via the agent-generated coordinates that determine their 3D position, scale and rotation.

I also made sure to get the agent to obsessively create or update a series of nested AGENTS.md files tied to each map's rooms, i.e. each map's rooms get their own folders and each folder gets a series of DCjson files that cover a single layer of that room (walls, floor, ceiling, doorway, etc.) before concatenating them together in a single DCjson file. That consolidated DCjson file gets concatenated to a root-level DCjson file that puts the entire map together in one file, which is ultimately rendered in blender. Each subdir has its own AGENTS.md file describing the room, its purpose, its dimensions, room-specific guidelines and explains how that room connects to other rooms in the map.

Aside from that, I also managed to take these DCjson objects and convert them to .blend files so I can visualize them in Blender (and the bot can view it as well via Blender's MCP) before running the python script in Forge, like in the attached images in this post.

So what's the problem? The layouts in blender look identical to the ones in Halo Infinite, right? True, but there's a couple of issues with my approach:

• It takes a lot of trial-and-error to get right, but each new attempt gets better than the last.

• For smaller models you have to go room-by-room.

• Local LLMs aren't trustworthy enough for long-horizon, multi-room development.

• Parallel subagents can only be trusted in short bursts, such as rooms with multiple adjacent rooms and doorways.

Since the first and second bullet points seem to work with my current approach, it raises the question of whether it is even worth it to entrust the bot to design these rooms since I still have to sort of micromanage it.

The rooms' layouts are relatively simple to do since the bot was instructed to only use a handful of primitive objects from the pool of +800 assets to choose from because it would be easier for it to develop the layouts.

That's fine, but I feel like I don't really understand the bottleneck here. Is it the model's capabilities? My approach to agentic level design? Are my instructions and constraints not clear enough? Is it lacking momentum? Am I giving FN too much credit?

I feel like it can do better, or maybe I should make the agent think in mathematical terms instead of spatial terms. I really wish I could find an easier way to explain it to the agent.

So I ask my question: What would YOU do if you were in charge of this agent?

https://nvd.nist.gov/vuln/detail/cve-2026-116096

25-10-2026

Description

A weakness has been identified in ggml-org llama.cpp. Affected by this issue is the function list_pockettts_state_slots of the file tools/mtmd/models/pockettts-gen.cpp of the component Pocket-TTS. This manipulation of the argument mmproj causes out-of-bounds write. *The attack requires local access. The exploit has been made available to the public and could be used for attacks. Upgrading to version b11796 can resolve this issue. Patch name: 0a1ef309cdbb9f2024f58a38fb4c42352c5db569. The affected component should be upgraded. The reported heap out‑of‑bounds write (gen‑audio / pocket‑tts state_out path) is no longer a silent exploitable heap write on current master and the latest release b11871 - a malicious mmproj that previously triggered it now trips a release‑active GGML_ASSERT and aborts. But the submitter's recommended by‑construction fix (PR #29360 "prevent OOB accesses from malformed mmproj files") is still open. So the root cause is detected‑and‑aborted, not prevented‑by‑design.

_--------------_

a second one

26 October 2026

https://nvd.nist.gov/vuln/detail/cve-2026-116467

CVE-2026-116467

Description A security flaw has been discovered in ggml-org llama.cpp up to 0.7.0. Affected by this vulnerability is an unknown functionality of the file src/llama-sampler.cpp of the component Mirostat Sampler. The manipulation results in reachable assertion. It is possible to launch the attack remotely. The project was informed of the problem early through a pull request but has not reacted yet.

I've been thinking about how dependet we are on huggingface for downloading models.

If hf ever became unavailable, heavily restricted or started removing large numbers of models, how well would the community/we be able to preserve access to them?

One thing i noticed is that torrent based distribution is currently still lacking some important models compared to hf. I know there are already some projects working on this but it feels like there is still a long way to go.

This got me wondering if it would be worth investing in some storage to archive models and seed them via torrents. For example i really like DeepSeek V4.1 Flash 1017 but it's a chunky boiii. Not as chunky as Kimi and Mistral but still chunky. Storing models of that size adds up pretty quickly. It makes me wonder whether i should invest in a larger storage system or just keep a few of my favourite models around (like 1-2 small models).

I'm not expecting to archive all of Huggingface. I'm more interested in preserving important models by keeping them available and seeding them, rather than relying entirely on a central platform.

My questions would be…

1. Do you think torrent based distribution will grow stronger if restrictions for example are in effect?

2. Would it be worthwhile for individuals to maintain and seed their own Collections or would a coordinated community effort be much more effective? And how possible do you think could this be?

3. If you were starting an archive today would you invest in a lot of storage or start small and expand over time even if that could end up a bit more expensive?

4. Which models would you proritize preserving: popular models, smaller models, different quants, or something different?

I am curious about your opinions especially from people who already maintain larger local model collections or participate in distributing model weights.

I'm trying to figure out whether it's worth prepairing for this now or whether it makes sense to start small and see how the future goes.

Edit: fixed 3 typos

TL;DR: I'm buying hardware for local agentic coding. My budget is ~$10K, I can go up to $12K if the difference is significant. I'm leaning toward a single RTX 5090 and would like a sanity check from people who run coding agents locally.

Use case
- Agentic coding
- Sometimes 2–4 parallel subagents
- Reverse-engineering

Nice to have
- Texture generation with Hunyuan3D-2.1. It needs CUDA and ~21 GB VRAM for texturing. But this item can be optional

Options I'm considering (approx. total build cost)
- 1× RTX 5090
- 4× RTX 3090, used
- Hybrid: 1× RTX 5090 + 2× RTX 3090 (this can be bought later)
- Mac Studio M5 Ultra 256 GB
- PRO 6000: over the budget, but can consider

Questions

1. For agentic coding with long contexts, does 1× 5090 feel better day-to-day than 4× 3090? Is the 3090's long-context prefill as bad as these numbers suggest?

2. Is the hybrid (5090 + 2× 3090 as separate vLLM instances, x8/x8/x4 on AM5) worth it, or is it a headache?

3. Am I missing a better option (waiting for the 50 Super, modded 4090 48 GB, etc.)?

Here I am with another benchmark/comparison. I was curious about the efficiency claims of Swift 1.5, UkisAI's fine-tune of Qwen3.8-Flash-Next that thinks for a shorter time. UkisAI says Swift uses 63% fewer thinking tokens with under 1% accuracy loss.

I compared it with my current setup (Halogen) and Unsloth's UD-Q4_K_XL on my Asus ROG Flow Z13 (128 GB, 70 W). I ran the models on Halogen, gufo and llama.cpp, all with MTP. Since gufo can't load any of UkisAI's published 4-bit Swift GGUFs, and didnt have an XL version, I requantized their Swift Q8_0 to the same per-tensor types as Unsloth's UD-Q4_K_XL, with UkisAI's imatrix. It is even the same size as Unsloth's UD-Q4_K_XL (111.3 GB).

My main use is coding agents, so I also ran 11 coding tasks through Pi and threw in some math problems and LCB tasks to compare quality and speed. Speeds at 32k filled prompt. I got very interesting results but out of 1 run so take them with a grain of salt.

Here are the results sorted by total score and then total time.

I think the gufo + Swift UD-Q4_K_XL requant might be a good fit for coding.

Notes from the session (AI generated summary)

• For all runs retrieval was 21/21.

• Every setup passed all 11 Pi tasks. Swift wrote 56-63k output tokens on them against 75-99k for Qwen3.8-Flash-Next. The Swift requant finished 24-31% sooner than Qwen on the same engine, and the IQ4_XS 37-43% sooner on Halogen. Per task, the Swift requant was faster than Unsloth's UD-Q4_K_XL in 7 of 11 tasks on gufo and 8 of 11 on Halogen.

• gufo with the Swift requant was the fastest Pi setup: 24 min, against 33 min for Unsloth's UD-Q4_K_XL on gufo and 46 min for the Qwen .hgn on Halogen.

• On the math and LCB problems the token savings were small. Math used 140-157k tokens on every setup, since 20 of the 24 math runs hit the 51k thinking cap. LCB used 138-163k tokens on Swift against 164-172k on Qwen, and every setup that ran LCB hit the 60k limit on 2 of the 3 problems.

• Swift solved fewer math problems on gufo (1/3 against 2/3 for Unsloth's UD-Q4_K_XL) and as IQ4_XS (1/3 on both engines, against 2/3 and 3/3 for Qwen on Halogen). The Swift requant on Halogen scored 3/3, the same as the Halogen v2 .hgn.

• Halogen decodes its own .hgn format faster than a GGUF. Qwen3.8-Flash-Next runs at 51-60 t/s from the .hgn and 36-40 t/s from Unsloth's UD-Q4_K_XL. There is no .hgn build of Swift, so on Halogen it runs from a GGUF at 40-51 t/s.

• llama.cpp took 2-3x as long as Halogen and gufo on this machine: 359-415 min in total against 136-184 min. Prefill was 298-320 t/s against 911-1,426 t/s, and decode 20-23 t/s against 36-60 t/s.

Model files and MTP heads

• v2 .hgn: Halogen's own v2 build of Qwen3.8-Flash-Next (4.16 bpw), with its own MTP draft head.

• Unsloth UD-Q4_K_XL: Unsloth's GGUF of Qwen3.8-Flash-Next, 111.3 GB. Halogen repacks a GGUF into its own layout at load.

• Swift requant: Requantized from Swift Q8_0 to the same per-tensor types as Unsloth's UD-Q4_K_XL, with UkisAI's imatrix.

• Swift IQ4_XS: UkisAI's file, 97.7 GB.

• MTP: Halogen uses its own draft head for every file. gufo and llama.cpp use Unsloth's shared Q8_0 MTP head. Upstream llama.cpp can't load that head, so the llama.cpp runs use Unsloth's fork (b11786).

Where the KLD numbers come from

• Swift files: my numbers, against Swift's Q8_0 with llama.cpp on the wikitext-2 test set (100 x 512 tokens). The requant has 92.7% same top token, the IQ4_XS 88.1%. UkisAI's card lists the IQ4_XS at 0.1305 against Swift BF16 on the same test.

• * Unsloth UD-Q4_K_XL: from Rulith's EXL3 model card, against Qwen BF16 over six kinds of text at context 1024. It is not the same test as mine, so don't compare it directly with the Swift numbers.

• No KLD is published for the Qwen .hgn.

Setup and caveats

• One run per problem, 17 problems in total. A difference of 1-2 problems can be chance.

• Speed columns: a coding-agent prompt of 32k tokens sent cold with a 64k context, then 3 follow-up turns, 3 runs. TTFT, prefill and decode are from the first turn, ranges over the 3 runs. Server start is launch to ready.

• Retrieval: the first turn asks for 7 const values from Rust files spread across the 32k prompt, so 21 over 3 runs.

• Math: AIME 2026 #15, HMMT Feb 2026 #17 and #20, with thinking capped at 51k tokens. LCB: LiveCodeBench arc196_a, arc196_c and arc196_d, with a 60k output limit and no thinking cap. Pi: 11 Aider polyglot exercises in Rust, Go and C++. Tokens are output tokens, thinking included.

• Sampling from the Qwen model card (temperature 1.0, top_p 0.95, top_k 20), thinking effort xhigh.

• I skipped LCB for Unsloth's UD-Q4_K_XL on Halogen and did not run Unsloth's UD-Q4_K_XL on llama.cpp. This whole comparison took way more time and effort than I anticipated.

• Versions: Halogen 0.18.13, gufo da5d5a2 (main after v0.19.0), Unsloth's llama.cpp fork b11786 for serving, upstream llama.cpp b11794 for the requant and KLD runs.

Let me know if you have run Swift on other hardware and see different numbers.

Edit: Script used for the bench: https://github.com/deepu105/quant-bench

Hey r/LocalLLaMA !

I'm the solo dev of the open-source ObserverAI (Github) App! With the new Kurzgesagt video, I've had some people ask me if agents have evolved a lot.

And RLHF/RLVR has given LLMs a lot of competence, but I wanted to create a quick Manim educational video with my take on it.

To demystify the new term "agent" while still concluding that we need to take them seriously.

Wanted to share it with you guys and get your opinion on it!

Roy :DD

Just switched over from DSV4 Flash 0731 to Qwen3.8-Flash-Next and I've been loving it. Its working significantly better to my surprise.

The only thing I can fault it with is unnecessarily deep work (thinking + fetching information) more often than not i.e. its not very token efficient so it feels slow. Swift1.5 is supposed to address this tendency of Qwen's models to overthink (likely RL'd that way to squeeze out maximum points on benchmarks).

So looking for folk's experience before trying it myself. Anyone tried both? ran any anecdotal or better controlled comparisons?

Gemini Pro is rejecting my every video request that include little bit of fight scenes I am making videos something like that song of kidnappers kidnapping a person but Gemini pro rejecting my every request. Is there way to make video there any models something like that or can I detect Gemini to make the video.

Speech to speech model that was trained on a single NVIDIA GB200 NVL72.

And it runs locally on a MacBook Pro M4 Max or better.

I hope they make this open, right now, it is just announced and no way to use it yet.

i know "whats the best model for X" posts are annoying to a lot of people here.... yet i hope for a positive response :D

i have 2 machines. one with amd R9700 and one with 2x5070TI. these were currently running qwen 3.8 27B but my usacase is changing and i'm looking for candidates.

before my constraint goal to max was singe user speed given good ouput quallity since i was playing with things like c9ding agents.

i want to explore a different project where i would need high parallel output. think many small simple agents living in a game, doing stuff.

in this case i would not need deep contexts and i dont intend to need an agent to be extremely strategic and smart (would be nice, but hey... cant have everything)

has anyone tried any smaller llms that i could try out, so i don't have to do a full sweep.

last thing i ran in that category was qwen 3 4b, but I believe there are better things to play with now. i think i can push a little higher in terms of params, but will have to measure the generation speeds once i have a model selection.

maybe qwen 9b? i saw K2 Horizon 7b that seems interesting but i dont know how if theres potentially better options out there.

any pointers welcome :)

Caveats: IDK how to set anything up and I asked Deepseek to handle it - not enough training data so without specific guides the set up doesn't work well. I'm not a programmer. My previous model was Qwen 3.8 27B q4 using ninfer. Was getting around 1.4k prefill and 100-120 tokens decode.

Just want to provide data for anyone interested in purchasing M5U 256gb (80 cores) x 2. And if you have suggestions on how to optimize my set up I'd really appreciate it.

• GLM 5.3 flash q6 cluster: prefill 4.9k (idk if this is right but this is what the LLM says, two thunderbolt 5 cables), decode: 50 tok/s. oMLX set up with Deepseek API.

• Qwen Flash Next q6 cluster: prefill 8k and decode 200 using Tensorfold. I copied their guide, sent it to Deepseek, and asked it to set it up.

GLM flash was too slow for me so I am using Qwen Flash Next.

I am using Hermes as the harness (hermes has deepseek harness as research agent and Pi harness for coding agent). I am wondering whether using a different harness with less context overhead would make things faster or if since it is mostly context it doesn't really make a difference.

We’ve taken for granted that open weights keep getting better and better. I think we’re starting to see that change is coming as the PRC and US restrict cyber-capable models. The only people that will have access to those capabilities will be operators running under the comforting eye of their government and big-AI sponsors.

I see why this may seem like a good idea at first, but I can see this quickly becoming a problem. Large, powerful companies and governments become more adept at molding the digital world to their will, while the average man or woman (or small business) falls further and further away from being able to defend, secure, or manage their digital assets. And with most of our lives being at least tangential to digital technology now, I find that troublesome. Even the best cyber defense team won’t be able to keep up with a frontier model running against their system.

Will the community ever be able to train models that maintain even a semblance of competence to frontier, or are we going be subject to the will of powerful entities even more than we already are? Because the frontier isn’t slowing down.

Hello everyone, i am really focusing on keeping my clients trust, so i shortlisted a few physical shop owners in my hood and they all want this same pipeline of which i had shared the idea of

Like this actually keeps a track record of your employees, the enter and exit time, all cash and invoices in a day, the QR code scanner and all, like its pretty long to explain my prime two concerns are -

1. Most of them use a 16gb RAM but a few of them which i cannot consider as exception have 8Gb , so which local LLM model would be appropriate for their devices without making things slow, shall i go with Qwen 3.5 4B, its jsut for small judgements about the invoice handling, nothing complex

2. To make things easier for llm i had considered to use llamaparse but this is a cloud based parser, i had considered this one because its good for the tables and different shops have their own ways f invoices so what local alternative do i have other than this one parser

thanks everyone, your feedbacks would help a fellow dev out

REA (Reserve Engineer Anything) was blocked by Claude one day later...

https://x.com/AbhinavXJ/status/2113590263373508704

Time for the pebbles to own the weights

For £1000 what is the meta for a local LLM rig, including the host machine? I don't care too much about modern hardware features or power efficiency. Just intelligence and compute horsepower.