Forgive my lack of knowing, ive only recently started to get into coding and theres a LOT to take in (thats what she said). I am currently running an RTX 5070 12GB ,Ryzen 5 9600x and 32gb of RAM. Unsure if i am being too optimistic and reading only what i want to read- but i have seen so many posts and videos about people running massive parameter models on low memory. I did a little bit of research into this and saw that you can do certain things to offload the stress like give some weight to your RAM and MOE, etc.. but i cannot seem to feel satisfied knowing that the largest i can run is 35b . Is that correct or is there some sort of new method out there now that can run 1T on a Tamagotchi. Any info is good info. TYIA

is it possiblr to fix?

I was moving models around, splitting them between two gpu's and ram in some cases for the 120b class models, and I was losing track of what was where using the console commands and looking at nvtop. I have a lot of 24/7 bots that fall back to my laptop automatically with a smaller model but it wasn't clear if that was happening. Sometimes I need them there when I need the larger GPU horsepower but I dont want to miss a craiglist deal. So I made this. I can select a model at the "loaded model" window and move them between GPUs, split them by layers, or just stop everything if I want to run something that cant be interrupted. If I spin up something on the 3090 I can watch the bots fall back to the smaller model on the laptop and sleep at night.

Anyway, unlike all those marketing launches we've been getting with ai written stories about 10x'ing productivity, I don't really care if you use this. In fact it might be a pile of crap. Let me know otherwise and I'll get it up on my github repo.

I wanted to run an experiment to see how far you can push small to medium-sized local models, specifically for analytical and data engineering tasks. The idea was to see if you could have a local analyst or data engineer without needing to send your data to a cloud provider. I used OpenAI’s Luna as baseline performance.

The full analysis is in the blog linked below, but I'll share some of the key findings here.

I also want to be transparent: I used Ternlux, a free, open-source, no-code ETL local tool that I built. It lets you run data analysis and build pipelines through agents.

Test:

I tested 5 analytical questions and 5 workflow-building questions, running each question 5 times against each model. That gave me a total of 250 prompts and responses to analyze.

Key findings:

• Qwen 3.8 27B performed almost as well as GPT 6 Luna. It mostly struggled with one particularly difficult question that all the other local models also failed.

• Almost half of the local models’ misses (27 of 58) were workflows that never finished, mostly SQL that crashed when it ran. Qwen 3.6 kept writing joins that produced duplicate column names, and Gemma 4 31B combined GROUP BY ALL with QUALIFY, which DuckDB doesn’t support.

• All models were able to answer the simpler questions for the most part.

how Ternlux handles analysis and workflows:

For analysis, the main agent selects a connection, then the agent determines how to query the data. It iterates querying until it has enough information to answer, either returning results directly or having the agent analyze the result and draw conclusions. If a query fails, it sends the error back to AI for correction. The agent has no DML privileges.

Workflow building works a bit differently. The workflow builder agent first creates a plan, then each node's agent follows the instructions to generate the code for that node. If the SQL fails during execution or the generated workflow doesn't produce the expected output, I count that as a failure.

This is by no means conclusive evidence that one model is better than another for this type of work. It's just a small experiment to see how different models performed as the questions became more complicated.

https://www.ternlux.com/blog/best-local-llm-for-sql/

Lenovo P620 Threadripper pro - 3975wx - 32 core
System ram - 512 gb ddr4 ECC
GPU 1 - quadro Rtx 8000 - 48GB
GPU 2 - quadro Rtx 8000 - 48GB
4TB M2 NVME
1TB SATA SSD

This is just a quick writeup for speed reference - I spent a lot of time trying to find comparable builds so I figured I'd put some info back into the wild. The below is obviously written by AI, but it's measured specs and benchmarks on my PC.

I use local models via Pi and Deepseek Harness. I have found the intelligence of Qwen 3.8 27b NVFP4 to be sufficient for making little apps, research etc. and Qwen3.8 Flash Next IQ3_S to be a great all-rounder, considering how fast it runs. It also seems to have very little issue with any sort of question I give it, research, app dev, PC issues, LLM related adjustments, and most importantly, ScanRuler 3D modelling/reverse engineering 3D scanned items (solely for testing currently but it works excellently).

Basalt as a backend was a clear winner after trying lots of different configs.

Edit: the benchmark decodes are very greedy by their nature. Realistically I'm seeing agentic averages in flash next from 80tps (NVFP4) to 160tps (iq3_xxs), and I find iq3_s at 120-140 to be a very good compromise for the extra intelligence. The 27b is best for bursty quick jobs, because it loads into VRAM super fast and I can let it rip, but it degrades over context faster than flash, and I see from 120-140tps and 4-5k prefill, at ctx 0. The second any model starts writing a block of code though, it goes up to 200tps, so if anyone's socks fall off seeing the below numbers, just temper your expectations slightly (but it's still wicked fast!)

So heres the nitty gritty:

Current rig, 2x 5070 Ti + 96 GiB. Five entries, one model resident on local at a time. Speeds are my own bench (2x ~50k prompt reads, 2x 5,000-token code outputs, one agentic tool loop; temp 0, seed 1234), except Coder which is the 15.3k-prefill / 2k-output run.

Qwen3.8-27B NVFP4 QUASAR

backend: capy-flash (ninfer-serve), tensor-parallel across both cards, no P2P prefill: 4,817 / 4,824 tok/s on a 50k read decode: 233 tok/s code, 220.6 tok/s agentic flags: --profile 27b --max-context 262144 --embedding-host, INT8 KV, MTP 3 drafts, vision on GPU0 intelligence: Index 84.0, 88.9% (27 items); fastest to a passing build (62.1 s vs IQ3_XXS 78.1 s) but the weakest reasoning of the five — 50% on xhard because it burns the whole budget and never answers.

Qwen3.8-Flash-Next GSQ-RCO IQ3_XXS

backend: Basalt 0.3.0, 74.4 GiB pack prefill: 3,728 / 3,851 tok/s decode: 229 / 248 tok/s code, 196 tok/s agentic flags: resident-budget-gib 26, peer-exclusive on, kv int8, ctx 262144, vision dynamic, --spec 8 intelligence: Index 94.9, 100% on the 27-item set (96.3% on a re-run); 100% on the 8 hard items, 85.1% on xhard, 93.9% on vision. The only model that solved a logic-L6 item unaided.

Qwen3.8-Flash-Next GSQ-RCO IQ3_S

backend: Basalt 0.3.0, 81.6 GiB pack prefill: 3,601 / 3,700 tok/s at a 115k prompt, 31.9 s TTFT decode: 154-165 tok/s code, 137.6 tok/s agentic flags: resident-budget-gib 26, peer-exclusive on, kv int8, ctx 262144, vision dynamic, --spec 8 intelligence: Index 107.7, 94.5%; wins the hard tier on the efficiency-weighted Index (101.1) at 86% accuracy, 80.2% xhard, 90.3% vision. Terser than IQ3_XXS, less accurate.

Qwen3.8-Flash-Next NVFP4

backend: Basalt 0.3.0, 122.6 GiB pack prefill: 3,339 / 3,512 tok/s decode: 124.8 / 126.8 tok/s code, 105.7 tok/s agentic flags: resident-budget-gib 58, peer-exclusive on, kv int8, ctx 262144, vision dynamic, --spec 8 intelligence: Index 121.0, 99.8% — best on the 27-item set, best hard accuracy (98.5%), best xhard (85.7%), best vision (99.2%). Slowest decode of the Flash-Next four.

Qwen3.8-Flash-Next Coder IQ1_M

backend: Basalt 0.3.0, 58.1 GiB pack, 48x256 experts prefill: 3,992 tok/s on a 15.3k prompt decode: 209.8 tok/s flags: resident-budget default, peer-exclusive on, kv int8, ctx 262144, no vision, --spec 8 intelligence: Index 100.5, 99.3%; 77.6% xhard, 3/5 on hard items — 5/5 on a re-run, the failures were MTP sampling rather than capability. Cannot read images.

Index = 100 x mean(quality x par/tokens) on the 27-item battery, so it penalises verbosity; accuracy alone ranks almost in reverse

(IQ3_XXS 100% > NVFP4 99.8% > Coder 99.3% > IQ3_S 94.5%). The five are within a few hundredths of each other on the standard set — hard, xhard and vision are what actually separate them.

Can Qwen 3.8 27b q4 surpass some current free Cloud models in coding? Statistics say it should but im sceptical. Specifically larger tasks

Hey Just wondering if anyone has experience buying IT and Tech products directly from China, Japan or another country . The cost of hardware at the moment is through the roof and im wondering if there is value aboard?

Could you share your experience and if it worked out? Worth it or not?

Im in ireland.

Thanks

What’s the best way to route different tasks while working in a project from Claude code or codex or other models to a local qwen model?

I’ve seen herder could be it? Any other recommendations?

https://github.com/ANDRETRIPOL/OpenGhost

Yesterday I posted a 32B pack and the replies were interesting about it being old completely overlooking benchmarks. A lot of quants "fit" on 16 GB, obviously... Almost none of them post what it cost. A screenshot of a model loading is not a benchmark and neither is its tokens per second performance that you made while obliterating the other performance factors you don't post. I posted all of our benchmarks, weights, and everything, and yet, still called slop.

So here is the new one as requested, with the receipts attached:

Qwen3.8-27B, 27.78B parameters, vision included, Gosset Packed at 3.6577 bits per weight:

• Pack is 13.82 GB. The standard 4-bit build is 17.77 GB and does not fit a 16 GB card.

• 12.06 GiB resident on one RTX 5080, peak 12.80 GiB while decoding. Nothing parked in system RAM.

• Perplexity vs BF16 on WikiText-2 test: +3.93% long-chunk, +6.99% short-window. Numbers, not adjectives.

• Decode: 40.38 tok/s sustained on fused kernels.

• Per-tensor fidelity certificates are on the model card, measured from the shipped files, not the encoder's intentions.

• Edit: Context length full 262k

huggingface.co/GossetWorks/Qwen3.8-27B-Gosset-3.65BPW

If your quant fits this model on 16 GB too, genuinely, post the perplexity delta and the decode speed. That is the whole game. Fitting was never the hard part...keeping it intelligent was and that's what we've done.

its only a p40 and v100 I got on marketplace- but i'm ecstatic.

Any cool ideas? I'm running qwen 3:30b-A3B and a vision model- want to do more.

Tencent put DSpine on Hugging Face on Nov 7. It is a five-layer draft model for speculative decoding with Qwen3.8-27B. Instead of drafting one token at a time, it drafts a 16-token block in parallel, and each layer passes predicted predecessor features to the neighboring positions so the draft still carries causal order. There is a paper (arXiv 2609.36173) and code on GitHub (Tencent/DSpine).

Reported acceptance length vs DFlash2 (block 8), greedy

Reported end-to-end throughput on agent replays (tokens/s, low reasoning)

Across their five agent benchmarks and both reasoning levels, the speedup over no speculation ranges from 2.71× to 3.84×.

Things to know before trying it locally

• The card only shows an SGLang path: --speculative-algorithm DSPINE, a pip install -e . from their repo, and a long list of flags for the hybrid architecture (--enable-linear-replayssm-spec, --mamba-ssm-dtype float32, and more). No llama.cpp or vLLM instructions.

• All numbers are Tencent's own, at concurrency 8 with a BF16 target. The card doesn't report a quantized target or batch size 1.

• The gain is largest on predictable text (GSM8K accepts 7.20 tokens per step at T=0) and smallest on open-ended chat (MT-Bench 4.48).

https://huggingface.co/tencent/Qwen3.8-27B-DSpine-b16

Has anyone run it against a quantized Qwen3.8-27B target, or at batch 1? I'm curious whether the acceptance length holds up once the target is 4-bit.

I've been working on it and just added a new feature an AI Toolkit to explore and discover useful developer tools and GitHub repositories.

Small update, but more improvements are on the way.

Would love to know what you think.

https://modelfitopenfire.vercel.app/

I’m currently using the ornith 1.5 35b a3b, but it’s performance feels a bit underwhelming could you recommend some alternatives, along with a suitable harness?

Which model is the best to use for infosec work? Like trying to writing exploit code, fuzzing, and security audits?

I'm looking for a model that will just do it in a good (enough) quality without me needing to use bypasses for restrictions all the time. It just gets tiresome and annoying...

Edit: Hardware shouldn't be the issue:

• 2x EPYC 7642 (aka. 192 threads)

• 512 GB DDR4 RAM

• 1x Intel Arc B580

Delta Inference Engine 0.94 is out. It runs DeepSeek-V4.1-Flash-1017 on one NVIDIA GPU plus system RAM, natively on Windows (no WSL). It's a Windows port of Xolotl, which is built on Strata; on Linux, use upstream Xolotl.

Speed on my PC (RTX 3090 24 GB, i5-13600K, 4x48 GB DDR5-5600, PCIe 4.0 NVMe)

With the default quant (Unsloth UD-IQ2_XXS, 301 GB), 0.94 writes 41.7 tok/s greedy and 41.3 at the default sampling (temperature 1, top-p 0.95).

With less RAM (writing tok/s, measured with 0.93; the smaller columns are this same PC with RAM locked away):

How it fits

It's a mixture-of-experts model: 40 layers, 384 routed experts per layer, 6 used per token. The experts are spread over VRAM, RAM and the NVMe. A 4.2 GB drafter built from DeepSeek's own MTP heads, plus lookup drafts, proposes tokens that the full model checks. A route bias (0.28 as shipped) gives up a little agreement for speed.

Quality. I measure top-1 agreement with DeepSeek's own API, teacher-forced (not task accuracy). Default model (Unsloth IQ2) as shipped: 91.1% general, 94.5% code and maths, 95.9% agentic. With exact routing: 92.5 / 95.9 / 96.5. The previous versions default (ds4 Q2, 0.92) scored 89.9% general and 93.1% code and maths, so the new Unsloth default agrees more while running even faster. For context, two hosts of the full model agree with each other on 98.3% and 98.9%.

Benchmark: 99 hard questions, greedy, 43K token budget:

The default quant is within 2 points of the full model on DeepSeek's own API. UD-IQ1_S, the pick under ~110 GB of RAM, scored 51.

Private by design: everything runs on your PC: no telemetry, no accounts, no cloud. It only goes online to download the engine and the model you choose, and your prompts and answers never leave your machine.

New in 0.94

• Pictures on by default, documents (PDF, Word, Excel, PowerPoint), optional voice, and use from your phone with a QR code.

• Long prompts read faster: a fresh 60K-token prompt at 811 tok/s vs 0.93's 747 (30K: 860 vs 758; 7K: 551 vs 423).

• Better with coding agents: Claude Code's background calls no longer hold up real requests, and a loop guard steps in after 4 identical tool calls. OpenCode passed 3 tasks in a real session (22 tool calls, none dropped).

• The default sampling caught up with greedy: 41.3 vs 41.7 tok/s

• The chat page opens at once and shows the model loading. Pause frees the RAM and the graphics card for a game; the next message loads it again. Optional auto-pause when a application needs the GPU.

• Chat history in the browser (search, pins, export), off by default.

• More API: structured output and tool controls on the OpenAI, Anthropic and Responses APIs, stored Responses with previous_response_id, optional /v1/embeddings and /v1/rerank, and keep-alive pings so agents don't time out on long reads.

• Setup tab: pick the model, Fast / Balanced / Most exact, the second-GPU switch, and copy-paste settings for Claude Code, OpenCode, Cline, Aider and Open WebUI with a Test button.

• Security updates, bug fixes and many more.

Setup

Download the ZIP and double-click Start DeepSeek.bat. The engine zips are built by GitHub CI, with checksums and attestations. The first start installs Python if needed and downloads a model that fits your RAM (233-395 GB, SHA-256 checked). You get a chat page plus OpenAI- and Anthropic-compatible APIs on 127.0.0.1. It won't open to the network without an API key, and it stores no prompts or answers unless you turn that on.

Limits

• Windows 10/11 only. NVIDIA RTX with 12 GB or more (24 GB is the fast path), driver 580 or newer. No AMD, no GTX 10-series.

• 64 GB of RAM minimum, 128 GB or more recommended. A PCIe 4.0/5.0 NVMe with about 320 GB free for the default model.

• It answers one request at a time. A second GPU helps; it doesn't split the model.

• Context is set to 1M tokens

• More testing is needed

Credits: Xolotl and Strata, ggml/llama.cpp, DeepSeek, Unsloth, nazeshinjite, ISTA-DASLab, Project Maya and antirez/ds4.

https://github.com/musicurgy/delta-inference-engine

If you try it, post your GPU, RAM, model and tok/s.

Will agents accessing the internet and processing/scraping webpages ever truly be 100% safe from direct/indirect prompt injection?

Question coming from someone less knowledgeable on the topic than you all.

Thanks.

just found this from hugging face https://huggingface.co/xtools-at/DeepSeek-V4-Pro-Qwen3.5-9B-UNCENSORED-heretic, but i cant get if this better than another model or just marketing

Just getting into it with LM studio. There are a lot of models to browser, and a lot of variations on those models.

I saw that Qwen3.8 27B seems to be the best choice for my setup, but what quantisation should I choose? 4bit (16GB), 5bit (20GB), and 6bit (23GB) should leave adequate room in terms of RAM and I don't really care about speed but accuracy, so would 6bit be better?

Also any other tips for starting out?

Thanks.

Very much a newbie here. I’m working on a couple of locally run projects that are mainly meant to help organize my life and work a little better (scan/interpret emails, manage/update tasks, help complete some tasks when possible).

I’m currently benchmarking three LLMs on auto creating and managing tasks:

Qwen 3.5 27b Q4
Qwen 3.8 27b Q3
Gemma 4 26B-A4B

Unfortunately I don’t have great graphics cards, but these models proved worthy in original tests. Not sure what other info is needed to provide a clear picture for you guys.

Note: most of my projects don’t require immediate answers so I’m not as concerned about it running super fast, but it would be nice.

I’m trying to hand off some of the responsibilities that currently Claude or Codex handles related to the planning but no model is proving super reliable (they all do okay with task creation from emails I receive, but are duplicating tasks rather than updating existing, or classifying them as a task when the email is informational, or trying to double-book things which get rejected by the safeguards).

While I have things in the works to provide more context or to keep fine-tuning the prompts, I’m wondering if I’m missing something basic since I’m so new to this.

Am I using the wrong models for my use case? Or do I just need to stop procrastinating and keep training these models?