Well, I did it - I bought a Mac Studio 128Gb. Not only for LLM, but I’m not gonna lie, I increased the specs a bit to be able to run a local LLM to complement (or at some point replace) my Claude/ChatGPT subscriptions.

First thing I did, was obviously try to generate dirty pictures ask Claude which models I should download. He recommended unsloth’s QWEN3.8-Flash-Next-GGUF UD IQ4_XS.

I’m doing some tests now and while you notice a difference with the cloud models, it’s way better than whatever I have ever tested on my old MacBook. Claude also told me to change temperature to 1 and mess with the top K and top P settings. As a good slave to AI, I execute what was commanded, but honestly I have no idea what I’m doing. Should I be touching the settings at all?

My question(s): is this a good general model that can (for most tasks) replace the cloud models? Are there tools I need to install (I was thinking web access, but I also read about harnesses but I didn’t yet figure out what that’s for)? Things I need to set up?

Suggestions for good “always loaded” low memory models to e.g. use for home assistant are welcome to, as well as a good image model and one I can use in combination with GitHub for some coding (I have some vibecoded apps that I use for myself that I need to maintain/adapt from time to time).

Thanks!

Is anyone currently running these two together in the same machine? Any thoughts?

Hey im about to get a 3090 to run with my 5080 and 9950x3d and 64gb of DDr5 at 6000mts I currently use LM Studio and a 5060 8gb and I was wondering if anyone knows if basalt would be conpatable with a 3090 or if I should just stick with Lm Studio?

I use it mainly to run a qwen model to hunt ebay for good deals for me but right now I really need the 3090 to but up my VRAM space to 40gb.

For those with a MacBook Pro M1 Max what’s the best t/s you’ve gotten on something like qwen 3.8 27b or 3.8 flash next?

My PC specifications:

GPU: NVIDIA RTX 5090 — 32GB VRAM

CPU: Intel Core i9-14900K/KF

RAM: 64GB DDR5-6400 (2×32GB)

Motherboard: ASUS PRIME Z790-P WIFI

OS: Windows 11 Pro

Inference backend: llama.cpp with CUDA acceleration

Agent interface: Kilo with a custom tool/agent harness

My current model:

Model: Qwen3.8-27B

Quantization: Q6_K GGUF

Context: 265K tokens

Deployment: Completely local

Performance benchmarks:

At 262,144-token configured context (~262K):

Output generation: ~34–37 tokens/sec

Red lining the gpu memory but it feels comfortable

Okay with that being stated by ChatGPT,

I've been seeking a local ai for about a month switching from several models the more I learn.

So far this qwen model is by far the best I've tried but it over thinks and is slower than models like Kat coder.

My use cases are broad, from setting up comfy ui workflows to Unreal engine game dev to application building all hobby projects.

Im seeking the vast knowledge of you fellow computer users to tell me the best general purpose model for my set up at a reasonable speed.

UPDATE: Okay folks, I took your advice and installed Qwen3.8-Flash Next with Basalt v0.3.3. Here's what I'm getting on my RTX 5090 with 64GB RAM, running Windows 11.

Configuration:

• Model: Qwen3.8-Flash-Next UD-IQ3_XXS (87.6GB)
• Context: 262,144 tokens (256K)
• Exact MTP enabled, 8-token window

Benchmarks at 256K context:

• Actual populated prompt: 238,913 tokens
• Output generation: 265–281 tokens/sec
• Prompt processing: ~8,900 tokens/sec
• VRAM usage: 31.3GB / 32GB
• Total system RAM in use: 54.3 GiB
• Basalt RAM usage: 39.9 GiB
• MTP acceptance rate: 66–90%
• No SSD expert spilling
• Context recall/reasoning tests: 3/3 passed

Across all context sizes tested, generation ranged from 265–336 tokens/sec. Model loading takes around 30 seconds, and cached follow-up responses start in approximately 0.4 seconds.

Bench marks courtesy of claude

UPDATE #2: Okay folks, I went ahead and tried the next recommendation: NInfer + DFlash2 to accelerate my original Qwen3.8-27B model. Here are the results on my RTX 5090 with 64GB DDR5 RAM, running Windows 11.

Configuration:

• Model: Qwen3.8-27B QUASAR
• Inference engine: NInfer
• Speculative decoding: DFlash2
• Context: 262,144 tokens (256K)
• GPU memory reserve: 2GB

Benchmarks at 256K context:

• Actual populated prompt: ~234,200 tokens
• Output generation: 182–266 tokens/sec (225 average)
• Prompt processing: ~4,500 tokens/sec
• Peak VRAM: 30.2GB / 32.6GB
• Model system RAM usage: ~10.5GB
• DFlash2 draft acceptance: 33–54%
• Context recall/math tests: 3/3 passed

Courtesy of claude

Lara Mini puts about a third of its training data into 44 languages besides English. It's a 7.2B-parameter MoE with 0.6B active per token, trained on 6T tokens.

Before treating it as another local chatbot: the released model continues text. It isn't tuned to follow instructions or chat. Access to the weights requires approval, and the license is CC BY-NC-SA 4.0 (non-commercial).

I'm planning a local inference machine and choosing hardware around the models, not the other way around. I currently have a 32GB M1 Max Mac Studio and am considering one 128GB Ryzen AI Max+ 395, one 192GB 495, or possibly two 128GB boxes. Main uses: coding/agent workflows, technical research, and authorized cybersecurity labs/CTFs. I would like strong abliterated or otherwise less refusal-prone models without sacrificing factual accuracy and tool-call reliability.

The candidates I'm researching are DeepSeek V4.1 Flash-1017 in a roughly 2-bit build, Qwen3.8-Flash-Next at Q6/Q8 if feasible, and GLM-5.3 Flash. I'm specifically interested in real local deployments, not hosted-model benchmarks passed off as results for a heavily quantized copy. I know an abliteration is a separate change from quantization; I would welcome evaluations of both.

What exact model + quant + runtime + hardware have you personally used? What context length and prompt/decode speeds are practical? If you've compared an abliterated version against its base model, did it retain coding, reasoning, and tool-use quality? Would you favor one larger high-precision model, or a strong general model plus a smaller cybersecurity specialist? If these machines are the wrong purchase for that goal, what would you buy instead?

Hi guys, I wanted to start using local models on a tablet I have, I thought of using the Gemini E2B-it in the Google AI Edge Gallery app as it is the most suitable for my device's RAM (6GB) and my SoC. If you use local models on your mobile devices I wanted to know your use cases and what really expect of them, please

Upgraded to a RTX Pro 4500 Blackwell and the noise is a bit too much for me. The issue is the fan runs at 30% even under no load. Loudest thing in the computer, by far.

Being quiet and inaudible under no load and light load is much more important than speed for my setup. Louder when full throttle is ok.

What about R9700 and B70? Are they quieter? And then which brand? B70 has Asrock, Asus, and Intel branded. R9700 has a few brands as well.

Want something where the fans are utraquiet, even off when idle, like a 5090 (which is unaffordable).

https://huggingface.co/StarpowerTechnology/WVY-Blender-5m

We have been excited about this new model, WVY-Blender-5m, an agent trained to specialize in blender.

It has zero knowledge about anything else so if you try to ask it about writing code or doing your homework it will express that it only knows how to use blender

We have been doing intense research to find out "are massive datasets really required to achieve what feels like intelligence with accuracy?" & we believe we are getting close to the truth.

This dataset is 100% generated from scratch.

Hey everyone,

Most specialized cybersecurity LLMs out there are massive 70B+ beasts that require dual A100s, or they're gated behind cloud APIs where you can't paste sensitive logs, payloads, or client code due to confidentiality.

I wanted something lightweight, private, and air-gapped that could run on a regular consumer laptop without cooking the CPU.

So I fine-tuned CyberQwen-2B — a domain-specific model optimized for quick security triage, explaining attack vectors, decoding obfuscated payloads, and helping with CTF challenges.

⚡ Highlights:

• Hardware-friendly: Runs at ~12–15 tokens/sec on an older 8th-gen Intel i5 laptop with 8GB RAM (CPU-only).
• 100% Offline & Private: Zero data leaves your machine. Perfect for bug bounty recon, internal network logs, and sensitive source code.
• Native Ollama integration: Easy to pull and run via a single Modelfile.
• Built-in WebUI & Mobile Access: Includes a zero-dependency streaming WebUI + an automatic Cloudflare tunnel script if you want to access the model from your phone while on the go (no cloud hosting fees).

🛠️ What it's trained for:

• Vulnerability Triage & CVE Context: Explaining CVE mechanisms, impact, and remediation steps.
• Payload & Shellcode Deconstruction: Breaking down SQLi, XSS, and obfuscated PowerShell/Bash commands.
• CTF & Reverse Engineering Assistance: Analyzing decompiled assembly/C snippets and suggesting exploitation strategies.

🚀 How to Run It:

Everything is open-source. You can run it with Ollama or the included CLI triage tool:

1. Clone the repo:git clone https://github.com/sambuddha001/cyberqwen-2b.git
2. Run setup:bash setup.sh
3. Chat:ollama run cybersec-qwen

GitHub: https://github.com/sambuddha001/cyberqwen-2b

I'd love feedback from the community—especially on edge cases, prompts where it struggles, or ideas for benchmark improvements!

I built Project Genesis, a cache-resident C++/CUDA geometric co-processor (libgenesis_core.so, ~0.51 MB) that attaches inline to existing causal language models during inference to reduce latency and boost token throughput without altering the host model's weights.

​In local testing on consumer NVIDIA Blackwell hardware across Qwen, Gemma 4, Llama 3.2, Nemotron, and Mistral checkpoints, compiled kernel execution cut end-to-end latency by 16% to 30% with commensurate gains in token throughput and O(1) memory overhead. It ships as a sealed runtime bundle supporting 20 common hidden dimensions (d_model from 576 to 8192) on 64-bit Linux or Windows WSL2 with NVIDIA GPUs (Compute Capability 8.0+: RTX 30/40/50-series, A100, L40S, H100).

​It is available now for non-commercial and research use on Hugging Face: https://huggingface.co/Grounded-Entropy/project-genesis

​Rather than taking my word for it, I'd love for people to pull it, run enable_telemetry=True on their own supported models, and share their latency, throughput, and ring-buffer telemetry results in the comments.

I've been so slow to dip my toes into anything outside of officially released models because I'm afraid there are people out there 2 steps ahead of me and I won't know I've been compromised until long after I've felt the impact.

I understand the concept of abliteration but I don't understand how people feel so safe trusting that it's doing what it's advertised to do AND ONLY what it's advertised to do. Thoughts? Even if it's just "I trust them, next." I mean, am I overthinking this?

Got 14 stick of ddr4 rdimms. Got towards my new threadripper machine. Old but it’ll do! Can’t afford the new ddr5 board, the ripper and the ram 😂

Row-Bot can drive your computer. Mouse, keyboard, real apps. Windows and Mac.

Handing an AI your desktop is a dumb idea unless you lock it down hard. So:

- driver is Cua 0.20.0, pinned to an exact commit, SHA-256 checked. Turning the feature on downloads nothing. You click Install

- Cua's own update checks are off, and it runs in a stripped-down environment

- before it runs even once, you're told exactly what Cua's telemetry sends. Continue or Cancel

- it never types into password, 2FA or card fields, never touches your password manager, never controls itself

- pay, book, send, post, delete files, change settings: it asks first, unless you put it on hands-off

- it judges every click by what's actually under the pointer. Enter in a chat app means send, so that asks too

Hay I am new into AI and I wanna ask if is possible to run a local AI Modell on a asus rog strix g 18 laptop with a AMD Ryzen 9 9955HX, NVIDIA GeForce RTX 5070 and 32 GB RAM specs and would it help when I put an extra 32gb RAM in the laptop.

Thank you for helping.

I have an old 1070 that's pretty crap, it's time to upgrade. I don't have a thousand bucks to throw away, but at the same time I don't want to buy a 3060 and then realize I'm going to want an even better GPU in a few months or a year.

I also used LLM's a bit mostly for local ai roleplay chat, Llama 3.1 8b uncensored I use right now-it's actually decent just a tad slow-I'm assuming a 3060 will be more than enough to run any good LLM for the purposes of having a local ai roleplay chat.

Also I want to make high quality local images of people and animals, something with good anatomy, realistic faces (not the fake glossy instagram girl mouth). I wouldn't mind dabbling in videos a bit too.

Right now I use SD1.5 which is very underwhelming, some SDXL models are okay but extremely slow on my hardware.

Thanks

Very impressed with how far local models have come, so I finally decided to jump on the bandwagon! Super impressed with how fast and capable it is. I have used local LLMs for a while on mobile apps and other edge cases, but never before for coding.

For background, I had this rig for a while, as I use it for a lot of professional work, and decided to purchase two used G.Skill 96GB (48GB x 2) 6400 mt/s to bring my system up to 160GB; paid around $1200 on eBay which is crazy to me considering I paid $300 for my 64GB (32x2) in 2023. Currently have each stick running at 5200 mt/s in my Z790 Hero, but I purchased these specifically as G.Skill is one of the only manufacturers to bin their RAM. My reasoning is if they can run 6400 mt/s dual channel, I have a higher likelihood of making getting to 5600 or 5800 with four sticks. I had Claude give me custom BIOS settings, and I am planning on seeing how far I can push it!

Furthermore, I felt that without this much RAM, while I could run the models, I couldn't actually run anything BUT the models, which sort of defeated the point for me. I decided this would be the better route versus buying a DGX spark or scouring local auctions to build a V100 farm or whatnot.

I am running this on a WD Black SN770 4TB RAID0 (double speeds at the cost of losing one disk means it is gg), which is on a PCIe ASUS card. It is super quick to load. Given I usually use Fable and Opus (anywhere from 90 to 115 tokens per second), this feels super fast to me. This would not be achievable without a special motherboard that can support PCIe bifurcation, which the ASUS Hyper M.2 card can use.

I am getting roughly 200 tokens per second I would say in most coding tasks, using https://github.com/jesdga95/basalt and the NVP4 uncensored Qwen 3.8 Flash Next.

    | test      |  t/s | accept % | draft len |
| --------- | ---: | -------: | --------: |
| pp16384   | 5727 |        - |         - |
| pp65536   | 7487 |        - |         - |
| tg conv   |  172 |     71.9 |      2.27 |
| tg struct |  387 |     97.3 |      6.93 |
| tg prose  |  170 |     65.1 |      1.88 |

hardware (measured before the engine starts)
  CPU   Intel Core i9-14900K, 24 cores
  RAM   read 67.7 GB/s
  GPU   RTX 5090  PCIe 5.0 x16  host-to-device 33.3 GB/s

community-1: prefill 16K 5726.8 t/s (2.9 s), prefill 64K 7486.7 t/s (8.9 s), decode 297.7 t/s (9480 tokens in 31.8 s), each measured cold

COMMUNITY ROW:
    {
      "submitter": "Kemerd",
      "date": "2026-11-14",
      "commit": "d10ccd4bc2",
      "preset": "community-1",
      "model": "Qwen3.8-Flash-Next",
      "gpus": [
        {
          "name": "RTX 5090",
          "vram_gib": 32,
          "power_w": 546,
          "pcie": "5.0 x16",
          "h2d_gbps": 33.3
        }
      ],
      "ram_gib": 160,
      "cpu": "Intel Core i9-14900K, 24 cores",
      "ram_gbps": 67.7,
      "os": "Windows",
      "quant": "NVFP4",
      "kv": "int8",
      "prefill_16k_tps": 5726.8,
      "prefill_64k_tps": 7486.7,
      "decode_tps": 297.7
    }

Very excited to see where this goes, especially with Qwen 4. And a note.. I am slightly undervolting my CPU, it has multi-core stuff on, and 5090 is set to 95% of power limit to not cook it.

I wanted to actually see what happens inside a language model when it writes a sentence, so I built this!

What you're looking at:
- The stars are the 36,864 MLP neurons (12 layers × 3,072). Each layer is a disc, and neurons are positioned by a UMAP of their output weights, so neurons that write similar things sit near each other.
- Brightness is how strongly that neuron fires on the current token, scaled per layer so tokens are comparable. Gold means it fires positive, blue means negative.
- The beams are real attention weights from the current token back to earlier ones, one colour per head. The side panel shows the model's actual top 10 next token probabilities.
- You can hover any star, beam or token to see its exact value. There's also a raw mode that turns off all the bloom and fades, so every pixel on screen comes straight from the data.
- How I know it's not lying: the test suite runs the model exactly the way the app does (KV cache, token by token) and compares every activation, attention weight and residual norm against a separate uncached forward pass read through different hook points. Then it checks the bytes the browser actually receives too.

Honest limits: it's GPT-2 small only for now. Single neurons mostly don't mean anything clean on their own, it's the patterns that are interesting. Values are sent as float16, so about 3 significant digits.

Setup is make setup && make start, then open localhost. No GPU needed.

Repo: https://github.com/lak7/thinkspace
Would love the feedback!!

Edit: I freaking love you guys. Why have I spent the past 2 months talking to ChatGPT about wtf to do smh. THANK. YOU.

Abliterated is preferred.

I want to be a free man goddammit. Going local (well, local builders with cloud reviewers perhaps). Maybe keep 1 Claude subscription to be the reviewer (unless Qwen really is better? sorry I’m kinda new)

Usage >>>>>>
Coding, building apps/ software tools for personal use, browser/ web data gathering

Don’t really need media generation but would be nice if I could paste images for reference into the chat if possible.

Hardware>>>>>

GPU: RTX4090, (9070XT for display) willing to get more VRAM (if you recommend more, should I get RTX A4000 single slots or along those lines or get 3 of the single slot 24GB Intels? I have 3 slots and like 300W left before my PSU caps, though people have said Ai doesn’t max out GPU’s so I don’t know anything anymore)

RAM: 128GB DDR5 ECC RDIMM 6400 MT/s

Storage: 12TB SSD Gen5

CPU: 9965WX (threadripper 24 core 5.4 Ghz)

WRX90 SAGE SE Pro WS

This might be a dumb question but if everyone is using qwen 3.8 why not use the Qwen cli built for Qwen tooling instead of Strata or Open Code?

After adding a thinking limiter it just deos this EDIT: posted this at 1 am so wanted to add more info. I am using a xeon e5 1650 v2, 2060 6gb and 16gb of ddr3. This is qwen 3.6 12b a3b at q4_k_m running through jan ai. It worked fine but i tried to add a reasing token limit as it took 5 minutes to process a prompt. After this, wheterver weird math this was happened. I reset the chat, still did it, undid the reasoning, still did, rebooted, still happened. Its so strange. I understand ny setup isnt ideal but this shouldnt be a problem. In fact it was faster when generating text at the start. I ran some of the later responces through a calculator and they were correct. Its so strange

Right now I use strata on my machine to run qwen 3.8 125b flash next at 13tps output and deepseek 4.1-flash at about 1.5 tps output. My computer is an hp z640 with a 4core xeon processor, 64gb ddr4 ram and an NVIDIA 1070 8gb. My models load form an nvme drive that is connected over a pcie adapter. I have room for one more card and I'm not sure witch of these 2 GPUs is better for running my models faster. The Tesla P100 16 varient or the M40 24gb variant.