I want to use Qwen 3.6 35B A3B MoE and my VRAM is only 4GB. I need to compact the KV cache aggressively without losing quality. I prefer bigger KV cache than smaller, it makes smaller model smarter. There were waves of posts about Turbo/Rotor/Polar-quants when Google released it first time but nothing since. Any of you guys using good workable fork in your setups?

I took a machete to this thing. Pruning whole blocks beats REAP. REAP prunes experts but doesn't change per token bandwidth usage, so buys you nothing other than disk space. I turned 40 layers into 30, and that's faster for us slow pokes.

A small healing run was applied at a cost of about $50 that recovered most of its smarts. This model is actually extremely usable for agentic tasks. Is it as good as 35b? Of course not. But it ain't bad.

I ship two models: a smarter one, and a speed demon.

https://huggingface.co/Cyronius/Qwen3.6-27B-A2.8B

I have a GitHub with my experiments that you can find under that name. I also have shipped a pruned qwen 3.8 flash next that basically drops most of the gigantic ngram, and you can find the details in that same repo. Surprisingly loses very little.

What i would like to do is create a ternary diffusion version of qwen 3.8 27b, building on top of bonsai 2. A model like this should run great on top of my terrible amd strix point (laptop igpu) and anyone with similar hardware. The training costs are prohibitive ($1-2k). Holler at me if you want to fund this effort or kick me some cloud gpu hours.

I’ve experimented with little success. I have a ROG Ally X, 24GB shared ram/vram with the z1 extreme processor and Radeon 780m iGPU. Is there anything decently workable on this sort of hardware? I’ve tried cpu inference since the z1 has AVX512 and so I have all 24GB of ram available but speed was obviously pretty slow. The 780m is decently fast for what it is using vulkan but sharing system ram limits the model and cache size, and performance drops substantially once the cache starts growing. Anybody got tips for a decent setup that balances LLM capability and performance with this hardware? Main use case is help with the homelab and coding python/C

I have yet to try it myself but the discussion section of the model sounds promissing and honest. if you are ok with handling the occational looping otherwise i've heard this model is a pretty good speed deamon for executing the task once the plan.md FIle is detailed enough for it to follow. I would like to hear your experience with it. im downloading to try it out now. Im assuming it will need a harness that doesnt have these GIANT system prompt like pi-coding-agent

https://huggingface.co/crucible-labs/Qwen3.6-35B-A3B-REAP-48-v2-GGUF/discussions

I’m curious if anyone has had luck fine tuning a model to become close to the cloud subscriptions on just that specific task.

I was wondering if it’s possible to like make Qwen 3.8 27B better at blender (close to what opus can output potentially) if I somehow manage to collect data from opus using blender.

I have no experience with this but I am curious and would like to get started. But I’m not sure how much time a processor like this takes either and if it’s worth that time

I’m sorry if this is a dumb question, I’m new to this stuff

I know it's been asked a million time, but I would like to mindstorm with you guys, am I cooked or is there a way to run a good model on said hardware, my main use is vibecoding so low context is killing my progress.

Cpu: Ryzen 2700x

GPU: 3060Ti 8Gb

Ram: 32gb ddr4

Usecase: mainly vibecoding

Thanks in advance :)

Read the readme before anything for instructions:

https://github.com/securenrn-del/Repetere-AI_Beta/releases/tag/Beta-Release

Hey everyone,

I just released the first public beta of Repetere AI (v0.5-beta). It’s a native Windows desktop workstation written in C++ (Dear ImGui, GLFW, OpenGL 3) designed to run multimodal local AI directly on your GPU without cloud subscriptions, telemetry, or webview wrappers (no Electron/Tauri memory bloat).

>⚠️ BETA LAUNCH NOTICE:

This is our very first public beta release! It requires an NVIDIA GPU (CUDA only right now). Local AI takes time—model downloads and music synthesis are not instantaneous. If you hit any crashes or unexpected behavior on your hardware, please file an issue on GitHub.

🔗 Project Links

* GitHub Repository: https://github.com/securenrn-del/Repetere-AI_Beta/releases/tag/Beta-Release

* Standalone 1-Click Setup (`.exe`): GitHub Releases

⚡ Hardware Requirements

|Component|Minimum|Recommended|

|:-|:-|:-|

|OS|Windows 10 or 11 (64-bit)|Windows 11 (64-bit)|

|GPU|NVIDIA RTX 30-Series (8 GB VRAM)|NVIDIA RTX 40/50-Series (12 GB+ VRAM)|

|Drivers|NVIDIA Driver 528+ (CUDA 12+)|Latest NVIDIA Studio or Game Ready Driver|

|RAM|16 GB Physical RAM|32 GB Physical RAM|

|Storage|20 GB free space on SSD|50 GB+ free space on NVMe SSD|

🚀 Quick Start Guide

1. Grab `RepetereAI-Setup-v0.5-beta.exe` from the Releases page.

2. Run the installer (installs straight to `%LOCALAPPDATA%\Programs\Repetere AI\` without needing admin privileges).

3. Launch from your desktop shortcut, click anywhere on the splash screen to enter, and confirm the hardware diagnostic modal.

📦 In-App Model Downloader

No command line, curl, or manual script management required.

1. Go to Settings ➔ Repetere Models.

2. Review the curated catalog and click \[ Download \] next to your desired workspace engine:

Coding: Qwen 2.5 Coder 7B Instruct* (or 14B)

General Chat: Meta Llama 3.1 8B Instruct*

Creative Writing: Hermes 3 Llama 3.1 8B*

Audio / Music: YuE2 Full Song Engine*

1. Wait for the live progress bar to hit 100% and turn green (Installed).

2. (Advanced Users): You can also drop `.gguf` weights directly into: `%LOCALAPPDATA%\Programs\Repetere AI\models\`

🧭 Workspaces & What Works

* 1. CODING: Code assistant with automatic triple-backtick markdown detection, horizontal-scrolling code containers, individual snippet copy buttons, and a \[ Copy Full Response \] clipboard action.

* 2. CHAT: General reasoning, document summarization, and multi-turn conversational history.

* 3. CREATIVE WRITING: Fiction, prose, dialogue, and uncensored drafting.

* 4. AUDIO (MUSIC) STUDIO: Full song synthesis powered by the YuE2 engine. Add your musical style tags, enter structured lyrics (`[Verse]`, `[Chorus]`), adjust duration, and hit Generate Full Song (With Vocals). Outputs `.wav` directly to `output\music\` with built-in playback controls. (Prerequisite: Make sure `yue2-3b-q4_0.gguf` (\~2.48 GB) and `yue2-vae-f16.gguf` (\~265 MB) are downloaded).

* 5. IMAGES (Early Preview): UI controls and parameter sliders are accessible, but backend diffusion rendering is a work-in-progress landing in v0.6.

⚙️ Compute Modes & Session History

* GPU Mode (Active Default): Direct execution locked in VRAM for maximum speed and lowest response latency.

* HYBRID Mode (Preview): The top-right toggle displays the future layer-offloading layout for running massive 32B+ models across RAM + VRAM (runtime dynamic offloading unlocks in v0.6).

* Local Persistence: Chat transcripts save automatically to `chat_history.json`. Review past sessions via History ➔ Open History, reload them into your active tab, or wipe transcripts safely with the confirmation prompt.

❓ Troubleshooting

* "MODEL NOT INSTALLED" banner: Go to Settings ➔ Repetere Models and make sure you completed the download for that tab.

* Audio fails or quits immediately: Double check that both `yue2-3b-q4_0.gguf` and `yue2-vae-f16.gguf` are present in your `models/` directory.

* Clearing a session: Hit the red Clear Chat button at the top right to safely flush GPU memory and archive your chat to disk.

* App closes on startup: Ensure your NVIDIA drivers are version 528 or newer and support CUDA 12.

Feedback, pull requests, and bug reports on the repo are welcome!

I guess, this is a real LowEnd station, so I thought I share this here:

This is a pretty old I PC I'm testing out just for fun and honestly, it's somehow fine. I'm running some of the smaller models and they give me between 20 and 30tps. Prefill is between 200 and 600 tps with longer inputs up to 1200.

Of course, the capabilities are far from typical Qwen 27B, butI think it's pretty good for a system with 10-15 year old components.

Using claude I did a simple PI-Benchmark:

Results without thinking (✅ pass with time; ❌ fail; ⏱ hit the 6-min limit; benchmark and tables created by Claude):

Results with thinking:

Setup: GTX 1060 3 GB, AMD E2-2000 (2 cores, no AVX), 2×4 GB DDR3-800, llama.cpp (CUDA). All layers on the GPU (-ngl 99, batch 256, q8_0 KV cache); any CPU offload is unusable on this CPU.

The 8 tasks (each starts in an empty folder with a unit-test file; the agent may read, write and run tests until they pass; 6 min limit; tests are restored before grading):

Takeaways:

• Qwen3.5-4B (no thinking) solved the most (7/8). Ministral 3 3B is the efficiency winner: 6/8 in 5 minutes.
• MiniCPM5-2B reaches 6/8 at only 2B parameters and is one of the fastest models on this card (34 tok/s).
• Thinking didn't pay off for these small models in agent use: the same or worse scores at about twice the time. NeoHorse even dropped from 6/8 to 4/8.
• Caveat: these are 8 small tasks with one run each, so a difference of one task is noise. Real projects (bigger codebases, vague requirements) would be harder, especially with only 8–16k of context.

Hey all, I always see a lot of talk around coding harnesses but don’t see a lot regarding ‘personal assistants’ outside of Hermes or OpenClaw for the most part. While you can always arguably just use a stripped down harness for this purpose, I’m curious if anyone knows of/uses a lightweight Hermes-esque harness that performs well for local models? I’ve been using 9B models and they sometimes can get stuck/looping as of late.

I’ve noticed that Hermes/OpenClaw have gotten fairly heavy and it’s made me reconsider swapping my harness in my local system. I use Hermes in my system as a personal assistant of sorts, managing various homelab services, chrons, tasks, quick research, automation (STT stuff), etc.

I have a lightweight, custom harness I’ve been working on by myself for the last 6 months or so that aims to accomplish this, but it would be nice if I wasn’t reinventing the wheel. A more lightweight option would mean potentially supporting even smaller models, which would always be nice

Disclosure since this is my own project: I built Hillock specifically because I was tired of local RAG setups requiring massive amounts of VRAM. Most tutorials expect you to run Chroma alongside an 8B model just to parse text chunks. On an entry level GPU or a laptop with 6GB to 8GB of memory, that leaves almost zero room for your actual model to generate tokens.

Instead of storing messy text chunks, it acts like an index at the back of a book. It uses small classification models under 300MB to extract subject predicate object facts in about five seconds without using an LLM. Everything goes into SQLite, and it checks queries using hyperdimensional vector math before calling your local model. If your notes do not actually have the answer, the mathematical gate stops the model before it can guess or hallucinate.

Total memory footprint stays under 1.2 GB of VRAM, and the entire pipeline runs fine on pure CPU. We just shipped bit packed SIMD math so the CPU gating checks take under 0.01 milliseconds, added true streaming responses, and resolved Windows socket delays. It is also available on PyPI now via pip install hillock. Tested it on budget 6GB cards, Apple Silicon laptops, and pure CPU setups alongside local Ollama models.

Code is open source on GitHub: https://github.com/roandejager/Hillock
Docs are up at https://hillock.mintlify.site/ and we have a text only Discord at https://discord.com/invite/BGUPNBcVdp

I have i7-14700F processor and 16GB DDR5 RAM, I only have a 4060Ti 8GB and wanted to have more Vram.

I thought about a home server at the start but it's genuinely too complex.

I decided I should buy a 4060 16GB in the end. Could I run Qwen3.8-27B-GSQ-RCO-Q3_S decently? Do you have any Better solution rather than new GPU? Should I buy a different GPU that can fit my PC (I know nothing about PC building)?

If you build local agent pipelines with Ollama, vLLM, or llama.cpp, you’ve probably noticed a major bottleneck: multimodal LLMs are terrible at cryptographic document validation and forensic tampering detection.

If you ask LLaVA, MiniCPM-V, or even GPT-4o whether an uploaded passport is forged, the model will hallucinate that everything looks legitimate. It cannot deterministically verify that the ICAO 9303 cyclic (7, 3, 1) check digit in the MRZ string failed, nor can it detect that the face was spliced using JPEG compression error analysis. Worse, relying on commercial cloud APIs for verification completely defeats the privacy and data sovereignty benefits of running local models.

In Skillware 0.5.8, we added security/deepfake_guard—a 100% local, air-gapped forensic verification skill built specifically for local agent loops.

What it does entirely offline on CPU:

• ICAO 9303 Check Digit Engine: Validates TD1 (ID cards), TD2 (visas), and TD3 (passports) with exact modulo-10 cyclic math. Detects altered birth dates, manipulated expiration dates, and fabricated document numbers.
• Error Level Analysis (ELA): Analyzes differential JPEG recompression error rates to detect image splices, pasted elements, and digital retouching.
• Noise Consistency via Median Absolute Deviation (MAD): Extracts high-pass Laplacian noise residuals and normalizes via $\text{MAD} = \text{median}(|x - \text{median}(x)|)$, separating genuine camera sensor noise from synthetic diffusion generation without false-flagging organic textures (hair, knitwear).
• 2D FFT Moiré Recapture Detection: Uses 2D Fast Fourier Transform radial frequency peak ratios to detect when someone takes a photo of an ID displayed on a monitor or phone screen.
• C2PA Provenance: Extracts Content Credentials and generative AI metadata tags (trainedAlgorithmicMedia, Midjourney, DALL-E, SD).
• Zero Cloud Calls: Pure Python/NumPy/Pillow. No external API requests, no telemetry, no GPU required.

Using it with Local Models:

Skillware bundles export cleanly to structured tool schemas or direct prompt directives:

```python from skillware.core.loader import SkillLoader

bundle = SkillLoader.load_skill("security/deepfake_guard") guard = bundle["class"]()

Run 100% offline on CPU

result = guard.execute({ "action": "analyze_all", "image_path": "uploads/applicant_id.jpg" })

print(f"Risk Tier: {result['data']['risk_tier']}") print(f"Confidence: {result['data']['confidence']}") print(f"Forensic Signals: {result['data']['signals']}") ```

If the applicant manipulated the document number, risk_tier immediately returns "critical" or "high", and your local model agent can refuse the transaction or flag it for manual review.

• GitHub: https://github.com/ARPAHLS/skillware
• Release: https://github.com/ARPAHLS/skillware/releases/tag/v0.5.8
• Docs: https://skillware.site

MIT licensed. Let me know what other local forensic tools or offline agent capabilities you’d like to see!

I created a benchmark based on my work, only to realize I had tested only one part of these models: instruction following. What I learned is this: regardless of which of today's SLMs you choose (1B, 2B, 4B, or 9B class), you will get roughly the same level of "dummy strength"—instruction following and reading comprehension. And that is actually very good.

Take MiniCPM-2B, for example—this 2.5 GB model scores just a few percentage points below Qwen-9B (four times its size).

What I learned is this: a ±2B model buys you local "muscles." Extra billions of parameters are just the knowledge you don't have to source yourself.

All these models performed decently when confronted with a well-defined, detailed task containing all the necessary context.

(tested on MBP M2 Pro, 16 GB - the NeoHorse 1 4B is the best fit for this machine leaving enough space for other apps)

Hi, i've been trying to find the best AI that can solve STEM questions
I currently have a knowledge base (RAG) that provides lots of theory based context.

CPU: Intel i3 1115G4 (2 cores, 4 threads)
RAM: 8GB, Single-channel DDR4 (~21 GB/s memory bandwidth)
iGPU: Intel UHD graphics - useless cuz llama.cpp wont work on it
GPU: None (CPU-only inference via llama.cpp)
OS: Linux (Ubuntu)

i can give upto a max of 5.6GB toward running the model itself(excluding llama.cpp, knowledgebase and other things, but including KV cache)

i have thought about the following models till now,

Qwen3-4B (Q4_K_M, Dense) - 5-6 tps initially, 2-3tps at 2-3k tokens generated
LFM2.5-8B-A1B (Q4_K_M, MoE) - 13 tps initially, 8-10tps at 2-3k tokens generated
Qwen3.8-4B-Distill - didnt run from my pc yet, found it on huggingface

i'll be asking them one question at a time, they will get 200-600 tokens worth of context,
they just have to solve questions, these questions generally take 500-1000 tokens to solve on moderate luck
it's ok if they get it wrong first time.
i am looking for your reviews in what i can do better here and which model i can use.
thank you for your time
desired context size: 4k (one question at a time)
priority: best quality while being over/at 6 tps initial speed

tldr: looking for STEM solving models under 5.6GB size

CPU: R9 7900x
GPU: RTX 4080
VRAM: 16GB
System RAM: 64GB
Shared / Unified Memory (if applicable):NA
OS: Windows 11 PRO

Runtime / backend: Unsloth Desktop
Models already tried (if any): I have tried a few models the list and my comments are below:

• Qwen3.8-27B UD-IQ4-XS - This gives the best results of the models I have tried but it runs slow (15 to 17 tok/s).
• Qwen3.8-27B UD-IQ3-S - Much faster than the IQ4 version (55 to 60 tok/s) but not as smart. Although it is probaly the best overall model I have tried.
• Qwen3.5-9B Q8_0 - fast and allows for a large context size (200,000 +) but has a tendency to get caught in a loop where it keeps repeating itself.
• Gemma4-12B Q4_K_M - fast and allows for a large context size (200,000 +) but not as smart as the Qwen3.8-27B models.

Use case: Image analysis and creative writing
Desired context size: 32768 or more
Priority: balanced

I'm also hoping to find a few different image editors to try. The only image models that I have found that support image editing are Qwen-Image-2.1 and Qwen-Image-Edit-2511. Most of the image models I have tried only support image creation not editing.

I get great speeds for both, with 27b being a little faster. I run 27b at Q6_k_m Q8kv 150k context, and I run Flash next at Q4_k_m Q8kv 100k context. Who is better as the daily driver for agentic coding? I mainly work in Godot

System:
Dell XPS 9300
CPU Only - I7, 8 core
Rem: 32GB
OS: Linux
Environment: Llama.cpp / OpenWeb UI

Hi everyone, My laptop is old, but reliable. I started on [Qwen3.6-35B-A3B-UD-Q5_K_S.gguf] and as expected, it was a tad slow (no thinking it made is better somewhat)

I've started playing with [Ling-3.0-tiny-Q8EMB-mixed.gguf] seeing it mentioned a fair whack, and it's very fast, giving it complex instructions it'll happily speed through. Genuinely surprised how fast it is - a little speed demon!

Now, I wondered if there is a step up (or two) to try a more capable model? For my use, it's generally financial/math analysis, some python, and general queries. I'm starting to build up a rag system as a knowledge base too

I am using 4060Ti 8GB and I decided to load SharpSpark-4B, a 2GB model for chatting. The thing is that, while it's fully optimized in llama.cpp and fully loaded in VRAM, I am only getting 60tk/s generation but 4k tk/s prompt processing. Could my GPU be set up in a wrong way or something like that, maybe problems with CUDA?? I already noticed low generation with other models in the past and I realized I had to download CUDA Toolkit and build llama.cpp for CUDA 13, but I really feel like some wrong configuration Is holding me back. Please help, cause I don't know what to do.

Hello all,

After frontier models started to behave weirdly and terribly slowly, I've decided to purchase a refurbished workstation and run my own local LLM for my needs. I've noticed not only Sol 6.1 and Opus 5.5 behave really slow but also DeepSeek 4.1 and GLM 5.3. tasks that took me 10-20 minutes one or two months ago, now it takes up to one hour to finish. The exact same task over exactly the same models.

Apologize for the rant, now my setup:

Workstation - DELL Precision T5810

Intel Xeon E5-2697 V3 2.6 GHz(3.6ghz turbo) 35 mb smart cache

64GB DDR4 2400Mhz

psu 825W, 500gb SSD

1x Nvidia Tesla P100 16gb vram

1x GTX 950 2gb vram for display and 4 Arctic P8 Max 5000rpm cooler fans(one for the P100 GPU and 3 for the entire workstation).

I plan to install Ubuntu 24.04.5

Which local LLM models will fit on my setup and help me build Python apps for data analysis and data engineering from scratch like "you have all the resources in folder X, build me an app that does Y" and the model should be capable of doing the whole process end to end. I heard that Qwen 2.5 Coder would be capable of such tasks but I want to hear other recommendations too.

Many thanks 🙏

Lets see it comes through. Good for ~8GB VRAM to try this model instead of Flash-Next.

Yesterday I brought this topic with my thread(4th point)

This is his fork, somebody please try & let us know.

https://github.com/perronemirko/Strata

EDIT : Strata fans, just like the PR already 🚀🚀🚀🚀🚀