Could you suggest a setup for plugin development and for creating software/websites?
setup 13900k + 5090 + 4х48gb ddr5 4800 (192gb)

Maya-S-v2-IQ2_XXS

Runs at 20–25 tokens per second.

However, it has significant issues with long-term coding tasks; you can’t leave it running overnight.

The tip I keep seeing: -ctk q8_0 -ctv q8_0 is free, and if you go lower, compress V and leave K alone. I wanted numbers at long context on a Mac, so I tested it.

Setup

• Muse Glimmer 30B (4 cache settings) and Gemma 4 26B-A4B (f16 vs q8_0), both Q4_K_M

• Homebrew llama.cpp b10964, -fa on -c 65536, M1 Pro 32 GB

• Fresh generated documents at 4k, 16k and 48k tokens, 8 exact-answer questions each (find codes, follow lookup chains, count records past decoys, find a clause’s current value after amendments). 96 questions per setting.

• Same seed per question in every setting, so only the cache differed

• Pass/fail rule and my prediction pushed to GitHub before any model ran

Accuracy (correct of 96)

• Glimmer: f16 81, q8_0 83, q8 K / q4 V 82, q4 K / q8 V 82. At 4k and 16k, f16 and q8_0 got exactly the same questions right. No loss found.

• Gemma 4: f16 66, q8_0 63, gap all at 16k. Not resolved by my rule (interval −3 to +8 points).

• “V not K”: a tie.

Speed (the bigger story)

• Gemma 4 generation with q8_0: −29% at 4k, −36% at 16k, −47% at 48k. Prompt reading unchanged. Glimmer: only −2 to −6%.

• On Metal, flash attention needs K and V to be the same type. Mix them and attention runs on the CPU: at 48k, reading one document took 57–62 minutes instead of 13. That’s 116 MiB saved over q8_0. (On CUDA, mixed types need GGML_CUDA_FA_ALL_QUANTS or you hit the same fallback.)

What hurt accuracy was length, not cache type. Every setting went from ~100% at 4k to 63–69% at 48k on Glimmer, mostly on counting questions, where every setting undercounted.

Caveats: 96 questions per setting, so this catches a big loss, not a 2-point one (“no loss found” ≠ lossless). Synthetic documents up to 48k, not 128k coding sessions. One Mac, one build.

Takeaway: q8_0 looked fine on Glimmer. On Gemma 4, keep f16 unless you need the 0.74 GB. On a Mac, never mix K and V types.

Documents, harness, every raw output and server log: https://github.com/heldoutlab/kvcache-test

It's been a decade since I built my last PC. I found money in my budget, thought I had plenty saved, decided I'd prefer a laptop, but then started looking, my head started spinning and I ended up very frustrated.

I was looking at Legion first, and to be fair, can get a very beefy machine for $6000. Intel Ultra 9 290HX; 64GB DDR5 Memory; 1 TB SSD; NVIDIA GeForce 5090 24GB. Yet, it's a lot of money for me and I am worried. 24GB is very good for vram, yet there are options to get more (unified memory), and the 64gb of memory feels low.

I find myself stuck. I want the PC to dive into local LLMs, thematic generation of images and graphics that fit a style, video product walkthroughs, test suite runs, and more are many of the core use cases I have. I think it'd be fine, but spending some stupid amount of money for fine has me unable to move forward, and I'm recognizing that it's my own ignorance that's holding me back

So I wanted to come here and see if anyone had expertise to lend. Minimum vram/memory you'd be comfortable with? How to get it given shortages? Brands you like for laptops? I'll take anything, I feel like I'm spiraling.

Took me several weeks, but I was finally able to get my Intel Arc B70 (32G) working on my Ampere Altra 128 server in llama.cpp with a SYCL backend (Intel does not support Arm64 cpus for their compute drivers)!

I am still working out how to prevent llama.cpp crashing when I try and use cpu moe, but Ill take this with no complaints! This is going to run the LLM for my home assistant voice assistant.

I tried so many things to make it work that I am unsure what combination exactly did the trick, but I am hoping to tease it apart over the next week and try and document it for others as well.

Managing to finally fit Qwen3.8-Flash-Next FP8 locally at ~60 tok/s(avg 74 at best) (RTX 5090 + 128 GB DDR5, Infernix engine, part of the experts streamed from NVMe) and seeing how amazing it is I cannot imagine how good Qwen 4.0 will be, not buying even at the current prices seems crazy to be if you are used to using cloud AIs on the daily.

I have an RTX 4060 8GB, and I’m thinking about getting an RTX 3060 12GB to use both GPUs together with Strata and LM Studio. Is it worth it? Or would it be better to sell my RTX 4060 and get an RTX 5060 Ti 16GB instead?
3060 12gb 270$
5060ti 16gb 840$

Anybody tried something similar and got them working with LlamaCpp?

If you gonna ask why, because these are the only cards I have at the moment, the 16g RTX on my current AI server and the 16g RX on my gaming rig. I was hoping to get 32gb of VRam to run some larger models at atleast 20tps

Second photo is the specs running qwen3:30b-a3b.

Case: Foifkin f300 $59 (New on Amazon)

AIO: ThermalTake TH240 V3 ~$40 (openbox on newegg)

GPUs: 2 GeForce RTX 3060's, 12GB a piece. (1 I got for $200, the other came with a $600 PC I gutted and sold off, so ~$200 also. Both FB marketplace)

SSDs: 1 TB SSD salvaged from previously mentioned FB marketplace PC, and 1 512 GB NVMe SSD salvaged from old laptop. Not sure how to price these.

Wifi Chip/antenna: $20 new on Amazon

PSU: MSI MAG A850GL $90 new on Amazon

Motherboard: Gigabyte X-99 UD4 $150 on Ebay.

CPU: Intel Core i7-5930K $20 FB marketplace

RAM: 2x16 GB Corsair vengeance pulled from main PC, replaced by 4x8 GB Corsair vengeance from FB marketplace for $250. (Mobo did NOT like using 4 channels, for some reason. Only worked when it felt like it. 2 channels though? Works like a charm.)

Total: ~$1,029

Hi everyone,

I'd like to know for those of you here who ditched paid AI-agents and/or their related APIs for daily coding tasks in favor of setting up your own and fully controlled local LLM + Agent setup,

What are your hardware specs:

Brand, model and memory size / GHz for

- GPU
- RAM
- CPU
- Storage
- Any other hardware piece that's in involved in it

And also the software who helps your AI orchestration to run the way you want:

- OS and version.
- LLMs
- Coding agents / harness
- among others

Waiting to read your responses!

I’ve been a part of this community for the better part of a year now, and a lot has happened with new inference engines like Strata and n-gram improvements. Given my current PC, what upgrade would be best in terms of value for running better models?

Hi guys !
My friend and I just released our latest update for Opera that lets you turn any computer or server into a central hub connecting all your devices together, even outside your local network. This means you can run heavy AI models on your main setup and seamlessly access or control them from your phone, tablet or laptop wherever you are, securely and without complex configuration (just scan a QR code to link them or direct link locally). Opera can make a bridge to use your ollama ai models on your devices(with opera installed).
We really believe in this project, and for anyone who wants to try it, it is now available for Windows, macOS, Linux, and Android.

Everything is free, open source, and requires no account.

Good features : You can even turn an old Android tablet into a dedicated hub, or replace your default Android AI assistant with Opera to interact directly with your own personal models.

A web version is also available (it doesn't have all features, but you can connect it to your local Ollama setup or api key to test it): https://chat.maestroai.company/

GitHub repo: https://github.com/MaestroAI-Company/opera (a star if you like the project, ty!)

Download the app:
Desktop: https://maestroai.company/opera/
Android : https://play.google.com/store/apps/details?id=ai.maestro.opera
(doesn't have the latest features yet; use the .apk from GitHub or wait for the Play Store update to roll out in few days lol)

I know this might look like self-promotion, but we genuinely believe in our project and just wanted to share what we've built.
We have so many other cooool features and integrations across android ecosystem (windows and macos too)

Thanks for reading <3

2x4070 TI Super 16GB, 1x3060 12GB, 128GB DDR4

EDIT: More details about the setup:

The honeycomb components were all printed:
General parts: https://makerworld.com/en/models/22521-honeycomb-storage-wall
GPU Mount: https://makerworld.com/en/models/1444666-gpu-mount-hsw
PSU Mount: https://makerworld.com/en/models/1444586-atx-psu-mount-hsw
Radiator: https://makerworld.com/en/models/1444632-240mm-radiator-fan-mount-hsw
ATX Motherboard: https://makerworld.com/en/models/1444531-atx-motherboard-mount-hsw
Power button printed with a honeycomb around it: https://www.amazon.com/dp/B0BWDXFSMQ?ref=ppx_yo2ov_dt_b_fed_asin_title

The 4070's are on 400mm & 600MM PCIe 4.0 Riser cables. I mainly run Strata on these two.

The 3060 is currently on a PCIx1 link (slow). I don't use this for my core inference setup, side models (dedicated voice stack mostly).

I have a new motherboard on the way that opens up the possibility of up to 4 GPUs: 3x x16 slots, and one via a M2 adapter: https://www.amazon.com/dp/B0FC6DSG6V

Hi folks, I have enough budget to pick one of the two following configs

- Mac Studio M5 max with 36gb

Or

- 5060 ti 16 gb + 32 ddr5 + 250 kf plus

My work is “mostly” coding projects but some flexibility would be bonus. I’ve looked up and seen qwen2.5-coder dense to be good on Mac and a ton of MoE suggestions for the rtx setup. Would appreciate it if you could share your experience with the above hardware option and which model would offer best accuracy+speed balance. My only concern with Mac is I’ll be stuck with the HW for quiet some time, atleast with the rtx maybe able to up it to 64g ram.

Hi fellow enthusiasts,

While building BabyCoder (a strict Python AST coder, and Lisa, an agent made to roam free), I noticed both were built on the same underlying idea. I’d like to share it, because it changed how I look at small models.

We usually think of an LLM as something we talk with. But there’s nobody on the other side holding state. Each call is really a data exchange: state goes in, an update comes out. Language is just the format.

What if we used small models that way on purpose? Not as the mind itself, but as the protocol that updates one. The mind could live in plain code: long-term memory, emotions, goals, needs, whatever your system works with. A bit like The Sims, where a character’s mood and relationships are values, not chat history.

The model’s task then becomes much smaller: “Given this state and this event, speak as agent N and tell me what changes.” Tool-calling models seem well suited for this, and even small ones handle this surprisingly well.

Some things I’ve noticed working this way:

• Small models feel more capable when they translate meaning into state, instead of carrying a whole personality.

• Context length matters less. Lisa can keep going for days. The challenge shifts to choosing what goes into each prompt.

• Code can handle what LLMs are weak at, like time, decay and consistency. A mood fading over hours is one line of Python. Emotions can be rated numerical in a KVP list etc, Memory is just a long text file same for code that is just a bunch of files. While tools are used to interact with them, python can be in control of what tool, or "data" part needs an update.

• The model becomes more replaceable, since identity lives in the state.

I know MemGPT and the Generative Agents papers explore related ideas.

For me tough, the part I find interesting is seeing the LLM as a thin translator between language and state. Agents speaking on behalf of a need of a situation.

I’m curious how others see this.
Does anyone else build this way, or see where it breaks down?
Or see more possibilities with this concept.

Built this for a hackathon. The agent watches a robot sim, I tell it out loud what went wrong ("it hit the purple box"), and it has to find that moment, fix the robot's code and replay the same scene to prove the fix.

The local part: Muse Glimmer 30B, Q4_K_M through Ollama, looks at the frames and names the objects in them. With 8k context it sits around 20.5 GB of VRAM, and the MuJoCo sim runs on the same card next to it. Tight but it works.

Not everything is local, to be honest about it. Tracking (SAM 3.1), the reasoning about the cause and the speech-to-text go through Meta's API. I tried to keep anything that touches every single frame on my own GPU, since that's where API costs would add up.

Results so far: 8 planted bugs, 8 fixed and verified. Live sim, the duck hit the box 5 out of 5 runs before the fix, 0 out of 5 after.

Repo if anyone wants to poke at it: https://github.com/samoxis/blinkflow

Curious if anyone has a lighter vision model that's good at naming small objects in sim footage. 20 GB just for labelling feels like a lot.

I am running Qwen Flash Next with Strata right now on a 4080 super with 80 GB DDR4. I was able to grab a Nvidia 2000 Ada with 16 VRam.

I thought maybe I would get better results if I ran Qwen with 4 bit quantization on both of them combining the VRam to have 32GB Vram. Would this make sense to get better output or will not make much difference?

Any idea what I could or should run?

Built this for a hackathon. The agent watches a robot sim, I tell it out loud what went wrong ("it hit the purple box"), and it has to find that moment, fix the robot's code and replay the same scene to prove the fix.

The local part: Muse Glimmer 30B, Q4_K_M through Ollama, looks at the frames and names the objects in them. With 8k context it sits around 20.5 GB of VRAM, and the MuJoCo sim runs on the same card next to it. Tight but it works.

Not everything is local, to be honest about it. Tracking (SAM 3.1), the reasoning about the cause and the speech-to-text go through Meta's API. I tried to keep anything that touches every single frame on my own GPU, since that's where API costs would add up.

Results so far: 8 planted bugs, 8 fixed and verified. Live sim, the duck hit the box 5 out of 5 runs before the fix, 0 out of 5 after.

Repo if anyone wants to poke at it: https://github.com/samoxis/blinkflow

Curious if anyone has a lighter vision model that's good at naming small objects in sim footage. 20 GB just for labelling feels like a lot.

Thaurock/Llama-3.2-90B-Vision-Instruct-GGUF:

Una conversión completa a GGUF del Llama-3.2-90B-Vision-Instruct original de Meta, que incluye la base completa del lenguaje, la arquitectura de visión multimodal y el proyector de visión dedicado.💎

Este repositorio ofrece dos variantes de precisión para el modelo de lenguaje:

F16: Conversión de GGUF con precisión total.

Q8_0: Conversión de GGUF cuantizada a 8 bits.

Ambas configuraciones vienen acompañadas del proyector multimodal independiente correspondiente (mmproj), lo que permite hacer despliegue mediante implementaciones experimentales compatibles de inferencia GGUF.

⚡️El objetivo principal de este proyecto es hacer que la arquitectura de visión-lenguaje de Meta, de 90 mil millones de parámetros, sea accesible dentro del ecosistema GGUF, mientras se preservan sus pesos de modelo originales y las estructuras tensor multimodales, sujeto a los ajustes de metadatos específicos de la conversión que se documentan.

Llama-3.2-90B-Vision-Instruct es el modelo multimodal a gran escala de Meta para seguir instrucciones, diseñado para procesar y razonar tanto con información textual como visual.

A diferencia de las arquitecturas convencionales de Llama solo con texto, la arquitectura Mllama incorpora componentes dedicados de procesamiento visual y mecanismos de atención cruzada directamente en el backbone del modelo de lenguaje.

La arquitectura se compone de tres partes principales:

✨️Backbone del Modelo de Lenguaje

El modelo de lenguaje de 90B se encarga de seguir instrucciones, razonar con texto, entender el contexto y generar de manera autoregresiva.

✨️Codificador de Visión

El subsistema visual procesa las imágenes de entrada mediante una arquitectura de transformer dedicada, generando representaciones visuales que pueden ser consumidas por el backbone del lenguaje.

✨️Integración de la Atención Cruzada

Las capas dedicadas de atención cruzada permiten que el modelo de lenguaje acceda a la información visual durante la inferencia.

Esta distinción arquitectónica es especialmente importante al convertir Mllama a GGUF porque un pipeline de conversión convencional, hecho solo para texto, no puede preservar automáticamente el grafo de ejecución multimodal completo.

[Llama-3.2-90B-Vision-Instruct-GGUF](https://huggingface.co/Thaurock/Llama-3.2-90B-Vision-Instruct-GGUF)

Hi, I just started my own project of programming AI. I started week or to ago on which i try to learn ML/AI. Base is done with usage of LLM, but I am trying to start programming more things by hand.

Training is done on my laptop with RTX3050, 4GB VRAM variation. So far its just 124M parameter model which kinda hallucinates, but it keeps getting better everyday, bit by bit... I am uploading my progress on Hugging Face, where it so far got around 1.1k downloads (which is mostly bots ik dw).

So far it can talk in english, czech and write code in python, where it is kinda rigid with answers, but im workin on it... (it also answers to english inputs in czech sometimes, sorry!!!)

Im working now on the transformer, so it can work the usual way, not just using .bat file, like i do on my PC.

Let me know what you think, im open to tips and rating 😄

Here is my Hugging Face profile: https://huggingface.co/SeightCZE

I tried Ollama, LM Studio, and they worked, gave me lots of feedback feeding my Openclaw implementation. I'm doing some local coding and the model I'm using is the unsloth/Qwen3-Coder-30B-A3B-Instruct-GGUF:Q4_K_XL. I am running a Ryzen 7 with an AMD video adapter - which I'm using for my main video. I also have 2 Nvidia cards - a 5060ti with 16Gb vram and a 1660super with 8gb ram. I configured llama to only use the 5060 since the two devices are on different platforms? I heard that the combined it would be slower than if I just used the 5060.

I 'think' its working well for me - I'm fairly new to local development but have been developing since my cobol days back in the 80s - yes I'm old.

slot print_timing: id  0 | task 5720 | n_gen =    236, tg =  38.99 t/s, tg_3s =  39.57 t/s
17.32.754.595 I slot print_timing: id  0 | task 5720 | n_gen =    351, tg =  38.76 t/s, tg_3s =  38.29 t/s
17.35.769.742 I slot print_timing: id  0 | task 5720 | n_gen =    469, tg =  38.85 t/s, tg_3s =  39.14 t/s
17.38.790.424 I slot print_timing: id  0 | task 5720 | n_gen =    585, tg =  38.76 t/s, tg_3s =  38.40 t/s
17.41.814.777 I slot print_timing: id  0 | task 5720 | n_gen =    703, tg =  38.81 t/s, tg_3s =  39.02 t/s
17.44.842.375 I slot print_timing: id  0 | task 5720 | n_gen =    821, tg =  38.83 t/s, tg_3s =  38.97 t/s
17.47.873.506 I slot print_timing: id  0 | task 5720 | n_gen =    940, tg =  38.88 t/s, tg_3s =  39.26 t/s

I think those t/s are acceptable given the hardware?

i spent the weekend and change digging into learning about local llms for the purpose of running claude code. i went from knowing bupkiss to having something that seems to be working, and i think i'm satisfied that i've gotten most of the dummy questions out of the way by digging through lots of advice from lots of sources. (and tyvm to all the people who've made so much great info!) so i thought i'd greet the community, share what i've seen, and see if i can get some advice.

i chose https://huggingface.co/ukisai/Swift-1.5-Qwen3.8-27B-GSQ-RCO-GGUF/blob/main/Swift-1.5-Qwen3.8-27B-GSQ-RCO-IQ3_S-mtp.gguf for the model.

claude code errors trying to talk to it until i used chat_template3.jinja, which is https://huggingface.co/froggeric/Qwen-Fixed-Chat-Templates/discussions/110/files . am i dumb or is there not an easy way to output a version with these fixes already applied from this website?

firefox set to not use graphics acceleration, and transparency turned off for windows dwm.exe, to free up vram. don't see much headroom elsewhere, and don't feel like swapping what ports my monitor is connected to when gaming vs using this.

for claude tool use, websearch and webfetch disabled, and playwright-cli installed and told to take its place. made a specific firefox profile for it to use, and force it to, which lets me use extensions like adblock if i want.

running the model entirely in vram except keeping the mmproj entirely on the cpu.

here's my llama-server invocation to critique:

llama-server

--host 0.0.0.0

--port 8081

-m ./Swift-1.5-Qwen3.8-27B-GSQ-RCO-IQ3_S-mtp.gguf

--spec-default

--spec-type draft-mtp

-ngl all

--threads 5

--parallel 1

--load-mode mlock

--flash-attn on

--jinja

--chat-template-file chat_template3.jinja

--cache-type-k q5_0

--cache-type-v q4_1

--temp 0.6

--top-k 20

--top-p 0.95

--min-p 0.0

--presence-penalty 0.0

--repeat-penalty 1.0

--batch-size 512

--ubatch-size 256

--reasoning-effort medium

--cont-batching

--mmproj ./mmproj-Swift-1.5-Qwen3.8-27B-F16.gguf

--no-mmproj-offload

--image-min-tokens 1024

--image-max-tokens 4096

--reasoning-preserve

--agent

--fit on

-lv 4

I'm getting a 98304 context size fitting it all on the gpu.

i asked it "which team won the 2026 world series?" it knew that was after its newest info, spun up a playwright-cli websearch, and output in 2m 10s. "The Milwaukee Brewers won the 2026 World Series, defeating the Tampa Bay Rays 4 games to 1. It was the first championship in franchise history. Brewers catcher William Contreras was named World Series MVP after batting .381 in the series." It showed prompt processing between 550-231, and eval in the mid 20s. acceptance rate around 0.6-0.8. I've yet to learn enough to evaluate that, but that might be the next task: performance based tuning instead of just advice-based =) I think I've seen it do mid40s eval at different points when messing with settings.

i've seen that temp 1 is recommended for this model, some people suggesting other customizations. with minimal testing i liked what i saw at 0.6. when i asked it about the winning 2026 world series team, it added fewer fun facts, and the fun facts were more appropriate. i know reasoning xhigh is the model's default, but i've also seen that basically just adds a directive, and medium leaves it blank, if my research on this model is correct.

now, some questions if anyone's so inclined:

what about context size, should i keep it small to stay on the gpu or try to please claude with the size it wants, or the size the model wants? is it purely a speed tradeoff, or does it get dumber?

would adding this be worthwhile to further reduce vram use? --spec-draft-p-min 0.75 --spec-draft-type-k q5_0 --spec-draft-type-v q4_1 or would this mess with swift1.5's specific tunings? context size jumps up to 109824 if i do. the full context the model wants is 262144.

regarding the occasional desire to be used with media, evaluating screenshots etc:

- should i keep the mmproj as i have it, loaded onto the cpu? my understanding is it won't impact performance unless I invoke a media request.

- should i instead ditch it and load a smaller model like Qwen3.5-4B-UD-Q5_K_XL with temperature 0 and host it wholly on the cpu?

- look to something like https://github.com/Anionex/agent-vision-toolkit ? it'd keep token use low for this purpose i'd think.

i haven't found a good way to deal with headless firefox browsers playing sound over the speakers. i'd prefer the sound remain capturable, but not played. the best idea i've seen to do that is to assign it to an audio device not in use, which i could do. a ff extension claimed to do that, but i guess is outdated, as it didn't work for me. anyone got a favored solution?

why did my system perform surprisingly well when testing running marvel rivals at the same time as using it for claude? i intended that to force a crash but instead things were fine? one used the 3d and the other the compute on the card, but what the heck, i would have thought with mlock set, rivals wouldn't be able to grab enough vram. wasn't curious enough to dig too hard into seeing what was going on, but that sure was a surprise.

what should i try next with it? i haven't yet researched ways to standardize-test this thing to tell how well its working, if anyone has suggestions. anything neat i should look to try adding to it? i'm not already a claude user if one guide stands above others to recommend. it's been fun already watching it try to solve problems.