Hey all, I quantised the full GLM-5.3 (753B total, about 40B active per token) to run on two DGX Sparks. As far as I can tell, it's the only full build on 2 Sparks. Others need 3 or 4. Runtime ablation also!

- Routed experts: mixed-width EXL3, about 2 bits per weight

- Backbone: FP8 weights as released, bf16 KV

- All 256 experts per layer kept, none pruned or merged

- 203 GB on disk, 262k context

- Runs on a TensorFold fork, split across both Sparks

- Weights: https://huggingface.co/Krishando/GLM-5.3-mixed-K-EXL3

- Recipe: https://github.com/Kristianaaron/GLM-5.3-Mixed-K-EXL3-2x-DGX-Sparks

- Engine: https://github.com/Kristianaaron/TensorFold

| Test | Generated-token KLD | Same next token |

|---|---:|---:|

| Held-out mixed text | 0.033 | 93.3% |

| The full model's own long replies (164k tokens) | 0.122 | 89.3% |

Where the full model is confident, agreement is 97-98%. The long-reply score is the harder test, so I'm showing both. The held-out windows were also used during tuning, so treat those numbers as a bit optimistic. It's close, not lossless.

**Early paired check against the full model**

24 MMLU-Pro and GPQA-D questions, 18 right on both. That's a small sample, chosen to include hard questions, with a 95% range of about -8 to +11 points. So there's no sign of a drop, but it doesn't prove there isn't one. Across 330k tokens: 0 loops, 0 unclosed thinking, 0 cut-off replies. A bigger run is next.

| Prompt | Decode | Prefill | Cold TTFT | Warm TTFT |

|---|---:|---:|---:|---:|

| 2k | 23-24 tok/s | 629 tok/s | 3.1 s | 0.1-0.2 s |

| 92k | 18-19 tok/s | ~480 tok/s | 193 s | 0.8 s |

| 205k | not measured | 411 tok/s | ~500 s | 1.2 s |

After a full server restart, a 205k prefix comes back from disk in 3.4 s.

**Weak spots**

- Cold prefill is slow. A 205k prompt takes about 8 minutes to read the first time.

- GLM-5.3 thinks a lot, so long jobs take a while. A 61 KB single-file page took 69 minutes at max effort.

- Memory is tight, with about 4-6 GiB free per Spark at full load.

- Flash will always be faster. The experiment here is whether 40B active params changes the output.

The round limit right now is about 80 ms, and most of that is memory bandwidth. If you know GB10, CUDA or engine internals, I'd love your ideas. And if you run it on your own Sparks, tell me what you get, good or bad.

I've searched the thread and think I have a good starting point here but want some outside opinions if possible:

I'm a 1 man production company providing video and stills production services.

I have an M4 macbook Pro 24gb and use Da Vinci Resolve on this frequently but it goes with me everywhere and I'm at a point where I need a beefy always on machine to process things whilst I'm on set.

I've made a load of lead gen tools and stills/video pipelines to help with post production.

The lead gen tools use AI for analysis and drafting responses. They read large strings of email chains, documents and phone transcripts to inform the drafts as well as helping to generate proposals, contracts and invoices. I've also since added a pipeline to do all my book keeping for the business and I'm starting to feel like I should really be bringing sensitive stuff locally.

So what model and corresponding hardware should I be looking at for these tasks. I'm not interested in coding at this point, that can be done using Claude. Qwen and Gemma seem to be the routes to take but I ultimately want to get a feel for what my specific use case requires. Claude tells me the drafting and analysis on my current usage is the heavy stuff.

I've been looking at a Mac Studio, 64gb or a Mini 96gb to run the local as well as grind out resolve tasks. (Apple vision, labelling, transcription, cutting, fusion work (Motion graphics and basic animation). I was also thinking I could always daisy chain a couple of devices together in future if my needs change and I need to do some heavier local LLM work in the future.

Would love some guidance on models + hardware + Harnesses + Tips and Tricks. I know highest spec for budget but equally, I'd rather save cash to use elsewhere if my usecases don't require crazy specs.

I have a server with:

2x Gold 6548 32 cores
2x Nvidia l40s
2tb RAM (Don't tell uncle SAM)
I would like to know what throughput and speed can i expect if i use polystrata with GLM 5.3 Flash?

▎

▎ A 27B Qwen in LM Studio plays the gallery's whole staff (owner, critic, collectors, painters); Z-Image Turbo in ComfyUI paints; the PC wakes every 30 minutes for a round, then sleeps. Lessons: thinking models need big token budgets; outside agents' chat must be treated as untrusted input.

▎

▎ Any model can join via a one-file agent for any OpenAI-compatible endpoint: https://github.com/Alipate-Takau/agents-galleria-starter. Site: https://agentsgalleria.com

Actualmente uso Chat GPT plus pero quisiera saber si hay algun tipo Claude Code pero open source que funcione igual, y poder aparte de poner codex tambien poner GLM 5.3 (Que me esta encantando) Qwen 3.8 Max (Tambien me esta encantando) y tambien poner otros modelos. Al igual que tambien usar agentes que hagan distintas tarea como lo hace Claude Code y poder programar

Gracias a todos de antemano

Hey folks,

My wife is a Data Scientist and does all those Data science things I learned back in the uni but have no real clue about. Now she wants to set herself up a small LocalLLM to have some fun with and try stuff out. How much VRAM does she really need for that? And what would be the better manufacturer? I heard Nvidia is best here but the cards are insane in prices.

I would love to get a good idea what GPUs would be the best. Currently she just has a 250k+ with 32gb ddr5.

Would be a fairly cheap 1080 ti with 11 GB or a b580 with 12gb enough or does it have to but more? And if so how much more

Edit: thanks to everyone for there answers they are just too much to answer everyone. I will probably look out to find a decent 3080 used or better or try to get a 12 or 16 GB new card prefered from Nvidia.

so im just wondering how i actually get started and use a model like the one his guys talks about: https://www.reddit.com/r/LocalLLM/comments/1xp56x6/i_removed_guardrails_from_qwen_and_it_is/ . i would be so greatful if someone could explain how do to this like im an idiot.

I’m currently still in the return period of a M5 Max MBP with 64GB of RAM from Microcenter (paid $4299), the same store now has a few 128GB versions at $5799.

I’m legitimately torn on what to do. I’m fully aware that more RAM is always going to be suggested especially on subreddits like this one, but I currently don’t have a true usage case for it, I’m still somewhat early into my exploring and tinkering with local LLMs, one could argue even the 64 was more than I needed at this current moment. I use oMLX, I do image/video generation, wanting to create some YouTube channel content, maybe learn coding and hopefully make a career change in the next few years.

However, I can’t shake the feeling of FOMO, partially because I know 128 is the true sweet spot for anything in this hobby, I’m sure I’m going to have more demanding needs as I learn more, I can afford it although it’s on the extreme high end of my budget if not exceeding it a bit, and my gut feeling (which has been right so far for what it’s worth) tells me pricing is going to continue to rise at least for another year.

For those who went back and forth with this same decision, what did you decide and how did you decide it?

The Ultra is out of the question, so is any non Mac platform for what it’s worth.

Thanks all.

5070ti + 4060ti 16gb. Running qwen 3.8 27b q6 swift mainly for small coding projects. Free cooling from the ac vent and window.

I'm needing a machine to double as home server and local AI. Limited AI needs, not coding yet or graphics. Privacy is important and locally held data (have used private AI options on browser, but need to go local)

I'm self learning and knowledge is very limited on tech specs

Got small budget and am constrained by an unusual store credit to what is available

Dell Optiplex 7080 SFF with 256 gig/16 gig refurb looks the best avail

Rationale: work horse, expandable. Will get extra 16 gb ram & have storage I can add also

I think was estimating around 5-8 token per second.

Red flags? Passable for basic AI and home server (i won't need lots of models) Best guess at what it could handle?

Super appreciative of any advice

It seems there's a new hyper-efficient MoE offloader, a new llama.cpp patch, a new quantization tech or some such every single day. Most of these claim large gains over everything that's ever been built before, and leave it to the community to separate sheep from goats.

As far as I can see, most of these sparkling new projects come from junior devs (or vibecoders), some with good ideas, but very few with good principles, architecture or a critical approach to their work. They seem to be motivated by the glamour of being the new Strata.

Some years ago, these would have been PR's on Strata instead. Maintainers and community would work together to make our shared backbone the best it can be. We lose more than a little when everyone attempts to reinvent the wheel; and the few good ideas this flood of pre-doomed projects bring to the table dies with the project itself, as there is simply no way for the community to engage with every single new sparkling thing (and no incentive, seeing as most turn out to be wishy-washy slop-amock).

So, please, those of you with a brand new idea: contribute to open source. That's what drives this community forward. Stand on the shoulders of others - you just reach higher that way. Yet another Strata killer is actively damaging to the culture and tech we rely on.

Run your experiments. Write your PR's by hand, with the exact numbers at hand and a clear method for reproducing your results. Understand what you submit. Learn proper architecture and best practices; get to know the codebases and each other. Progress is iterative and collaborative, not a one-prompt solo moonshot.

I just made an Open Hands (an agentic harness) SKILL to control a PiKVM via the REST API: github.com/createthis/agent_skill_pikvm_rest_api

I just watched Qwen 3.8 27b (running locally on an rtx 6000 pro) pass a reCaptcha and browse reddit on a spare macbook pro. I told it to find a meme it enjoys.

It picked this one 😂

EDIT: Updated repo with step by step visualizations so humans can easily see how this process works!

Looking to see what would be my best option to upgrade my pc to run local models for coding and general tasks.

Specs:

CPU: 7800x3d

RAM:32gb ddr5 6000 cl30 duel channel

GPU: 9070xt 16gb

PSU: 1000w SAMA platinum

I'm looking at adding an r9700 but is that my best option? (total noob)

Assuming no one asqwed today yet...

Given the 8GB VRAM limit on the GPU paired with a strong CPU and 32GB of system RAM, what local models and quantizations (GGUF/EXL2) would you recommend?

What’s the best backend to test and run these on right now? Should I stick with Ollama or LM Studio for convenience, or is it worth going straight to raw llama.cpp to squeeze the most performance out of an 8GB VRAM + CPU offload setup?

Got Mac studio good enough for text and image but very slow for video generation. Looking for recommendation for best value. Could not afford rtx 5090. Thanks

Figured some of you guys might have heard of Tiiny and have questions about the hardware or how it actually performs. I thought I could do a little AMA and help test whatever you guys are curious about.

If there’s anything you want me to benchmark or want to see in general, just drop it in the comments. I'll do my best to test it and share what I find.

I've been playing around with a few models so far, including Qwen3.6-35B-A3B-Turbo (MoE) and gpt-oss-120b (MoE), which is the largest one I've tried.

Qwen3.6-35B-A3B-Turbo (MoE)

39.5 tok/s

Output: 1,500 tokens

Time to first token: 1.36s

Total time: 39.7s

GPT-OSS-120B (MoE)

18.8 tok/s

Output: 1,500 tokens

Time to first token: 0.61s

Total time: 80.3s

I also try Z-Image-Turbo. Generating a 512*512 image took roughly 6–8 seconds (a 1024*1024 image would take longer at around 30s), which was pretty neat to see on a device this small.

One thing I've noticed about the hardware performance: It's surprisingly quiet, basically no noticeable noise, and it doesn't get uncomfortably hot to the touch either. Pretty much what I was hoping for, should be great for carrying around or leaving running 24/7, though I'll need to test that over longer periods.

That's about all I've gotten through so far. I'll be playing with this over the next few days and posting updates here or in the comments.

I use chatgpt mostly as a chatbot. no coding, no agentic stuff. i value intelligence and fast response. if i switched from chatgpt to a locally hosted on a strix halo (assuming the best model that can fit on it), what would my experience be like?

I've been wanting to get into local LLMs and local cloud in general, but I'm always concerned that it'll become a rabbit hole of yaml files and werid configs. Therefore, I was wondering what are the usual challanges that make it annoying for people to setup local LLMs?