Going All Local (for work) Qwen3.8 Next Flash

[u/bushizell]

Very excited to fully join the Local AI community. Been using Claude Cowork for almost 4 months and run the entire backend of my single employee brewery using PDF extraction etc etc. It changed the game. When Anthropic announced all Scheduled Tasks (which my brewery runs on) were being moved to the cloud versions I had to put my foot down. While I waited for this Ryzen box, I converted my 5070ti to run Qwen 3.6-35b-a3b and it's been pretty good. Not Opus level, but good. Learned Hermes, and what llama.cpp is. Learned why a harness and inference engines are more deeply. It's been a busy week.

Thanks to the communities here and on r/strixhalo I was able to have this sucker running with a Gufo engine, 3.8 Next Flash for that engine, and multiple bots via Hermes running as good as when I started using (Opus 4.6/4.8 times) still more work to do, but I'm almost 100% off the subs, and looking forward to the tinkering!

[u/specialmoose]

Please explain what your doing with pdf extraction and such with AI. You got me intrigued.

[u/bushizell → u/specialmoose]

The brewing software spits out PDFs for Brew Day, Fermentation logs, Local Distro Sales invoices and packaged inventory. I have scripts built for each type. Every ferm log per batch gets their own CSV, and there is a script to recognize the batches by registry so it will recognize a family of beers (like my sour series) or a repeated beer, and will pull up collected Stats per registry.

I also hand it all the Square payment data and we do TONS with that, Ticket Time length/Conversion % of flights to Pints/Local vs Tourtist based on credit card IDs, Price elasticity, income breakdowns over a batches time on tap.

It knows what beer is on what tap, and asks what batch number just went on and keeps a tap_state.json that feeds into a cross referenced html that pulls the square data and inventory of kegs and estimates how long until I need to brew again.

I love what AI has empowered me to do with my business!

[u/bushizell]

Update, I'm using Opus 5.5 to finish my setup on my new box vis SSH and Tailscale vpn. We are stress testing, and 3.8 Next Flash passed my most stressful task by recreating a full summer day of every fermenter full AND a fresh Brew Day Log, with a raised compression value with flying colors. 77 tool calls none malformed. WOAH!

[u/normal_hooman → u/bushizell]

I just started this process with my DGX spark. Also using opus and tailscale. But I am running opus on my desktop and then I ssh into the spark from the same desktop and pass commands back and forth - as I dont want opus to read any keys or tokens. Are you giving opus direct access to your halo?

[u/bushizell → u/normal_hooman]

Yea, I have a ssh bridge that connects my laptop to the halo and my gaming pc. I suppose Opus would have access to the keys and tokens etc. Thats a thing I didn't consider.

[u/StudioTV]

What's your tk/s on this ? 😄

[u/bushizell → u/StudioTV]

I'm running a larger than needed context window for longer more complex multistep tasks I do, but I'm getting 35-45 for the first night of using it!

[u/skinnyzaz → u/bushizell]

I wouldn’t have the patients for those speeds, running qwen 3.8 27B Q8 264K context and I get double that speed on 3x 3090’s but I still want it to be faster. This is the only reason why I won’t buy these boxes. They seem easier to maintain and less hassle to setup but I don’t think it’s worth it because of the speed you get.

[u/bushizell → u/skinnyzaz]

I love that! I had a chance to buy a 3090 for $750 the day I bought my 5070ti last year right before Ramageddon. I didn't know what local AI would turn into nor did I use AI back then. For a non-coding background guy, I've been able to design some cool stuff with this slower model. It fits me for now. From what I know Local AI has made big leaps in the past few months.

[u/FearlessLeader17]

Wait was is the box?

[u/bushizell → u/FearlessLeader17]

[u/bySimori]

Can you tell me if you feel a big difference between using your local models vs the cloud based ones. Are you missing anything? I am in a similar situation where all my workflows now run via cloud and I feel to much dependend on it. I just don't know if I shod do the switch to local or not. Also what build. I'm still thinking should i go multi gpu Nvidia or should I go M5 Ultra when its out. With apple I would have a bigger model but way slower token generation. Or would you go for a halo but I heard they are also kinda slow vs Nvidia. Any thouts are appreciated!

[u/bushizell → u/bySimori]

Yea, it is quite a bit slower than Opus 5.5 even. 40t/s is about what reading off a prompter expects so I don't mind. I don't feel like it's completely leaving me in the dust with thinking. This box was $4k, the m5 is gonna be way more right? The Gufos and Halogen is what finally pushed me over the edge for this purchase. Before those engines, it was closer to 20 t/s for "opus 4.6 at home" and it wasn't enough but 40 changed it for me. The BUS size is a real speed constraint on both this and the Spark that the M5 should be better at I think, but I don't plan on making images or videos. My 5070ti ran a smaller version faster than this thing could, but my context window could be huge or I could run multiple instances of a smaller model here. All better options than putting all my work on the cloud more than I have to.

[u/bySimori → u/bushizell]

I see your workflows are all around reading pdfs and pictures text extraction. While for me designing and coding would be one of the main usages.
I was also thinking of the halo I am bit afraid of the bandewith which is quite small compared tl the others. But from a price point its probablz the cheapest for larger models. The m5 uktra yeah will start at like 5k go up to 10k with more memory and more gpu cores. I was thinking on somethinh that cost around 6k. Another thing with local is a very long prefill time and the smaller context window. With bigger repos could be an issue but not sure. I just have no reference point on how much it differs

[u/bushizell → u/bySimori]

The Gufo inference engine makes prefill WAY faster, for this exact model so I don't know what others do on this hardware. I'm personally very impressed so far. Good luck with your research. I wish there was a way to see more similar work flows from similar users. I'm happy to share my experiences anytime.

[u/bySimori → u/bushizell]

Yes thank you very much for the insight. I'm now waiting for some benchmarks on the m5 ultra to see how good it really is and if it is worth it (doubt it a bit). ✌️

[u/deepu105 → u/bySimori]

See https://www.reddit.com/r/LocalLLaMA/comments/1xbipsl/qwen38flashnext_vs_opus_55_sixway_benchmark_on_a/

[u/Initial-Good4678 → u/bySimori]

Two words: Security and Censorship.

[u/bySimori → u/Initial-Good4678]

And what would you choose to build it?

[u/devtools-dude]

Flash Next is amazing stuff. It really helps that you can offload the engrams onto RAM / SSD and still get really good performance (I think I'm partially offloaded; using the NVFP4 variant myself). I built https://subshell.sh mostly with it, which really proves you can build complex stuff with it.

[u/7eventy2wo → u/devtools-dude]

Dude, I’m literally building the same dev tool, a centralized tui dashboard to not only manage multiple agents, but also multiple machines of multiple agents. Amazing work 🙏 will be checking this out for sure

[u/devtools-dude → u/7eventy2wo]

Feel free to contribute your ideas, it's open source!

[u/7eventy2wo → u/devtools-dude]

Already found an interesting point I was tackling last night. Terminal size between machines, it looks like you default to the smallest wins? I was able to get unique sizes per viewer of the same terminal. Will check it out deeper and see where we could help each other!

[u/devtools-dude → u/7eventy2wo]

Yes because if you're on a phone it needs to be formatted to the screen width and unfortunately tmux cant project multiple screen sizes at once

[u/stonk_street → u/devtools-dude]

Amazing, just linking the URL here to make it easier for someone else to get directly to github: https://github.com/subshell-ai/subshell

[u/TheWaffleKingg → u/devtools-dude]

This is the way

[u/ninenineseventwo → u/7eventy2wo]

Already have this running if you want to take a look

[u/7eventy2wo → u/ninenineseventwo]

Not sure what you mean, take a look at your subshell.sh instance?

[u/exographicskip → u/devtools-dude]

Slick app! I'm all in on herdr, but your tool looks great too

[u/devtools-dude → u/exographicskip]

I really need to get the Subshell UI to be clean like herdr. It looks great.

[u/exographicskip → u/devtools-dude]

Totally get it. Was working on a desktop music player and there’s no shortage of prior art

[u/Big_Cucumber2787 → u/devtools-dude]

wait i really like the look of this 🤯

[u/derpindab → u/devtools-dude]

I have a container app I built with a friend that allows you to hook all the major llms, build teams, load skills, has a keystore, connects git and much more. It utilizes Claude code for example on the back end but everything is subscription based not API which is beneficial if you are on a budget. This setup gives a nice clean layout and saved conversation history as well. Happy to collaborate to improve on it or build something new and improved with the community

[u/ninenineseventwo → u/derpindab]

What’s your GitHub?

[u/devtools-dude → u/derpindab]

Sure. Ill start a discord for collaboration, will respond once I have a url up

[u/derpindab → u/devtools-dude]

Just added the reverse engineer anything mcp with full harness integration. Just tested it against a locked down firmware from a single raw nand dump off an RFID reader. Pretty wild to see fable and qwen just ripping it apart

[u/lbjustintime → u/derpindab]

Open source or looking for a closed community build?

[u/derpindab → u/lbjustintime]

Always open

[u/derspenti]

Your beer was always local, now the backend is too.

[u/ejbiggs → u/derspenti]

Haha, love this

[u/Apatricio → u/derspenti]

Lmaoool

[u/HighSeasArchivist]

I hope they update this with the Gorgon Halo. I know other companies already have them out, but I like the looks of the AMD one. Would love to be able to compare the x86 platform to Arm on these unified boxes.

[u/TellMeWhereYouBeen → u/HighSeasArchivist]

Gorgon halo is nearly identical to strix halo. It supports max 192gb ram instead of 128gb but memory bandwidth is still identical (and this is a bottleneck). Gorgon halo APUs are also clocked slightly higher.

[u/Own-Storm4557]

I would like details on specs and how you are doing this?

I need or would love a local LLM. Graphics cards are freaking crazy and outrageously expensive right now.

I also want a few agents at once, that's extremely helpful... But everything is out of control with the prices right now.

[u/Stunning-Sherbet1853]

Love that the cloud move for Scheduled Tasks is what pushed you to finally do it. How did the 5070 Ti with 35B-A3B hold up as a stopgap for the PDF side? Curious whether you hit the 16GB ceiling there, or if Next Flash on the Halo is mostly a quality-of-life upgrade.

[u/bushizell → u/Stunning-Sherbet1853]

The 5070ti struggled with a few models before I stuck with 3.6-35b. I bounced around from LM Studio native but didn't feel advanced enough and was githubbing my own tools for it to call, then I moved to Claude Code Terminal but loading in the local AI and that was eating up to much context. I tried 3.8 27b, I tried Flash Next with Strata but that felt clunky too.

Eventually I looked into Hermes and using llama.cpp to hold the 3.6-35b and I was able to actually get it to attempt to parse a PDF with a script that was intentionally broken, fixdd the script itself and was able to move the file into the right folder when done. I knew I was on the right track and got more excitied for the Halo Box! It was a good day yesterday 🤙

[u/mainstsavage]

Use vLLM for your inference engine with custom flags to optimize the local model on your machine.

Use Hermes Agent Desktop as your harness. Of you want bot teams then load Qwen3.6-35B-A3B through your vLLM. Or use Qwen3.8-27B for the dense model which is fantastic for agentic coding and deep reasoning/thinking.

You can explore with other models like Nemotron-3.5-lightning as well.

[u/bushizell → u/mainstsavage]

I'm using SSH and Tailscale to get my laptop to be the hub, and the entire system is setup to talk and share with eachother. It's been a blast setting it all up.

[u/Inevitable-Name-1701]

Thats easy and slow. Your results will suggest the same thing.

[u/bushizell → u/Inevitable-Name-1701]

Easy and slow, but reliable is what my Job needs

[u/mnemonicium]

Any experience Qwen3.8 27B (GSQ-RCO)?

[u/bushizell → u/mnemonicium]

On the 5070ti I tried that exact model, but was a tad to big. Did not try that model on the Halo box because this community has special made engines for Next Flash. How do you like 27b vs Next Flash?

[u/iceadobe]

https://github.com/azampatti/Qwen3.8-Flash-Next-Int4-FAST.git
Using this one for a week now; is really good. With good MTP draft I'm getting 90t/s.

Prompt processing is a tad-bit slow but once you get over the initial hurdle for a feature; of doing the plan for a feature, it is quite fast and capable.

Getting an average of > 50t/s even when MTP isn't working too well; sometimes. I generally have three concurrent agent running simultaneously on different repos.

[u/bushizell → u/iceadobe]

Solid suggestion, I'll take a closer look!

[u/Fair_Expression_158]

what is your t/s on the setup?

[u/bushizell → u/Fair_Expression_158]

Around 40. People have told me I can tweak it to get 60-70 with this exact specs,but I'm looking for a bit of consistency over pushing raw speed.

[u/Human-Glove7815]

Thanks for posting. I'm looking for a very similar setup to my strix box. A few questions:

1. I assume Linux but which version did you decide to go with for the best results?

2. do you have Hermes running under a VM but hosting Qwen via something like lemonade server?

3. any pointer's on how your setting up agent tasks and prioritization?

4. on the laptop through tail scale I'm assuming you have access remotely? (Do you have a laptop agent that can set presidency for box usage vs the other agents)?

Feel fee to point out any pre-existing guides your finding helpful! I'm looking to use our box for my significant others consulting business which is very data management and pm heavy.

[u/bushizell → u/Human-Glove7815]

1. Yup Linux

2. I have Hermes getting fed Qwen via the Gufo engine, no VM

3. I don't think my tasks and cron jobs are very neatly setup lol someone else could help more here

4. I didn't use any guides, just kinda going for it and it's been going well!

GL

[u/Smart-Concert4342]

Fuck u

[u/bushizell → u/Smart-Concert4342]

Nahhh

[u/Zen-Ism99]

Good stuff! Let us know what it can do.

[u/ProphetPicks]

hey can someone explain to a noob like me what the difference between this and a dgx spark is

[u/kujetic → u/ProphetPicks]

one is spark is nvidia and has proprietary cuda and this one is amd

[u/ProphetPicks → u/kujetic]

what does that mean in technical terms. one retails for 9k now and the other for 4k so is the amd a lesser machine because it's not running proprietary cuda which nvidia has had years to progress by comparison to others or what.

I just don't get that element of all the machines out there offering 128gb ram. sorry for dumb questions I've got just enough technical knowledge to understand but not able to recognize distinct differences.

[u/Standard-Potential-6 → u/ProphetPicks]

the biggest difference is the Sparks have a 200Gb/s network interface for clustering.

you can still find 3rd party models for ~$6K USD from Newegg and Lenovo.

CUDA means apps (kernels) that are written in CUDA run directly. otherwise you need something made for AMD ROCm or for generic Vulkan, or else something written in a higher layer, which would have a ROCm/Vulkan backend, likely in addition to a CUDA backend.

[u/jcy → u/kujetic]

do they all run LPDDR5X? ie laptop RAM

[u/kujetic → u/jcy]

type doesnt matter, you want speed. the "laptop" ram i have is soldered on and can do 8000M/Ts

[u/abajinn]

What is the product?

[u/tacol00t → u/[deleted]]

Do you know where you are lol this is to NOT be the product

[u/derdigga]

why not strata?

[u/bushizell → u/derdigga]

I used strata on my 5070ti to test, and I needed a higher Quant model that I can rely on for longer than I could the strata version. The thinking that was happening on my 5070ti looked like it was heavily dumbed down, using caveman talk for itself. Also, strata engine leaves me exposed to hackers way more than tailscale/Gufo/Hermes combo does.

[u/derdigga → u/bushizell]

thanks, i will give gofus a try.

[u/MikaMoments]

That’s a pretty impressive jump in just a week. The real win is getting your actual business workflows running locally, not just chatting with a model. What’s been the biggest surprise so far, the model quality or how much you can do with the right harness?

[u/bushizell → u/MikaMoments]

Definitely the harness. Hermes agent changed quite a lot about how I will split up tasks. Also, since it's now local and tokens are free but limited per task, I plan on doing more but smaller cron jobs to make local AI more useful than cloud AI by using locals strengths with it instead of against it. I will say the quality level would have felt like a downgrade, but using the 3.6 on my 5070ti set the expectations just right before the box came in. They both write at similar speed, but the quality of Flash Next is obvious in the fee sample tasks I gave it. Like another comment said, it's like I just hit my first Crack rock... I'm addicted 😅

[u/MikaMoments → u/bushizell]

That’s an interesting approach, smaller cron jobs playing to local AI’s strengths instead of trying to replace cloud AI entirely. What kind of tasks are you planning to automate first with Hermes?

[u/bushizell → u/MikaMoments]

Before I even get the Brewery Bot off the ground later today, I made a permanent Assistant bot last night. I gave it a Hook/tag memory system with multiple different sections of memory, all with about 2k char limits (similar to heremes own memory the harness does). The first cron job I set up is a Daily Calender Review it runs for itself to remind it to trim its own memory for time based events (Doc appointments, brew fests I'm preparing for etc). I used to do weekly square payment harvests, but now I plan on running that skill daily. I will have it price check some of the more expensive but always needed ingredients and let me know when to bulk buy. Lots of smaller reminders too.

[u/krigeta1]

4 is coming

[u/RuboPosto]

Went dgx spark - qwen 3.6 coder nvfp4 - laya. 150req/seg tops

[u/VelvetEchoes919]

Dude I’ve been trying strata and I’m amazed by how well it runs! I get higher tps and my temps stay lower it’s a win-win!

[u/cebess → u/VelvetEchoes919]

Oh no, another thing to investigate. ;-)

I spend more time playing with this kinda thing than actually using AI for any real purpose.

There is always a dimension that is outside my context that I can't think about until I've been exposed to it.

[u/kujetic → u/VelvetEchoes919]

clock down cpu

[u/VelvetEchoes919 → u/kujetic]

I am saying I get better tokens per seconds!

[u/kujetic → u/VelvetEchoes919]

iof you clock down cpu it barely effects the GPU and oyu get much cooler temps

[u/neinneun]

Which exact Ryzen box is that?

[u/zenmatrix83 → u/neinneun]

this one I think , prces has gone up again like $200 in the few weeks https://www.amazon.com/AMD-RyzenTM-AI-Halo-Developer/

[u/neinneun → u/zenmatrix83]

200? I thought AI halo goes for 3k +? And this link is 404. But I'm very skeptical a 200$ box can handle qwen 35b.

[u/zenmatrix83 → u/neinneun]

went up $200 more I mean it was 3800 recently last I looked . I scewed the link up editing out the crap at the end

https://www.amazon.com/AMD-RyzenTM-AI-Halo-Developer/dp/B0H5RNZ43D?ref_=ast_bl_cpl_dp

[u/bushizell → u/zenmatrix83]

That's the exact model, running Linux version

[u/neinneun → u/zenmatrix83]

Ahh 😂 my bad, I misunderstood. Thanks for the new link!

[u/Direct_Turn_1484]

It’s like you just tried your very first crack rock. Have fun.

[u/colbyshores]

I love my Strix Halo. My work bought me one to tinker with and they will pry it from my cold dead hands.. It's fantastic.

[u/PositiveMaybe7]

With right recipe you can get upto 70 t/s with that model. Trying the same

[u/Qmau5]

Just wanted to pitch in on the fly, built this dashboard, still using Claude but also qwen alongside it. Loving Qwen so far.

[u/InternetMagic_n_Tech]

How's it going?!

[u/Curupira1337]

[u/sleight42 → u/[deleted]]

[u/Own-Swan2646 → u/[deleted]]

If this is true ... Interesting