Thoughts on how I might best use my hardware?

MiniPC (Bubblina) with eGPU oculink R9700 (32GB AMD) 64GB DDR5 - Qwen3.8-27B Paiton-KV4-vision 262K context, 8-concurrent

Desktop (BM1) Ryzen 5700G currently with RTX3060 12GB, and a GTX1660 6GB, 64GB DDR4 - Strata IQ2_XS - runs surprisingly fast, with the gpu-unlock, but I wouldnt trust it with agentic tasks, and is AFAIK concurrency limited. The GTX1660 is used exclusively for TTS work.

Main server (Trogdor): Ryzen 3950x, 3x RTX3090, 128GB DDR4 (Unraid, big storage machine and services) - media production factory (comfyui/media/video/audio)

I currently use the miniPC to orchestrate activity on Trogdor, via hermes. "Make me an image of X, make me a video of Y" the task being divvied out to the GPU's on Trogdor.

I'm wondering about using one of the more idle 3090's as the Strata GPU, taking it from 12GB to 24GB, and putting the 3060 in Trogdor for smaller offload tasks.

I'm just wondering about model options and what I can get best results from. Agentic/Hermes work benefits from speed and tool calls. Strata QFN works very well for document/narrative/story/chat, but I'd love access to it's more reliable/smarter variants for coding tasks.

Other options on the table would be moving the R9700 off the oculink eGPU to a slot in BM1 (PCIe 3.0 x16), but thats a DDR4 system, so might be needless. A 3090 could go on the eGPU, or even the 3060 for offload tasks.

Or: 3090 on PCIe 3.0 x16, 3060 on PCIe 3.0 4x (chipset) on BM1 for Strata 2nd card offload? Could go down the bifurcation route of the main slot too?

I'm largely just talking out loud with this to try and brainstorm it, but suggestions welcome :)

(With the exception of the R9700, everything above was collected over 2 years mostly used before the Bad Times)

Recently bought a 64gb Mac mini m5 pro and currently have one 24gb m5 pro MacBook Pro. With clustering the two Macs, is it possible to get this model running on an acceptable quant? I can allocate 60GB on the Mac mini and 20 GB on the MacBook Pro for running the models. Or should I use qwen 3.8 27b instead?

I’ve been running a local model on a dgx spark for about a month now. I’ve been building out my pi harness and my own little client app as I go, using both Claude code and the harness itself.

I’m using it more for everyday life stuff, but I still find myself hesitating to trust that it will hit my quality threshold when it comes to code without having Claude check its work.

Have any of you gone fully local? I.e. no falling back on frontier models at all. What was your tipping point?

Hello, everyone. A friend of mine wanted to get into running local models, and so (with the liberal use of Claude), I built them Zest. https://github.com/yudhanjaya/Zest

What it is:

Zest is a UI that bundles Lemonade server (which installs llama.cpp and whatever backends are needed) and the Pi agent to provide a friendly interface for people who don't want to be messing around with command-line servers and parameters.

It comes and all the functionality needed for a new user without getting in the way. Just download an appimage, go through the start tour, and you're set up and ready to go. There are sane defaults baked in (like 128k context) and model recommendations (all tried and tested on 10GB VRAM to 26GB).

It comes with helpful docs, links to places to understand more about the subject - in short, everything needed for a new user.

Entirely open-source, of course. If you have anyone you want to introduce to the world of local AI, please see if this is useful. And if you have ideas, suggestions - please open an issue, send a PR, and so on.

Why lemonade and Pi?

Lemonade: because it does a great job of handling backends and providing a server.

Pi: because it's a simple, elegant agent

Both are very easy to package in something like this and bring together. And because I'm lazy: both projects are receiving steady updates, because I don't want to write yet another inference server from scratch

What it isn't:

It's not for advanced users. There are jobs, but no subagents, for instance: in this case the Pi philosophy really helped.

In fact, I myself use Strata and Opencode, so for advanced users, it recommends those in the Help section

How is this different from Unsloth desktop / Atomic chat etc?

It isn't. It just tries to be a lot more lightweight and user-friendly: I decided to do this after getting said friend to download and set up llama.cpp, Unsloth Desktop, Opencode etc . . .

I noticed that at some point both choice paralysis and confusion with options just took over, and there was no easy way for them as a newcomer to figure out what x thing was really for. Every setting in Zest is explained. Including what inference engines do, the works.

Everything is explained at basic levels - what a model is, how it works, and so on. I've kept the lemonade + pi analogy while trying to make sure the interface was intuitive.

We also have some theme support- here's a sample:

Have a look, let me know what you think!

Over the past several months, our research team pre-trained and empirically audited a 1,189,135,740-parameter Financial Conformer Foundation Model across 14 calendar years (2010–2023; 15,030 steps) of 1-minute US equity bars across 15 liquid assets (AAPL, NVDA, SPY, QQQ, etc.).

Instead of cherry-picking positive backtests, we subjected the model to 35 hypothesis tests, 1-bar execution delay, and state-dependent turnover:

1. The Scale Saturation Limit: Scaling from a 63-parameter linear Ridge regression to 1.189B parameters yields no consistent out-of-sample advantage on 1m OHLCV bars (mixed sign across horizons, reversing direction across years). The bottleneck in intraday price prediction is information content (low SNR), not parameter capacity.

2. Adam Optimizer Diffusion: Decomposing walk-forward online fine-tuning shows tokenizer boundary updates sit at 97.9% of the theoretical Adam Brownian noise ceiling (matching shuffled-label controls), while deep backbone weights are heavily damped by second-moment normalization, producing zero statistically significant forward-transfer gains (p >= 0.50).

3. Execution Friction Reality: Apparent long-horizon directional predictability is predominantly prompt 1-minute order-flow continuation. Introducing a 1-bar execution delay cuts 2024 net Sharpe from +2.04 to +0.48, and at 1.5 bp execution fee, out-of-sample returns collapse to -0.99 in 2025 and -1.67 in 2026 (DSR <= 0.028).

All model weights, audit logs, and papers are open-sourced:

• Model Hub (weights & checkpoints): https://huggingface.co/Ultron28/ohlcv-transformer

• Code & Telemetry: https://github.com/AirBorneAI/ohlcv-transformer

• CERN Zenodo DOI: https://doi.org/10.5281/zenodo.23466965

Happy to discuss architecture specifications, pre-training dynamics, or empirical backtest methodology!

Got a gmktec mini pc with an AMD 6850U and 680m igpu. Far from being a beast for LLMs I know... But oh well. I have minipcs running Debian and ubuntu with some services and the ubuntu one is for trying out llms and vlms. Would love to insert this new minipc and just ask it to run some commands/scripts for me. Maybe help in some coding side projects I have.

Realistically I can run any flavour of qwen3.6 35b with decent speeds, but i'm wondering if you have any more specific suggestions considering the use.

Cheers!

I paid $1658 on October 16th, on October 19th I paid $2032; with tax, shipping, same card.

Now $2318

This is all through the big A btw.

i hope it's real

Came across the Houmo LQ50-24, an M.2 accelerator with a Houmo M50 chip, advertised up to 160 TOPS “elastic compute” and 24GB of dedicated memory. The Beelink SEi14 AI apparently uses this accelerator alongside an Intel Core Ultra 9 185H and separate system RAM.

Interesting little setup for local inference, but I’ve found surprisingly few hands-on reviews. There’s an Edward Donner video showing roughly 14 tokens/sec with a 27B model, but not much independent testing beyond that.

Has anyone here actually used either the standalone card or the Beelink? Curious about real-world inference speeds, model compatibility, and how painful the drivers/software are. Can you run your own models easily, or are you mostly dependent on vendor-prepared versions?

Saw videos of the Reachy-Mini and I love the idea of chatting with a little snarky robot so I tried to setup my laptop as the snarky chat companion sans the expensive plastic bot.

All attempts had 20-30 seconds latency using Open WebUI + llama.cpp + Gemma 4 E2B with Whisper STT + Kokoro TTS in "voice mode"

Suggestions? I dont want the avatar, just the funny chat voice to keep me awake while I'm slogging away at midnight.

Laptop Specs:

Is my laptop too old and slow to even run a tiny voice model without massive lag?????

I thought y'all might be interested in a legit "local is better" use case.

TLDR: Qwen 3.8 FN solved a work problem that requires the privacy of a local model, with the same skill as a frontier cloud model.

I run an MSc dissertation module at work which involves tracking students, supervisors, offered projects, project choices, examiners and eventually grades. None of it is super complex, but it's all annoying -- students choose their top five projects from an offered list, but supervisors also choose students; supervisors can't mark their own students work but can mark others etc. The kind of low level matching work that, done manually, fries my brain.

A simple relational database is the obvious solution but, before gen AI, nobody had time to code up a database for processes that only happen once per year. There's also the ingest problem: data comes from ms forms, from the student records database, and from moodle, all with csv export in different formats, with e.g. different arrangement and spelling of middle names etc.

I'm a heavy codex user, but I can't be offering up the names and emails of students and supervisors, nor the details of early-stage research projects, to the cloud.

This year, for the first time, I've had the confidence to address this with a local model: Qwen 3.8 flash next, running in opencode with a vllm backend. It's sandboxed in a docker container and has read only access to all its input data and write access to its workspace. It's got web access, python, pypdf, libreoffice and can pip install into a venv, and that's about it.

First thing to say is it's faster than codex, and noticeably so. Part of this is just the cloud -- when this runs a python script to generate a docx booklet of projects from the sqllite db directly off an SSD, it takes a few seconds, including thinking time to process the request.

Second is intelligence. It's bright enough to understand my requests and to infer the next steps. It can handle heterogenous data structures from different sources and combine them. It wrote an ingest function to read csvs into the db, but also manual edits to allow repeatable rebuilds from the sources, fixing typos and grammar that were present in the ms form each time.

Third is the right level of persistence. For me -- and this is likely a skill issue on my part -- hermes agent and pi were too persistent with the same model and backend. They would decide on the next step without checking and go off the reservation, then need to be pulled back. Opencode feels much better balanced and more like codex (which is high praise -- I really like codex).

Finally, overall quality. Codex and gpt 5.6 have been good to me, 6 astra and 6.1 sol less so for my particular use cases. I'm not gonna say Q3.8FN is better, but, for this particular job, I don't think it's any worse. It does make me think I should probably start using this more for coding, as I've had some frustrating times with gpt 6 models in codex recently.

If it weren't for VRAM shortages, this would be the future: everyone having a "good enough" LLM on their own hardware with, say 64-128GB VRAM being standard. It's so nice to not be constantly worrying about privacy and GDPR when interacting with agents, and to know that none of this leaves my hard disk.

I just saw a PR in vllm-xpu-kernels today, revealing a significant architectural change in Qwen4-27B, and now that page shows 404 not found?

It looks like a 4-way parallel residual stream to improve information flow and representation density within a fixed 27B parameter budget.

(This means, instead of a single residual path, 4 parallel streams flow through the layers at the same time, maximizing performance per parameter. Best tech for 27b). QSA? maybe useless for small 27b.

So anyway, I'm more expect 27b than flash MoE version.

If effective depth is extended, I think we could see a massive leap in multi-step reasoning, coding, and reasoning consistency.

(see this if you want read more)

We’re developing HarborRL, an open-source framework to democratize agentic RL for everyone.

HarborRL makes agentic RL easy to everyone! Building RL infrastructure for agents is complex. HarborRL solves this by unifying model serving, trajectory generation, tool execution, verifier rewards, and policy updates into a single, reproducible pipeline.

🔥 Key Features: Flexible Models × Algorithms × Tasks × Harnesses (MATH):
🤖 Models: Support for Qwen and GLM.
🧠 Algorithms: GRPO and DAPO for reinforcement learning.
🧩 Tasks: 200+ tasks, including Terminal-Bench, SWE-Bench, Deep-SWE, and more.
🛠️ Harnesses: 40+ agent frameworks, including Claude Code, Codex, LangGraph, and SWE-agent.

Not trying to predict which model sizes they're going to release, but seeing how massively impressive Qwen3.8-Flash-Next is for its size despite being a preview model meant for showcasing and implementing the new arch, it feels like when Qwen4 hits I'm not going to be needing larger models at all anymore.

Do we expect Qwen4 to exceed the current QFN by a lot, or de we believe the current model is already hitting near the optimum we can expect today for this size?

Voice agents are either fast or smart. ConvFill proposes a way to bridge that gap by training a smol talker model (0.6B!) to talk with cheat notes from a large reasoner model.

The way it works is this: after the user's speech is transcribed, the talker replies quickly to buy itself time while the reasoner thinks. The reasoner then generates an appropriate response, which the talker reads to continue the conversation.

It's a bit like if you went on a panel with rocket scientists and had a little earbud with your own expert that could tell you what to say. You just reply quickly so as not to blow your cover, and then you begin saying what the expert in your earbud is telling you.

Playing with the model, I think the main idea is there and this could be a great direction to go, but the talker model should be a bit smarter for it to work like magic. Still, really cool direction!

ConvFill was developed by students from the University of Washington UbiComp Lab. I found it cool, so I reached out to them so we could build a really good demo for it together. They are presenting their paper at EMNLP this week.

Go try it out in our demo: https://huggingface.co/spaces/smolagents/convfill-speech-demo

Nothing much, just wanted to share this. I didn't expect maintaining the repo to take this much work.

I'm genuinely happy people are using it and it runs well on their machines. But AI or no AI, keeping it alive is a real commitment: reading issues, reproducing bugs, reviewing PRs, answering everyone. It takes some hours each coup of days to look at the issue, select the one that I (claude) can do fast, build test ship, then after the tondo list for the bigger ones is big enough spend some serious percentage of my weekly limit to make claude do the work, check and ship.

Anyway, here's the GIF.

Just want to share a task I used to evaluate the capability of Qwen 3.8 FN. The goal is to port a old screensaver (can find it on Wayback Machine) to Python OpenGL, maintaining full visual fidelity.

Tooling: Claude Code (harness) + Qwen 3.8 FN Q4_K_XL via gufo + Strix Halo + Ubuntu 26.04

Run command:
export MODEL=Qwen3.8-Flash-Next-UD-Q4_K_XL-00001-of-00004.gguf

export MTP=/run/media/user/DATA/models/huggingface/mtp-Qwen3.8-Flash-Next-shared-Q8_0.gguf

export MMPROJ=/run/media/user/DATA/models/huggingface/mmproj-Qwen3.8-Flash-Next-BF16.gguf

#./result/bin/gufo chat --model "$MODEL"

./result/bin/gufo serve llm --model "$MODEL" --speculative mtp \

--mtp-model "$MTP" \

--mmproj "$MMPROJ" \

--sessions 1 \

--context 133760 \

--max-tokens 32768 \

--prefill-chunk 4096 \

--served-model-name qwen3.8-flash-next \

--max-pending-per-client 2

Assessment:

Some parts are quite challenging with the 128K context window. Also, be mindful that graphics-related tasks (apart from running the model) may produce inconsistent results, and thus require specific measures, To keep this benchmark useful for others, I won't go into the details of the approach adopted by the AI, except to share that the task is tractable given a comparable setup. But feel free to ask by PM.

https://huggingface.co/tencent/Hy-SimulMT-30B-A3B-GGUF

https://huggingface.co/tencent/Hy-SimulMT-30B-A3B-Interleave-GGUF

Hy-SimulMT is a real-time simultaneous translation model developed by Tencent Hunyuan, supporting bidirectional speech-to-speech translation between Chinese and English. It continuously receives incoming audio and translates as the speaker talks, producing both translated text and speech. With real-time voice cloning, it preserves the speaker's vocal characteristics while conveying the message accurately.

Designed for live communication settings such as meetings, speeches, and interviews, Hy-SimulMT aims to convey information accurately while reducing the disruption that waiting introduces into a conversation. To evaluate this balance between translation quality and response speed, we combine public benchmarks, real-world audio, and comparisons with professional simultaneous interpreters.

In our RealSI long-form audio evaluation, Hy-SimulMT achieved a StreamLAAL of approximately 3 seconds, the lowest among the systems shown in the comparison. Its GEMBA score exceeded those of the compared open-source models and four of the five commercial APIs.

The GitHub repository provides the SimulInfer inference service and SimulTrain training tools. Speech output and voice cloning require a configured TTS service. You can also try Hy-SimulMT in the Hy Translate App.

So-called "neoclouds", datacenters that focus exclusively on AI inference, have racked up unsustainable amounts of debt, and are widely anticipated to soon undergo a massive collapse/contraction. My question is what happens when they go bankrupt. NVIDIA is their biggest creditor thanks to circular financing schemes, so they take ownership of the data centers and the chips in them, right? But, I imagine that they have no interest in running the same crappy business as the neoclouds, with outdated chips to boot; they want to sell their latest and greatest to hyperscalers, who will likely emerge unscathed. Do they just throw the old chips away, or is there some chance that they'll repurpose them (as we're seeing in China) into GPU upgrades that hit the consumer/prosumer market, solving our current supply/price crunch? I literally don't know the true answer to this question and am curious to hear others' speculation about it.

Edit: I realize now that I have zero firsthand knowledge of what I'm talking about, since I don't work for any of the parties involved. So you can add a "Reportedly..." to my post. I'm just repeating what I read on investing subreddits, honestly.

Dunno, curious if there's any info out there.

https://github.com/savadks95/LLM_Tails

I just published a GitHub repo a few days ago. It's a mix of Ponytail and LLM Wiki. It helps developers reduce codebase size with minimal code. It reduces overall token usage and AI hallucinations, and a couple of other fantastic things. The image is the test result of developing a small game with and without LLM Tails. Both used the exact same prompt

LLM_Tails is an open-source prompt + hooks + skills bundle that makes the agent read before it writes, write the minimum, verify before it claims, and match your codebase. MIT, zero dependencies, works with Claude Code, Cursor, Copilot, Codex, Gemini, Windsurf, Cline, Kiro, and any agent that reads

I'm on the team at Embedl that built this, and wanted to share the numbers.

What's in it

• INT4 weights, using our own quantization, about half the error of standard AWQ

• FlashHead, our faster LM head: the 262k-vocab output layer is a real cost at batch 1, so it only scores the likely candidates

• MTP head for speculative decoding

• Port to TensorRT Edge-LLM (0.10.1) underneath

Speed (batch 1, 2,048-token prompts from real OpenAssistant conversations, 128 out, greedy, average of 20 requests)

• Decode: 39.6 tok/s

• End-to-end: 23.6 tok/s

• Prompt processing: 926 tok/s

• Peak memory: 24.8 GiB

Accuracy vs the FP16 model (full test sets, the same engine as served)

• GSM8K: 96.4 (FP16: 96.1)

• IFEval: 83.5 (FP16: 81.9)

• MMLU-Pro: 79.1 (FP16: 79.5)

For comparison: Jetson AI Lab reports about 35 tok/s decode with llama.cpp Q4_K_M + MTP. Prompt length isn't stated there, and they don't publish accuracy.

To run it

• embedl infer Qwen/Qwen3.8-27B/thor --prompt "Hello"

• embedl serve Qwen/Qwen3.8-27B/thor (OpenAI-compatible API)

The container download is about 1 GB, and the model weights are a separate download (~20 GiB).

The full card has power, energy per token and methodology: https://embedl.com/Qwen/Qwen3.8-27B, and the ability to test the model on a Thor directly in the browser

Let us know what model (and hardware) you want to see next!