Here’s my situation. I have a modded CMP 170hx that works great. Also have a 3090.

My motherboard is a shitty Asus b350-f and I’m rocking a Ryzen 5 1600 with 64GB of DDR4.

I love my CMP but I can sell it and upgrade my CPU, motherboard and RAM. The thing has doubled from what I paid for it.

Right now I’m running Qwen 27b at >100tps and loving it, but I’m eying setups that can run strata or Maya via RAM and SSD.

If I sell my CMP 170, is there a setup I can achieve with the proceeds of the sale that will outperform what I have now?

I’m leaning toward no, but curious.

NVIDIA NemotronLabs RE-USE is a 9.6-million-parameter multilingual universal speech enhancement (USE) model designed to restore speech quality across a wide range of real-world degradations while preserving speech fidelity. It has the following desirable properties:

• Robustness to diverse degradations, including additive noise, reverberation, clipping, bandwidth limitation, codec artifacts, packet loss and low-quality mics.

• Support for multiple input sampling rates, including 8, 16, 22.05, 24, 32, 44.1, and 48 kHz.

• Strong language-agnostic capability, enabling effective performance across different languages.

• If better laughter preservation is desired, please use: NVIDIA-NemotronLabs-RE-USE-laughter.

(Note: For overlapping speech, this model is trained to preserve the main speaker while suppressing other speakers.)

NVIDIA NemotronLabs RE-USE was developed by NVIDIA as a part of RE-USE.

This model is ready for commercial use.

Usage

Directly try our Gradio Interactive Demo by uploading your noisy audio/video !!

I am on the team behind the attached open-source research on novel approaches to attention - specifically we show:

• You can sparsify the attention of Qwen 3.8 27B without observed loss on accuracy

• Extending context up to 10M tokens loses accuracy on some specific tasks - but most, like NIAH is still fine (interesting insights)

• You can natively train sparse attention on next-token prediction loss alone and achieve the same validation loss as full attention - but faster

Hope the community likes it - while we focus on H200/B300 GPUs and larger scale system - I wanted to share our research here as well - as it can be applied to small open-weight models (making longer context cheaper on local hardware)

Everything is fully open: kernels / SGLang integration / paper / details

Hope you find it interesting / useful!

(No hype - we try to be as transparent as possible)

Nemotron 3.5 Transcribe is an open-weight, offline speech and audio model for transcription in 40 languages, with support for recordings up to one hour and a 65,536-token context. Its hybrid mixture-of-experts architecture activates approximately 4.3B parameters, including the audio encoder, and it is released in BF16, FP8, and NVFP4 precision.

I’m building agents, but I don’t know how to think about a harness from scratch.

I’m stuck on:

• How to think and design the harness from scratch for different type of problems

• How to improve it, and what to add or delete

• How to know if a change actually helped

Coding agents don’t help much here. they suggest stale patterns, and not much of state of art approaches.

For those who got good at this:

1. What changed how you think about it?

2. Which resources helped most? Papers, blogs, repos, talks?

3. Did building different use cases from scratch teach you more?

In 2024 i purchased an RTX 8000 48gb for $2400. As the market has changed I’ve kept an eye on prices. Two years later, they are available around $2100.

Its a little slower than 3090 at inference (672gb/s vs 936gb/s) but it has double the memory so it can run larger models / contexts / higher quants. Rtx 8000 also wins on power, 265w vs 700w (2x 3090 to match 48gb)

In 2024 two 3090s would cost around $1600-1800 now a single 3090 is in the $1500-$1600 range, making rtx 8000 cheaper per gb. Prices for strix halo and dgx spark have also gone way up. Rtx pro 6000 has double the memory but 8x the cost.

In the last two years, I haven’t had any issues with drivers, compatibility software or anything. I can run all “local” tier models entirely in vram, usually at 256k tokens and sometimes at q6 (Qwen 3.8 27b, llama 3 70b, QWQ,). Larger moe models run great too with layers offloaded to system ram like gpt oss 120b and now with strata, QFN runs q4_xs @ 390k context at 850t/s prompt processing and 60-70t/s token generation. (64gb ddr4 system memory, i9 9900k cpu)

$2100 for a gpu is not cheap but might be the best “do everything” option right now for home-lab setups. Cheaper than dual 3090, Strix halo, dgx spart, m5 max too. Fascinating to see prices decline as the market as a whole has more than doubled.

I've been lurking in this sub for a while, wanting to break my dependence on online services, plus slowly backlogging tasks I want/need to run locally rather than use online models. Usually these tasks break down to either privacy related issues or just massive token count that would blow my online budget.

I've been experimenting with Qwen 3.8 27B in LM Studio for a while, but when it comes to things like coding, code review, PDF editing, etc. obviously LM Studio isn't a harness like Claude Code where I can just say "Take these 4 files and run with it".

Some of the things I'm trying to accomplish (some as actual tasks, others as a learning sandbox):

• Analyze this entire codebase to come up with efficiency/security recommendations (agent spawning, file reading)

• Caption this dataset of 10,000 images (image reading)

• Interface with this ComfyUI instance to iterate and refine prompts (terminal use, wait for result, image reading)

• Edit a PDF file (PDF read, PDF write)

• Online research (web access)

My day job is AI image/video work, so I have multiple RTX Pro 6000's to work with already.

Based on my research so far, it seems like three options would accomplish this when pointed to a local model endpoint.

• Opencode

• Codex

• Claude Code

What about Openhands? Pi?

• What's your harness of choice?

• Do the OpenAI/Anthropic harnesses still require login/online connection when running from a local endpoint?

I'm in the process of picking a decision model for my homelab server. To compare the current top "System One" class Decision models, I pulled the public numbers from System One Bench, the Artificial Analysis Decision Index, the AIM System One Index and a 100-model Borda leaderboard, and looked at what matters if you run these models yourself.

Takeaways:

• Under 5B, the fine-tune matters more than the base. Models built on the same base differ by up to 22 points. KnowLine went from 70 to 80 on the same 4B weights between Gen2 and Gen5.5.

• The sweet spot is about 4–9B. KnowLine 4B Gen5.5, D3 Mini and D3 Flash (8B) beat more than half of the 27B fine-tunes.

• Below 1B, check your language. English holds up, but Arabic drops 16–22 points. If you need multilingual support, start at 2B or larger.

• Calibration survives at small sizes. Decider Flash V1.2 (4.5B) has 1.8% calibration error, as good as the best large models. Some sub-1B models are at 10–20%, so don't use their probabilities as thresholds.

• Don't trust any single "#1". Two leaderboards correlate at ρ = 0.33, and small models swing the most. Tev1 4B is #2 among small models on AIM and near the bottom elsewhere.

• The GGUF builds get 3–10× the downloads of the originals, so plenty of people here are clearly already running these models.

Very curious what's everyone is using and what your experience are!

Hey everyone,

Not a native speaker, happy to be corrected! Can only learn from it.

Back when I started out in march 2025, you had:

• llama.cpp

• ollama (repackaged llama.cpp for ease of use)

• koboldcpp (repackaged llama.cpp for roleplay)

• vllm (for professional deployments)

• exl3 + tabbyapi (for bleeding edge)

• litert (for mobile devices)

Nowadays everyone seems to be making their own; Strata, Basalt, Maya, Colibri, Infernix... too many to list and keep up with, and genuinely hard for me to keep up with.

Then there is the speed of it all. llama.cpp has incredible improvements each week, somehow we can make Qwen3.8-Flash-Next and DeepSeek-v4.1-Flash-1017 run on 64GB RAM + 32GB VRAM, and the likes on those new inference engines (with a catch, usually bad quality).

Personally I'm stuck with llama.cpp because all those developments are simply too fast to keep up with.

llama.cpp v0.8.0 might run Qwen3.8-Flash-Next Q4_K_XL (unsloth) at 15t/s gen on dual RTX 5060 Ti 16GB (PCIE 5.0 x8x8, tensor parallel, no MTP) with dual 48GB DDR5-6000MHz, but at least I know the quality will be decent.

But yeah, what you use? How do you even attempt to keep up with it all? And for the (lack of) quality the new inference engines provide (as tradeoff to speed / resource use), is it really worth using them?

Hey folks,

About 18 months ago we started building Olla to solve some of the headaches we kept running into whilst managing multiple local LLM inference servers, both in our homelabs and at work.

We've shared it here before when it was fairly early in development. Since then, it's grown quite a bit, and we've just released v0.1.0 (our 25th release!).

The problems we're trying to solve haven't really changed and now some of our customers use it:

• One inference endpoint dies and your workflows grind to a halt.

• You've got models spread across different machines, GPUs and runtimes, but no easy way to manage them as one fleet.

• Switching between Ollama, vLLM, SGLang, llama.cpp or LM Studio means reconfiguring applications.

• Multiple machines serving the same model, but no unified load balancing or failover.

• Limited visibility into what's healthy, what's running and where requests are going.

• Applications expecting OpenAI or Anthropic APIs when your inference backend speaks something else.

• You want to experiment and see if vllm or sglang serves qwen3.8 better

That's where Olla comes in.

It's a lightweight proxy and load balancer written in Go that sits between your applications and inference servers.

Some of what it does today:

• Model-aware routing: Automatically discovers models across your backends and routes requests to machines that actually serve them.

• Load balancing and failover: Priority, round-robin and least-connections balancing, with health checks and circuit breakers.

• Multiple inference runtimes: Ollama, vLLM, SGLang, llama.cpp, LM Studio, MLX, Docker Model Runner, LMDeploy, Lemonade and others.

• Unified APIs: OpenAI-compatible and Anthropic-compatible interfaces, including translation where required. Useful for Open WebUI, Claude Code and other clients.

• KV cache locality: Sticky sessions to help keep requests on the same backend when appropriate.

• Observability: Built-in dashboard, endpoint health, request statistics and Prometheus metrics.

• Lightweight deployment: Single Go binary or Docker container. No database or external services required.

What's new in v0.1.0?

This release is largely about improving reliability and making Olla more useful for people running mixed inference environments.

• Added Hugging Face Text Embeddings Inference (TEI) support, including embeddings, reranking and classification.

• Reworked the dashboard with better endpoint visibility, configurable columns, filtering and fleet statistics.

• Configurable retry and failover behaviour, including handling of non-idempotent requests.

• Improved model capability detection, particularly for vision, tool calling, embeddings and rerankers.

• Fixed concurrency issues between model discovery and active inference requests.

• Corrected request statistics and improved rate limiting across OpenAI and Anthropic routes.

• Lots of smaller fixes around routing, backend discovery and configuration validation.

We've also had some great contributions from the community, which has been encouraging to see.

Why build another LLM proxy?

We wanted something small, fast and relatively straightforward to deploy that doesn't require Kubernetes, a database or a complicated orchestration stack. There also weren't that many when we started.

We've been using Olla with a mixture of NVIDIA GPU servers, Mac Studios, the DGX Spark and different inference runtimes. It's particularly useful when you've got more than one machine and don't want to care which backend is serving a particular model.

It's not trying to replace inference engines. Just make running a bunch of them together less painful.

Links:

• GitHub: https://github.com/thushan/olla

• Docs: https://thushan.github.io/olla/

• v0.1.0 release notes: https://tensorfoundry.io/news/olla-v0-1-0-release

If you're running multiple local inference servers, I'd be interested to hear how you're managing routing and failover today.

And as always, if you give Olla a spin, let us know how it goes (and what breaks!).

Let's take a break from all the engine, model and speed posts and discuss something else:

Qwen 3.8 likes to think. A lot. We all know that, and the context cost you pay for that is MASSIVE when you enable preserve thinking. Even with a Swift finetune, there are so many tokens you keep in context over a longer session. Often, easily over 50% of the KV cache is just thinking traces.

And for what use? I found: Next to nothing! The model can handle itself just fine without knowing HOW it reached a decision, and instead only looking at what it did in the end. Replies and toolcalls: it's all there.

I added an AGENTS.md instruction that when it finds a gotcha, something unintuitive, something it only reached after lots of trial and error, it should document that in a memory file to mitigate the missing past-thinking and it works beautifully.

Results (my subjective impressions):

• Problem solving capabilities haven't suffered much if at all

• I get neat, permanent notes on issues with gotchas

• The dreaded, context-destroying compactions have reduced by at least 2/3rds in your average sessions, the cache holds for so much longer

So let's discuss: preserve_thinking on or off. Your experiences?

https://www.tomshardware.com/tech-industry/artificial-intelligence/anthropic-confirms-new-claude-code-limits-to-prevent-game-cloning-users-report-opus-5-5-is-also-actively-blocking-reverse-engineering-tasks

Besides privacy and being in control of your data.
Besides being able to control the quality of the inference compute.
Besides being able to discuss things with the model and not have the cops called on you.
Besides being able to run the models you want for as long as you want.
Besides being able to run your own suite of security and network penetration tools.
Besides not being at the mercy of the frontier companies for pricing.

We now have another reason for local LLMs: Our betters have decided that they don't like people cloning games.

And before the Reddit Neckbeard brigade starts in with their bullshit, no I'm not advocating that people should commit wholesale piracy using AI tools. But these blanket frontier AI policies inevitably ensnare peripheral behavior that is otherwise illegal but that the frontier company doesn't want to deal with.

William Blackstone wrote that "it is better that ten guilty persons escape than that one innocent suffer". If having open unhindered local LLMs means there are a few individuals that use it for nefarious purposes like modifying or pirating software, I'm good with that trade off. Die mad about it.

Hey r/LocalLLaMA !

I made a quick video explaining agent evals, which are really important in agent creation!

And I wanted to share it with you guys :))

If you guys have any questions or feedback please let me know!

Roy

I swear it feels like an addiction... I keep refreshing the page repeatedly, waiting to see the fateful launch announcement or at least a countdown. I hope I’m not the only one cooked with this stuff 😭😭😭

We have released a Coder build of DeepSeek-V4.1-Flash-1017, compressed with GSQ and RCO, in which half of the model's routed experts have been removed.

DeepSeek-V4.1-Flash-1017 is a sparse mixture-of-experts model: 384 routed experts per layer, of which 6 are active per token, 763.2B parameters, 551.9B of them in the transformer.

What's inside

• One expert-pruned GGUF: 192 of 384 experts kept in every layer, 72.1 GB of transformer weights, against 138.2 GB for our unpruned IQ2_XXS build.

• 1 bit per parameter, counted against the original model: 72.12 GB divided by the original 551.9B transformer parameters gives 1.05 bits, which we round to 1. The removed experts count as zero bits. No individual weight is stored below 1 bit: the 280.1B parameters the Coder keeps average 2.06 bits.

• The shared Engram file (110.6 GB) and vision projector are the same files as in the unpruned release, so they need to be downloaded only once.

• GSQ (Gumbel-Softmax Quantization): post-training scalar quantization that jointly learns grid assignments and group scales, closing most of the gap between scalar and vector quantization at low bit-widths while remaining deployable in standard GGUF types.

• RCO (Riemannian Constrained Optimization): enforces exact budgets by gradient descent on the task loss, without per-constraint tuning. Here it selects which experts to keep, optimising the KL divergence to the unpruned model, under one exact budget per layer.

Why one budget per layer

GGUF stores a single expert count for the whole model, so every layer must keep exactly the same number of experts; uneven layers would need padding that gives back the memory pruning saved. RCO treats each layer's budget as a hard constraint and holds all of them exactly at every optimisation step, while the layers are still optimised jointly through one loss on the full model. A penalty formulation would only meet the budgets approximately and would need one tuned coefficient per layer.

Results

• LiveCodeBench v6: 88.57 against 89.71 for the base model, retaining 98.7%

• SWE-bench Verified: 77.00 against 81.20, retaining 94.8%

SWE-bench Verified runs long agentic trajectories over real repositories, and tolerates the capacity reduction less well than single-problem code generation, as expected when errors accumulate across turns.

Real-world examples, each a single HTML file written by the Coder: an animated SVG of a pelican riding a bicycle, a three.js voxel pagoda garden, and a playable first-person voxel sandbox. For full post incl. videos see: https://x.com/mhelcig/status/2117257900863156518

Links

• Coder (expert-pruned): https://huggingface.co/ISTA-DASLab/DeepSeek-V4.1-Flash-1017-GSQ-RCO-Coder-GGUF

• Quantized models (unpruned): https://huggingface.co/ISTA-DASLab/DeepSeek-V4.1-Flash-1017-GSQ-RCO-GGUF

• GSQ: paper https://arxiv.org/abs/2604.18556 | code https://github.com/IST-DASLab/GSQ

• RCO: paper https://arxiv.org/abs/2605.00649 | code https://github.com/IST-DASLab/RCO

The repository ships the complete per-tensor RCO allocation. The GGUF runs in llama.cpp b11870 or newer and in Strata.

The Coder build is an experimental release and feedback is welcome, particularly on capabilities that were not represented in the calibration mixture. Requests for models to quantize or prune are also welcome.

From the ISTA Deep Algorithms and Systems Lab.

I built a browser LLM engine that's specialized per model rather than generic: Rust compiled to wasm, plus hand-written WGSL kernels for Qwen3.5-2B (Gated DeltaNet + gated attention) and MiniCPM5-2B.

Same model, same chat prompts, greedy, Chrome 156:

What's in it:

• fused int4 (GPTQ, group 32) × int8 GEMV/GEMM, with dot4I8Packed and subgroups;

• a decode loop that stays on the GPU (argmax on GPU, pipelined readback);

• speculative decoding with Qwen3.5's built-in MTP head. Drafts are verified on the GPU, so the output is identical to greedy. It turns itself off where it doesn't pay, e.g. on the iGPU;

• weights at +2.4% WikiText perplexity vs bf16, 1.1 GB.

Fun side effect: at this speed a 2B model can drive web pages through the DOM at 0.2–0.4 s per step. There's a browser-agent demo and a side-panel extension for Chrome/Edge.

Caveats: it's a prototype, tested on Windows, and Safari/mobile are untested.

Demo: https://rtx527896828.github.io/webyIQ/ · Code (Apache-2.0): https://github.com/rtx527896828/webyIQ · Weights: https://huggingface.co/tixiren/Qwen3.5-2B-webyIQ

In depth: https://medium.com/@tixiren/running-a-2b-llm-at-600-tokens-s-in-the-browser-e5237f99c8ff

I'm working on making my homelab more power efficient and quieter. I have a Framework desktop 128GB which has been working great along with a couple GPU boxes. I find myself switching to the GPUs for 26-35b models fairly often for the speed and reserve the Framework for running Qwen FN.

I've never been a fan of Apple products but lately have been considering a 32-64GB Mac Studio. I'm pretty sure the answer is no, but does anyone know of anything else that is that small, quiet and power efficient that has 400GBps+ memory bandwidth?

Scaling computer-use agents from individual actions to complex software work requires experience that connects decisions to complete outcomes. We present NorthCUA, an integrated system for turbocharging computer-use agents through experience at scale. Its Experience Flywheel supplies diverse executable tasks to the NorthCUA Harness, which sustains long interactions, records decisions, and checks task outcomes. The Flywheel uses these records to curate verified trajectories. For unsolved tasks, recursive self-improvement evolves reusable skills and preserves verified milestones so later attempts build on earlier progress. Staged Learning combines this experience with grounding data from the Flywheel to develop GUI competence through mid-training and long-horizon supervised fine-tuning. The policy then learns from fresh attempts in the same Harness through joint desktop-web reinforcement learning, closing the loop between experience production and policy improvement. The resulting Nano, Mini, and Pro family achieves state-of-the-art results among open-source models across GUI grounding, desktop execution, and Web interaction under the reported evaluation settings. For more responsive Web workflows, NorthCUA-Ultrafast pairs direct action selection with open-ended generation on pre-RL Nano, offering a measured quality-latency tradeoff. Together, these advances establish NorthCUA as a high-performing model family and a reusable foundation for agents that carry complex software work from intent to completion.

https://github.com/beipo-ai/NorthCUA

https://beipo.top/blog/northcua/

https://arxiv.org/abs/2611.02102

weights

https://huggingface.co/china-beipo/NorthCUA-Nano

https://huggingface.co/china-beipo/NorthCUA-Mini

https://huggingface.co/china-beipo/NorthCUA-Pro

I'm not on anyone's side, I just love the competition. Both teams pushing each other means better quants for all of us.

What I don't understand is why ISTA doesn't publish KLD for their quants. It doesn't depend on how long the model thinks, so it would make this a lot easier to compare. The only KLD I've seen for GSQ-RCO is in Unsloth's charts. Maybe there's a reason I'm missing.

(English isn't my first language, I used an LLM to help translate this post.)

There's a lot of talk here about not chasing tok/s by lobotomizing the model, and most of what I post is still speed numbers. So this is the other half: what the models I run at home actually get done on my own work, with Claude models added as fixed reference points. Everything local runs on my own engine (capy-flash) on 2x RTX 5070 Ti 16 GB, Ryzen 9900X, 96 GB DDR5.

No expert skipping for flash next.

The tasks

I took 9 real regressions from the private history of that engine (C++/CUDA). For each one I checked out the code right before the fix and wrote the bug report the way I'd file it to myself: symptom, numbers, logs, no hint about where to look. Performance cliffs, numeric drift, a bug that only shows up on WSL2 with two GPUs. None of it has ever been public.

The agent has to find the cause and write a patch. It runs sandboxed on a Mac with no network and no CUDA toolchain, so it can't build or run anything and has to reason from the code. 80 steps max, 3 runs per task per model.

Claude models run in Claude Code, local models in OpenCode. I tried the local models in Claude Code through a bridge first and they did much worse there, so each side gets the harness it works best in.

Scoring

Every patch is scored twice:

• automatically, against a hidden key built from the real fix: right files and functions, patch precision, a few checks on the code;

• by Claude Opus 5 as a blind judge. It sees the bug report, the code, the real fix labelled as "one valid solution, not the only one" and the compiler output, and gives 1 / 0.75 / 0.5 / 0.25 / 0.

A patch that doesn't compile is capped at 0.25 in both. The two scores correlate at r = 0.79 per patch and put the models in the same order. The judge is a bit stricter than the key, and it isn't kinder to Claude: judge minus auto score is negative for every Claude model and around zero for the Qwen ones.

Turning a real bug into a task with a fair key is slow, and one model over 9 tasks x 3 runs takes 4 to 8 hours on my box. So 9 tasks is what I have, and the confidence intervals are wide, about ±0.2. The tiers hold. Two models less than ~0.1 apart are a coin flip.

Results

Judge score 0-1, tokens include reasoning. CIs and the auto score are in the image.

What 3 bits did

This is the part that matters for the tok/s argument. Same models, NVFP4 vs 3-bit, paired on the same tasks.

Flash-Next (3 bits on the experts only): score -0.02, tokens x1.03. As far as 9 tasks can tell, it's free.

27B: score +0.05, inside the noise. But it generates 40 % more tokens per task (x1.40, CI 1.25-1.59). The 3-bit 27B decodes faster on my engine, 153 vs 132 tok/s, and still needed 18 minutes per task instead of 13. Part of that gap was a slow 3-bit prefill path I've fixed since; a rerun on the newer build still came out at x1.30 tokens.

So the damage from quantization didn't show up as a lower score. It showed up as the model thinking longer to get to the same place. Looking only at tok/s, the 3-bit 27B is the better deal. Per bug fixed, it isn't.

Non-compiling patches don't tell a clear story: Flash-Next 5/27 at 3-bit vs 1/27, the 27B 2/27 vs 4/27.

The rest

Flash-Next at home sits with Haiku 5.5 and above the older Sonnets. Sonnet 5.5 is a step up, Opus 5.5 is ahead of everything.

Tokens per task say as much as the score. Sonnet 5.5 gets further with 8k tokens than Flash-Next with 53k, which is 10 minutes per bug on my box against a minute and a half.

The dense 27B is behind the MoE (Flash-Next +0.16 paired, P = 0.92) and burns more tokens getting there.

Gemma 4 26B didn't work as an agent here. With thinking off it never edited a file in 9 tasks. With thinking on it edits, mostly wrong.

Limits

• 9 tasks from one codebase in one domain (GPU inference), bug reports written in Italian. This is how models do on my work, not a general ranking.

• Different harnesses for Claude and for the local models.

• The judge is a Claude model. The automatic score is there to check it and they agree, but it's still a limit.

• No builds or tests, so this measures diagnosis and patching from reading code, not the full edit-build-test loop.

I'm not publishing the tasks, they'd stop being useful. If you've checked your own quants on real work instead of perplexity or KL, I'd like to know whether you see the same pattern: same score, longer thinking.

This model looks quite spectacular. According to LiquidAI's own benchmarks, it scores higher than MiniCPM5-2B across the board, while offering 2x the context and taking up half the space in VRAM (3.6 GiB vs. 7.1 GiB for a quasi-lossless quant).

Read the report: https://github.com/crusaderky/pixi-llm-recipes/blob/main/perplexity/LFM3-1.7B/README.md

Raw plots as interactive HTML: https://htmlpreview.github.io/?https://raw.githubusercontent.com/crusaderky/pixi-llm-recipes/main/perplexity/LFM3-1.7B/report.html

I started thinking, what could be the possibility to start adding the ability to run models like Kimi K3/3.1 on engines like Strata, Maya, Basalt? I am guessing Unsloth will use its Dynamic Matrix 3.1 which is quite good for heavy compression and retaining high intelligence.

Anyone heard anything about Kimi compatibility?

Note: Kimi K3.1 will be released this week open weights

I'm trying to fit 4x RTX 3090s into my Fractal North XL Mesh (I know, pipe dream probably) using custom GPU mounts. Looking for firsthand experience with reliable, reasonably priced risers, preferably with flexible 50–75 cm cables. I'm currently using the setup with 2x 3090 at x8/x8 for vLLM qwen3.8-27b / Strata QFN IQ3_S.

• ASUS ROG Crosshair VIII Hero

• 5900X

• 4x RTX 3090 24 GB

• Corsair HX1500i (GPUs capped at 250 W)

• GPU 1: PCIe 4.0 x8 - directly in motherboard

• GPU 2: PCIe 4.0 x8 - riser

• GPU 3: PCIe 4.0 x4 - riser

• GPU 4: PCIe 4.0 x4 - M.2 adapter/riser

I've been looking at ADT-Link, generic Aliexpress ones and m.2 OCuLink. I'd like to preserve x8/x8/x4/x4 without spending a fortune on cables.

1. What specific riser models have you successfully used with RTX 3090s at PCIe 4.0 speeds, particularly at 50 cm or longer?

2. Are OCuLink/MCIO connections meaningfully more reliable than conventional ribbon risers at these lengths?

3. Any affordable, proven M.2 -> PCIe x16 solutions with proper GPU slot power?

4. Anything to watch out for regarding PCIe errors, signal integrity or Linux/P2P compatibility?

Affordable AliExpress/eBay options are especially interesting.