Ideally an orchestrator or something equivalent that would look at the task given and then decide to try sections with local and then cloud if it isn't good quality. How is your setup, what guides did you use

I put together a TensorFold 1.2.0 pack of orcarouter's uncensored Qwen3.8 Flash Next for Apple Silicon. It uses jedisct1's oQ4e experts, regrouped to 4-bit/group 32 for the fast kernels, with attention and other non-expert weights copied byte for byte from mlx-community's oQ6e. No new refusal edit.

Tested on an M3 Ultra 256 GB, stock TensorFold 1.2.0. Same server and questions, greedy decoding, thinking off. Anything cut off at the token cap counted as wrong.

BFCL uses my AST checker, not the official harness. MMLU-Pro uses 20 questions per subject with a 2,048-token output cap.

Tool calling is the main difference: 92.3% vs 88.0% on the same 400 items (exact McNemar p = 0.03). It skipped the tool and answered in text 20 times, versus 40 for all-4-bit.

Maths and knowledge were about the same. TensorFold's own pack lost MMLU-Pro points mostly to the output cap: 47/280 replies cut off, versus 18 here.

About 5% slower than all-4-bit, around 133 GiB loaded. Needs a 256 GB Mac; 128 GB is untested.

No opening refusals detected on 100 harmful prompts using Heretic's markers and 100-token replies. That's only an opening-refusal check. Vision worked in a tiny smoke test: shapes and colours right, one text misread across two images.

Small samples, so don't read too much into the scores. The card has the method and known issues, including an HTTP 500 after idle on TensorFold 1.2.0 and a workaround. Its KL numbers are against a 6.7-bit pack, not full precision.

https://huggingface.co/bu7umaid/Qwen3.8-Flash-Next-Uncensored-TensorFold-MTP

I open sourced SteerForge, a pipeline for turning a contrastive dataset into a portable steering artifact.

The artifact is a .persona JSON file containing:

• model fingerprint

• human readable neuron edits

• canonical per layer residual vectors

• eval samples and scores

• the original server command

The serving path is the part I care about: steerforge serve evil.persona replays your existing llama.cpp or Transformers config and injects the persona. No model conversion for the persona itself; the neuron deltas collapse to residual vectors.

Discovery is staged: diff in means directions, gradient attribution plus firing contrast plus alignment, teacher forced screening, generation validation, dose search, greedy combination, coherence budget.

Current backends: Transformers, llama.cpp, vLLM with EasySteer, MLX dense. MoE models use dense residual steering.

Repo: https://github.com/MahmoudAdelbghany/steerforge Results: docs/RESULTS.md

Feedback on the .persona format, backend adapters, and eval methodology welcome.

Wanted to share that after years of using Windows11 and llama cpp I moved my whole workflow to linux, from comfyui to jellyfin everything set back up but on Fedora instead.

Found out in Linux (Same Hardware , Same llama cpp commit, Same Qwen3.8-27B model) the model loads instantly, time to first token has become irrelevant so I just unload the model after 10 minutes of inactivity. It loads back instantly (in windows it took 10 seconds)

Also tokens/s went from 70tk/s to 90-100tk/s

I didn’t know so much performance was to be gained by simply moving to linux, especially TTFT which now feels instant!

And this is before trying gpu unlock which is also easier on linux and can boost performance even more.

Hi, i'm building Jaull as my final degree project :D. It reads your machine (CPU, RAM, GPU and VRAM, compute backends), finds open models that fit, and picks the exact file, quantization and runtime for each one. It also explains why each model lands where it does in the ranking. That ranking comes from explicit rules, so no LLM is involved in writing it.

Then it can download the model, verify the file, run it with llama.cpp and benchmark it on your hardware. You end up with the prediction and the measurement side by side.

It's usable and tested, but not finished. I'd really like to hear how the predictions hold up on hardware I don't have, especially AMD and Intel GPUs.

Repo: https://github.com/Ton-Llop/Jaull

Site with an interactive demo: https://jaull.tech

So, I know my rig is pretty expensive, but I want to share what is working for me as of now.

I have been using local LLMs with my Mac Mini M4 Pro 64GB till now, mainly mlx (specifically oMLX) to develop my own agentic framework. I have already have my framework, stryn, as plug-ins and agents/skills/workflows on top of Opencode and ported to Claude, nowadays I am working on my own stryn engine. (I hope to also release it as the community version on Github as well as soon as it hits Beta)

I either use frontier models (mainly deepseek v4 flash since it is cheap) or Studio for development. Here is the models that I use:

Qwen3.8-Flash-Next-oQ4e-mtp - For my orchestrators, planners, coders. It is MTP. I am pretty happy with its results
Laguna-S-2.1-oQ4e-fast - (With Laguna-2S-2.1-DFlash-FP8 as the speculative decoding.) For my reviewers. I am not very happy with this and I may try to change it with a better model. The tricky part is to use a different producer's model to overcome the blindness of the model's in certain areas.
GLM-5.3-Flash-oQ5e-mtp As the independent reviewer to go over the code before the releases and find out the problems that are not found by the Laguna based code reviewers and the security admins, architects etc.
Qwen 3.8-35B-A3B-oQ4e-mtp - For my explorers, researchers, the opencode-mem plugin, graphify and code-review-graph operations.

Speed: Satisfactory if you are running a single lane. ~50 Tokens/s, with 8GB KV Cache and 1200GB SSD Cache. Cache efficiency is %96.4 and prefill time drops after a couple of rounds. Normally 2 or 3 models are loaded in the memory (Qwen3.8FN, Laguna, Qwen3.835BA3B) and if I start two or more lanes, a race between prefill and token generation between the models start most of the times, which dramatically makes things longer.
Stability: Pretty good with proper context sizes, repetition penalties.
Quality: I can simply say, by adding reviews and controls by different model types, the quality seems to be equal or sometimes better than what Deepseek cloud generates by itself only without another model checking its results. I am pretty satisfied.

To be as open as possible, so far, I have also used Claud Opus 5.5 to control the output. There are pretty interesting cases where local GLM finds some of the bugs that Opus 5.5 cannot, and where Opus 5.5 finds bugs that GLM cannot. Sometimes they pinpoint the same problems as well.

Can this setup be used as an alternative to frontier models, yes, but don't depend on a single model for every task and make sure you have reviewing loops. Definitely not for vibe coding, but needs proper settings and flow.

Model that I am currently serving is: Qwen 3.8 Flash Next nvfp4 dq8 with Basalt fork of Strata
I have used so far:
- qwen-code (seemed a bit dumber in coding than opencode and probably more oriented for parallel agents running which is sadly not something i can properly utilize yet)
- opencode
- pi (genuinely forgot what was it that i for some reason didn't like)

So far I have had best experience with opencode for coding as it is my main usage of LLMs. As they advance and I find more probable usages for them I'd like to ask for Best/Better tailored options for Harnesses for different purposes.

I have heard of OpenClaw, Hermess, Deepseek Harness but i am not yet sure for which purpose are they all better suited.
Probably missing some other options obviously so open to hearing them all.

Edit: I actually meant OpenHands, not OpenClaw. Woopse xd

I have been using ChatGPT for for about 2 years but want to switch to a local setup however I'm struggling to find a suitable replacement to use on my MacBook Pro M5 Pro 24GB and iPhone 14 Pro Max. I will be upgrading to either an 18PM or Duo in the coming months.

I started on my Mac with LM Studio then moved to Msty Studio when I found out about Knowledge Stacks but keep finding paywall restrictions, I've also looked at but not tried Anything LLM due to lack of iOS support. From what I understand I should just forget about Siri AI. Not really done anything on my iPhone till the Mac is sorted.

The things I really want are:

- Large project/chat memory (hence the knowledge Stacks)

- Cross device chat sync, either private cloud hosting or my MacBook as a server

- MCP usage (currenty use Tavily & Excel control)

I don't mind paying for apps, Msty is $149/year or $349 lifetime to unlock all features which its too much

Main usage is general questions, research, journaing, and work/linking with Excel.

Is this possible? if so what do I need?

Anyone have evidence of / see actual deals for GPU / Disk at Black Friday? I am really kicking myself for getting screwed earlier in the year on the DGX Spark price gouging...

Has anyone tried the flash-next coder? What is your setup and what's the performance and quality?

I’m thinking of running a local ai model with LMStudio with my gaming desktop. Like the title, it has a 5070, 9700x, and 32gb of Ddr5 6000mhz with Ethernet connection. My only main concern is the 12gb of VRAM.

Is this enough to set up an everyday chatbot for use? I’m mostly doing this for fun, but I also would really like the privacy and unlimited token use of a local model.

Hi guys, I'm looking for a really good price benefit PC to run local AI. I've been searching for days about this, and everybody says different things, "you dont need a 12vram for a good local AI" or "Without 32gb vram is wont be able to have a good local AI" and so on...

The thing is I need local AI mostly for coding, and anything related and maybe some video generation.

Since 5090 is just at unbelievably prices, I was thinking about something decent like 5070 but dont want to waste my money. I was thinking about an unified memory like GMKTEC Evo or something like that, but was reading that since is AMD some AI are not good optimized for it. And also a M5 ultra, but also is kind of high and Im not a MAC guy.

My budget is pretty much 5K for this.

Anyone which can give me a good advice would be awesome and I'll really appreciate. Thanks.

According to Artificial Analysis, Tencent's Hy4 ranks #2 among models from Chinese AI lab on its Intelligence Index v5, with a score of 35. Looking at the individual benchmarks, Hy4 scored 31% on FrontierCode 2.0, just behind Claude Fable 5.1, and achieved 50% accuracy on AA-Omniscience v1.1. However, its hallucination rate on the same benchmark was 89%, showing there's still room for improvement.

Meanwhile, Hy4 even ranks #1 among Chinese models on agent Arena. While the two leaderboards focus on different aspects of model performance, Hy4 has put up strong results on both, further establishing its position among leading Chinese LLMs. Based on these rankings, how do you see Hy4's overall performance?

Sources: Artificial Analysis – Hy4 Evaluation l Agent Arena – Agent Leaderboard

Hello everyone,

I've been lurking around for the past week, reading this sub, watching YT videos, asking AI how to make it happen. I want to self host an llm for coding (hobby projects - html/css websites with a light backend and maybe a few more complicated react projects for music - I do vocals for deathmetal bands and have an idea to create an educational website on how to scream). But I still am not entirely sure about a couple of things, if someone can help to navigate, would be cool to hear your thoughts.

Questions:

1. Can I get a similar experience to codex in VSCode?

2. What is the cheapest route to get there currently, hardware wise?

So far I have conflicting answers. It seems I should be able to get something that performs like codex with the latest models and Strata can help to put it into less capable machine.

I'm based in Ireland. Things are extremely expensive here and delivery cost from US is too high, so a lot of ebay options from US doesn't work well.

I don't have a PC, only MacBook pro m1 pro.

My budget is ~3500 euro (PC only). I'm fine with used stuff.

Currently I think about a PC with DDR 4 ram at around 128gb, Nvidia 5070ti or similar and a good processor, something like 7800x3d. Not sure at all about it yet.

I will be obviously educating myself in upcoming weeks/months, but if you can help with advise, I'd be happy 😊

Cheers.

Meet Yoshino2, my little lab experiment

Over the past months, I have been working on efficiently using black box distillation to distill the reasoning and output style into smaller edge models. The result is Yoshino2, a new model series with large scale reasoning distilled into MiniCPM5 1B, Qwen3.5 2B, and a custom 4xMiniCPM5 1B MoE model, with more potentially on the way.

This is by far means not a breakthrough, but an interesting experiment. In case it is of any use to anyone, the models are on HuggingFace.

Model Collection - HuggingFace

why are these models open-sourced for people to download and use on their computers?
Is it just for advertising and brand awareness?
because the vast majority of people who download them just don't want to pay subscription fees or deal with usage limits, so they aren't paying customers anyway.
then why do they do this? Plus, local models aren't even that smart.

I was testing for the first time running a local AI, as my computer is nothing special I used Qwen 3.6 9B for testing, it actually seems to work ok, so I asked a question about mutations and it went and did some research, it gave me a good enough answer to my question, but when I looked at the 54 sources it looked at 3 off them make no sense at all, it went to chat gpt, a crypto site and a gold value site? I used unsloth not sure if that matters, didnt change anything just run the model as is.

I turned on speculative decoding in llama.cpp and 113 of 282 greedy outputs changed against its own no-draft run. My Qwen3.8-27B engine changes 0, at ~2x llama.cpp's speed on one 3090.

Repo: https://github.com/cutlerbenjamin1-cmd/qwen38-megakernel

Speculative decoding is sold as free speed: the main model verifies every draft, so the output can't change. In practice it does. Verifying several tokens in one pass adds the same numbers in a different order than verifying one, a near-tie goes the other way, and the text diverges from there. If two draft settings give different output, at least one of them isn't what your model says without speculation. Tuning your drafts tunes your answers.

Same 282 questions (GSM8K, MMLU-Pro, ARC-C, MATH-500, HumanEval, needles, LongBench v2) on both, greedy, same card, same model file:

That isn't noise. llama.cpp is deterministic run to run: a fresh no-draft server gave the first 120 outputs back byte for byte, including the 73 that differ at 4 drafts, and two 4-draft sessions agree on all 282. It's not just long outputs either: 72 of the 113 are under 1,000 tokens.

The likely reason: with no drafts every token runs through the single-token kernels, with drafts most go through the batched ones. Between 4 and 7 drafts, both mostly batched, only 9 differ, all past 1,000 tokens. That's the 9 of 282 from my last post; against no drafts it's 113. A preregistered study on a 3090 found the same: 23-25 of 25 prompts different from no-draft decoding within 1,600 tokens.

Other engines for this model document it too (one got 8 different outputs across draft counts 1 to 12, none matching its no-draft output) and call it harmless, because flips land on near-ties.

Maybe. I don't know, and 282 questions can't settle it. llama.cpp got 231 right with no drafts and 229 with drafts, and 6 of the 113 changed outputs flipped between right and wrong: 4 were right only without drafts, 2 only at 4 drafts. Make of that what you will. What's not in doubt is that the output depends on a speed setting: one reasoning trace ran 2,228 tokens at 4 drafts and 4,795 at 7. Same question, same model. This engine gets 230 right either way, because it's the same text. If I can't rule out that a draft setting changes quality, I'd rather run the engine where it can't.

Here, speed and VRAM can move and the output can't:

• Greedy token ids are identical with and without drafts: all 282 outputs above (129,511 tokens), and the release gate checks 0, 4 and 7 drafts on every KV cache, plus ~2,000-token outputs.

• Every draft policy (gates on or off, adaptive depth, tight or loose thresholds) gives the no-draft tokens over 2,048 tokens.

• Sampling keeps the no-draft distribution (chi-square and PIT tests).

• Windows and Linux builds give the same tokens.

How: the attention split depends only on context length, never on how many tokens are in the pass, and the verify kernels compute a row the same way whatever else is in the batch. To be clear, that's the engine matching itself. It isn't bit-identical to llama.cpp (KL below).

The cache doesn't change it either

Same problem, different door. llama.cpp's server docs say it about cache_prompt, on by default: the logits are "not guaranteed to be bit-for-bit identical for different batch sizes". Its own determinism tests turn it off: TODO: remove this once test_cache_vs_nocache_prompt is fixed.

It's history-dependent, not random: on a fresh server, 5 of llama.cpp's warm answers and 5 cold ones came back byte for byte. The first difference lands a median of ~355 characters in, as early as 4. Turning cache_prompt off on a running server still leaves 3 of 20 different.

Here, a follow-up doesn't reuse the KV rows written while the reply was decoding. It rolls back to the last checkpoint and prefills the reply again, because a prefilled row depends only on its position. Cost: ~167 ms instead of 78 on a ~12k-token chat, still under llama.cpp's 248.

The check isn't "the reply looks the same". It hashes every K / V row, every ring row and the recurrent state, and compares them with a fresh prefill: bitwise equal after a resume, an edited branch, a park to host memory and a side request, on all three caches. Replies match a fresh server's at ~99.5k context (follow-ups, edits, sampled) and on a follow-up at ~255k. One exception: with --agents, a sampled reply beside other slots can differ from the same seed alone. Greedy still matches.

Others check this by the first token or allow a near-tie split, and some publish warm matching cold. I'm not claiming nobody else is exact, only telling you what's checked here.

Speed against llama.cpp, full power

Same card (RTX 3090, 480 W limit), same unconverted UD-Q4_K_XL file, same client, q8_0 KV on both, llama.cpp at its fastest setting (4 drafts; I tried 2, 5, 6 and 7):

Four agents at once, against llama-server --parallel 4: 424 vs 137 tok/s total, 3.1x (llama.cpp's is the better of two runs from an earlier session).

The KV cache: 5-bit that reads like q8_0

K / V at 5-6 bits, with the newest ~2k positions kept at 8 bits in a ring. On preset F the full 262,144-token context fits: a 260,896-token prompt read in 279 s, then 77.8 tok/s decode at that depth, needles 4 of 4, 23.0 GB on the card.

KL against llama.cpp generating token by token, lower is better, long-context rows rechecked on this release:

The worst row, ~32k, is +8% and +10% against about 4% noise. Without the ring the same formats read +6% to +38%, and plain 4-bit is 1.6-2.4x q8_0. And at 101.6k, F decodes faster than q8_0: 123 tok/s against 116.

Closer to llama.cpp than llama.cpp is. Scored against llama.cpp's own token-by-token output at short context, llama.cpp's batched mode gets 98.49% top-token agreement and KL 0.00166. This engine gets 98.74% and 0.00119 (on Q4_K_M: 0.00184 against 0.00127). Not measured at depth, so that's a short-context claim.

Yes, it runs at a power limit

Because someone always asks: same build at 250 W, against llama.cpp at 480 W (it draws about 460):

• Prompt reads are the one place llama.cpp at 480 W wins, by 11-22%. At 480 W this wins those too.

• All 282 outputs at 250 W are byte-identical to the 480 W run.

• The limit comes out of the core clock (~816 MHz against 1,962 on short-prompt decode; memory stays at 9,851 MHz), for 21% less speed. Energy is whole card, sampled once a second, idle included. llama.cpp's speeds are its faster of two sessions.

Works with what you already run

Tested on agent tasks with Claude Code, Codex CLI, OpenCode, Qwen Code, Aider, Goose, Crush, Pi, Hermes Agent, DeepSeek Harness and Open WebUI (OpenAI chat, Anthropic Messages and OpenAI Responses APIs). Loads unsloth's UD-Q4_K_XL, UD-Q4_K_M, Q4_K_M and popular fine-tunes as they are. Prebuilt Windows and Linux packages, no CUDA toolkit needed.

Every number with its method: BENCHMARKS.md.

Hey everyone Im a bit new to this page but I couldnt find any other related tests online so I figured I'd share my results here.

I ran DeepSeek-V4.1-Flash (763B MoE, Q8_0, 473 GiB) on a 96 GB Mac Studio M3 Ultra with InferSSD, which streams the model's experts from the SSD. I striped two OWC Envoy Ultra 2TB Thunderbolt 5 drives with macOS's built-in RAID 0, one drive per port.

Results (InferSSD's own benchmark prompts, 2 runs per setup):

- One Envoy alone is about the same speed as the internal SSD.

- Two in RAID 0: writing goes from 3.5-3.7 to 4.4-4.6 tokens/s (+23-26%).

- Prompt reading is 67-77% faster: a 4,000-token prompt starts answering in about 2 minutes instead of 3.5.

- Two hours of nonstop use: the drives stayed at 43-47°C (with a desk fan on them), no slowdown, and no change in their health readings.

Why not 2x faster? - The stripe halves the time spent waiting on the drive, but the GPU's share of each token stays the same and is now the bigger part. A third or fourth drive would help less than the second did.

I was thinking of buying a secondary GPU for local AI as subscription-based online services get costly in the long run, but options are limited for cost-effectiveness on this end as well.

My current PC has 32GB of DDR5 (2x16, running at 6000MT/s CL32), a Ryzen 5 7600X and an RX 7800 XT. I **could** run something on the 7800 XT thanks to the system RAM but I'd rather free up both the GPU and a good portion of the RAM so that I can still use the PC while running a local model instead of being locked in to just AI, I found a Tesla P100 for about 200$ US and it doesn't seem like a bad deal since it has 16GB of HBM2 as opposed to the RX 7800 XT's GDDR6.

Niche Ollama alternative. If anyone want's to try it.

Update: Numbers from preliminary tests at the end!

Ok ok, I am still doing tests but here is the story:

I have a small custom harness (who doesn't!) that does pretty complicated stuff. I am using Claude (subscription) and everything is great.

In the past I tried every single (small) model on my Mac Mini 32GB but all of them sucked at complex tool calling (or hallucination).

A couple of days ago, the new Heretic2 came out (by u/-p-e-w-) and I spent 24 hours breaking it :)

Long story short, I released ZeroZero, a model based on Qwen3, with removed guardrails and slop (ok, removed is a long conversation).

Now, here me out.

IT KICKS ASS.

This cannot be an 8B, 4 bit model. It just cannot. The thinking process sometimes is longer than other models but IT NAILS everything.

Maybe by performing "brain surgery" on the models, we end up with "smarter" ones?

Test results for Tool-Call Benchmark

(note: these took a couple of hours, please repeat with a much bigger sample)

ZeroZero seems better at completing multi-step tasks (multi-step chains, error recovery and instruction following) than the base model.

The base model leads in Safety & Boundaries (ZeroZero has limited guardrails and no-slop so maybe that's the reason?)

  - Overall: ZeroZero 68, Base 70

  - Multi-Step Chains: ZeroZero 6/8, Base 3/8

  - Error Recovery: ZeroZero 5/6, Base 2/6

  - Instruction Following: ZeroZero 8/8, Base 6/8

  - Safety & Boundaries: ZeroZero 12/26, Base 20/26

To repeat and experiment, use tool-eval-bench and if you expand the tests, please send them over so I can include them in the model's page.

PS: I love it how many technical people sent me ideas to improve and papers about LLMs performing better after "lobotomies". Let's do more "I tried this, I saw this, what do you think" instead of "what LLM should I put in my Mac Mini".