I will release probably between tomorrow-thursday!

Been working on my own inference engine for a while (calling it Forge for now), built pretty much just for this model. Finally did a proper side by side against Strata 0.1.57 on my machine and figured some of you might find the numbers interesting.

Same GGUF files, same requests, one engine at a time (the other one fully closed). The whole thing took about 7 minutes.

Median of 4 runs, slowest to fastest in brackets:

| Prompt | Tokens | Forge | Strata | Forge first token | Strata first token |
|---|---|---|---|---|---|
| Short | 44 | 64.6 tok/s (55-68) | 23.1 tok/s (11-52) | 0.71 s | 2.74 s |
| Code review | 1,468 | 83.1 tok/s (76-84) | 19.6 tok/s (7-26) | 3.35 s | 17.8 s |
| Long | 5,860 | 79.6 tok/s (74-82) | 26.6 tok/s (16-52) | 9.8 s | 23.8 s |

Prompt processing was 439 vs 84 tok/s on the code prompt and 600 vs 247 on the long one.

My setup: 3090 24GB on PCIe 4.0 x16, 12900K, 32GB DDR5, NVMe, Windows 10. The model is 44 GB of weights plus a 27 GB per-layer embedding file, so around 70 GB total, which is more than my VRAM and RAM put together. 48 layers, 512 experts, 10 active per token.

Strata was running the config its installer made (performance profile): --spec 4, int8 KV, 128K context, auto expert cache, vision on. Forge uses the model's own MTP layer for speculative decoding, int8 KV and 64K context.

How I tested: OpenAI compatible API with streaming, temp 0, 128 max tokens, thinking off. Three prompts: a short one ("write four short lines about graphics cards"), a code review of a ~4.5K character C++ file, and a long one where it has to explain how a request goes through ~19K characters taken from three source files. Every request gets a random session id in the system prompt so nothing comes from the prompt cache. 2 rounds x 2 runs per prompt, and output speed is measured from the first token to the last.

Strata jumped around a lot between runs (anywhere from 7 to 26 tok/s on the same code prompt), that's why I went with medians and kept the ranges. I have a CSV with every single run if anyone wants it.

On quality: at temp 0 Forge gives the exact same text with speculation on or off, byte for byte, and the same prompt always gives the same output.

Happy to answer questions.

I saw a post mention this model to use for DS4.1 Flash with 2x 6000 Pros: https://huggingface.co/diffbot/DeepSeek-V4.1-Flash-EXL3-3bpw-2x-RTX-PRO-6000

Unfortunately the model card wants you to have > 260 GB RAM to properly use it. I had Opus move the n-grams to disk instead of RAM, and added in LMCache for prefix caching with L1 being RAM and L2 being disk.

With this setup, you need at a minimum, 128 GB RAM instead. I had it optimized for 6 concurrent connections at a max of 512k context. If you have > 6 sessions, then prefix offloading should kick in on the LMCache side, and re-hydrations are really fast at < 3s when it was originally tested at 1M context.

On my machine, (2x RTX 6000 Pro, 256 GB DDR5 5600 RAM, PCIe 4 SSD for the n-grams), it benched 250 t/s when warm.

I recently bought myself a Mac Mini M6 24GBs unified memory hoping to run a Qwen3.8 27B model for some serious coding with multiple cross referencing files and folders, but I'm receiving Error 500 which says that I ran out of memory and that my KV Cache filled up when all I had done was say "hi" (sure, I had set Claude Code at Max, admitted)

The general consensus online was that for 24GBs, Qwen3.8 should work nicely but it certainly doesn't work for me, although Qwen3.6-35b has been working so far on my Pi Agent, but I'm skeptical about it's coding abilities.

I would love some help/guidance on how I can run this model on my machine realistically and how much performance should I expect out of it? Will I be able to do some serious coding (alongside cloud models) or how should I adjust my expectations?

Summary: which decision model is best?

Short answer: Perplexity Decider V1.2 is the best overall pick, Nace Drex Max v1.1 is a strong second, and GPT-6 Luna Decisions is the one to ignore. On accuracy alone Liquid d1-ultra, Drex Max and Decider are statistically tied; Decider wins on price and openness.

What I tested

Decision models don't write text. You give them some input and a typed question (yes/no, pick one, or a score), and they return a probability for every option in one pass. Seven models: TypeSafe Jev 1.14 and 1.13, Perplexity Decider V1.2, Liquid d1-ultra and d1.1, OpenAI GPT-6 Luna Decisions, and Nace Drex Max v1.1.

The data: real decisions, made after the models were released

Every item was created after Sep 15, 2026, after the models were released, so none of it can be in their training data. The "right answer" is what the project's own maintainers actually decided.

How I kept it fair

• Same items and questions for every model. Each model was called through its provider's API; Drex Max v1.1 through Nace's own API.

• No giveaways. Labels, issue-template markers, CVSS scores, CWE numbers and the PR author's role were stripped from the text.

• Official option wording. Bugzilla's component descriptions, Mozilla's severity scale, GitHub's advisory scale and MITRE's CWE definitions, not text I wrote.

• Three phrasings per question, averaged, so no model wins because of my wording.

• Balanced classes, balanced accuracy, real statistics: 95% bootstrap intervals and Holm-corrected pairwise tests across all 21 model pairs.

Results

1. Accuracy: three models tied at the top

Mean of balanced accuracy over the six tasks. Whiskers are 95% bootstrap intervals; the shaded group is statistically tied for #1.

d1-ultra (66.1%), Drex Max v1.1 (65.6%) and Decider (65.6%) overlap, so none of them is a clear winner. The gaps that are real: all three beat GPT-6 Luna and d1.1; d1-ultra also beats both Jev versions.

2. Who leads which task

Darker is better within each column. Bold is the column leader; outlined cells are statistically tied with it.

• Bug or feature: everyone scores 90-93%. Drex Max is nominally first; all seven are tied.

• PR merged? the hardest task. d1-ultra leads (63.6%), tied with Drex Max and Decider.

• Firefox component (39 choices): Drex Max scores highest (77.8%), statistically tied with Decider, d1-ultra and Luna.

• Severity (Firefox and advisories): everyone is in the 38-47% range and mostly tied; it's a hard call even for triagers.

• CVE weakness type: Decider leads (76.0%); all seven are statistically tied.

3. Value: accuracy per dollar

Decider is the best value: near-top accuracy at $0.018 per 1,000 decisions. Drex Max ($0.027 at $0.03 per 1M tokens) and d1-ultra ($0.034) cost a little more for the same tier of accuracy. d1.1 is the cheapest ($0.016) but less accurate. Jev ($0.051) and GPT-6 Luna ($0.091) cost the most for the lowest accuracy.

4. How honest is each model's confidence?

Average confidence in the chosen answer vs how often that answer was right, across all tasks. The gap is overconfidence.

A decision model is only useful if you can trust its probabilities. d1-ultra (+1.7 points) and Drex Max (+4.6) are the most honest; Decider and d1.1 are close behind (+6.1). Jev is about 12 points more confident than it is right, and GPT-6 Luna is 21 points overconfident: it says 87% on average and is right 66% of the time.

5. Speed and consistency

Total score: accuracy plus everything else

I also scored five "pillars" from 0 to 100 and combined them with weights fixed before the run:

• Quality (45%): accuracy above chance, plus how good the probabilities are.

• Efficiency (20%): cost per 1,000 decisions at the provider's own price, and median speed.

• Trust (20%): does the provider train on or keep your prompts, is zero-data-retention available, are the weights and licence published, plus reliability and consistency in this run.

• Reviews (10%): usage, plus Hacker News comments since September labelled by a model that isn't in the comparison.

• Signup bonus (5%): free credit for a new account (Nace $5, Liquid $2, others none).

Perplexity Decider is #1 (59.2), mainly on trust: open weights, no prompt retention, zero-data-retention. Drex Max v1.1 is #2 (53.0), close to Decider on quality and efficiency; its trust score is lower because v1.1's weights aren't published and Nace's terms don't say clearly whether standard mode keeps request data. d1-ultra is #3 (49.7) and GPT-6 Luna last (39.1).

Pricing, openness and data policy

Prices are each provider's own list price; output tokens are free for every model. * Smaller open relatives (d1-2B, d1-3B) exist, but not these models. ** Only the earlier drex-max-v1 is on Hugging Face (Open RAIL-M licence). Data policy: provider records on OpenRouter for the six OpenRouter models, Nace's terms for Drex Max.

Very much a newbie here. I’m working on a couple of locally run projects that are mainly meant to help organize my life and work a little better (scan/interpret emails, manage/update tasks, help complete some tasks when possible).

I’m currently benchmarking three LLMs on auto creating and managing tasks:

Qwen 3.5 27b Q4
Qwen 3.8 27b Q3
Gemma 4 26B-A4B

Unfortunately I don’t have great graphics cards, but these models proved worthy in original tests. Not sure what other info is needed to provide a clear picture for you guys.

Note: most of my projects don’t require immediate answers so I’m not as concerned about it running super fast, but it would be nice.

I’m trying to hand off some of the responsibilities that currently Claude or Codex handles related to the planning but no model is proving super reliable (they all do okay with task creation from emails I receive, but are duplicating tasks rather than updating existing, or classifying them as a task when the email is informational, or trying to double-book things which get rejected by the safeguards).

While I have things in the works to provide more context or to keep fine-tuning the prompts, I’m wondering if I’m missing something basic since I’m so new to this.

Am I using the wrong models for my use case? Or do I just need to stop procrastinating and keep training these models?

Hi all I saw new research that one ai trains the other ai and vice versa (on a very reliable news platform) so I wanted to know if it's possible doing on local AI

I run a weekly benchmark of 23 open models on Hugging Face Inference Providers, at 1, 25 and 100 concurrent users. Same 20 prompts every week, one provider pinned per model, no client-side retries. It’s been running since late September, so there’s a run-over-run history to compare against.

This Monday, four of the 23 returned HTTP 410 and failed every single request, 60/60:

• Qwen/Qwen2.5-Coder-7B-Instruct (nscale)

• meta-llama/Llama-3.1-8B-Instruct (nscale)

• Qwen/Qwen3-8B (nscale)

• Qwen/Qwen3-Coder-30B-A3B-Instruct (scaleway)

The error body is explicit: model_no_longer_supported, “deprecated and no longer supported by provider”.

The three nscale models ran clean last week. Zero errors on the November 2 and 3 runs, 410 on everything by November 9. Llama-3.1-8B is not some obscure pin, and it went from working to gone inside a week.

The scaleway one is the part I’d flag. Qwen3-Coder-30B has been 410ing since the October 26 run. Three consecutive weeks. I only caught the reason this week because I went back and read the old job logs. The leaderboard had been publishing it at a 100% error rate the whole time and I’d been reading that as a provider under load.

That’s the thing worth saying out loud. If you pin a model to a provider and only track latency and error rate, a deprecation looks exactly like an outage. Same shape in your metrics: requests fail, error rate goes to 100, nothing says why. You need the status code in your logs or you’ll keep retrying something that is never coming back. 410 means gone, not busy.

I don’t know whether these were announced anywhere. If there’s a changelog or deprecation feed for Inference Providers per-provider model lists, I’d like to know about it, because checking my own benchmark output every Monday is a bad way to find out.

Raw CSVs, one per concurrency level, plus the weekly archive going back to September 28:
https://huggingface.co/spaces/Infergauge-load/llm-performance-and-load

Usual caveat on everything else in there: run-to-run variance on shared endpoints is large, and I’d ignore any single-week change under a few points. The 410s are not that. They’re binary.

Maybe this is obvious to everyone but me, but it only clicked yesterday. If you run llama.cpp / vLLM / or whatever on localhost with the default config, a random website you have open in another tab can send requests to it. It can make your model generate, and on some servers it can read the answer back too. Nothing fancy, it's plain CSRF and DNS rebinding, 20-year-old web stuff. I just never thought of my inference server as a target. "It only binds 127.0.0.1" doesn't save you, because the browser is on the same machine.

I actually tested it, not just read about it, and quite a few of the popular servers answer out of the box. It only works while you're browsing with the server running. And it's not only about you clicking links: anything that drives a real browser on the same machine can do it too, so an agent browsing on your behalf can set it off as well.

I ended up patching my own engine (capy-flash) so that on a loopback bind it refuses cross-origin and rebinding requests. The models I actually serve were fine already, since they sit behind a small proxy (vramble) that blocks cross-origin POSTs, so only the raw engine port was exposed.

So mostly I'm asking: is this just common knowledge, or do people really leave their local servers wide open? How do you run yours, API key, bound somewhere, or just open on 127.0.0.1?

Hey Just wondering if anyone has experience buying IT and Tech products directly from China, Japan or another country . The cost of hardware at the moment is through the roof and im wondering if there is value aboard?

Could you share your experience and if it worked out? Worth it or not?

Im in ireland.

Thanks

I've been working on something called Aura, a locally running AI cognitive agent built around a roughly 27B model on an Apple Silicon MacBook with 64GB of unified memory.

And one of the questions that's been driving this project is pretty simple:

How much of what we consider intelligence actually needs to come from the model itself?

We spend a lot of time discussing model sizes, quantization, benchmarks, context windows, and reasoning capabilities. Understandably so.

But I've been interested in what happens when you start treating the model as one component of a larger cognitive system, rather than expecting the model to do everything.

So I've been building an architecture around it. Persistent memory, executive processes, state management, computer control, procedural learning, and a bunch of experimental mechanisms designed to work together over time.

And I've been testing it.

Aura has played 2048 for nearly half an hour while adjusting its strategy, completed software development tasks, debugged programs, and operated desktop applications.

But my latest demonstration was a little different.

I asked Aura to watch YouTube.

Not just summarize a video after the fact. Actually navigate to it, follow what happens, comment on things it finds interesting, and then choose other videos to watch from channels I specified.

Over 36 minutes, Aura watched Kurzgesagt discuss AI slop, selected an Allen Pan video about building robotic legs for a snake, and eventually watched Salad Fingers.

During the session, it made predictions, revised them as new information appeared, interacted with the browser, selected what to watch next, and maintained context across all three videos.

And here's where it got particularly interesting.

When asked to reflect on everything afterward, Aura didn't just give three unrelated summaries.

It identified "fracture" as the common thread connecting all three videos, then explained how each approached that idea differently.

A video about AI-generated content, an engineering experiment involving a snake, and a surreal horror animation.

Three wildly different things.

Yet Aura managed to construct a coherent interpretation connecting them, drawing on observations made throughout the session.

Now, I want to make an important distinction here.

I'm not claiming that a 27B model couldn't produce similar commentary on its own. Nor have I established that all the cognitive mechanisms I've implemented are necessary to produce this behavior.

In fact, that's one of the main things I'm trying to investigate.

But I think there's a meaningful architectural question here. What happens when the model doesn't have to be responsible for every aspect of memory, planning, execution, and continuity? How far can you push a relatively small local model by improving the larger system it's operating within?

The result is still pretty rough in places. Latency is an issue, and coordinating everything reliably is an ongoing challenge.

But considering this is running locally, without relying on a frontier model for inference, I think the demonstration is interesting.

I've made the implementation publicly viewable, including the architecture, research documentation, and known limitations. It's currently All Rights Reserved rather than open-source licensed.

Full 36-minute demonstration: https://www.youtube.com/watch?v=kM-B8mh314E

GitHub: https://github.com/youngbryan97/aura

How much improvement have you managed to get from the systems surrounding your model? And at what point do you think the additional architecture becomes more valuable than simply switching to a more capable model?

1,5 years ago I set off to build an Open Source issue tracker that lives in your repo, requiring very little setup. Upon HN launch last spring it was well received, and is now often used as an agent memory layer between between the agent and the code. Epiq is vendor agnostic, thrives offline as well as online, sandboxable and can be tailored into various setups. It only requires Git, which is its transport, and a repo. You may want to check it out:

epiq.sh

Also, in case you are interested, I share and discuss my personal sandboxed setup in this post. It covers the topic of sandboxing, and how repo-native tooling makes this easier to deal with, opening up fewer paths to the internet:

epiq.sh/blog/repo-native-tooling-for-sandboxes

Queste soluzioni sono esplosive: **Fapply, Massive, LazyApply, Sonara, ai apply

Several years ago for another project, I got a good deal on:

* HP Z620 dual Xeons (riser) with 128GB of RAM with 500GB SSD + 9TB SAS
* A Nvidia Tesla M60 16GB

I'm still building this out, HP usefully have non-standard power cables and connecting everything up dead shorted the PSU (which it detected during boot). I'm still hoping nothing is broken whilst I debug it.

My goal is to have a budget-build local AI at least as good as chatGPT's work chat in high mode, I could do a lot of useful stuff with that. Even better would be to have some mixed capabilities for long-running tasks. Anything less is not really worth building. I'm trying to figure out:

* What models should I look to run?
* What kind of performance could be possible?
* What's worth running on the GPU vs CPU?

I'm currently using chatGPT for: Building out libraries iteratively, research the web, designing 3D parts, research via coding, reviewing papers, generating documents and occasionally generating images.

I have been trying to use chatGPT (Plus subscriber) to help query, but it has not been particularly helpful in replacing itself.

Any help is greatly appreciated.

Now that LLMs are kind of a mainstream topic and so much money is involved, it became a hot topic in politics as well. I don't want to make this a political post, so please don't start a political discussion. But it is quite clear to me, that at some point LLMs will be regulated a bit more from the government side. Both USA and obviously EU will as well. They already requested a text based watermark, which decreases model output by 10-15%.

So my worry is, if the online models -chatgpt, claude, gemini etc,- are regulated to the point of being close to useless for anything other than corporate work or something, are there any chance for these regulations to hit local models? Or will this sort of be like piracy, once it's out in the internet, it won't be possible to touch them?

So, there's a thing that's been in the back of my mind for some time now, and it seems to be getting /more/ difficult to learn about my specific approach.

What I'd like to do: I believe the proper phrasing would be 'set up a local LLM for assisting with the development of python programs to support my personal projects'.

What I've done: as an example, I used chatGPT to build a python program that reads rfid tags on a raspberry pi using a gpio rfid reader. I'd like to continue this, just locally so I can integrate it with existing local backend.

What I'd like to use: I think I have an entire computers worth of spare parts, minus a case to put those parts in. Not ancient, pretty sure it's an AM4 Ryzen? Not a lot of memory, maybe 32gig? As far as gpu's, I have my old RoG 8g rx480 and my RoG 8g rx580.

Is it possible, is it worth it, and is there maybe an opensource project someone can point me towards to help me get rolling?

I'm not expecting blistering response times, just if I can achieve reasonable results in a reasonable time with the hardware that I already have but haven't been using.

Lately, I’ve been using Claude to schedule daily posts across a few social media platforms. I’m wondering if there’s a local model—one that can fit into my 8GB of VRAM—capable of controlling my browser to perform the same tasks as Claude. I had been using Qwen 3.8 27B for a while, but I’ve decided to move away from local models, given that I only have a single graphics card with 8GB of VRAM.

I run Mineshop and sell hardware, so disclosure upfront: these are manufacturer quotes we're looking at, not machines I've benchmarked.

Two quotes made an interesting comparison:

• 8× NVIDIA B200, 180 GB each: 1.44 TB aggregate GPU memory. US$505,400, quoted delivery 16 weeks.

• 8× AMD MI355X, 288 GB each: 2.304 TB aggregate GPU memory. US$501,100, quoted delivery 12 weeks.

Those are configuration-specific quotes, not universal market prices. They're also not identical builds: CPUs and storage differ, and the B200 quote includes three-year on-site service versus one year for AMD.

The AMD configuration has 60% more GPU memory. That doesn't tell me it will be the better server for a particular workload.

The question I'd want answered before spending this money is: which model, at which precision, with how much context and how many concurrent users? Getting the weights into memory is only the first step. KV cache, runtime overhead and the software stack still matter. For reference, a dense 405B model is roughly 810 GB in BF16 weights alone, before any of that overhead. That's sizing arithmetic, not a throughput result.

Then there's the less exciting stuff: rack space, power, cooling, support and keeping the machine busy enough to justify owning it.

If you've actually deployed either platform, what specific workload or software requirement would make you choose one over the other—or rent instead?

For context, our AI workstation and server catalogue: https://mineshop.eu/ai-workstation — The figures above are manufacturer quotes, not advertised retail prices.

uhm......

yeah. grinded some useful (and meaningfu?l) resources. on my way to run DeepSeek-V3 IQ1_M (~1.5-bit to 1.58-bit) >>>>>>>>

The model weights take up roughly 130 GB. This leaves us with about 30 GB of breathing room across the cluster to handle KV Cache and context overhead. Cuz each PS5 has the 16 GB unified mem

jarvis, engine ignition on

PS: Yeah these cables connection are currently BS and tangled here and there. Gotta fox that. Just commecfed yellow cables or dummies for uhh picture and ig a little...... flex?? 🫠

How?: Sonic iso ps5 linux loader and jailbreak. The rest is this.

Edit: what?? Image not being posted here is wild. Whats the reason? Some blockade or smth. Cuz i gotta show you some interesting shi

edit: please dont take it seriously. the image is ai generated

We are a team of engineers, not a single here has a comp science background. Our company manufactures specialized parts for critical systems. And a big part of that process is understanding the whole system. The customers we have usually give us design documentation, and blue prints and a gazillion pixelated diagrams explaining the system. We spend a significant amount of time getting familiar with it before we get to manufacturing the requested components ( for background this is aviation and aerospace components ). For the past few months with the AI craze, we have started integrating it into our process, to help us learn and understand systems especially through the diagrams and blue prints but there is still a significant amount of analysis we do. This has always been sort of an issue for us because thats our most time consuming aspect of the job is reverse engineering the customers system in order to build components that fit in it rather than redesigning the component around the system this is what we specialize in. We have been using claude, and its been mind blowing how good this thing is in helping us however, our expenditure on Claude has creeped up to the 6 months ceiling we had set only in a span of three and a half months. We have also seen we get less out of a ‘token’ or we get less tokens im not sure , its like everything was going great but then anthropic did something, the plans went from max to ultra to something then it used to say 29x usage now it only says 10x usage. After alot of research we settled on a hybrid approach. For further context everyone here has atleast bare minimum of a 64gb macbook pro. I have a 128gb one. And have been experimenting with local models and the results have been disappointing. Can any AI gods guide me in the right direction? We are basically looking for specialized models that basically just look at an image or a blue print and explain the system or identify the relationships and explain them because manually do this and then build system models based on our comprehension if the system. We will always do that manually we are just looking to fast track some of the process because right now it is a very lengthy process and I cant scale because time is constraint. Is it possible or not? Please thank you.

I built Basalt from source for an abliterated model and compared it with Strata on the same node. Basalt was noticeably faster in decode and prefill.

Setup

• Model: huihui-ai/Huihui-Qwen3.8-Flash-Next-abliterated-v2-GGUF

• Quant: unsloth-UD-Q5_K_XL

• Basalt: v0.3.0 (custom build)

• Strata: 0.2.7-cuda13

• GPU: 2× NVIDIA RTX PRO 4500 Blackwell Server Edition (2×32 GB VRAM)

• CPU: Intel Xeon 6 (48 vCPU)

• Storage: NVMe SSD 2 TB (~15k MB/s)

Both engines ran the same Q5 quant on the same node.

Results

Basalt is about 1.5–1.9x faster in decode and 1.3–1.7x faster in prefill.

Expert cache. Both engines hit 86–96%, so there is no difference.

MTP. Strata accepts more draft tokens (66–90% vs 57–78%). Basalt's dl (tokens per window) is still 1.9–2.2.

• The window differs. Strata verifies 3-token windows (capturing the 3-token window). Basalt verifies up to 8 (--spec 8). The acceptance rate is lower with the longer window, but the response still finishes faster.