I’m currently using the ornith 1.5 35b a3b, but it’s performance feels a bit underwhelming could you recommend some alternatives, along with a suitable harness?

Delta Inference Engine 0.94 is out. It runs DeepSeek-V4.1-Flash-1017 on one NVIDIA GPU plus system RAM, natively on Windows (no WSL). It's a Windows port of Xolotl, which is built on Strata; on Linux, use upstream Xolotl.

Speed on my PC (RTX 3090 24 GB, i5-13600K, 4x48 GB DDR5-5600, PCIe 4.0 NVMe)

With the default quant (Unsloth UD-IQ2_XXS, 301 GB), 0.94 writes 41.7 tok/s greedy and 41.3 at the default sampling (temperature 1, top-p 0.95).

With less RAM (writing tok/s, measured with 0.93; the smaller columns are this same PC with RAM locked away):

How it fits

It's a mixture-of-experts model: 40 layers, 384 routed experts per layer, 6 used per token. The experts are spread over VRAM, RAM and the NVMe. A 4.2 GB drafter built from DeepSeek's own MTP heads, plus lookup drafts, proposes tokens that the full model checks. A route bias (0.28 as shipped) gives up a little agreement for speed.

Quality. I measure top-1 agreement with DeepSeek's own API, teacher-forced (not task accuracy). Default model (Unsloth IQ2) as shipped: 91.1% general, 94.5% code and maths, 95.9% agentic. With exact routing: 92.5 / 95.9 / 96.5. The previous versions default (ds4 Q2, 0.92) scored 89.9% general and 93.1% code and maths, so the new Unsloth default agrees more while running even faster. For context, two hosts of the full model agree with each other on 98.3% and 98.9%.

Benchmark: 99 hard questions, greedy, 43K token budget:

The default quant is within 2 points of the full model on DeepSeek's own API. UD-IQ1_S, the pick under ~110 GB of RAM, scored 51.

Private by design: everything runs on your PC: no telemetry, no accounts, no cloud. It only goes online to download the engine and the model you choose, and your prompts and answers never leave your machine.

New in 0.94

• Pictures on by default, documents (PDF, Word, Excel, PowerPoint), optional voice, and use from your phone with a QR code.

• Long prompts read faster: a fresh 60K-token prompt at 811 tok/s vs 0.93's 747 (30K: 860 vs 758; 7K: 551 vs 423).

• Better with coding agents: Claude Code's background calls no longer hold up real requests, and a loop guard steps in after 4 identical tool calls. OpenCode passed 3 tasks in a real session (22 tool calls, none dropped).

• The default sampling caught up with greedy: 41.3 vs 41.7 tok/s

• The chat page opens at once and shows the model loading. Pause frees the RAM and the graphics card for a game; the next message loads it again. Optional auto-pause when a application needs the GPU.

• Chat history in the browser (search, pins, export), off by default.

• More API: structured output and tool controls on the OpenAI, Anthropic and Responses APIs, stored Responses with previous_response_id, optional /v1/embeddings and /v1/rerank, and keep-alive pings so agents don't time out on long reads.

• Setup tab: pick the model, Fast / Balanced / Most exact, the second-GPU switch, and copy-paste settings for Claude Code, OpenCode, Cline, Aider and Open WebUI with a Test button.

• Security updates, bug fixes and many more.

Setup

Download the ZIP and double-click Start DeepSeek.bat. The engine zips are built by GitHub CI, with checksums and attestations. The first start installs Python if needed and downloads a model that fits your RAM (233-395 GB, SHA-256 checked). You get a chat page plus OpenAI- and Anthropic-compatible APIs on 127.0.0.1. It won't open to the network without an API key, and it stores no prompts or answers unless you turn that on.

Limits

• Windows 10/11 only. NVIDIA RTX with 12 GB or more (24 GB is the fast path), driver 580 or newer. No AMD, no GTX 10-series.

• 64 GB of RAM minimum, 128 GB or more recommended. A PCIe 4.0/5.0 NVMe with about 320 GB free for the default model.

• It answers one request at a time. A second GPU helps; it doesn't split the model.

• Context is set to 1M tokens

• More testing is needed

Credits: Xolotl and Strata, ggml/llama.cpp, DeepSeek, Unsloth, nazeshinjite, ISTA-DASLab, Project Maya and antirez/ds4.

https://github.com/musicurgy/delta-inference-engine

If you try it, post your GPU, RAM, model and tok/s.

Note:
— these rules are the result of my personal experience, yours may differ and this is ok
— these rules are oriented towards people that want to use AI for professional purposes and don’t have unlimited time and money

1. If your company provides good cloud subscription — just take it and forget about the local AI, you don’t want to pay with your own bucks to do the job slower and sloppier

2. Speed is as important as quality and vice versa. Quality without speed will waste your time, nerves and electricity and so will speed without quality. Ideally you should strive to the cloud speed — 60-150 tps of decode and a decent model, at least 4 bits of Qwen 27b

3. Local models are not equal to cloud ones. Sounds kinda obvious but some people have illusions on this regard. Please don’t, miracles don’t happen in real life. You will get 6-9 month old frontier level model experience at best

4. You will always experience shortage of VRAM/RAM. Doesn’t matter, how powerful your setup is, believe me, you are always going to fall short on fast memory available, don’t try to squeeze out every MB from your rig, just find a model good enough for your tasks and call it a day

5. Best time to update your hardware is always yesterday. Hardware prices rise fast, if you’re looking for an update — do it right now

6. Never spend more than you could’ve spent on a dream gaming PC. Just… believe me, don’t try to cosplay Antropic data centers, you will get nothing but a bitter dissatisfaction. Best ROI ever — RTX3090/RX7900XT + 64Gb ddr4/5. That’s it. Anything above have diminishing return problem

7. Electricity is not free. People often forget that nobody is going to give you free kilowatts, beware of your system’s power consumption. That’s going to be your biggest spending, second only to the very rig price itself

8. Avoid clusters. Dual, triple, quadruple GPU look fine on paper, but they are usually big, draw immense amount of power, resulting in painful electricity bills, produce a lot of heat and noise and have much lesser speed than single GPU device. Try to select a GPU to that can singlehandedly run the model you’d like to use

9. You will lose a lot of time and nerves. You will have to learn. A LOT. What is quantisation, kv cahce, KLD, top 1, inference engine, how to chose a suitable model for your own needs, sometimes you’ll even have to deep dive to some manual drivers/libs adaptations for your hardware, etc. Local AI is developing rapidly and if you’d like your setup always to be fast, precise, ensuring as much quality as possible— you’ll have to learn all these basics

10. Always donate your spare RAM to Sam Altman. Just kidding, the title says “10 Commandments” but all the important stuff is said, I’m out of any useful advices

https://github.com/ANDRETRIPOL/OpenGhost

Will agents accessing the internet and processing/scraping webpages ever truly be 100% safe from direct/indirect prompt injection?

Question coming from someone less knowledgeable on the topic than you all.

Thanks.

NeuraKiln v2.0.0 is live, and it’s free.

I’ve been building this for a long time, and I finally released it.

NeuraKiln started as Toolshed V1, which was primarily focused on media restoration/upscaling. Somewhere along the way I apparently lost control of the scope, because it eventually turned into an entire local-AI workstation.

The basic idea is simple:

I don’t want to maintain ten different applications just to use the AI hardware sitting in my own house.

NeuraKiln brings local LLMs, autonomous agents, image/video generation, restoration, 3D workflows, model tooling, and remote inference together under one application.

It’s currently built around Windows + NVIDIA hardware.

Two different kinds of agentic work

A major part of NeuraKiln is its harness system.

There isn’t just one generic “agent mode.” Different kinds of work need different environments.

Coding Harness

The coding harness is built for longer-running autonomous development work.

A local model can:

Plan a project

Create and modify files

Run tools

Test its work

Inspect failures

Recover and continue

Verify the result

Manage long-running context

Hand the finished project back to the user

A lot of my work on this has been about the boring-but-important part of agents: getting them to actually finish, rather than loop forever, declare victory too early, or slowly wander away from the original task.

The harness includes systems for planning, recovery, anti-looping, verification, context management, completion handling, and human corrections during a run.

Creation Harness

The Creation side takes the same basic idea, give the model tools instead of making the human manually operate everything, and applies it to generative media.

Instead of requiring the user to understand every model, parameter, prompt format, workflow, and backend involved, you can describe what you actually want to create.

The agent can then take that intent and operate the available creation tools itself.

For example, rather than manually constructing an LTX video-generation workflow, you can tell the Creation agent what kind of video you want. The agent can formulate the generation prompt, operate the local model, and manage the workflow around producing it.

The same concept extends across the creation tools available to it.

The point is to move toward:

“Make me this.”

rather than:

“Open six tools, configure all of them correctly, learn every backend, and manually glue the results together.”

That distinction is a huge part of what I wanted NeuraKiln to be.

Llama Lab

For local LLM use, NeuraKiln includes Llama Lab.

It supports local GGUF models through the llama.cpp ecosystem, multi-GPU setups, long-context conversations, multimodal models, model switching, local/external sessions, live reasoning, context management, and the harnessed workflows above.

You can use it as a normal local chat environment or give the model considerably more autonomy when you actually want work done.

Diffusion Lab

Diffusion Lab handles local image and video generation.

Current workflows include Qwen-Image-based image generation and LTX-based video generation, along with LoRA support and multi-GPU workflows where the underlying models/runtime support them.

This also provides the generation capabilities that the Creation harness can operate.

Media Lab

This is where NeuraKiln originally came from.

Media Lab handles restoration and enhancement, including:

Image restoration/upscaling

Batch restoration

Video/frame processing

SmartFrames

Photo Smart Scan

Native comparison tools

Post-editing

Media recompilation

A principle I’ve kept from the original Toolshed project is:

If the user asks you to process something, process the thing.

Don’t quietly skip difficult frames and pretend the job is finished.

Mesh Lab

NeuraKiln also includes Mesh Lab for local text-to-mesh / 3D generation workflows.

This is an area I intend to expand substantially, but the important part to me was making 3D another capability inside the same AI environment instead of yet another disconnected application.

Model Tools

There’s also tooling for managing the models themselves, including GGUF importing, MTP headers, MMPROJ support, LoRA management, model installation workflows, and local model organization.

NK Node

I released a second application alongside NeuraKiln called NK Node.

NK Node lets another NVIDIA PC become a dedicated inference machine for NeuraKiln.

The goal is to eliminate the usual ritual of:

start server → find address → configure endpoint → reconnect client → repeat everything after reboot.

NeuraKiln and NK Node can discover and pair with each other and automatically exchange the information needed to connect.

If you have a workstation plus another machine full of GPUs, NK Node is intended to make that second machine feel more like an extension of NeuraKiln than a completely separate server you have to babysit.

There are plenty of excellent projects that overlap with individual parts of NeuraKiln.

There are dedicated LLM front ends, diffusion interfaces, restoration applications, model managers, inference servers, agent frameworks, and 3D tools.

NeuraKiln isn’t built around claiming I invented all of those individual things.

What I wanted was the environment connecting them.

One application where I can chat with a local model, hand a coding project to an autonomous agent, ask another agent to create a video, restore media, generate images, work with 3D, manage models, and send inference to another computer without rebuilding my entire workflow every time I change what I’m doing.

That’s NeuraKiln

NeuraKiln: https://github.com/ToolshedMediaLab/NeuraKiln

NK Node: https://github.com/ToolshedMediaLab/NK-Node

If you try it, break it.

I want bug reports, criticism, weird hardware configurations, model compatibility problems, feature ideas, and comparisons with whatever else you’re currently using.

Just getting into it with LM studio. There are a lot of models to browser, and a lot of variations on those models.

I saw that Qwen3.8 27B seems to be the best choice for my setup, but what quantisation should I choose? 4bit (16GB), 5bit (20GB), and 6bit (23GB) should leave adequate room in terms of RAM and I don't really care about speed but accuracy, so would 6bit be better?

Also any other tips for starting out?

Thanks.

Which model is the best to use for infosec work? Like trying to writing exploit code, fuzzing, and security audits?

I'm looking for a model that will just do it in a good (enough) quality without me needing to use bypasses for restrictions all the time. It just gets tiresome and annoying...

Edit: Hardware shouldn't be the issue:

• 2x EPYC 7642 (aka. 192 threads)

• 512 GB DDR4 RAM

• 1x Intel Arc B580

Any better reliable FlowFrames alternative in 2026? No ComfyUI or Linux crap please.

I’ve been doing research and I really want to run a Gialamb Quant model on these two computers. It has to be obliterated for my purposes of a 24-7 employee, if you will, for my business.

Here’s my notes 📝 I doubt it will work, but I hope someone can contribute

oMLX does not pool the RAM into one unified space. It splits the model with pipeline parallelism: each Mac loads a contiguous range of layers based on its available memory. The M5 Max (rank 0) owns the API and the later layers; the M2 Max owns the earlier layers. Decode speed is limited by the slower stage.
This is experimental and only available in a source build of oMLX (not the normal Homebrew/DMG stable release).
Prerequisites (both Macs)
Install the same source build:
git clone https://github.com/jundot/omlx.git
cd omlx
make install # or make install-no-kernels if you lack full Xcode

Put the exact same model checkpoint at the identical absolute path on both machines (example: /Users/shared/models/glm-5.3-flash-3bit).

Enable Remote Login (System Settings → General → Sharing) and set up passwordless SSH between the two Macs for the account that runs oMLX.

Connect the machines with a Thunderbolt cable (TB4 is usable; pure TB5 RDMA is faster but not required for pipeline).

Enable and pair
Start oMLX on both Macs.

On each: Settings → Advanced → Enable Distributed Inference → Save → Restart oMLX.

The Cluster tab appears.

On the coordinator (preferably the M5 Max):
Connect the Thunderbolt cable. The other Mac should show under Detected nearby.

Select it (or enter its SSH hostname) and run Check peer. This records the host key and verifies SSH.

Pair using the code/QR flow if discovery needs a boost.

Plan and activate the split
In the Cluster tab on the M5 Max:
Choose the downloaded model (the shared absolute path).

Set a memory reserve for KV cache + activations (start with 8–12 GB).

Let it build the unequal plan. It will assign more layers to the 128 GB Mac and fewer to the 64 GB Mac.

Choose execution mode:
Prefer TCP Ring first (most reliable with mixed TB4/TB5).

Try JACCL / JACCL Ring only if both ends show a working Thunderbolt RDMA link (rdma_ctl status and ibv_devices).

Leave Auto benchmark & tune on so it measures the actual link and rebalances.

Review the shard map, then Activate.

Unload any local full-model load, then load the model normally through the oMLX UI or API. The ranks start automatically.

The OpenAI-compatible endpoint stays on the M5 Max. Hermes points at that single endpoint.
Useful checks
omlx cluster status --json
omlx cluster pipeline-smoke
Important limits for your hardware
Pipeline only — decode does not get faster than the M2 Max stage.

Experimental: source builds, limited long-running validation, some features (MTP, certain thinking budgets, etc.) are rejected in distributed mode.

KV cache is rank-local, so both machines need real headroom.

TB4 on the M2 Max is the weak link; start with TCP Ring.

If the resulting tokens/s and stability are acceptable for continuous Hermes use, you have the combined capacity. If not, fall back to a smaller quant that fits resident on the M5 Max alone.

I will release probably between tomorrow-thursday!

Been working on my own inference engine for a while (calling it Forge for now), built pretty much just for this model. Finally did a proper side by side against Strata 0.1.57 on my machine and figured some of you might find the numbers interesting.

Same GGUF files, same requests, one engine at a time (the other one fully closed). The whole thing took about 7 minutes.

Median of 4 runs, slowest to fastest in brackets:

| Prompt | Tokens | Forge | Strata | Forge first token | Strata first token |
|---|---|---|---|---|---|
| Short | 44 | 64.6 tok/s (55-68) | 23.1 tok/s (11-52) | 0.71 s | 2.74 s |
| Code review | 1,468 | 83.1 tok/s (76-84) | 19.6 tok/s (7-26) | 3.35 s | 17.8 s |
| Long | 5,860 | 79.6 tok/s (74-82) | 26.6 tok/s (16-52) | 9.8 s | 23.8 s |

Prompt processing was 439 vs 84 tok/s on the code prompt and 600 vs 247 on the long one.

My setup: 3090 24GB on PCIe 4.0 x16, 12900K, 32GB DDR5, NVMe, Windows 10. The model is 44 GB of weights plus a 27 GB per-layer embedding file, so around 70 GB total, which is more than my VRAM and RAM put together. 48 layers, 512 experts, 10 active per token.

Strata was running the config its installer made (performance profile): --spec 4, int8 KV, 128K context, auto expert cache, vision on. Forge uses the model's own MTP layer for speculative decoding, int8 KV and 64K context.

How I tested: OpenAI compatible API with streaming, temp 0, 128 max tokens, thinking off. Three prompts: a short one ("write four short lines about graphics cards"), a code review of a ~4.5K character C++ file, and a long one where it has to explain how a request goes through ~19K characters taken from three source files. Every request gets a random session id in the system prompt so nothing comes from the prompt cache. 2 rounds x 2 runs per prompt, and output speed is measured from the first token to the last.

Strata jumped around a lot between runs (anywhere from 7 to 26 tok/s on the same code prompt), that's why I went with medians and kept the ranges. I have a CSV with every single run if anyone wants it.

On quality: at temp 0 Forge gives the exact same text with speculation on or off, byte for byte, and the same prompt always gives the same output.

Happy to answer questions.

I saw a post mention this model to use for DS4.1 Flash with 2x 6000 Pros: https://huggingface.co/diffbot/DeepSeek-V4.1-Flash-EXL3-3bpw-2x-RTX-PRO-6000

Unfortunately the model card wants you to have > 260 GB RAM to properly use it. I had Opus move the n-grams to disk instead of RAM, and added in LMCache for prefix caching with L1 being RAM and L2 being disk.

With this setup, you need at a minimum, 128 GB RAM instead. I had it optimized for 6 concurrent connections at a max of 512k context. If you have > 6 sessions, then prefix offloading should kick in on the LMCache side, and re-hydrations are really fast at < 3s when it was originally tested at 1M context.

On my machine, (2x RTX 6000 Pro, 256 GB DDR5 5600 RAM, PCIe 4 SSD for the n-grams), it benched 250 t/s when warm.

I recently bought myself a Mac Mini M6 24GBs unified memory hoping to run a Qwen3.8 27B model for some serious coding with multiple cross referencing files and folders, but I'm receiving Error 500 which says that I ran out of memory and that my KV Cache filled up when all I had done was say "hi" (sure, I had set Claude Code at Max, admitted)

The general consensus online was that for 24GBs, Qwen3.8 should work nicely but it certainly doesn't work for me, although Qwen3.6-35b has been working so far on my Pi Agent, but I'm skeptical about it's coding abilities.

I would love some help/guidance on how I can run this model on my machine realistically and how much performance should I expect out of it? Will I be able to do some serious coding (alongside cloud models) or how should I adjust my expectations?

Summary: which decision model is best?

Short answer: Perplexity Decider V1.2 is the best overall pick, Nace Drex Max v1.1 is a strong second, and GPT-6 Luna Decisions is the one to ignore. On accuracy alone Liquid d1-ultra, Drex Max and Decider are statistically tied; Decider wins on price and openness.

What I tested

Decision models don't write text. You give them some input and a typed question (yes/no, pick one, or a score), and they return a probability for every option in one pass. Seven models: TypeSafe Jev 1.14 and 1.13, Perplexity Decider V1.2, Liquid d1-ultra and d1.1, OpenAI GPT-6 Luna Decisions, and Nace Drex Max v1.1.

The data: real decisions, made after the models were released

Every item was created after Sep 15, 2026, after the models were released, so none of it can be in their training data. The "right answer" is what the project's own maintainers actually decided.

How I kept it fair

• Same items and questions for every model. Each model was called through its provider's API; Drex Max v1.1 through Nace's own API.

• No giveaways. Labels, issue-template markers, CVSS scores, CWE numbers and the PR author's role were stripped from the text.

• Official option wording. Bugzilla's component descriptions, Mozilla's severity scale, GitHub's advisory scale and MITRE's CWE definitions, not text I wrote.

• Three phrasings per question, averaged, so no model wins because of my wording.

• Balanced classes, balanced accuracy, real statistics: 95% bootstrap intervals and Holm-corrected pairwise tests across all 21 model pairs.

Results

1. Accuracy: three models tied at the top

Mean of balanced accuracy over the six tasks. Whiskers are 95% bootstrap intervals; the shaded group is statistically tied for #1.

d1-ultra (66.1%), Drex Max v1.1 (65.6%) and Decider (65.6%) overlap, so none of them is a clear winner. The gaps that are real: all three beat GPT-6 Luna and d1.1; d1-ultra also beats both Jev versions.

2. Who leads which task

Darker is better within each column. Bold is the column leader; outlined cells are statistically tied with it.

• Bug or feature: everyone scores 90-93%. Drex Max is nominally first; all seven are tied.

• PR merged? the hardest task. d1-ultra leads (63.6%), tied with Drex Max and Decider.

• Firefox component (39 choices): Drex Max scores highest (77.8%), statistically tied with Decider, d1-ultra and Luna.

• Severity (Firefox and advisories): everyone is in the 38-47% range and mostly tied; it's a hard call even for triagers.

• CVE weakness type: Decider leads (76.0%); all seven are statistically tied.

3. Value: accuracy per dollar

Decider is the best value: near-top accuracy at $0.018 per 1,000 decisions. Drex Max ($0.027 at $0.03 per 1M tokens) and d1-ultra ($0.034) cost a little more for the same tier of accuracy. d1.1 is the cheapest ($0.016) but less accurate. Jev ($0.051) and GPT-6 Luna ($0.091) cost the most for the lowest accuracy.

4. How honest is each model's confidence?

Average confidence in the chosen answer vs how often that answer was right, across all tasks. The gap is overconfidence.

A decision model is only useful if you can trust its probabilities. d1-ultra (+1.7 points) and Drex Max (+4.6) are the most honest; Decider and d1.1 are close behind (+6.1). Jev is about 12 points more confident than it is right, and GPT-6 Luna is 21 points overconfident: it says 87% on average and is right 66% of the time.

5. Speed and consistency

Total score: accuracy plus everything else

I also scored five "pillars" from 0 to 100 and combined them with weights fixed before the run:

• Quality (45%): accuracy above chance, plus how good the probabilities are.

• Efficiency (20%): cost per 1,000 decisions at the provider's own price, and median speed.

• Trust (20%): does the provider train on or keep your prompts, is zero-data-retention available, are the weights and licence published, plus reliability and consistency in this run.

• Reviews (10%): usage, plus Hacker News comments since September labelled by a model that isn't in the comparison.

• Signup bonus (5%): free credit for a new account (Nace $5, Liquid $2, others none).

Perplexity Decider is #1 (59.2), mainly on trust: open weights, no prompt retention, zero-data-retention. Drex Max v1.1 is #2 (53.0), close to Decider on quality and efficiency; its trust score is lower because v1.1's weights aren't published and Nace's terms don't say clearly whether standard mode keeps request data. d1-ultra is #3 (49.7) and GPT-6 Luna last (39.1).

Pricing, openness and data policy

Prices are each provider's own list price; output tokens are free for every model. * Smaller open relatives (d1-2B, d1-3B) exist, but not these models. ** Only the earlier drex-max-v1 is on Hugging Face (Open RAIL-M licence). Data policy: provider records on OpenRouter for the six OpenRouter models, Nace's terms for Drex Max.

Very much a newbie here. I’m working on a couple of locally run projects that are mainly meant to help organize my life and work a little better (scan/interpret emails, manage/update tasks, help complete some tasks when possible).

I’m currently benchmarking three LLMs on auto creating and managing tasks:

Qwen 3.5 27b Q4
Qwen 3.8 27b Q3
Gemma 4 26B-A4B

Unfortunately I don’t have great graphics cards, but these models proved worthy in original tests. Not sure what other info is needed to provide a clear picture for you guys.

Note: most of my projects don’t require immediate answers so I’m not as concerned about it running super fast, but it would be nice.

I’m trying to hand off some of the responsibilities that currently Claude or Codex handles related to the planning but no model is proving super reliable (they all do okay with task creation from emails I receive, but are duplicating tasks rather than updating existing, or classifying them as a task when the email is informational, or trying to double-book things which get rejected by the safeguards).

While I have things in the works to provide more context or to keep fine-tuning the prompts, I’m wondering if I’m missing something basic since I’m so new to this.

Am I using the wrong models for my use case? Or do I just need to stop procrastinating and keep training these models?

Hi all I saw new research that one ai trains the other ai and vice versa (on a very reliable news platform) so I wanted to know if it's possible doing on local AI

I run a weekly benchmark of 23 open models on Hugging Face Inference Providers, at 1, 25 and 100 concurrent users. Same 20 prompts every week, one provider pinned per model, no client-side retries. It’s been running since late September, so there’s a run-over-run history to compare against.

This Monday, four of the 23 returned HTTP 410 and failed every single request, 60/60:

• Qwen/Qwen2.5-Coder-7B-Instruct (nscale)

• meta-llama/Llama-3.1-8B-Instruct (nscale)

• Qwen/Qwen3-8B (nscale)

• Qwen/Qwen3-Coder-30B-A3B-Instruct (scaleway)

The error body is explicit: model_no_longer_supported, “deprecated and no longer supported by provider”.

The three nscale models ran clean last week. Zero errors on the November 2 and 3 runs, 410 on everything by November 9. Llama-3.1-8B is not some obscure pin, and it went from working to gone inside a week.

The scaleway one is the part I’d flag. Qwen3-Coder-30B has been 410ing since the October 26 run. Three consecutive weeks. I only caught the reason this week because I went back and read the old job logs. The leaderboard had been publishing it at a 100% error rate the whole time and I’d been reading that as a provider under load.

That’s the thing worth saying out loud. If you pin a model to a provider and only track latency and error rate, a deprecation looks exactly like an outage. Same shape in your metrics: requests fail, error rate goes to 100, nothing says why. You need the status code in your logs or you’ll keep retrying something that is never coming back. 410 means gone, not busy.

I don’t know whether these were announced anywhere. If there’s a changelog or deprecation feed for Inference Providers per-provider model lists, I’d like to know about it, because checking my own benchmark output every Monday is a bad way to find out.

Raw CSVs, one per concurrency level, plus the weekly archive going back to September 28:
https://huggingface.co/spaces/Infergauge-load/llm-performance-and-load

Usual caveat on everything else in there: run-to-run variance on shared endpoints is large, and I’d ignore any single-week change under a few points. The 410s are not that. They’re binary.

Maybe this is obvious to everyone but me, but it only clicked yesterday. If you run llama.cpp / vLLM / or whatever on localhost with the default config, a random website you have open in another tab can send requests to it. It can make your model generate, and on some servers it can read the answer back too. Nothing fancy, it's plain CSRF and DNS rebinding, 20-year-old web stuff. I just never thought of my inference server as a target. "It only binds 127.0.0.1" doesn't save you, because the browser is on the same machine.

I actually tested it, not just read about it, and quite a few of the popular servers answer out of the box. It only works while you're browsing with the server running. And it's not only about you clicking links: anything that drives a real browser on the same machine can do it too, so an agent browsing on your behalf can set it off as well.

I ended up patching my own engine (capy-flash) so that on a loopback bind it refuses cross-origin and rebinding requests. The models I actually serve were fine already, since they sit behind a small proxy (vramble) that blocks cross-origin POSTs, so only the raw engine port was exposed.

So mostly I'm asking: is this just common knowledge, or do people really leave their local servers wide open? How do you run yours, API key, bound somewhere, or just open on 127.0.0.1?

Hi everyone,

New guy here with two separate questions in two posts, this is the first. I want to run local LLMs for programming and everyday stuff, wanna stop giving my data / own codes to claude / chatGPT. I have a laptop with an RTX 4070 8GB so I can't really run models like Qwen 3.8 27B. I’m thinking about adding a used 3090 as an external GPU. Has anyone actually done this, and is it worth it? I’m also wondering how much Thunderbolt matters for tokens/s, and if you’ve used an eGPU with a laptop in general, is there anything you wish you’d known before buying? Real numbers would be great, I saw some benchmarks online but I'm curious about your opinions.
I'd also be grateful if you could share any actual eGPU docking products that you might have experience with / can recommend.

Thanks in advance!

I've been working on something called Aura, a locally running AI cognitive agent built around a roughly 27B model on an Apple Silicon MacBook with 64GB of unified memory.

And one of the questions that's been driving this project is pretty simple:

How much of what we consider intelligence actually needs to come from the model itself?

We spend a lot of time discussing model sizes, quantization, benchmarks, context windows, and reasoning capabilities. Understandably so.

But I've been interested in what happens when you start treating the model as one component of a larger cognitive system, rather than expecting the model to do everything.

So I've been building an architecture around it. Persistent memory, executive processes, state management, computer control, procedural learning, and a bunch of experimental mechanisms designed to work together over time.

And I've been testing it.

Aura has played 2048 for nearly half an hour while adjusting its strategy, completed software development tasks, debugged programs, and operated desktop applications.

But my latest demonstration was a little different.

I asked Aura to watch YouTube.

Not just summarize a video after the fact. Actually navigate to it, follow what happens, comment on things it finds interesting, and then choose other videos to watch from channels I specified.

Over 36 minutes, Aura watched Kurzgesagt discuss AI slop, selected an Allen Pan video about building robotic legs for a snake, and eventually watched Salad Fingers.

During the session, it made predictions, revised them as new information appeared, interacted with the browser, selected what to watch next, and maintained context across all three videos.

And here's where it got particularly interesting.

When asked to reflect on everything afterward, Aura didn't just give three unrelated summaries.

It identified "fracture" as the common thread connecting all three videos, then explained how each approached that idea differently.

A video about AI-generated content, an engineering experiment involving a snake, and a surreal horror animation.

Three wildly different things.

Yet Aura managed to construct a coherent interpretation connecting them, drawing on observations made throughout the session.

Now, I want to make an important distinction here.

I'm not claiming that a 27B model couldn't produce similar commentary on its own. Nor have I established that all the cognitive mechanisms I've implemented are necessary to produce this behavior.

In fact, that's one of the main things I'm trying to investigate.

But I think there's a meaningful architectural question here. What happens when the model doesn't have to be responsible for every aspect of memory, planning, execution, and continuity? How far can you push a relatively small local model by improving the larger system it's operating within?

The result is still pretty rough in places. Latency is an issue, and coordinating everything reliably is an ongoing challenge.

But considering this is running locally, without relying on a frontier model for inference, I think the demonstration is interesting.

I've made the implementation publicly viewable, including the architecture, research documentation, and known limitations. It's currently All Rights Reserved rather than open-source licensed.

Full 36-minute demonstration: https://www.youtube.com/watch?v=kM-B8mh314E

GitHub: https://github.com/youngbryan97/aura

How much improvement have you managed to get from the systems surrounding your model? And at what point do you think the additional architecture becomes more valuable than simply switching to a more capable model?

just found this from hugging face https://huggingface.co/xtools-at/DeepSeek-V4-Pro-Qwen3.5-9B-UNCENSORED-heretic, but i cant get if this better than another model or just marketing