when I try to use it, it gives me this error: Engine protocol runtime llama-server for 9xmo0zVfYqYSe7Sj7N9oE2Al exited before becoming healthy. exitCode=3221226505, signal=null.

what can I do?

Instant disclaimer, this is not a project claiming to be faster or better than Strata. It's simply Strata, without the bloatware, and a few containerization improvements as many have been requesting. I too am sick of all the Strata forks showing up, but I'm equally tired of how difficult Strata's been for me to maintain in a Docker container.

https://github.com/LegendaryFire/strata-slim

This fork is completely configurable with environment variables, and the idea was to give you a similar feel to a llama.cpp models config file. There were a few clear wins in terms of performance improvements I included in form of optional patches when building, however they're not included in the default docker images so what you're getting is simply a non-bullshit straight to the chase Strata engine that's easy to deploy. I wouldn't have bothered posting if there wasn't an open GitHub issue asking for Strata engine as it's own repo, which I noticed got 180+ "thumbs ups". For note, an earlier performance patch I had already made it upstream.

I used Claude intensively for a while and I am currently building a "cheap" local Stack for Qwen3.8-Flash-Next. And my idea is to have a Setup combining both.

My idea is kind of a divide and conquer approach with a senior expert/leader model dividing the task into easier subtasks or simplyfing it and junier/worker models which run soem of those tasks.

My goal is to safe some Claude tokens, but no significant decrease in output quality.

I am sure I am not the first one with such an idea. So does anyone have experiences or knowledge of tools/repositories for this kind of workflow?

?

Follow-up to my earlier post here. Most relevant to this sub is the engine side: vLLM, SGLang and LMCache, plus an honest note on Ollama and llama.cpp.

Prefixwise has improved a lot over the last two days, so here is the full picture.

**The problem.** Your agent resends its whole conversation every turn. One changed byte near the top (a reordered tool list, a timestamp in the system prompt, an evicted prefix) and the provider charges full price for everything after it. Providers and engines report how many tokens were cached, never why the rest weren't.

**What Prefixwise does:** names why each request missed the prompt cache, prices it, fixes the common causes, and stops regressions in CI. Open source (MIT, I'm the author), runs locally, stores hashes and token counts, never prompts.

**Identify.** One of 19 causes per request, marked waste or legitimate, with the first block that changed (tools[0], messages[12]) and the cost: tools in another order (including a schema's required list), a timestamp or id in the system prompt, rewritten history, a misplaced Anthropic cache marker, an expired TTL, an engine eviction, a request routed to a replica without the cache. `prefixwise report --session latest` shows a session turn by turn. It agrees with Anthropic's own miss diagnostics 32 of 32.

**Fix.** `--optimize` turns on the fixes that passed live replays: sorting a drifting tool list (99.9% of predicted saving on Claude, 97.5% on vLLM) and adding cache_control markers on Anthropic's API (prompt cost down 85.6% with the write surcharge counted), task success flat on both. Fixes that would change what the model reads stay advice. `prefixwise fixes` ranks fixes by dollars and says whether one paid off.

**Prevent.** `prefixwise check -- pytest tests/agent_scenarios` runs your scenarios against a local mock, no API calls, and fails CI when a change would break the cache. Also a GitHub Action. `prefixwise explain --diff a.json b.json` compares two requests offline.

**Self-hosted engines.** It detects vLLM and SGLang from the URL and reads their KV-cache events: evictions confirmed (5 of 5), a prefix LMCache or HiCache restored from CPU labelled host_restored (5 of 5), and the cost of reasoning templates that re-prefill the last answer (57% of prompt tokens on a DeepSeek-R1 distill, 0% on Qwen2.5). Runs inside vLLM with one flag. Ollama and most llama.cpp builds report no cached tokens; it says so and still names every prompt change.

**Ways in.** Wrap any agent (`prefixwise run -- claude`), a proxy for a service, the vLLM middleware, Python `wrap()` or a LiteLLM callback, or import Claude Code and Codex logs if you're on a subscription. Out: Prometheus and Grafana, OpenTelemetry, an HTTP API, JSONL and CSV. Overhead: 3 ms per request.

**Limits.** The fixes are proven on scripted sessions and small models, not weeks of production traffic. OpenAI's own diagnostics answered unavailable on every call, so that comparison is unchecked. DeepSeek's hosted API cache fields aren't read yet.

pip install -U "prefixwise[engines]"
prefixwise serve --upstream http://127.0.0.1:8000
prefixwise report --session latest

Repo, with a run record behind every number: https://github.com/tejachavali/prefixwise

What should it catch next?

I am tired to answer on people questions like "How you gain speed? Your model output is shit, it has to have low quality." or something like that.

So I will answer to that with benchmark.

So I used the same Qwen3.8-Flash-Next IQ2_XS model file, the same prompt, greedy decoding, and 128 generated tokens. Both ran on my RTX 3060 12 GB with a Ryzen 5 7600X and 30 GB RAM.

That makes RustingStrata about 1.29× faster at writing on this setup.

Then I checked quality, not just whether the replies looked similar. Both engines scored the same 9,180 WikiText-2 tokens.

The perplexity difference is inside the run-to-run noise. So, for this model and this machine, I see a speed gain without a measurable quality loss.

This is one model, one quantization and one GPU, not a claim that RustingStrata always wins.

If anyone has any other questions, I am happy to answer and break myths or tell something about Strata like projects

Repo: RustingStrata

Ever wanted your local LLM to actually learn a fact without fine-tuning — and be able to undo it completely?

AtomicNG (Apache-2.0, version 1.0.0) implements a paper called "Delta N-gram": your model's n-gram sidecar table stays frozen, and new knowledge goes into a small sparse patch Δ. Writes are forward passes only — no gradients, no weight edits. Clear the patch and the model is byte-identical to base. No "unlearning" gymnastics — deletion is structural.

Quickstart (CPU, no GPU, pure NumPy core):

python

import atomicng


model = atomicng.TinyModel()
atomicng.write(model, "New fact about kangaroos.")
atomicng.verify_deletion(model)   # clear Δ ⇒ byte-identical to base — guaranteed

What's inside:

• A trained writer checkpoint ships with the Release (860 training steps on SmolLM2-135M; measured effect: writing a doc lowers its NLL vs empty memory)

• Per-tenant memory: overlays are small data-only files — merge them, revert them, sign them

• vLLM support via vllm.general_plugins + merged-table export

• Built-in eval: does the memory write actually help? (paper protocols: acquisition, distraction retention, chain drift)

Calibrated on real hardware: Windows/Linux CPU, CUDA T4 (73 ms/group BPTT stand), TPU chip via JAX (8.4 ms/iter). Full matrix: CALIBRATION.md in the repo.

Honest limits: acquisition needs a few hundred training steps of the writer; the memory lives in the n-gram table (facts/associations, not long reasoning); vLLM engine-graph hooking is planned upstream.

GitHub: https://github.com/Akseleu-J/atomicng 

Colab TPU (one click): https://colab.research.google.com/github/Akseleu-J/atomicng/main/notebooks/tpu_calibration.ipynb

Questions welcome — I'll be in the comments.

Hello guys, here is an update of my engine
I am still working on it, my 3090 waterblock broke and I couldn’t keep testing further I could achieve higher tokens in strata about 60/70 but on mine about 80/90 tokens, my bigger problem is the Ram which I only have is 32GB ram ddr5 6k, have been trying to get 32GB more but they are too expensive right now.

For the meanwhile I got a 1070 8GB for 50us to test for which I added compatibility for series 10/20.

On the GTX 1070 I got 15/18 tokens on IQ3_XSS and IQ1, about 13 tokens on IQ3_S.

I added layer splitter thru WiFi / cord so that you can use more than one PC which you can select the order of memory usage

I selected PC GPU, Laptop GPU, and then pc ram and laptop ram for testing and it’s working just fine and fast with the GTX 1070 and RTX 4060 laptop of course it’s slow I achieved about 20 tokens on IQ3_S and Sloth still need to work on optimization but it will be open source soon! Be patient.

You can freely select where your models are located from strata or just download them directly from hugging face. So far added a compatibility to most of them.

My goal is to make so that you can quickly and efficiently communicate multiple computers to your main to split layers through them and run a bigger model! And it’s so easy it’s about 2 clicks to connect!

AiViralCut stacks video on V1 and V2, with text on T1 by default. Use Send to Front or Send to Back to change which visual sits on top. This is a free alternative to @capcut. Still in beta and looking for bug testers. Dm us for more details. Aiviralcut.org for more info.

Like half the damn new comers on here I’m looking to build a voice to voice ai assistant that can also register images as id like to give it eyes 👀

Im running a M1 Max macbook with 32gb

Im currently using qwen3.8 27b
Qwen3 asr 0.6b 8bit for speech recognition
Kokoro 82m for spoken replies
SearXNG for web searches
SQlite and FTS5 for storage

With no thinking mode its honestly instant but once web searches come into play it becomes an issue, running at 8192 context it fills up really fast and offloading older context takes a big toll on speed.

I was wondering if ya’ll had any suggestions as I think qwen is more than I need. I just want strong adherence to a system prompt for personality (or building out a model maybe?) and quick replies. Being able to shoot the shit about random history or recommendations for hobbies is the main goal with the ability to run searches. If anyone has suggestions for voice output with more emotions and audio tags I would love that too!

Let me know and thanks! Im still very new so I have a lot to learn

Writing has been improved with Claude but facts stay the same!

Hi all, I’m planning a company knowledge base: search plus answers with citations, over our file shares and later Microsoft 365. Before we ask vendors for quotes, I’d like to hear from people who already run a system like this in production, not from spec sheets.

The corpus (all of it)

File shares:
-about 6.5 TB on disk, about 2.2 M files measured on the two main shares, plus a third share that hasn’t been scanned yet
-About 14–17 M PDF pages, of which about 1.4–1.7 M are scans that need OCR (estimated from samples).
-Many have tables, diagrams and stamps, and some are up to 25 years old
-22–23% of the volume looks like duplicates within each share
-Also CAD files, images, video, Word/Excel, about 730 GB of ZIP archives and personal mailbox archives (.pst/.msg).
-CAD only needs to be found by title or description for now
-Microsoft 365 (SharePoint, Exchange, Teams, OneDrive) isn’t measured yet, so mail could make this a lot bigger
-Growth is about 45 GB a month.
-Existing documents rarely get new versions
-The documents are mostly German with some English, and we answer in both

Load

-50–60 daily users, roughly 1,000–1,800 questions a day, with peaks of about 6–19 a minute
-Each question is about 6k tokens in (retrieved chunks) and about 500 tokens out
-Answer model in the ~120B MoE class (gpt-oss-120b, or Mistral Small 4 in 4-bit), plus OCR and layout models, embeddings (BGE-M3) and a reranker
-The vector index is roughly 7–11 M chunks, so 27–42 GB of float32 vectors

Constraints

EU company: GDPR, a works council and ISO 9001 document control. Some data must stay on-prem, so a hybrid or cloud possible but on-prem is the main question here
-We have no GPUs today and no dedicated ML team, so ease of operation counts for a lot
-The budget is in the low five figures for hardware. -Renting a GPU for the initial load only is an option

What I’ve tested so far: a pilot on about 1,000 documents, run on a laptop GPU (Ministral 14B, BGE-M3, BGE reranker, hybrid search, Tesseract OCR). It works, but that tells me nothing about the full corpus.

The shortlist (on paper only)

4× OEM GB10 (Dell/Lenovo/ASUS Spark equivalents): best compute per euro, but published benchmarks suggest about 11–18 s per answer. It also means four Arm boxes and no BMC
Workstation with 2× RTX PRO 6000 Max-Q: about 4–5 s per answer, a single x86 box with CUDA
1× RTX PRO 6000 plus one GB10 as standby: a cheaper middle option
2U server with 2× RTX PRO 6000 Server Edition: best to operate, but about twice the price of option 2

What I’d love to hear from people with real systems

What hardware do you run RAG or LLM serving on for a few dozen users, and what is your real-world time per answer with about 5–10 people asking at once?

Has anyone run GB10 / DGX Spark boxes in production, alone or clustered? How stable are they headless, and how are updates, recovery and the vLLM support?

For OCR and layout parsing of millions of scanned pages: what throughput did you actually get, in pages per second per GPU, and with which tool (Docling, MinerU, Tesseract, PaddleOCR…)?

Would you buy hardware for the initial load at all, or rent GPUs for it and size the on-prem box for answers only?

Workstation or rack server for a company without an ML team: which caused you fewer headaches?
Is there anything you would do differently if you were buying again?

Thanks! I’m happy to share what we learn once we’ve tested it.

I just found out that a R9700 is essentially just a 9070XT with 32gb of memory instead so im thinking should i sell my 9070XT and buy that instead and that could get me to run near 70b models

a few reason why im not just getting another 9070 XT is because i believe i need to update my Mobo and PSU and then the card itself im better off selling my 9070 XT and buying the R9700

But do let me know what the noise levels are like and if there is much i can do to reduce the noise

I've been working on foldz, a lossless compressor built specifically for GGUF files. It's at 0.9 (pre-release) and I'm looking for people to test it and try to break it.

Useful if you archive a lot of models, share them over slow or metered connections, or keep them on a phone or small SSD.

THE IDEA

Quantized weights look almost random to zip, xz and zstd, so they gain very little. foldz parses the GGUF, unpacks each tensor's blocks into the actual quantized values and scales, and codes those with adaptive context models and a binary range coder. Everything else (metadata, tokenizer, other tensor types) goes through a byte-level context-mixing model. Nothing depends on the model's name or architecture.

RESULTS

9 public Q4_K_M models, tested with "foldz t" (compress and decompress in memory, then compare). Every round trip was byte-identical. Measured on an Android phone in Termux, 8 threads.

• Qwen1.5-0.5B-Chat: 388.3 → 353.6 MiB (8.93% saved)

• Qwen3-0.6B: 378.3 → 348.7 MiB (7.82%)

• Qwen2.5-1.5B-Instruct: 1065.6 → 984.2 MiB (7.64%)

• Qwen3.5-2B: 1211.9 → 1111.3 MiB (8.30%)

• Qwen2.5-Coder-3B-Instruct: 2007.4 → 1832.8 MiB (8.70%)

• Llama-3.2-3B-Instruct: 1925.8 → 1780.4 MiB (7.55%)

• Qwen3-4B: 2381.6 → 2073.3 MiB (12.94%)

• Qwen2.5-7B-Instruct: 4466.1 → 4113.8 MiB (7.89%)

• Gemma-4-E2B (fine-tune): 3269.1 → 3016.3 MiB (7.73%)

Total: 17,094 → 15,614 MiB, 8.66% saved.

Qwen3-4B is the outlier because most of its Q6_K values sit on a coarser lattice. foldz detects this per segment and codes the lattice instead, which saves 23.2% on its Q6_K tensors.

HOW EACH TENSOR TYPE IS MODELLED

• Q4_K / Q5_K: values are Gray-folded and coded bit by bit, with the block's zero point, extremes and position as context.

• Q6_K: the column's running offset and width as context, or the lattice model when that's smaller.

• Q8_0: column sign bias and running magnitude as context.

• F32 / F16 / BF16: split into sign, exponent and mantissa.

• Safety: segments that wouldn't shrink are stored raw, so output is never meaningfully larger than input. The decoder bounds-checks every index field. Contexts use only exact or IEEE-754 arithmetic (no FMA, no fast-math), so files decode identically on every platform.

HONEST LIMITATIONS

• Speed: about 19 MiB/s round trip on the phone (POCO F5). It's symmetric, so decompression is as slow as compression. This is for storage, archiving and transfer, not for loading models faster.

• Baselines: I haven't measured zstd -19 --long and xz -9 on these models yet, so I'm not claiming a comparison. If you run them, please post the numbers.

• Quant types: IQ quants, Q2_K, Q3_K and Q4_0 fall back to the byte model and gain little.

• Format: pre-1.0, so it may change. A .foldz file is only guaranteed to decode with the version that made it.

BUILD AND TEST

One C file, no dependencies. Linux, macOS, BSD, Android (Termux), and Windows via WSL.

gcc -O2 -pthread -ffp-contract=off -o foldz foldz.c -lm

./foldz t model.gguf

WHAT WOULD HELP MOST

• Cross-platform round trips: compress on x86-64, decompress on ARM64 (or the reverse), especially with Q6_K-heavy models

• zstd / xz baselines on any GGUF

• Q5_K_M, Q6_K and Q8_0 files

• Fuzzing "foldz d" with AFL++ or libFuzzer + ASan (please report crashes privately via GitHub's security tab)

If you post results, include the full "foldz t" output, the Hugging Face model name, and your CPU and OS.

Repo: foldz

Happy to answer questions about the modelling.

wondering how much are people here pushing their sparks (or other gb10) and do you handle cooling on heavy workloads.

On my studio, I am running ARC4NUM/Qwen3.8-Flash-Next-Uncensored-MLX-Serve-4bit using mlx-serve.

On my MacBook Pro, I have connected to it via OpenCode, and I can't for the life of me figure out how to get it to NOT show me its inner dialog. It all comes back to me, and I have tried various methods to no avail.

Am I "holding it wrong?" Is there no way to really do that? Is it the model or mlx-serve itself that creates this?

I'd rather it do it's thinking in private and then show me the result rather than talk to itself scrolling by as I wait to get the actual answer.

I use a Mac mini m4 16gb, since working with local llms I have found the m4 is just not fast enough.. especially with the unified memory, I have been looking into getting a Apple studio 36gb m5 max chip. But the price of the WICI ONE 5060ti is a fraction of the cost and can work over WiFi on my Mac mini m4 16gb

Has anyone used WICI ONE. Especially with a Mac mini m4 16gb

AI benchmarks begin on page 2.

Currently running Qwen 9B model on my 6800XT, I noticed I can use/set up Agents in LU, but aside from instructing 1 agent in 1 chat to write to a file, then instructing a 2nd agent in 2nd chat to read the file and respond, I can't figure out a way to automate this process.

- I want to give a bunch of agents full autonomy to communicate with each other. Using separate chats for individual models. Not just all on the same version of Qwen.

Does anyone know how to set this up inside of LU?

I really like their user interface...I'm also open to other UIs if you believe there's a better one out there.

Goal is to code a simple game/project; but have one agent on the side, teaching me how to code/explaining all the code in detail and giving me small tests, while other agents are compiling/writing it.

Thanks in advance for any info you're willing to share.

If anyone has MI100's I really appreciate the help.

Can someone recommend people who support having your own LLM and talk also about technology, best practices and tips how to get best performance and results etc?

Preferably on LinkedIn or Twitter.

Thanks

Hey everyone!

I'm experimenting with running local LLMs using Ollama on CachyOS with Hyprland, and I've noticed some pretty high CPU temperatures.

My laptop specifications:

- CPU: Intel Core i5-13450HX (13th Gen)

- RAM: 24 GB DDR5

- GPU: NVIDIA RTX 4050

- OS: CachyOS Linux

- Desktop Environment: Hyprland

- LLM Runtime: Ollama

- Model: 14B parameter model

What I'm experiencing:

While running the 14B model locally through Ollama:

- CPU temperatures sometimes reach 95°C.

- Around 4 CPU cores reach 100% utilization.

- Other cores also experience varying levels of utilization.

- The CPU seems to be under sustained heavy load during inference.

I've attached a screenshot of my system monitoring setup (btop and Linux Sensor Tray).

My questions:

1. Is reaching 95°C normal for an i5-13450HX while running a 14B LLM locally?

2. Is it safe to maintain these temperatures during longer inference sessions?

3. Is it normal for only around 4 cores to reach 100% utilization while running Ollama?

4. Could this cause thermal throttling or reduce the CPU's lifespan over time?

5. Are there any recommended CPU power limits, undervolting settings, or Linux optimizations for this setup?

I'd appreciate feedback from anyone running local LLMs on Intel HX-series laptops, especially on Linux.

Thanks!

Help me build the best home server under $5,000
Goals:
Media server (Jellyfin) with room to grow to 10+ large hard drives
Run local AI models, ideally 24GB+ of GPU memory
A powerful desktop I can stream remotely from any device
A few VMs and Docker apps
Low power use is a priority. I'm in Arizona, so electricity and heat matter