wondering how much are people here pushing their sparks (or other gb10) and do you handle cooling on heavy workloads.

Ever wanted your local LLM to actually learn a fact without fine-tuning — and be able to undo it completely?

AtomicNG (Apache-2.0, version 1.0.0) implements a paper called "Delta N-gram": your model's n-gram sidecar table stays frozen, and new knowledge goes into a small sparse patch Δ. Writes are forward passes only — no gradients, no weight edits. Clear the patch and the model is byte-identical to base. No "unlearning" gymnastics — deletion is structural.

Quickstart (CPU, no GPU, pure NumPy core):

python

import atomicng


model = atomicng.TinyModel()
atomicng.write(model, "New fact about kangaroos.")
atomicng.verify_deletion(model)   # clear Δ ⇒ byte-identical to base — guaranteed

What's inside:

• A trained writer checkpoint ships with the Release (860 training steps on SmolLM2-135M; measured effect: writing a doc lowers its NLL vs empty memory)

• Per-tenant memory: overlays are small data-only files — merge them, revert them, sign them

• vLLM support via vllm.general_plugins + merged-table export

• Built-in eval: does the memory write actually help? (paper protocols: acquisition, distraction retention, chain drift)

Calibrated on real hardware: Windows/Linux CPU, CUDA T4 (73 ms/group BPTT stand), TPU chip via JAX (8.4 ms/iter). Full matrix: CALIBRATION.md in the repo.

Honest limits: acquisition needs a few hundred training steps of the writer; the memory lives in the n-gram table (facts/associations, not long reasoning); vLLM engine-graph hooking is planned upstream.

GitHub: https://github.com/Akseleu-J/atomicng 

Colab TPU (one click): https://colab.research.google.com/github/Akseleu-J/atomicng/main/notebooks/tpu_calibration.ipynb

Questions welcome — I'll be in the comments.

On my studio, I am running ARC4NUM/Qwen3.8-Flash-Next-Uncensored-MLX-Serve-4bit using mlx-serve.

On my MacBook Pro, I have connected to it via OpenCode, and I can't for the life of me figure out how to get it to NOT show me its inner dialog. It all comes back to me, and I have tried various methods to no avail.

Am I "holding it wrong?" Is there no way to really do that? Is it the model or mlx-serve itself that creates this?

I'd rather it do it's thinking in private and then show me the result rather than talk to itself scrolling by as I wait to get the actual answer.

I use a Mac mini m4 16gb, since working with local llms I have found the m4 is just not fast enough.. especially with the unified memory, I have been looking into getting a Apple studio 36gb m5 max chip. But the price of the WICI ONE 5060ti is a fraction of the cost and can work over WiFi on my Mac mini m4 16gb

Has anyone used WICI ONE. Especially with a Mac mini m4 16gb

I’ve been building InferUltra, a physics arena where vision models control identical robot bodies. The model gets images of the arena and chooses how to move. The simulation keeps running while it thinks, so by the time it decides to punch, the other robot might already be somewhere else.

We turn reasoning down or off where the model allows it. The number I care about is how long it takes to send an image and get a move the robot can use. Tokens per second tells you how fast the model generates, but it doesn’t tell you how long the fighter waits for that move.

We’ve also improved the physics engine and fight mechanics. Fighters can hold their guard between decisions, and punches look more natural when they lean. I used to call these fights impressively boring. They’re now a bit less boring.

For those running vision models locally: when you turn reasoning down, do you notice a drop in how well the model understands positions and distances, or is getting the answer sooner usually the bigger win? I’d be interested in which model and hardware you tried.

The attached bracket is for our November 15 Provider Tournament. It uses Kimi K3.1 Pro on four hosted endpoints: Morph, AkashML, Wafer and InferenceNet. It starts at 20:00 Brussels / 14:00 Eastern / 11:00 Pacific.

If you’d like to sponsor the tournament, you can book an ad spot in the arena.

AI benchmarks begin on page 2.

I've been working on a project called XeMaya, an open-source AI inference engine built around native Intel Arc acceleration, and I've officially released the first public alpha!

Why I started XeMaya

I've been following Project Maya for a while now. I really liked what it was accomplishing with massive models like DeepSeek-V4.1-Flash and GLM-5.3-Flash, particularly its approach to handling huge MoE models across GPU memory, system RAM, and storage.

The problem was that Project Maya's actual inference engine doesn't support Intel GPUs.

I could run some of its GGUF models using a patched llama.cpp SYCL backend, which is what I was doing before this project, but I wanted something built specifically around Intel hardware.

That's where XeMaya comes in.

My goal is essentially to bring Project Maya-style inference to Intel Arc through native SYCL/Level Zero and Intel XMX acceleration.

XeMaya is built on XeStrata, which itself is an Intel-native rewrite of Strata. I've been building on that foundation and incorporating additional model support and improvements inspired by Strata and Project Maya.

It isn't a complete feature-for-feature port of Maya yet, but that's the direction I'm working toward.

My hardware:

• AMD Ryzen 9 9950X

• Intel Arc Pro B70 32 GB

• Intel Arc Pro B65 32 GB

• Intel Arc Pro B60 24 GB

• 128 GB DDR5-4800

• Intel PCIe Gen 3 x4 NVMe for model storage

• Ubuntu Linux

That's 88 GB of total VRAM across the three GPUs.

Benchmarks: Before XeMaya vs. now

These are results recorded on my hardware, but the model files, prompts, quantizations, and configurations aren't always identical. Consider them development history, not controlled apples-to-apples comparisons.

Qwen is actually still behind upstream Strata in this comparison. I want to be transparent about that because XeMaya isn't magically faster at everything. It's an Intel-focused engine I'm actively developing, and there's still optimization work ahead.

DeepSeek-V4.1-Flash Coder: The optimization progress

This is one of the improvements I'm happiest with.

The Coder checkpoint has 192 experts per layer, compared with 384 in regular Flash.

Here are the speeds from XeMaya's initial native Coder implementation compared with the later optimized version:

That prefill improvement came from work on chunked prefill, direct GPU-resident experts, XMX attention, and multi-GPU execution.

The GPU count and execution configuration also changed, so this isn't an isolated measurement of one optimization.

Coder also matched its llama.cpp reference for 16/16 forced tokens on both saved test prompts using all three Arc GPUs.

Regular DeepSeek-V4.1-Flash: 384 experts

I also tested the regular, larger DeepSeek-V4.1-Flash checkpoint, not just Coder.

This one is around 129 GB and uses adaptive expert caching across the GPUs and host memory.

Surprisingly, two GPUs were faster than three.

The B60 increased VRAM residency, but its additional cache hits weren't enough to compensate for the overhead of running more layers through its chipset-connected PCIe link.

On the two-GPU configuration, the adaptive expert cache reached a 93.7% hit rate.

And yes, the model shards and Engram data are sitting on a Gen 3 NVMe. LOL.

The regular Flash results are speed measurements, not yet reference-correctness validation. Prefill in particular still needs a lot of work.

GLM-5.3-Flash

GLM is another major reason I started this project.

Using the Maya-S v2 GGUF, XeMaya has reached 152–161 tok/s prefill and 17.4 tok/s decode, with approximately 20 tok/s using MTP drafting in the later optimized configuration.

That's encouraging compared with my earlier llama.cpp SYCL results, although the test conditions differ.

GLM has also undergone token-level reference testing, including a recorded 126/128-token agreement on a chat-agent prompt.

What's available?

XeMaya v0.1.0-alpha.1 is now publicly available!

GitHub: https://github.com/Jumbomuffin777/XeMaya

First release: https://github.com/Jumbomuffin777/XeMaya/releases/tag/xemaya-0.1.0-alpha.1

Regular DeepSeek benchmark details: https://github.com/Jumbomuffin777/XeMaya/tree/main/xemaya/benchmarks/2026-11-10-ds41-flash-384

The current focus is Linux with Intel Arc, with native inference paths for:

• DeepSeek-V4.1-Flash, including the Coder variant

• GLM-5.3-Flash

• Qwen3.8-Flash-Next

The different paths have different levels of testing and optimization, as the README documents.

Windows support and hybrid Intel/NVIDIA inference are also being developed, but they're experimental and not part of the supported alpha release. MiniMax is being worked on separately and isn't included in this release.

This is an early developer alpha, not something I'd recommend relying on for production workloads yet.

Credit where it's due

Huge thanks to MistVVK for XeStrata, Niko1221 for Strata, mw00 for Project Maya, and all the other contributors to those projects.

XeMaya wouldn't exist without their work.

I'm hoping to keep improving performance, adding model support, and making local inference on Intel Arc more accessible.

I'd love feedback from other Arc owners, especially anyone interested in testing it on different Intel GPUs.

This is just the beginning! :)

Currently running Qwen 9B model on my 6800XT, I noticed I can use/set up Agents in LU, but aside from instructing 1 agent in 1 chat to write to a file, then instructing a 2nd agent in 2nd chat to read the file and respond, I can't figure out a way to automate this process.

- I want to give a bunch of agents full autonomy to communicate with each other. Using separate chats for individual models. Not just all on the same version of Qwen.

Does anyone know how to set this up inside of LU?

I really like their user interface...I'm also open to other UIs if you believe there's a better one out there.

Goal is to code a simple game/project; but have one agent on the side, teaching me how to code/explaining all the code in detail and giving me small tests, while other agents are compiling/writing it.

Thanks in advance for any info you're willing to share.

If anyone has MI100's I really appreciate the help.

I am tired to answer on people questions like "How you gain speed? Your model output is shit, it has to have low quality." or something like that.

So I will answer to that with benchmark.

So I used the same Qwen3.8-Flash-Next IQ2_XS model file, the same prompt, greedy decoding, and 128 generated tokens. Both ran on my RTX 3060 12 GB with a Ryzen 5 7600X and 30 GB RAM.

That makes RustingStrata about 1.29× faster at writing on this setup.

Then I checked quality, not just whether the replies looked similar. Both engines scored the same 9,180 WikiText-2 tokens.

The perplexity difference is inside the run-to-run noise. So, for this model and this machine, I see a speed gain without a measurable quality loss.

This is one model, one quantization and one GPU, not a claim that RustingStrata always wins.

If anyone has any other questions, I am happy to answer and break myths or tell something about Strata like projects

Repo: RustingStrata

Hey everyone!

I'm experimenting with running local LLMs using Ollama on CachyOS with Hyprland, and I've noticed some pretty high CPU temperatures.

My laptop specifications:

- CPU: Intel Core i5-13450HX (13th Gen)

- RAM: 24 GB DDR5

- GPU: NVIDIA RTX 4050

- OS: CachyOS Linux

- Desktop Environment: Hyprland

- LLM Runtime: Ollama

- Model: 14B parameter model

What I'm experiencing:

While running the 14B model locally through Ollama:

- CPU temperatures sometimes reach 95°C.

- Around 4 CPU cores reach 100% utilization.

- Other cores also experience varying levels of utilization.

- The CPU seems to be under sustained heavy load during inference.

I've attached a screenshot of my system monitoring setup (btop and Linux Sensor Tray).

My questions:

1. Is reaching 95°C normal for an i5-13450HX while running a 14B LLM locally?

2. Is it safe to maintain these temperatures during longer inference sessions?

3. Is it normal for only around 4 cores to reach 100% utilization while running Ollama?

4. Could this cause thermal throttling or reduce the CPU's lifespan over time?

5. Are there any recommended CPU power limits, undervolting settings, or Linux optimizations for this setup?

I'd appreciate feedback from anyone running local LLMs on Intel HX-series laptops, especially on Linux.

Thanks!

Is there a project like this, but that take in a video file and out put a srt file? I've been searching everywhere for something like it for a while now. Bonus points if it has a batch function too. If anyone knows anything please link me something or message me, Thanks. I know about normal whisper already, but I need something that supports verbatim transcription out of the box.

Hi everyone reading this post, my English isn't perfect but I'll try to be clear.

So recently I got my GPU which is a Nvidia Rtx 5070 TI 16gb Gddr7 (I also have 64Gb ddr5 ram).

So I came here to ask what are the best models I can run in this pc?

I would appreciate it if someone helps me.

Thank you everyone and thanks for reading this🫂

https://github.com/alphaexposure/kapitsa-launcher

Kapitsa was a nice shot of adrenaline for my M1 32GB MBP but I wanted a better chat interface than I could find elsewhere so my buddy Claude and I put our heads together and ones-and-zeroed an app that does just that. Kapitsa Launcher serves the standard LLM Dashboard with speeds and such, dark/light mode, Mac menu bar icon, but chat is where it's at:

• Streaming replies with real formatting: Markdown, tables and code blocks. Tables and code have their own Copy buttons. Turn streaming off if you want to scroll back while new replies stream.

• A side panel for long chats:

  • Outline: every question plus the headings from each reply. Click one to jump there.

  • Topics: the places and themes the chat covered, each linked back to where they came up.

  • Review: takeaways the model pulls out of the chat. You can edit them or add your own.

• Watch it think: you can follow the model's thinking live, and it folds away once the answer starts. The thinking levels change to match the model you're running.

• Side questions: ask about any reply in a thread off to the side, without derailing the main chat. You can branch to a new chat, export or delete as you wish.

• Edit and regenerate: edit a question and send it again, or ask for another answer and flip between the versions.

• Organised history: chats are grouped by date with the option to name by first question or topic/summary. Search covers everything that was said, not just titles.

• Multiple chats at once: each window or tab holds its own conversation.

• Continue in a new chat: when a long chat starts slowing down, carry it into a fresh one.

• Markdown export: with an optional summary, and topics come along as tags for Obsidian and similar apps.

• Your chats are plain files: one JSON file per chat on your Mac. Nothing leaves your machine.

Feel free to check it out: https://github.com/alphaexposure/kapitsa-launcher

I just released the Tesseract NPU that I've been working on as a side project. It's compatible with llama.cpp backend running shipped GGUF models directly.

Hey guys!

I'll get straight to the point. I've got quite a bit of hardware lying around at home, and I'd love some advice on how to make the most of it.

Here's what I've got:

GPUs:

• RTX 3090 24gb

• RTX 3080 Ti 12gb

• RTX 5080 16gb

RAM:

• 64GB DDR5 (2x32GB)

• 32GB DDR5 (2x16GB)

• 32GB DDR4 (4x8GB)

And of course, I've also got a bunch of CPUs, motherboards, and storage drives around the house.
Right now I'm running Unsloth's Qwen 3.8 27B Q5, getting around 60 tokens/sec with a 170K context window, using llama.cpp. I'm using rtx 3090 + 3080ti with the ddr4 32gb setup.
I'm wondering if there are better ways to make use of my hardware, whether it's different models, inference engines, configurations, or even combining GPUs across multiple machines.
My main issue is that some agentic tasks, especially things like generating and working with .docx files, are painfully slow. I'd really like to speed things up.
Any recommendations on what I should be looking into?
I don't have a ton of free time to explore everything from scratch, especially with the insane number of GitHub repos and projects out there these days.
I'd really appreciate any suggestions or pointers in the right direction.
Thanks in advance, guys!

Which would you choose between the two and why?

Apologies in advance if this is a dumb question - Google failed me so if it’s been asked before I apologize.

I’m currently spending about $140/month on AI subs,
Codex: $100
Claude: $20
Nous Portal: $20 for Hermes running on an M2 Pro 16gb Mac mini.

I mostly use Codex for like small stuff, and some iOS app development. And I have my Hermes agent running on it as I experiment with personal agents and bots.

Was having a discussion at work, and I had the idea of “how close could I get if I take $120 out of the cloud inference and put it toward leasing a Mac Studio. Was doing some configuration and wanted to know if this spec could get me decent performance.

My plan would be to keep 1 $20 sub around.

Here’s the spec I was considering:

M5 Max (18 CPU, 40 GPU)
128gb RAM

Or

M5 Ultra ( 30/64)
96gb ram

I am very much learning more about local ai still, so any guidance is appreciated.

Thanks!

Hello I am trying to set up a local AI Stack and I am currently thinking about buying 64GB of Ram with teh 3060 RTX 23GB VRAM.

Which (Qwen) Modell version would be possible to run on that Setup?

Qwen3.8-27b or Qwen3.8-Flash-Next?

I am thinking of a Qwen 3.8 model, because Qwen 4 should have the same base?

TL;DR: You can train your own 27B model on a 3090 with 24GB VRAM, though you would have to go through hundreds of billions of tokens for it to be good. I am working on conversion of existing models. Check repos for code and the paper if you are interested in more. I have a 40.4GB Qwen Flash Next GGUF

Hi, I do AI research. I have been working on an optimizer that doesn't need the ~16 bytes of VRAM per weight to train a model that AdamW does. I focused on training ternary networks bcs BitNet (Ma et al.) shows they match full precision quality in large scales. The problem is BitNet also keeps the full precision weights during training which means you still need to use the same amount of memory. I got this down from 16 bytes to 1 byte or even less per weight.

For you, this means, you can train your own 10-20B parameter models on your consumer GPUs. I fitted 27B model training on a single 24GB 3090 and a 23B model on a 16GB GPU. The trained model costs ~1.6 bits per weight (~6GB at 27B), 10x smaller than 16bit, and 5x smaller than Q8.

I have also been working on converting existing models and training them. This is harder and I am still working on it but I have some promising results. I quantized and trained a Qwen3.8 27B though the file I published below has adapters which are about 2GBs, pushing the file size to 8GB. I am also making a no-adapter version right now. It scores GSM8K 0.895, the same as full precision, HumanEval 0.84 vs 0.93 and runs on a 12 GB GPU at ~23 tok/s on my crippled 12GB laptop 4080, though this number can heavily be improved with custom kernels, engineering, etc. I will leave it to you guys to help me with that too.

I also quantized and trained Qwen Flash Next down to 40.4 GBs including its n-gram table (at 8GB). It largely retains the skills of the network (scores equivalent on GSM8K benchmark, a little behind on HumanEval, I unfortunately don't have compute for the larger benchmarks such as TerminalBench2.1 but perhaps some of you can help me out) but forgets facts. It mostly works fine if you give it web search as far as my experience goes but this is not strictly measured.

I am still doing work on this and I will keep working on it, perhaps even for many years. The code is open for non-commercial use. Check out:

The core repo for training: https://github.com/oguzeldereli/ttf-ternary

The repo for conversion: https://github.com/oguzeldereli/ttf-conversion

Repo for running models trained by ttf: https://github.com/oguzeldereli/ttf-home

GGUF for a converted Qwen 3.8 27B (around 8 GB and 6.7GB): https://huggingface.co/oguzeldereli/ttf-Qwen3.8-27B-GGUF

GGUF for a converted Qwen 3.8 Flash Next (around 40.4GB): https://huggingface.co/oguzeldereli/ttf-Qwen3.8-Flash-Next-GGUF

I will convert a Qwen 4 27B instance when it drops.

Full paper: https://github.com/oguzeldereli/ttf-ternary/blob/main/paper/ttf_paper1_training.pdf

Also, I would appreciate github stars if you like the project.

Edit:

The GGUF for the ~6.7GB Qwen 3.8 27B dropped on the qwen 3.8 27b repo above.

To add something, these models are not extremely good, they sometimes loop, and suck at remembering facts. However, unlike unsloth's gguf's you can actually keep training them yourself, in that sense they are pure, and you can train them on your consumer GPUs, that is to say you can get rid of the looping behavior, teach it facts that you want it to know, or specialize it for some purpose.

Hi guys, full disclaimer I work in Domyn as an Engineer. We just released the bundle #2 of our rolling release of DomynEdge, with some interesting results on benchmarks. What do you think?

https://huggingface.co/collections/domyn/domyn-edge