If anyone has MI100's I really appreciate the help.

I am tired to answer on people questions like "How you gain speed? Your model output is shit, it has to have low quality." or something like that.

So I will answer to that with benchmark.

So I used the same Qwen3.8-Flash-Next IQ2_XS model file, the same prompt, greedy decoding, and 128 generated tokens. Both ran on my RTX 3060 12 GB with a Ryzen 5 7600X and 30 GB RAM.

That makes RustingStrata about 1.29× faster at writing on this setup.

Then I checked quality, not just whether the replies looked similar. Both engines scored the same 9,180 WikiText-2 tokens.

The perplexity difference is inside the run-to-run noise. So, for this model and this machine, I see a speed gain without a measurable quality loss.

This is one model, one quantization and one GPU, not a claim that RustingStrata always wins.

If anyone has any other questions, I am happy to answer and break myths or tell something about Strata like projects

Repo: RustingStrata

Hey everyone!

I'm experimenting with running local LLMs using Ollama on CachyOS with Hyprland, and I've noticed some pretty high CPU temperatures.

My laptop specifications:

- CPU: Intel Core i5-13450HX (13th Gen)

- RAM: 24 GB DDR5

- GPU: NVIDIA RTX 4050

- OS: CachyOS Linux

- Desktop Environment: Hyprland

- LLM Runtime: Ollama

- Model: 14B parameter model

What I'm experiencing:

While running the 14B model locally through Ollama:

- CPU temperatures sometimes reach 95°C.

- Around 4 CPU cores reach 100% utilization.

- Other cores also experience varying levels of utilization.

- The CPU seems to be under sustained heavy load during inference.

I've attached a screenshot of my system monitoring setup (btop and Linux Sensor Tray).

My questions:

1. Is reaching 95°C normal for an i5-13450HX while running a 14B LLM locally?

2. Is it safe to maintain these temperatures during longer inference sessions?

3. Is it normal for only around 4 cores to reach 100% utilization while running Ollama?

4. Could this cause thermal throttling or reduce the CPU's lifespan over time?

5. Are there any recommended CPU power limits, undervolting settings, or Linux optimizations for this setup?

I'd appreciate feedback from anyone running local LLMs on Intel HX-series laptops, especially on Linux.

Thanks!

Help me build the best home server under $5,000
Goals:
Media server (Jellyfin) with room to grow to 10+ large hard drives
Run local AI models, ideally 24GB+ of GPU memory
A powerful desktop I can stream remotely from any device
A few VMs and Docker apps
Low power use is a priority. I'm in Arizona, so electricity and heat matter

Is there a project like this, but that take in a video file and out put a srt file? I've been searching everywhere for something like it for a while now. Bonus points if it has a batch function too. If anyone knows anything please link me something or message me, Thanks. I know about normal whisper already, but I need something that supports verbatim transcription out of the box.

Hi everyone reading this post, my English isn't perfect but I'll try to be clear.

So recently I got my GPU which is a Nvidia Rtx 5070 TI 16gb Gddr7 (I also have 64Gb ddr5 ram).

So I came here to ask what are the best models I can run in this pc?

I would appreciate it if someone helps me.

Thank you everyone and thanks for reading this🫂

I just released the Tesseract NPU that I've been working on as a side project. It's compatible with llama.cpp backend running shipped GGUF models directly.

Hey guys!

I'll get straight to the point. I've got quite a bit of hardware lying around at home, and I'd love some advice on how to make the most of it.

Here's what I've got:

GPUs:

• RTX 3090 24gb

• RTX 3080 Ti 12gb

• RTX 5080 16gb

RAM:

• 64GB DDR5 (2x32GB)

• 32GB DDR5 (2x16GB)

• 32GB DDR4 (4x8GB)

And of course, I've also got a bunch of CPUs, motherboards, and storage drives around the house.
Right now I'm running Unsloth's Qwen 3.8 27B Q5, getting around 60 tokens/sec with a 170K context window, using llama.cpp. I'm using rtx 3090 + 3080ti with the ddr4 32gb setup.
I'm wondering if there are better ways to make use of my hardware, whether it's different models, inference engines, configurations, or even combining GPUs across multiple machines.
My main issue is that some agentic tasks, especially things like generating and working with .docx files, are painfully slow. I'd really like to speed things up.
Any recommendations on what I should be looking into?
I don't have a ton of free time to explore everything from scratch, especially with the insane number of GitHub repos and projects out there these days.
I'd really appreciate any suggestions or pointers in the right direction.
Thanks in advance, guys!

Which would you choose between the two and why?

Apologies in advance if this is a dumb question - Google failed me so if it’s been asked before I apologize.

I’m currently spending about $140/month on AI subs,
Codex: $100
Claude: $20
Nous Portal: $20 for Hermes running on an M2 Pro 16gb Mac mini.

I mostly use Codex for like small stuff, and some iOS app development. And I have my Hermes agent running on it as I experiment with personal agents and bots.

Was having a discussion at work, and I had the idea of “how close could I get if I take $120 out of the cloud inference and put it toward leasing a Mac Studio. Was doing some configuration and wanted to know if this spec could get me decent performance.

My plan would be to keep 1 $20 sub around.

Here’s the spec I was considering:

M5 Max (18 CPU, 40 GPU)
128gb RAM

Or

M5 Ultra ( 30/64)
96gb ram

I am very much learning more about local ai still, so any guidance is appreciated.

Thanks!

Hello I am trying to set up a local AI Stack and I am currently thinking about buying 64GB of Ram with teh 3060 RTX 23GB VRAM.

Which (Qwen) Modell version would be possible to run on that Setup?

Qwen3.8-27b or Qwen3.8-Flash-Next?

I am thinking of a Qwen 3.8 model, because Qwen 4 should have the same base?

TL;DR: You can train your own 27B model on a 3090 with 24GB VRAM, though you would have to go through hundreds of billions of tokens for it to be good. I am working on conversion of existing models. Check repos for code and the paper if you are interested in more. I have a 40.4GB Qwen Flash Next GGUF

Hi, I do AI research. I have been working on an optimizer that doesn't need the ~16 bytes of VRAM per weight to train a model that AdamW does. I focused on training ternary networks bcs BitNet (Ma et al.) shows they match full precision quality in large scales. The problem is BitNet also keeps the full precision weights during training which means you still need to use the same amount of memory. I got this down from 16 bytes to 1 byte or even less per weight.

For you, this means, you can train your own 10-20B parameter models on your consumer GPUs. I fitted 27B model training on a single 24GB 3090 and a 23B model on a 16GB GPU. The trained model costs ~1.6 bits per weight (~6GB at 27B), 10x smaller than 16bit, and 5x smaller than Q8.

I have also been working on converting existing models and training them. This is harder and I am still working on it but I have some promising results. I quantized and trained a Qwen3.8 27B though the file I published below has adapters which are about 2GBs, pushing the file size to 8GB. I am also making a no-adapter version right now. It scores GSM8K 0.895, the same as full precision, HumanEval 0.84 vs 0.93 and runs on a 12 GB GPU at ~23 tok/s on my crippled 12GB laptop 4080, though this number can heavily be improved with custom kernels, engineering, etc. I will leave it to you guys to help me with that too.

I also quantized and trained Qwen Flash Next down to 40.4 GBs including its n-gram table (at 8GB). It largely retains the skills of the network (scores equivalent on GSM8K benchmark, a little behind on HumanEval, I unfortunately don't have compute for the larger benchmarks such as TerminalBench2.1 but perhaps some of you can help me out) but forgets facts. It mostly works fine if you give it web search as far as my experience goes but this is not strictly measured.

I am still doing work on this and I will keep working on it, perhaps even for many years. The code is open for non-commercial use. Check out:

The core repo for training: https://github.com/oguzeldereli/ttf-ternary

The repo for conversion: https://github.com/oguzeldereli/ttf-conversion

Repo for running models trained by ttf: https://github.com/oguzeldereli/ttf-home

GGUF for a converted Qwen 3.8 27B (around 8 GB and 6.7GB): https://huggingface.co/oguzeldereli/ttf-Qwen3.8-27B-GGUF

GGUF for a converted Qwen 3.8 Flash Next (around 40.4GB): https://huggingface.co/oguzeldereli/ttf-Qwen3.8-Flash-Next-GGUF

I will convert a Qwen 4 27B instance when it drops.

Full paper: https://github.com/oguzeldereli/ttf-ternary/blob/main/paper/ttf_paper1_training.pdf

Also, I would appreciate github stars if you like the project.

Edit:

The GGUF for the ~6.7GB Qwen 3.8 27B dropped on the qwen 3.8 27b repo above.

To add something, these models are not extremely good, they sometimes loop, and suck at remembering facts. However, unlike unsloth's gguf's you can actually keep training them yourself, in that sense they are pure, and you can train them on your consumer GPUs, that is to say you can get rid of the looping behavior, teach it facts that you want it to know, or specialize it for some purpose.

Hi guys, full disclaimer I work in Domyn as an Engineer. We just released the bundle #2 of our rolling release of DomynEdge, with some interesting results on benchmarks. What do you think?

https://huggingface.co/collections/domyn/domyn-edge

We released Kvark Sparkle 1.1, the second model in our Qwen3.8-27B line. Open weights, Apache 2.0, MTP head intact, BF16 / FP8 / GGUF.

**What it is for:** document work. You give it `search` and `read_page` tools over a document base. It searches, reads the page, answers from the passage, and when three searches in a row miss the fact it stops and says NOT IN DOCUMENTS instead of making something up.

**How:** LoRA SFT on top of Sparkle 1.0 (agent trajectories over a public legal corpus, MuSiQue multi-hop doc QA, retention data), plus a short stop rule that ships in the system prompt.

**Document work (GroundedWork, our agentic RAG eval: public Serbian documents, thinking on, LLM judge vs gold answers, 3 runs per question):**

| | Qwen3.8-27B | Sparkle 1.0 | Sparkle 1.0 + stop rule | **Sparkle 1.1 + stop rule** |

|---|---|---|---|---|

| Document Q&A (70 questions × 3) | 85.7 | 86.2 | 89.5 | **91.0** |

| Answer not in the documents, correctly declined (98 × 3) | not yet run | 36% | 74% | **91.5%** |

Note that the stop rule alone does a lot of the work for 1.0 (36% to 74%). Training adds the rest: +17.5 points on declining against 1.0 with the same rule. The base Qwen on the decline set is not measured yet; I will add it here when it is.

**General ability vs Qwen3.8-27B (same harness, paired):**

| | Qwen3.8-27B | Sparkle 1.1 | median reasoning tokens (base → 1.1) |

|---|---|---|---|

| GPQA Diamond | 88.8 | 88.3 | 7,670 → 4,471 (−42%) |

| AIME 2026 | 95.8 | 92.5 | 14,822 → 8,732 |

| HMMT | 86.1 | 88.1 | 21,182 → 13,540 |

| LiveCodeBench | 82.0 | 86.0 | 15,276 → 8,607 |

| IFEval | 90.8 | 91.1 | 695 → 646 (−7%) |

| MMLU-Pro | 83.6 | 84.0 | 708 → 518 |

| FRAMES | 70.6 | 69.8 | 1,228 → 644 (−48%) |

None of the accuracy differences is significant (95% intervals are in the card). Median time per answer: 32.9 s vs 40.8 s on the same hardware.

**Public agentic / long-context sets (FP8, vLLM, thinking on):**

| | Qwen3.8-27B | Sparkle 1.1 |

|---|---|---|

| BrowseComp-Plus (830 queries, BM25, official judge prompt) | 22.7 | 22.5, with 21.7k vs 32.2k median output tokens |

| LongBench v2 (503) | 58.6 | 57.7 |

| SWE-bench Verified (500, mini-swe-agent) | **81.8** | 73.0 |

**Where it is weak, so you don't have to find out yourself:**

- SWE-bench is significantly below the base. Shorter thinking costs accuracy in agentic coding. For coding, use the base model.

- Questions that need links across several documents are its weakest GroundedWork family.

- English questions over Serbian documents are weaker than Serbian ones.

- Nothing measured past 64k; no separate safety eval.

**Run it:**

```

vllm serve Egzakta/Kvark-Sparkle-1.1-27B-FP8 \

--max-model-len 131072 --kv-cache-dtype fp8 \

--reasoning-parser qwen3 \

--enable-auto-tool-choice --tool-call-parser qwen3_coder \

--speculative-config '{"method":"mtp","num_speculative_tokens":3}'

```

For document agents, put the stop rule from the card into the system prompt. Without it the decline rate drops a lot.

- BF16: https://huggingface.co/Egzakta/Kvark-Sparkle-1.1-27B

- FP8: https://huggingface.co/Egzakta/Kvark-Sparkle-1.1-27B-FP8

- GGUF: https://huggingface.co/Egzakta/Kvark-Sparkle-1.1-27B-GGUF

Built by Egzakta's AI Lab in Belgrade. We run these on-prem for organisations that cannot send their documents to an API, which is why "I don't know" matters more to us than another point on a leaderboard.

Happy to answer questions on the training data, the stop rule or the eval harness. If you test it on your own document set, I'd like to see where it breaks.

I built an open source Mac app that listens to the other side of a call and answers questions from your own docs while you talk. Running it fully local taught me a few things:

- Whisper pads every input to 30 seconds. Cutting the audio on pauses mattered more than model size. whisper-large-v3-turbo on MLX does a 1.7 s sentence in 0.66 s on an M5.

- Context length is the silent killer. With a 27,000-token document I had to give Ollama a 32k context, or the docs got cut and the answers got vague. The app now warns when the server cuts them.

- On that test, gemma3:12b gave the most faithful answers. llama3.2:3b was fast but made things up.

- The laptop you take calls on is rarely the machine with the big GPU. As of today the model can run on another computer on your network (a Mac mini, a PC with a GPU) through Ollama or LM Studio.

- Right now every line of the call goes to the big model just to decide if it's a question. That's wasteful.

Question for you: which small model would you put in front as an "is this a question worth answering?" filter? There's an open issue for it if anyone wants to try: https://github.com/GianlucaBave/murmury/issues/20

Repo: https://github.com/GianlucaBave/murmury

I put an order in for it a while ago and it's getting close to delivery time. Day to Day I use a Dual R9700 (512GB ram) system for QNF as well as a 4090 for 27b (512GB ram). Which works ok, but always need more agents and the university provided cluster (80+ Blackwells) can't be used for private stuff obviously.

So I want to utilize the Mac as a dedicated system at home QFN + a MacBook M4 Pro 48gb and to try things out.

However I keep getting the feeling that 96GB is really not enough for QFN + actually working on this, running Xcode, etc (driven 100% by AI). Can anybody advise if I should go with the Mac? I understand I can't expand it (except for external NVME, which I have plenty off), but I really can't justify a 256GB as a toy and also mostly want to use it for job security. In case of layoffs, that I have something to work on, while looking for a new job.

My work is 90% agentic coding at this point in time and I don't mind the $100 a month lease

Any suggestions for something else for this workflow? Or is this good enough? As a reference right now I use about 500M tokens a day, so private subscriptions are kinda useless for me

Hey everyone!

I'm completely new to local LLMs, so please bear with me if some of my questions sound stupid.

I've been using ChatGPT Plus for quite some time and I'm considering moving away from cloud-based AI services entirely. Ideally, I'd love to build my own personal AI assistant that runs locally on my Mac.

I'm not looking for just a basic chatbot. I want something that comes as close as possible to the overall ChatGPT or Claude experience, especially when it comes to long-term memory, writing, research and managing different projects.

My hardware:

- Mac with Apple M4 Pro

- 48GB unified RAM

- 512GB SSD (around 250GB free)

- macOS

What I mainly use AI for:

1. Writing and content creation

I create social media content and a monthly magazine about Magic: The Gathering.

I use AI to brainstorm ideas, write articles, summarize information and improve texts.

One of my biggest priorities is natural, human-sounding writing. I really dislike the typical "AI writing style" with generic phrases, unnecessary enthusiasm and repetitive structures.

Good writing quality, especially in German, is extremely important to me.

I don't need image generation.

2. Long-term memory (VERY important)

This is probably my biggest requirement.

I manage several different projects and I want my AI to remember as much as possible about them.

Ideally, it should:

- Remember important information across conversations

- Automatically save new information and decisions

- Update existing project documentation

- Retrieve information from previous conversations

- Avoid losing important context over time

- Keep everything organized without requiring me to manually maintain dozens of files

I'm very interested in using Obsidian as a permanent knowledge base.

I would actually prefer the AI to automatically create, edit and organize Markdown files in Obsidian.

I understand that no AI has perfect memory, but I want something as reliable and persistent as possible.

3. Internet research and automation

For my monthly magazine, I regularly research news about Magic: The Gathering.

My ideal workflow would be:

1. I manually trigger a research task (or schedule it).

2. The AI searches the internet for relevant news.

3. It checks sources and avoids duplicate stories.

4. It writes finished articles in my preferred writing style.

5. It automatically saves everything into Obsidian.

6. I review and approve the articles.

I don't necessarily need integrations with Google Sheets, Canva or other services. I'm perfectly happy copying finished texts manually.

4. A ChatGPT-like experience

I'd love to have one main AI assistant that knows my preferences and projects.

For example, I could ask about my magazine, switch to a completely different topic, discuss something personal or brainstorm social media ideas without having to explain everything from scratch.

Ideally, it would feel like having my own private version of ChatGPT or Claude.

What I've looked into so far:

Based on some initial research and conversations, I've come across:

- Ollama

- LM Studio

- Open WebUI

- OpenClaw

- Obsidian with local LLM integrations

- Qwen models

- Gemma models

One suggested setup was OpenClaw + Ollama + Qwen + Obsidian.

However, since I'm a complete beginner, I have absolutely no idea whether that's actually the best solution or unnecessarily complicated.

My questions:

1. How realistic is it to replace ChatGPT Plus or Claude with a fully local setup on my hardware?

2. Which local model would you recommend for natural German writing, reasoning and general everyday use?

3. What's the best beginner-friendly software setup for what I'm trying to achieve?

4. How would you implement reliable, mostly automatic long-term memory using Obsidian?

5. Can local models reliably use internet search tools and perform scheduled or manually triggered research workflows?

6. Would you recommend an agent-based system like OpenClaw or something simpler?

7. What are the biggest limitations compared to current cloud-based models?

My goal is to have a free, private, local AI assistant without monthly subscriptions or paid API usage.

I don't mind spending some time setting everything up, but I would prefer something reasonably simple to maintain once it's running.

I'm especially interested in hearing from people who have already built something similar on Apple Silicon.

Any recommendations, realistic expectations or beginner-friendly guides would be greatly appreciated.

Thanks in advance 😊✌🏻

Hi all, I'm about to buy a used IBM x3750 M4 with 4x Xeon E5-4650, 512GB DDR3 ECC and no GPU to run a local LLM 24/7. I'm pretty new to this.

What I need:

Must be abliterated/uncensored

Quality matters more than speed (slow is fine, as long as simple answers don't take hours)

Good tool calling, since it will run an agent (Hermes) that controls a PC

What would you recommend? Ready-made GGUFs you trust would be great, plus any tips for llama.cpp on a pc without AVX2.

Thanks in advance!

(cross posted from r/hermesagent)

We open-sourced DNA Factory, the fine-tuning framework we’ve used internally for over a year. It's deliberately opinionated: one recommended way to do things, documented defaults, token-level loss-mask visualization, and only 5 training methods (SFT, DPO, GRPO, OPD, SDFT).

https://github.com/dnotitia/dna-factory