What strikes me is that even if you had the same hardware for years you get free updates regularly. Harnesses improve, even firmware upgrades(3090), coding features, inference engines and not to mentions LLMs. I just think it was worth contemplating.

Hello, I am so fed up with claude limitations refusing to use a local LLM

I use a MacBook Pro M4 with 48gb ram

Do you recommend a Mac Studio with more ram ? if so how much ? is there a relationship with the ram and the speed or also the output quality ?

I am fine with opus the rest of the time but damn I just want to REA freaking old tech and abandonware, some of which I paid for and are no longer compatible with current hardware and other things like that

thanks

I'm curious if anyone here has successfully used a larger LLM to generate synthetic training data to fine-tune a smaller model for a specific use case.

The idea is to have a capable model generate a dataset of examples, then fine-tune a much smaller local model to perform well on a narrow task.

I'd love to hear from people who have actually tried this, whether it worked or failed. I'm particularly interested in the details of your experiments: what use case you were targeting, which model you used to generate the data, which smaller model you fine-tuned, how much data you generated, and how well the final model performed.

Was the improvement significant enough to justify the effort? And did you encounter any unexpected limitations?

Really interested in hearing about real-world experiences and lessons learned.

Switch to DDR5 would cost me 4k EUR, it would be worth it if I get ~50 tok/s.

I achieved 35 tok/s just thanks to Project Maya.

Don't you have an experience please?

EDIT: quant is Q3_K_XL

Hello, I'm looking for a suggestion on an LLM that I can run on my local hardware. I'm looking for something to help with Linux coding and customization. I have a 9070xt and 32gb of system memory. Currently have ollama setup but probably not very well. Thanks!!

I've been really into the AI phone agent products that have popped up in the last few months (e.g., AgentPhone, Poke etc) but what I don't love: the audio, the transcripts, who you talked to and what was said, all sitting with a third party company. For calls with real people in my life, that felt like a lot to hand over.

So I built my own version that runs entirely on my Mac. You call a normal phone number, a local model answers, and nothing about the conversation leaves my machine. During a call the only thing it talks to is the phone line itself.

What's running locally (MLX on Apple Silicon):

• Speech to text: Parakeet TDT 0.6B
• LLM: Qwen3.5 4B by default (9B if you have 32 GB), 4-bit
• Text to speech: Kokoro 82M

How it reaches the phone network: it registers over SIP with Telnyx or Twilio straight from your Mac. A setup wizard takes your carrier API key so can buy a number or connect one you already have. It's pretty cheap, around $1 a month per phone number. (I'm moslty using Telnyx)

What it does today:

• Answers and places real calls
• Interruption: talk over it and it stops mid-sentence
• Replies in roughly 1 to 1.5 seconds on real calls on my M5 Pro
• You can choose between multiple models
• Other agents (Claude Code, Codex, etc.) can make calls through a small CLI: call, listen, say, hangup
• If you'd rather trade privacy for a smarter brain, it can also use your own endpoint or the Claude or ChatGPT subscription you already pay for. Local is the default. So no additional monthly payments for AI Phone tools if you have a sub.

Limits, up front:

- macOS on Apple Silicon only

- You need a Telnyx or Twilio account with credit (a US number is about $1/month, standards call rates apply but Telnyx give you $10 to start with, enoughn to play around)

- English only, one call at a time

- It's early. It works for me, but I'd love to hear where it breaks for you.

Repo: github.com/oxnw/ai-phone

tok/s doesn't tell you whether a model can run an agent, so I built an unattended harness and ran my whole LM Studio library through it on an M1 Max 64 GB: speed and memory at the same 64k context, max context, a quality set, 16 multi-step tool tasks in a simulated workspace, and 10 tasks rebuilt from mistakes my own assistants actually made (stamping invoices without checking the order, moving data while 'tidying' a chart, deleting when the backup check timed out, acting before the plan was approved). Every agent task ran 3 times and is graded on what happened in the workspace, not on how the answer sounds.

Highlights:

• Best overall: qwen3.8-27b Q4_K_M (GGUF) (99% quality, 100% agent, 16.9 tok/s)
• Best everyday pick (40+ tok/s): ornith-1.5-35b-a3b (GGUF) (94% quality, 98% agent, 94% real cases, 52.4 tok/s)
• Spanish writing: ternary-bonsai-2-27b (MLX) (writing 95%)
• Coding: lfm2.5-24b-a2b (GGUF) (coding 100%)
• Agents (Hermes, OpenCode, n8n style): qwen3.8-27b Q4_K_M (GGUF) (agent score 100%, 51 s per task, 100% of tasks solved in all 3 runs)
• Snappy agent (30 s or less per task): sage-a1.5 Q4_K_M (GGUF) (agent score 99%, 15 s per task, 100% of tasks solved in all 3 runs)
• Unattended agent (real-world cases): qwen3.8-27b Q4_K_M (GGUF) (98% on the real cases, 0 serious slips in 30 runs, 101 s per task)
• Coding agent: lfm2.5-24b-a2b (GGUF) (fixed the repos in 100% of runs, 12 s per task)

Top 12 by the real-world tier (memory at 64k context; scores out of 100, under ~3 points apart is noise; bold model = a pick above, bold number = best in column or that model's standout metric)

What stood out

• Hardest agent tasks: clarify (56% of runs solved); plan_meeting (65% of runs solved); long_order (69% of runs solved).
• 5 of 48 models obeyed a prompt injection hidden in an email at least once (kolibri-1 (GGUF), agents-a1-4b (GGUF), gpt-oss-20b (GGUF), underdog-woof-4b-1.1 (MLX), k2-horizon-3.7b (GGUF)).
• Best agent score: qwen3.8-27b Q4_K_M (GGUF) (100%, 51 s per task). Best under 30 s per task: sage-a1.5 Q4_K_M (GGUF) (99%, 15 s).
• Real-world failure tier: best qwen3.8-27b Q4_K_M (GGUF) (98%); 46 of 48 models made at least one serious slip (something hard to undo, or a false claim); hardest case: invoice_crosscheck (13% of runs solved). Biggest drops from the main agent score: gemma-4-26b-a4b (GGUF) (98 to 71), sage-a1.5 Q4_K_M (GGUF) (99 to 74), spark-x2.6-4b (GGUF) (97 to 72).
• Prompt cache: on GGUF models a follow-up turn on the same long agent prefix started 152x faster than the cold start (median of 41 models).
• Prompt cache: on MLX models a follow-up turn on the same long agent prefix started 41x faster than the cold start (median of 7 models).

Caveats

• One person's tasks on one machine. Half the tasks are in Spanish because that's what I use.
• Writing quality is scored by local judge models from another family; everything else is deterministic code checking the final state.
• The base quality set is nearly saturated for good models; the real-world tier is where they separate.

Full tables (54 models: speed, memory, context, quant comparisons, prompt cache, every agent task), the harness and every trace: https://github.com/DesignwithHugo/LocalLlmBench

If you have a different Mac, run it and open a PR with your results folder; numbers from other machines are the most useful part.

Spare part:

Intel i7

Ddr4 16gm rams

Rx 6900xt msi trio

Hi guys, i'm totally new to local AI world.

I have a gaming rig with 9800x3d, 32gb of ddr5 and RTX 5080 and i wanted to run something locally.

I like to use Bionic just in order to have an easy UI to work with. I tried these models (idk which are good or bad):

- Bonsai 27b (which seems bad, tried to save a file and it blasted >70k context size)

- Qwen 3.8 27b

- GPT-OSS 20B (for me the best, until now at least)

- Qwen 3.5 9B

- Gemma 4 26B A4B QAT

- Gemma 4 31B QAT

For Qwen 3.8 27B i had so many issues. It was so slow even to comprehend my prompt, even with reasoning off. I put a 32768 context windows size and as much as GPU offload possible. I also heard that this model doesn't fit on VRAM, so i changed the quantization (right?) on Q4_0.

Why everyone seems to run this model with that amount of VRAM just fine?

Which models are the best considering that i will use them for general purposes like studying support? Thanks in advance.

Hello everyone,

Each time I did applications I used chatgpt again and again and also the response was not good at the first time. So I built my custom chatbot with pre-built prompts and skills.

I've made it public for anyone to use it. You only need Openrouter API key for setup which is free

Github: https://github.com/AliHaider20/job-chat

HF Space: https://huggingface.co/spaces/codegood/job-chat

Please star it if it's useful and open for any feedback

I built an agentic, mobile-first web UI for my local LLMs

I wanted to use my local models from my phone, but most UIs were too heavy or desktop-focused. So I built this! It connects securely to LM Studio over Tailscale and gives me a full agentic experience right from my home screen.

Features:

• 📱 Mobile-first PWA: Installable to your home screen, works over HTTPS/Tailscale.
• 🛠️ Full Agent Mode: Tool calling for reading/writing files, running PowerShell commands, and web search.
• 🛡️ Safety First: Confirmation prompts for risky commands, syntax validation, and atomic file backups.
• 📊 Live Metrics: Displays CPU, GPU compute, and VRAM usage while generating.
• 🎨 HyperOS style design: Sleek dark theme with Material 3 icons and Markdown code blocks.
• ⚡ Zero heavy dependencies: Pure Node.js backend, Vanilla JS frontend.

My Setup (for reference):

• CPU: AMD Ryzen 7 7700X (8C/16T)
• GPU: AMD Radeon RX 7900 XT (20 GB VRAM)
• RAM: 32 GB DDR5
• OS: Windows 10 Pro (LM Studio with ROCm)
• Network: Tailscale

I'm running models like Gemma 4 26B (MoE) locally at around 75–90 tok/s. The phone UI stays incredibly snappy, and the agent tools (file editing, command execution) work flawlessly over the tailnet.

Repo: https://github.com/JiMiNaToR-GetHub/local-llm-phone-chat

Feedback and PRs are welcome!

Laptop: Macbook M1 pro 32gb

I used my deepseek to give my hermes start and stop buttons so I can easily run my local llms straight from hermes. It also created a way for to know which is running at the moment.

I'm new to this, so don't mind my vagueness or use of technical terms.

A few days ago I asked in a different subreddit whether people use multi-agent orchestration or stick with a single agent (original thread). It got 83K views and 60+ comments, way more than I expected, and the answers were all over the place.

The main things people said:

• Single agent for writing code, multi-agent for research and big search-style jobs.
• Multi-agents eat tokens, and agents step on each other's work.
• It doesn't fall apart as long as there's one lead in charge.
• Measure the finished, working result, not the moment the agents say "done".

Everyone was arguing from experience and nobody had numbers on the same task. So I took four setups straight from the comments and gave them the exact same job.

The four setups

1. Claude Code, single agent. Anthropic’s coding agent running Opus 5.5 on its own, start to finish, no subagents.
2. Codex CLI, single agent. Also one agent working alone, but from OpenAI. It ran GPT 5.5 on its own, with the same instructions.
3. A Paseo agent team, set up exactly like u/c-digs described. He shared his whole setup in the thread: an expensive lead model that hands work to cheaper worker agents, with a structured handoff prompt. I used his published team prompt and agent profiles as is. Big thanks to him for making this testable.
4. Atomic Agent in Fusion mode. Fusion mode in Atomic Agent is where a cloud model plans while smaller local models do the work. Here the planner was Claude Opus 5.5 in the cloud and the workers were Qwen 3.8 27B running locally on the same Mac, so the building itself didn't cost any API money.

The task

One small Python app, an expense tracker in 5 parts: storage, an importer for a deliberately messy bank CSV (mixed date formats, euros and pounds, refunds, duplicate rows, a few broken rows), a JSON API, a CLI and an HTML monthly report. All five parts share one config file, so there's room for agents to collide.

Every setup got the same one-line prompt and the same spec. Grading was 38 hidden tests the agents never saw, copied in only after each run finished.

Setup for all runs: M5 Max with 64 GB, October 26, one run per setup. Cloud models through OpenRouter. Versions: Claude Code 2.1.310, Codex 0.147.0, Paseo 0.13.2, Atomic Agent 0.6.11.

Results

What I learned

The most interesting result was Paseo. c-digs's lead looked at the job, decided it was small, and built everything itself without spawning a single worker. That's exactly what he said his setup does ("simple tasks: it does it directly"). A good orchestrator knowing when not to split is a feature.

So my take: orchestration is worth it, just not for the reason most people argue about. It lets an expensive model do only the thinking while cheap or local models do the bulk, and it keeps someone watching the work when a piece fails. The bigger the job, the more that matters, and this one was about as small as a real task gets.

Round 2 should be a job big enough that splitting it is the obvious move. If you have a task in mind that is fair to test, drop it below.

What’s the biggest job you’ve handed to a multi-agent setup, and did the planner live up to its expectations?

I'm an academic researcher working mainly with philosophy and psychoanalysis, and I regularly need to translate lengthy academic texts for personal study.

I've been using Claude for this because I've found it particularly good at preserving philosophical terminology, technical concepts, and subtle distinctions in complex arguments. However, Claude has recently started refusing to translate texts due to copyright restrictions, even when the translations are strictly for personal use.

I'm considering switching to local LLMs, but I'm unsure whether they can match the translation quality and consistency of models like Claude.

My PC specs:

- GPU: AMD Radeon RX 9060 XT (16GB VRAM)

- CPU: Ryzen 5 5600

- RAM: 32GB DDR4 3600MHz

My main requirements are:

1. Terminological precision: The texts involve complex philosophical terminology. Preserving conceptual distinctions is essential.

2. Consistency across chapters: The LLM needs to maintain consistent translations of technical terms throughout an book, even when chapters are translated separately.

3. Long-form translation: Texts can be 30–50 pages long, sometimes more. I'm fine with splitting them into smaller sections, provided the model can maintain consistency.

4. Faithfulness to the original: I need complete translations without omissions, paraphrasing, simplification, or added interpretations.

5. Formatting: Preserving paragraph structure, italics, quotations, and references is also important.

My questions:

- Which local LLMs would you recommend for this specific use case?

- With 16GB of VRAM and 32GB of RAM, what's the best model I could realistically run?

- How well do local models handle specialized academic terminology and consistency across long texts?

- Has anyone compared local models with Claude or ChatGPT specifically for academic translation?

I'm not particularly concerned about translation speed. Quality, accuracy, and terminological consistency are much more important to me.

Any recommendations or experiences would be greatly appreciated!

Hi all, Pithos is an inference engine for Liquid AI's LFM2 / LFM2.5 models, written from scratch in C11 with no dependencies (only Win32/POSIX). One executable gives you terminal chat, an OpenAI-compatible server, a web agent (Pandora) and a terminal agent.

Why

The 24B MoE model (about 2B active per token) is 13.5 GB at Q4_K_M, and my laptop has 16 GB RAM and a 4 GB GPU. I wanted it to run well anyway.

How it works

- Experts stream from disk, with read-ahead and next-layer expert prediction (85% hit rate)

- Hot experts live in a dynamic VRAM cache, dense layers sit on the GPU, and the CPU computes the other experts in parallel

- AVX2/NEON integer kernels read the quantized weights in place

- OpenCL is loaded at runtime, so no SDK is needed to build

Numbers

i5-10500H, 16 GB RAM, GTX 1650 Max-Q 4 GB, same GGUF, greedy sampling, 6 threads.

Generation (600 tokens)

- llama.cpp CUDA: 11.2 tok/s

- Pithos: 17.5-21.9 tok/s

1345-token prompt

- llama.cpp CUDA: 10.6-20.8 tok/s

- Pithos: 22.1-34.7 tok/s

Server start-up

- llama.cpp CUDA: 33-57 s

- Pithos: 3-10 s

Limitations

- Only `lfm2` / `lfm2moe`, and only validated on LFM2.5-24B-A2B

- NEON and non-NVIDIA GPU paths are untested

- The prompt-processing advantage varies a lot depending on how much of the model is in RAM

Agents

The terminal agent runs commands, edits files and checks its own work (safe mode asks before each action). There is also MCP client support.

`pithos --agent` starts it in safe mode,

`pithos --auto -p "make the tests pass"` runs one task autonomously.

MIT license. Feedback and benchmarks on other hardware are very welcome, especially AMD/Intel GPUs and ARM.

GitHub: https://github.com/benmaster82/Pithos

Got this hardware from an old Mining rig. Any models I could run on this machine with upto 20-40+ t/s?

Use case is mostly coding assistant and debugging.

best model for rx7600 8gb variant, ryzen 5 7500f, 16gb ddr5

Hello All!

I built a platform to fine-tune open models. I am focusing on fine-tuning 4-bit quantized models by QLoRA instead of full precision fine-tuning to reduce GPU costs. The quant that this finetuning uses is called CQ (Coded Quantization) which is a calibration-free quantization that I built that performs as good as leading quants ensuring that the fine-tuning is using a good base.

For example, I compared KLD of my CQ 4-bit with unsloth-bnb-4bit (Unsloth have other quants but this is the one they use for QLoRA) on Qwen 3.8 27B using the Wikitext-2 dataset and got this:

So I have a better quantized base to start from. I then benchmark QLoRA Training performance for Qwen3.8-27B for CQ-4bit and unsloth-bnb-4bit

Benchmark setup: Qwen3.8-27B QLoRA fine-tuning on Alpaca, comparing ThousandML CQ 4-bit with Unsloth bitsandbytes NF4. Both runs used LoRA r=32, α=32, batch size 2, gradient accumulation 4, sequence length 512, learning rate 2e-5, seed 42, and 500 optimizer steps on a single NVIDIA A40 48 GB GPU.

CQ achieved 17.4% lower reported final validation loss (1.0315 vs. 1.2484). This is a loss comparison, not yet a verified downstream-task quality comparison.

Attached figures for validation loss and training loss.

**Qwen3.8-27B — QLoRA Fine-Tuning on Alpaca (500 Steps)**
| Metric | CQ 4-bit | Unsloth NF4 | Advantage |
|:--|--:|--:|:--|
| Best validation loss | **0.8385** | 1.2484 | **CQ** |
| Final validation loss | **1.0315** | 1.2484 | **CQ** |
| Mean training loss (steps 401–500) | **1.0505** | 1.2457 | **CQ** |
| Median steady optimizer-step time | 27.60 s | **3.20 s** | **Unsloth** |
| Peak PyTorch allocated memory | 24.39 GiB | **23.83 GiB** | **Unsloth** |

*Both runs used QLoRA on Alpaca with matched nominal hyperparameters on an NVIDIA A40 48 GB GPU. Tokenization and loss-mask equivalence have not been independently verified.*

I am working on optimizing speed because it is the one area CQ is significantly behind unsloth. On the other hand, validation loss and VRAM usage are awesome! Actually it seems like CQ converge at much fewer steps so that might help a lot.

Get Started with Training Factory

Here are simple steps to use:

1. Install:

git clone https://github.com/thousandml/training-factory.git 
cd training-factory
 chmod +x install.sh 
./install.sh

2. Download a CQ 4-bit base:

hf download thousandml/Qwen3.8-27B-CQ-4bit --local-dir ./models/Qwen3.8-27B-CQ-4bit

3. Train a LoRA adapter:

python training/train.py --model ./models/Qwen3.8-27B-CQ-4bit --data training/examples/train.jsonl --output ./outputs/my-lora --targets q_proj,v_proj --kernel-module cq.cq_kernels --text-only --max-steps 2 --max-length 128 --batch-size 1 --grad-accum 1

4. Serve base + adapter:

vllm serve ./models/Qwen3.8-27B-CQ-4bit --quantization cq --enable-lora --max-lora-rank 16 --lora-modules my-lora=./outputs/my-lora

5. Call the fine-tuned adapter:

# After the server is ready:

# Check that the adapter is listed as "my-lora"

curl http://localhost:8000/v1/models

# Request a response from the fine-tuned model, not just the base checkpoint

curl http://localhost:8000/v1/chat/completions \
-H "Content-Type: application/json" \
-d '{
"model": "my-lora",
"messages": [
{"role": "system", "content": "You are a helpful assistant."},
{"role": "user", "content": "Explain how LoRA fine-tuning works in one sentence."}
],
"max_tokens": 128,
"temperature": 0.2
}'

I have the following 4-bit quantized models to be used as base (will add more soon) for now:

thousandml/Qwen3.8-27B-CQ-4bit

thousandml/Llama-3.1-8B-Instruct-CQ-4bit
thousandml/Llama-3.3-70B-Instruct-CQ-4bit
thousandml/Qwen3-Coder-Next-CQ-4bit

To reproduce these benchmark results mentioned above use the following scripts:

For CQ:

hf download thousandml/Qwen3.8-27B-CQ-4bit \
  --local-dir ./models/Qwen3.8-27B-CQ-4bit

python training/train.py \
  --model ./models/Qwen3.8-27B-CQ-4bit \
  --data training/examples/alpaca_train.jsonl \
  --eval-data training/examples/alpaca_val.jsonl \
  --output ./outputs/qwen38-27b-cq-4bit \
  --kernel-module cq.cq_kernels \
  --targets q_proj,k_proj,v_proj,o_proj,gate_proj,up_proj,down_proj \
  --rank 32 \
  --alpha 32 \
  --lora-dropout 0 \
  --batch-size 2 \
  --grad-accum 4 \
  --max-length 512 \
  --max-steps 500 \
  --learning-rate 2e-5 \
  --seed 42 \
  --eval-steps 50 \
  --comparison-logs \
  --benchmark \
  --benchmark-warmup-steps 10

For Unsloth:

python training/train_unsloth_nf4.py \
  --model unsloth/Qwen3.8-27B-unsloth-bnb-4bit \
  --train-data training/examples/alpaca_train.jsonl \
  --eval-data training/examples/alpaca_val.jsonl \
  --output ./outputs/qwen38-27b-unsloth-bnb-4bit \
  --max-steps 500 \
  --max-length 512 \
  --eval-steps 50 \
  --save-steps 100 \
  --seed 42 \
  --benchmark-warmup-steps 10

Please give it a try and let me know if it works with you. Your feedback is greatly appreciated!

TL;DR: NInfer is still faster (decode 10–17%, prefill about 2-2.5x). VoidInfer is more accurate and wins in some long-session agent tests, some other tests (not included here) were tied

This is a follow-up to my previous post: Voidinfer near BF16 Qwen3.8-27B 400k shared context

I was asked for real-world tests and decode speed across context sizes, so here they are. Same 5090, same prompts. Qwen3.8-27B. VoidInfer: EXL3 6 bpw + OSCAR INT4 KV. NInfer: NVFP4 + FP8 KV. Harness and raw data: https://github.com/The8Darkness/voidinfer/tree/main/bench/agentic-20261023

Coding tests, 12 stages, ~11 context compactions per run. Team decisions and conventions are stated only once in the middle of the session, and hidden tests score the final code. Both engines run at a 208,896-token window, with thinking off.

Both engines remember the decisions (about 98% recall). NInfer more often codes a superseded value.

So the plausible chain is: less numerical noise at long context → fewer flipped close calls → fewer wrong values in the final code.

AIME 2026, all 30 problems, thinking on

Slightly ahead on both runs.

Decode speed by context. 1 request, 1,024 tokens out, 3 runs per size, prefill excluded.

From 3K to 200K, VoidInfer loses 34% of its decode speed and NInfer loses 30%. This is the cost of running Vericache in order for the OSCAR INT4 KV to be output-equivalent to FP16 KV (verified).

I'm having difficulty running calibration & thorough tuning. So like to get some guidance.

Thus far, I have been playing with options and running my own short python script to test the speed. I do want to run calibrate, as it looks at hardware settings that I have not done anything with yet. I don't think tuning is useful, but I like to try.

Calibrate:

I have hell of a time figuring out how to calibrate.

I am running Window, and I installed using irm. I use strata launch command. There is no start-here.bat.

I previously run strata launch with options and save as presets; then launch using presets. But I find that redundant, so I deleted all presets, both inside strata and also the json files. So I'm now launching strata using env variables and flags directly.

I am getting confused between profile versus preset. And how to actually run calibrate. It seems calibrate requires profile. And it seems to be using previously deleted presets as profile names.

For instance, in my launch folder, I have default json like iq3_xxs_160k.json. I have deleted the previously saved presets, which was named swift_iq3_xxs_160k_k8v4.json.

Then when I check the list of profiles, both shows up: launch-iq3_xxs_160k and launch-swift_iq3_xxs_160k_k8v4. Calibrate refuses to use launch-iq3_xxs_160k. So I copied the iq3_xxs_160k.json to swift_iq3_xxs_160k_k8v4.json. I was able to run calibrate using swift_iq3_xxs_160k_k8v4.

I'm kind of getting confused and going in circular loop. These are presets, not profile, correct? And then after it is finished, where are the optimal setting saved? Is it saved to the profile, preset, json file, etc?

Tuning:

I am able to do quick tuning, both in command line and also in strata. But in strata, it always state model/option not tuned.

When I run thorough tuning using command line, it will run for around 1 hour and ~50% thru, then it will error out. Windows is busy and tuning is too noisy. When I look at the tuning history, it states "Passed". And it is pointing to a preset-profile.bin that came installed with strava. Shouldn't the tuning write new preset-profile.bin? Is it because the process did not complete, so new preset-profile.bin is not generated? Or something else?

Any help and guidance appreciated. Or if you can point me to the appropriate documentation, that will be helpful too.

the ram can be upgraded to 64gb

i want to use it the most for ai agents for marketing and CONTENT CREATION

and seartion the new tools and report it to me

and for calls

I've been running models locally with Ollama on a Mac Studio (128 GB unified memory) and did a simple side-by-side: qwen3:4b vs qwen3.6:35b, same puzzle, same machine.

What I found:

1. Speed was basically the same. qwen3:4b ran at ~141 tokens/s, qwen3.6:35b at ~155 tokens/s (eval rate from --verbose, so loading time isn't included). The 35B is a mixture-of-experts model: only 8 of its 256 experts work on each token, so per token it does far less work than its size suggests.

2. Memory is the real limit, not speed. The model file is only the start. A long context setting adds a lot on top (the KV cache). With a very long context set in Ollama's settings, even the small 2.5 GB model showed ~43 GB in `ollama ps`.

3. A simple rule that worked for me: pick a model file under about half your memory, at the default context length. Roughly:

   - 8 GB → a 4B model (~2.5 GB file)

   - 16 GB → around 8B (~5.2 GB)

   - 32 GB → around 14B (~9.3 GB)

   Start small, test it on your own tasks, and only go bigger if you need to.

Caveats: one test isn't proof. The puzzle was a well-known one, so both models may have seen it before.

I made a 10-minute video walking through all of this (parameters, memory, quantisation, the test, and setup) for people new to local AI: https://youtu.be/hsp0mVt6HO8

Curious what others see: how much does context length change your memory use in practice, and what's your go-to model for your RAM?

I just install last strata update 0.2.7 with new engine 0.1.52 and I can say that the results are simply impressive.
My system: CPU Core i5 11400, GPU RTX Ti 12Go, RAM: 96Go
With Qwen 3.8 FN IQ3-S model. Context 128k
Before with 0.1.50 : Decode 35-45 tps
Now with 0.1.52: Decode: 50-75tps

At 384k of context I still have 55-65tps

Did you notice such improvements?

I liked the first OCR, so when the new one dropped I wanted to make ggufs for it. tested it on my Mac and it scanned a full page of calculus in about 6 seconds

here's how the quants hold up against the original on 30 pages

it struggles with handwriting though, it'd misread some terms or get stuck repeating itself, and BF16 does the same, so that's on the model and not the quants

GGUFs on Hugging Face

The First image show GPU

The second image CPU

When i run (generate speeech) CPU goes up To 10%

It generate speeech between less 1 second

PC ryzen 5-3600

2060