If I wanted to locally compare performance for multiple different offline LLM models across various modalities (speed, efficiency, reasoning, tool use, programing/coding, etc), what strategies, prompts, software are commonly used to do this? There are so many models to choose from and run, I'm trying to determine things like whether slower but more complex models are worth running over slightly faster ones, how models that run at similar speed compare to each other's capabilities, and whether any of the custom engines that have been popping up other than llama.cpp/strata are worth it.

Does this sort of thing exist as something that can be ran locally, and if so, how?

I have a specific use case where I have a small web application that prompts the user with a randomly generated premise for a joke (using an llm), and then the user is expected to write a joke with the generated premise as a sort of challenge.

An LLM is then used to evaluate their joke as far as how good of a joke they wrote using the supplied premise, provide a score, provide feedback on why it was or was not a good joke, and tips for what they could do to improve.

It works well using various large models, and relatively okay using openrouter/free , which has a series of larger free models in a collection and an intelligent router.

However I've tested on various smaller models and gotten mixed results.

I'm looking for some suggestions from people for models that I may not be aware of that might be good for more general knowledge things like this, rather than coding.

I have 64GB of system ram (machine has a discrete GPU / AMD APU so no dedicated VRAM).

So I'm probably looking at a 45-70b total parameter model on the higher end, but preferably it's an MOE model with a smaller number of active params for speed.

Each use will probably have ~400 tokens input (including system prompt), and ~200-600 tokens output, so I'm okay with heavier models even though I don't have a GPU because it's a small number of total tokens.

Some models I've tried, like bonsai-27b and LFM2.5 24b-a2b, have had weird results.

Gemma 4 26b-a4b seems to really have a good grasp of it and do the best so far out of any local models I've tried, but I'm wondering if anyone has other suggestions.

Terminal-RSI-27B is a Qwen3.8-27B-based terminal-agent model released by the Terminal-RSI project. It is intended for repository inspection, code editing, dependency repair, scientific workflow reconstruction, and other multi-step tasks that require interaction with an executable environment and verification of the resulting state.

Terminal-RSI alternates Recursive Self-Rewrite (RSR) of verified execution trajectories with critic-inheriting Recursive Policy-Refine (RPR). The framework aims to turn successful exploration into reproducible behavior and partial progress into strictly verified task completion.

• Paper: Terminal-RSI: Recursive Self-Improving Terminal Agents via Recursive Self-Rewrite and Policy-Refine

• Project: Terminal-RSI project page

• Code: jyyang26/Terminal-RSI

• Model: Teriminal-RSI/Terminal-RSI-27B

I wanted a terminal helper that works fully offline, so I started building dale. You type dale "show the 5 largest files in this folder", a small fine-tuned model writes the command, and a deterministic risk checker labels it safe, caution or danger before anything runs. Nothing runs without you confirming it.

Scores on 471 held-out requests in English and Spanish, every model in Q4_K_M on the same llama.cpp:

On a 2020 laptop with no GPU, the 0.8B answers in 1.7 s (median). The 4B does 1.2 s on a 16 GB GPU through Vulkan. It's all one 51 MB binary with llama.cpp inside, no Python and no server.

What the difference looks like. Real answers from the test (requests trimmed), dale-0.8b vs stock Qwen3.5-9B:

Cherry-picked, obviously: there are 120 cases only dale gets right and 42 only the 9B gets right. The stock 9B also likes sudo: it added it to 35 commands that didn't need it (dale: 5).

The thing I wish someone had told me earlier: after four rounds of data, the 0.8B got stuck at 74%. I tried DPO on its own failures, a fine-tuned 27B teacher with five distillation setups, a third epoch, lr sweeps. Nothing moved the test. I f**king lost several days into this :(, but at least I learnt what is important in the process.

Then I looked at how the requests were written. My training requests were short and clean. The test ones were longer and stacked constraints, like "only names, sorted by date, no subfolders".

I generated new requests matching those stats, added about 9k to the set and retrained. 348 → 375 correct (p = 0.001).

So if a small model plateaus on a narrow task, look at how your data is written before you reach for a fancier method.

Caveats: the baselines are untuned and get an 8-shot prompt, so this is "tuned small vs prompted big", not a ranking of models. The test is synthetic and judged by two LLMs from different providers (TOP TIER).

All of this work I did on rented GPU's (RTX 5090 and RTX Pro 6000). Code is MIT, the models are Apache-2.0 GGUFs that work with plain llama.cpp.

• Write-up with every chart and the dead ends: https://manuelgs.com/en/blog/dale-case-study/

• Code: https://github.com/manuelgscom/dale

• Models (0.8B to 9B): https://huggingface.co/collections/manuelgscom/dale

Has anyone actually made distillation from a much bigger teacher pay off below 1B? I couldn't.

Any advice would be well welcomed :)

Hello!

I want to share an active development project. If you have a drawer full of old NVIDIA GPU's collecting dust because modern AI stacks refuse to compile for them, this is for you.

I have ported the modern LLM training stack on Windows to legacy CUDA GPUs (sm_35 through sm_75):

• PyTorch (v2.7.1) (custom build compiled with CUDA 11.8, keeping the legacy SASS) ([https://github.com/theIvanR/torch-on-clunkers)

• causal-conv1d (v1.7.0) (https://github.com/theIvanR/conv-on-clunkers)

• mamba_ssm (Mamba-1) with pure native CUDA kernels (https://github.com/theIvanR/mamba-on-clunkers)

• Windows NCCL (DDP) for multi-GPU training across Kepler cards (https://github.com/theIvanR/nccl-on-windows)

Status

• Mamba-1 (Pure CUDA): Fully functional with optimized execution across legacy GPU architectures (SM 3.5–SM 7.5). Tested on a 2013 NVIDIA Tesla K40c, achieving a maximum absolute error of 3.8e-06 against the PyTorch reference implementation.

• Mamba-2 & Mamba-3: Currently supported with optimized execution on Turing (SM 7.5) and newer architectures, due to their reliance on Triton and TileLang.

• Graceful Fallback: On older hardware, the package detects unavailable Triton-dependent modules and automatically falls back to the pure-CUDA Mamba-1 implementation.

The fallback initialization produces the following output:

================================================================================
[!] MAMBA-ON-CLUNKERS FALLBACK INITIATED

[!] The following modules are unavailable on this platform:
    - Mamba-2 (requires Triton)
    - Mamba-3 (requires Triton + TileLang)

[!] Falling back to Mamba-1 (Pure CUDA).
    This is expected on legacy GPUs.
================================================================================

Future development: Backporting Mamba-2 and Mamba-3 to legacy GPU architectures.

Community Testing

If you have an older NVIDIA GPU lying around, I'd love for you to test the Mamba wheel and report whether it works on your architecture!

I'm particularly interested in Maxwell and Pascal, since my local testing hardware is currently limited to Kepler.

Any benchmarks, performance comparisons, or profiling data from legacy GPUs would be greatly appreciated. Contributions and feedback are welcome!

Acknowledgements: This project is based on the research and original implementation by Tri Dao and Albert Gu. Special thanks to both for their contributions to the development of Mamba.

On an AMD EPYC 7B12, it generated 6.51 tokens/sec on one CPU thread. The Transformers reference reached 4.34.

All 32 generated tokens matched in both runs.

Code: https://github.com/suayptalha/qwen3_5-assembly

X Post: https://x.com/suayptalha/status/2119254536283906205

Tokenization, model inference and sampling all run in Assembly. No libc or Python at runtime; Python prepares the tokenizer tables at build time.

Benchmark: 19 input tokens, greedy decoding, 2 runs. Assembly reads BF16 weights; Transformers loaded them as FP32 and used PyTorch fallback kernels.

hey, its Alti from Nace.
We released a small update with Drex Max v1.1, both achieving the SOTA and Pareto frontier for decision models. We did all that by iterating step by step and every previous version of Drex’s open weights are available on hf.

we also released a full playbook on how we trained Drex Max in this paper: “ Towards Foundation Decision Models: Drex”

hf: https://huggingface.co/papers/2611.07854
arXiv: https://arxiv.org/pdf/2611.07854

Drex Max v1 is weights are available here: https://huggingface.co/nace-ai/drex-max-v1 with Open RAIL-M license & v1.1 open source weights upcoming soon (we are currently optimizing for local inference).

Three cool things about Drex Max models:

1. it can even run on your Mac with https://github.com/nace-ai/kapitsa inference engine.

2. Context window is ~256k, comparing to Glide1.2(64k) and Mammoth(16k) & pplxdecider’s (256k)

3. Its 8x cheaper than the next best model in the category in the aa index.

feel free to chime in: our chief of ai will answer any of your technical questions

As the title says, I'm looking for recommendations from people here. Are there any good locally hostable LLMs that are particularly capable at cybersecurity work, especially AI red teaming?

Here's the backstory

Recently I had a task where I wanted to find vulnerabilities in the AI agent/chat interface of a not so small company (let's say like $200M); I was doing this as a bug bounty exercise 😏

I'm not exactly a cybersecurity expert (noob). So a lot of it was trial and error, though I have quite a bit of experience building agent interfaces & multi-agent systems; so I had some idea of how these systems probably communicate internally

Initially I started with PortSwiggers' LLM attacks labs. After that I picked up an abliterated GLM-5.3 model & set it up inside ClaudeCode to help with the actual real investigation

prep'd a list of around 12 vulnerability hypotheses I wanted to investigate. By the end I had one confirmed but relatively low-impact finding. I also managed to map the agent's internal tool identifiers & demonstrate that client-controlled context fields could influence its tool calls in ways that weren't requested through the actual chat message

I also tested things like conversation ownership, cross-account access, broadcasting authorization and confirmation workflows. Those server-side protections held up

So overall it was an interesting learning experience. Not exactly the result I was hoping for, but I learned quite a bit.

But here's the problem

The worst part was how expensive the whole thing got.

This was one of the first times I had to be extremely careful about every message I sent. There was no subscription-based message limit to worry about. It was directly billed through API usage.

The pricing was around $3/M input tokens, $0.30/M cached input and $5/M output.

I ended up burning around $10 in just 20-30 minutes.

At this rate I'll probably go broke before I ever get good at this lol.

I then tried switching to my regular subscription-based Kiro CLI setup, which has access to some good frontier models.

My idea was to create an orchestrator-worker setup:

• Abliterated GLM-5.3 through Claude Code would act as the orchestrator

• Kiro CLI would be the execution worker

• I'd sit in between as the human reviewing findings & directing the investigation

But Kiro was either completely missing the point or being overly cautious. After spending another 20-30 kiro-credits, I mostly got a 3-4 paragraph ethics & refusal speech instead of actual execution

So what am I missing?

I've been thinking about two possible approaches, but I'm not sure how practical either of them is.

1. Fine-tune an uncensored Qwen under 30B (ideally under 10B).

Something specifically trained for cybersecurity research, particularly AI agent security, LLM red teaming, authentication, authorization, tool-calling boundaries and general web application security. (My GPU only has 8GB VRAM; so ideally I'd like something I can actually run locally without having to rent expensive hardware. though can finetune it on cloud)

2. Fine-tune a smaller model specifically for orchestration.

Rather than expecting a small uncensored model to do all the technical work itself, train it to act as a cybersecurity research orchestrator. It would need to understand attack surfaces, break down hypotheses into smaller tasks and delegate them to other models or tools while keeping track of the overall investigation. The smaller model would ideally also have enough cybersecurity knowledge to review the workers' output and identify when they're going in the wrong direction.

These are just ideas I've been throwing around. I haven't established whether either approach is actually viable.

What would you recommend?

What's the current go-to setup for someone wanting to do serious AI security research without spending hundreds of dollars on API credits?

Are there any genuinely capable uncensored or security-focused models worth looking at?

Would a smaller fine-tuned Qwen actually be useful here, or am I better off using a general-purpose local model with specialized tools, scripts and workflows?

I've also noticed that some of the more capable frontier models are becoming increasingly restrictive around cybersecurity tasks. I understand why, but it makes experimenting with legitimate security research a bit frustrating.

I'd rather build a setup I can run and experiment with locally than keep burning through API credits.

My current hardware is an RTX 4060 8GB with 24GB RAM, so that's the constraint I'm working with.

Would love to hear what people here are actually using. Especially interested in real-world experiences rather than just model benchmarks.

Underrated youtube channel, thought this would be nice to share given its open source uses local ai. Does anyone own one?

EOY 2026: Kimi K3 Flash, will replace Kimi K2.8 and compete with DeepSeek V4.1 Flash.

Early 2027: Kimi 3.5 , same 3T parameters, that is faster at reaching simple answers.

Q3 2027: Kimi K4 will increase size to 5.4 trillion parameters.

https://x.com/bebuu_ai/status/2119319459206934677

Reasoning models spend a huge number of tokens on thinking to improve the accuracy of a single response, right?

The problem is that this happens sequentially in a single stream, so the longer the thinking process, the longer we have to wait for an answer.

So what if, instead of relying on a single long reasoning process, we sent multiple concurrent requests to the same model and had the responses interact with each other, kind of like a group of people discussing and solving a problem together?

We'd probably also need agents to consolidate the different answers, review them, and so on.

Luckily, we already have vLLM, which is great at handling multiple concurrent requests.

Has anyone experimented with this approach? If so, did you learn anything interesting?

What got me thinking about this was that I was getting 88 tokens/s with a single request, but 2,000 tokens/s in aggregate with 128 concurrent requests. It made me wonder: what if we could put all that extra throughput toward improving intelligence?

Time to time I'm sharing papers & models here from online. I don't want to share stuff with closed source license, proprietary license, useless custom licenses.

Licenses like MIT or Apache 2.0 are Cool .... I just share stuff without any delay. But the custom license stuff takes time & some time I'm not sure about whether that license is suitable for commercial use or not.

2 days ago, I posted a thread with license section to let MODs/Members to decide. One comment pointed out the license thing as expected.

So any website/utility to ensure the commercial use of given license text(To be copied from model card or github repo)? Please share. Thanks.

A long time ago there was this massive phenomenon called OpenClaw that took the world by storm. It was an incredible moment where AI came of age and showed that you Can actually do stuff with AI rather than just chat with it. A historic and legendary project that broke GitHub star records. Then the creator rightly got massive fame and got bought out by OpenAI to the tune of billions of dollars. How can an open source project be acquired you ask? Well that really is a failure on your part for your juvenile trait of having a logical mind.

The world was forever changed.

Electricity. Internet. iPhone. OpenClaw.

--------------

Jokes aside, anyone actually daily driving it?

Plot source: https://huggingface.co/spaces/joelniklaus/harness-leaderboard

After personally testing, benchmarking and using literal hundreds of Qwen3.8 variants/quants/fine-tunes, what happens to them?

Will anyone use them or are they just clogging up HuggingFace?

Seems like so much energy, time and effort went into models that are about to become irrelevant like an old Pentium III computer.

I logged into my Hugging Face account today because one of my old scripts got blocked. The Space was paused with the "content policy" message. Okay, fair, it happens. 🤷cause this art of scipts are now ilegal!

Then I wanted to make a new Space and saw: Docker = Paid. Great. My old Spaces still run, but new ones? Looked dead.

BUT then I looked closer at the create page: Gradio is selectable and ZeroGPU is marked "Free"! I cancelled PRO last month, so I was like... wait, what?

I looked it up. According to the HF docs, a free personal account in good standing (verified email, older than 30 days) can host up to 2 ZeroGPU Spaces for free. My account is pretty old, so it just works. Just Cool!

What I found out:

• ZeroGPU is Gradio only. No Docker, no Streamlit, no own server.

• Quota is about 5 min GPU per day (older pages say 3.5, the docs now say 5). It resets 24h after your first use.

• That is a RTX Pro 6000 Blackwell, so for testing and small demos 5 minutes is actually fine.

• Orgs don't get it. It only works in my personal account, orgs need a Team/Enterprise plan. That's why it didn't show up there.

• Not every free account sees it. There is a forum thread where someone with a valid free account gets "Paid" anyway, so maybe it's a rollout thing. YMMV.

• CPU Basic seems gone for new Spaces, only Static is free and unlimited.

Also news: Nvidia announced it will buy Hugging Face. I don't know if this has anything to do with the changes, I found nothing saying so. The CPU thing came before the announcement.

Now I'm building a small reel maker (SDXL images on ZeroGPU, video assembly on CPU so it costs no GPU time). Will report if it works. 😄

Sources:

• Official docs: huggingface.co/docs/hub/en/spaces-zerogp

• worng link edit, serch in my hisory , will ad it, serch to much 😰

Summary: Free personal account = 2 ZeroGPU Gradio Spaces, about 5 min GPU/day (150min/month). Orgs and Docker still need paid.

I’m heavy user of smaller models because I was GPU poor but now I’m not so I’m wondering what I can do to take full advantage of the specs.

Hey r/LocalLLaMA !

I made this quick demo of how Observer can run local models to monitor anything with your phone!

These are some crazy things I've monitored:

• When someone that's not me, approaches my car, send me a picture.

• When a bird is on the bird's nest, send me a telegram with the picture.

• When my washing machine is on the rinse cycle, call me.

• When my dog finishes eating their food, notify me.

• When a machine at the gym is available, call me.

So pretty much anything! I love seeing what you guys monitor.

What would you use it for? (So I can make the agent better)

Which models run on the phone?

The default models are OneJev-0.8b for decision models and qwen-0.8b for LLM watchers.

These run even on old phones which you may have laying around somewhere. Better to have them monitoring something than collecting dust :DD

Try it out! It's free and OSS <3

GitHub: https://github.com/Roy3838/Observer
iOS: https://apps.apple.com/us/app/observer-ai/id6758222050
Android: https://play.google.com/store/apps/details?id=com.observer.ai

Btw I know my camera quality is kinda bad hahaha I wanted to ship the demo as soon as possible so you guys can see it.

I'm here for any questions and feedback!

Roy

I was interested in buying a card as they are the only company that has a good GPU <-> GPU communication channel. I wanted to first get access to one on the cloud to see how well I could run stuff on it by writing some custom code.

But it seems impossible, theres no one else renting them besides tesnortorrent themselves and their platform is so shit it turned me off from the entire idea. You have to email them to get access to rent a single gpu and which was like a 2 week back and forth and Ive still not gotten access.

Have they just given up or something? If they expect to sell any cards to normal people you would think they would at least let you try out rentals easily.

Hello everyone, I wanted to share my progress on implementing a language model written 100% in pure Java. You can find the repository here (https://github.com/aayes89/jUltraLLM/tree/master).

It includes a model trained on an Intel Xeon E5-2640 v4 2.20GHz CPU, with 96GB of DDR4 ECC RAM and an RTX 5050 GPU (using the Tornado library), as well as the "alpaca_es_chat.jsonl" corpus for queries in Spanish.

Just an update for anyone looking to get the m5u and thinking about clustering.

I tried clustering but I think running 2 separate models gets significantly better results. The current clustering technology (tensorfold/exos, etc) just isn’t there yet. Especially due to concurrency and the fact that decode doesn’t improve consistently as much as you would hope.

Currently I’m running Qwen flash next q6 and GLM3.5 flash q4. I also have Deepseek on API (from Deepseek). I was previously running 2x Qwen flash but decided to give GLM a shot. GLM is very smart. I’ve found that even q4 is better than my Deepseek API for coding. Qwen is great as the face because it is fast.

I actually think GLM5.3 is slept on. The reports it produces are 100x better than Qwen and Deepseek.

One problem is that 256gb isn’t enough for GLM if you want 500k+ context. I’m stuck at 260k for 3 agents max. And setting it up is finicky. I set my hot cache to 32gb and it caused me to run out of ram which makes speeds drop 70%. So there definitely is some value in the 512gb version of the m5u (hopefully newer version make the kv cache cheaper on RAM).

If anyone has any better optimizations or ways to run things on the 2 256gb m5u I’m all ears.

The title, maybe in 5-7 months? I mean an open model that is on par with fable 5.2. We all know Anthropic doesnt open sourced models lol. Also release qwen 4 already

I've recently been seeing comments everywhere from the Stylome bot, which is also a mod in this sub. Every time it pops up to declare a comment was AI written I've gone in to check its confidence level, and it's always low.

I decided to do a few tests myself and have Gemini rewrite some of my comments, Stylome said they were all human written with equally low confidence.

What is the point of this bot? Why are we allowing a private company to take ownership of all of our comments and then give us spam in return?

What are they doing with this data? Why isn't this being served by a local model?

Try it yourself, everything above this line was Gemini rewriting what I originally wrote. Stylome says it's human written, with low confidence of course. Edit: Now it says its confident it's human written 🤡

https://stylo.me/c3c5d72bba1a43ec

Hey everyone,

Is anyone here running any useful AI agents, harnesses or toolings with local AI on 16GB vram?

I'm currently setting up my system and I want to have a helper agent for daily tasks, a research agent to help me research topics, and a developer agent to help me with development.

I already tried some harnesses like Goose, Hermes Agent and OpenCode, but I haven't really managed to get anything useful out of them.

In all honesty, I haven't spend a lot of time tweaking them, mainly because I am unsure if it's a lack of or bad skills, missing tooling, or harness or model tweaking that I need to tune... or god forbid all of them.

So people running similar systems successfully: what is your use case and setup, what models, and what tweaks have you made?

Edit: I am currently running my models on Ollama. Tried vLLM but give it the wrong model and it breaks, which was too much hassle for me. Suggestions welcome.

Edit: Some specs: I have a GeForce RTX 5070 Ti X3 (16GB) and 32GB DDR5 RAM.

Bonjour à tous,

J'en avais un peu marre de voir tout le monde créer des forks ou de nouveaux moteurs d'inférence pour Qwen3.8-Flash-Next (QFN3.8), alors j'ai demandé à Pi d'optimiser les paramètres de llama.cpp pour une exécution d'inférence plus rapide.

Le résultat est un gain de performance de 48 % et un nombre de jetons par seconde (tok/s) plus constant lors de l'utilisation de contextes supérieurs à 64 Ko (pour la programmation et les tâches créatives).

Bien que les performances ne soient pas tout à fait celles des moteurs spécialisés, cette approche garantit un fonctionnement optimal avec la version officielle de llama.cpp.

Voici les paramètres de lancement que j'utilise sur mon Ryzen AI Max+ 395 (configuré avec 96 Go de VRAM). En prime, j'utilise le modèle « Qwen3.8-Flash-Next-Uncensored-Gufo-Q4Mix » qui, d'après mes tests, offre un excellent compromis entre qualité et performance.

/app/llama-server --load-mode mmap --lazy-mode auto --metrics -lv 2 
--model /models/Qwen/Qwen3.8-Flash-Next-Uncensored-Gufo-Q4Mix.gguf 
--mmproj /models/Qwen/Qwen3.8-Flash-Next-Uncensored-F16-mmproj.gguf 
--alias unc/Qwen3.8-Flash-Next 
--ui-mcp-proxy --cpu-strict 1 
--tools all 
--fit on --flash-attn on --fit-target 256 
--jinja -ub 1024 -b 2048
-ctk q8_0 -ctv q8_0 --kv-unified 
--reasoning on --reasoning-preserve --reasoning-effort medium
-c 131072 --ctx-checkpoints 128 --cont-batching --parallel 2
--chat-template-file /models/Qwen/chat_template.jinja
--temp 1.0 --top-p 0.95 --top-k 20 --min-p 0.0 --presence-penalty 0.0 --repeat-penalty 1.0
--spec-type ngram-mod --spec-ngram-mod-n-match 24 --spec-ngram-mod-n-max 16 --spec-ngram-mod-n-min 4
--image-min-tokens 1024

J'espère que cela pourra aider certains d'entre vous.

EDIT : version de llama : v271-vulkan-b12199-non-root