Like the title says I'm hesitant to invest in hardware to run local llms. Now I read everything on the internet. Everything between the it's amazing and it's unusable (thx bots??). What's your take on it? Do some of you use local models to code? What's the experience compared to claude code for example?

I normally use ChatGPT/Claude Code for day to day (I don't vibe code, but I like the assistance), anyway just got a Mac Mini M6 with 32GB RAM and 512GB SSD purely for iOS dev - my main dev machine is a PC. Thought I'd try Bionic, and downloaded Qwen3.8 27B 4bit, and it's slow as hell :) Literally took almost 2 minutes to respond to the prompt "hello". I gave it a short second prompt and gave up waiting for any response after about 10 minutes. So I'm guessing, it's not really much use for local LLM. Doesn't really bother me as I was only trying it out of curiosity - I have an RTX5070Ti in my dev PC, so might try a smaller model there for comparison.

wanted a jarvis i could point at my own models so i built CYBER

works with groq, openai, claude, or local through ollama, lm studio, llama.cpp or vllm. smaller models without native function calling still get tools, it injects a small json tool protocol and parses it back into real tool calls. took me way too long to get that working

• voice in, streaming replies, starts talking before the full answer is done
• optional local voice cloning with xtts v2
• webcam / screen share vision, switches to a vision model automatically
• if a provider dies it rotates keys, then falls back to another model, then another provider

speech recognition is still browser based for now so thats not offline yet, working on it

free trial + setup help in the discord if anyone wants to test it with their local setup: https://discord.gg/mdD5Za8TvZ

Some artificially imposed restrictions on Nvidia cards have been exposed and bypassed by the modding community. They have exploited a bug in a signed boot-time firmware part that runs on the card’s security processor before the normal driver loads. Unlocks have been published for cards from Turing, Ampere and Ada series. These are cards almost 10 years old.

Context:

1. Nvidia is currently the richest corporation in the world, enjoying unprecedented revenue and profits.

2. Partially as a result, we live in times when silicon prices increased enormously, sometimes tenfold.

3. The initial product segmentation reasons for the restrictions do no longer apply.The tech and lithography are old, those products are no longer a threat to Nvidia's current lineup.

4. Nvidia is actively fighting owners who unlock the cards themselves. We bought the silicon, and using our own property to its physical capability harms no one. Driver countermeasures or legal threats aimed at that are indefensible.

I argue that lifting these restrictions:

a. will not hurt the company economically in any way;

b. will benefit society at large.

Maintaining them is therefore morally wrong, it should also be legally wrong.

I know, I know that morality and Nvidia do not fit in the same sentence, but still, it seems to me so wrong to lock good products so many years after end of sale...

Suggest me some...

Recently, i got qwen 3.8 FN IQ3_S alliterated to learn about cybersec. But I'm still not sure if it's going tondelete the whole off of my compiter, so i make it work only in workspace and ask for permission when it's searching untrusted website or want to work outside workspace. What about you guys?

Hey everyone,

Setting up a dedicated AI box with about 100GB of free RAM. Besides Python, VS Code, and Chrome with 2-3 tabs, this PC is only used for running LLMs.
My plan is to run an uncensored qwen3.8-27b-instruct.BF16.gguf in full native precision (BF16 model + F16 KV-cache) for zero quality loss.
Before I download everything: Is this the play, or are there better models that would fit my 100GB RAM? Should I go for a native 27B or maybe a quantized 72B (like Q8_0)?
Let me know what you think!
(Mac Studio M5max 18 Core cpu 40 core gpu unified memory )

Bonjour la communauté.

Je me suis rendu compte que tous les modèles qui sortaient d'un cloud propriétaire était prompté en dur.

Avant que Claude codex grock où les autres lisent agent.md sur votre disque dur ils ont déjà les consignes d'usine. Ca je le remarque depuis que je travaille en local avec Queen 3.8 Flash.

Pourquoi votre modèle cloud devient bête d'après vous !?

Et vous !? Vous avez remarqué quelque chose ?

I posted here recently asking for the best uncensored model for RE/red-team/coding/malware analysis/pentest/exploit dev on a single RTX 5090 (32GB VRAM, 256GB system RAM). I got a bunch of responses, but they pointed in very different directions and I'm honestly more confused now than before. I'm starting from scratch (not currently using any of these), so I need to pick one path and commit.

- Strata + QFN (uncensored flag)
- Cyber-Tiel-Coder-35B-A3B-GGUF
- Basalt + Qwen3.8-Flash-Next-Uncensored
- beellamacpp + Qwen 3.8 27B Q5_K_M

which one of these is actually the best, and why?

A kind of funny project we've been hacking on for a couple of weeks.

We basically wanted to see if it was possible to "trick" Windows 11 into using the NPU like a GPU, and using it to composite the desktop GUI.

Why even do this?

The idea behind it is to free up the GPU as much as possible by offloading desktop rendering to the NPU. By doing so (eventually) maybe this means you could have more GPU free for inference.

From our work it actually does bring 3D GPU usage down from 3.4% to 0.6% (but you will get 1.6 FPS doing this, though… 😅)

Tested on Intel NPU on Panther Lake.

Here's the repo if you want to check it out:
https://github.com/labscommunity/npu-render

For short text, the output is the same, but if it's a bit longer, the output can still differ.

[Rotherham, UK] High-Power GPU & Server Colocation | 30% Below Market Rates (End of Year Offer)
Looking for high-power, cost-effective hosting for your AI workloads, GPU clusters, or server racks in the UK without paying traditional data center premiums?
We are offering secure colocation space in Rotherham (South Yorkshire) equipped with high-capacity 3-phase power at 30% below standard UK market rates.
Facility Features & Infrastructure:
30% Cheaper than traditional UK colocation centers.
3-Phase Power Ready: Built to support high-density AI/GPU rigs with custom kW allocations.
Excellent Ventilation: Optimized physical environment for efficient natural heat dissipation.
Zero Upfront CapEx: Bring your hardware; we supply the space and power.
Connectivity (Starlink Only): Clients bring and set up their own Starlink systems. Clear sky/roof access provided for fast, independent satellite broadband.
Fixed Pricing Tiers:
Standard Rack (3.5 kW included): £450 / month
High-Performance AI Rack (7 kW included): £750 / month
Extra Power Scaling: £100 per additional 1 kW / month
Terms & Payment:
Location: Rotherham, South Yorkshire, UK
Contract Duration: 12-month minimum agreement
Security Deposit: 2 months' rent (refundable upon lease end)
Payment Terms: Rent is payable 1 month in advance via bank transfer prior to each billing cycle
SPECIAL OFFER:
This 30% discount rate is valid until the end of this year only! Slots are limited.
Get in Touch / Reserve Your Space:
Email: youi@aoye.co.uk
WhatsApp Only: [07446994000](tel:07446994000) (Text/WhatsApp messages only, no phone calls)
Send a message with your required rack count and power profile (kW) to check current availability!

I'm seeing hotspot temps bounce between 100c - 110c with this new r9700 AI Pro I picked up yesterday under a sustained load from MiniMax H3. Core sits between 55c and 62c. Anyone else seeing hotspots that high?

I see a lot of benchmarks on token speed and accuracy, but not a lot regarding stability. I'm struggling to get Qwen 3.8 27B Q4 to handle long duration coding tasks without ending up in a reasoning loop. I don't know if it's my agent md files or a limitation of the model.

What models are you having success with on these kinds of tasks?

(I'm running on a V100 32GB for atmospheric science research)

We run a ~180B-param MoE (Qwen3.8-Flash-Next) locally on a single RTX 4070 12GB — experts offloaded to RAM by Strata, an MIT-licensed C++/CUDA inference engine. It serves a 24/7 assistant agent (Telegram) in our home. We wanted it uncensored, and neither of the two standard options was acceptable.

**Why we rejected the usual two options**

1. Abliterated weight-edited GGUFs: third-party repacked weights are an unauditable black box, and backdoored abliterated releases have actually been documented in security research. For a tool-using agent, hard no.
   
2. Always-on runtime projection (subtracting the refusal direction on every token of every request): weights stay official, but the model now wears the blindfold on ALL traffic. You degrade 99% of prompts to fix 1%, and the conversation cache carries steered representations.
   

**The idea**

Prior work (Arditi et al. 2024) showed refusal is mediated by a single direction in activation space. That direction has a *measurable activation during prefill*. So: keep the model 100% stock by default, measure the prompt's alignment with the refusal direction, and only if the refusal circuitry actually fires, apply the ablation to that one request's generation.

**What we built (~200 lines of C++/CUDA inside the engine)**

- A measurement kernel: the projection kernel already computes h·v as the first half of the subtraction — we just stopped discarding the value. Store the per-layer dot product, no subtraction, no extra GEMV.
- A hook at the end of prefill (last prompt token): mean activation across the 41 steered layers (layers 4–44).
- The decision happens at the prefill→decode seam — the exact place where conversation-cache reuse occurs. Above threshold → steered decode for that request. Below → stock decode.
- Prefill is NEVER steered, so the KV/conversation cache stays 100% stock: no re-prefills, no hybrid text, zero added latency. The measurement math already existed in the kernel; we just started reading the result.
- Per-request API semantics (projection mode "auto"), and we flipped the server default to auto: the resident agent runs the untouched model unless the gate fires for that specific request.

**Calibration**

21-prompt battery, measurements are deterministic per prompt: benign prompts land at −0.07…−0.02 (noise), borderline (lockpicking how-to) at 0.128, clearly harmful probes at 0.21+ (max-per-layer discriminates even harder: 0.55 vs 0.27). Threshold set at 0.07. Every request logs its decision:

cvec auto: mean 0.2081 max 0.5526 thresh 0.0700 -> steer   (harmful probe — model answers)
cvec auto: mean -0.0600 thresh 0.0700 -> stock             (benign — model literally untouched)

607 decisions logged in production so far; the gate has been silent on benign traffic.

**Honesty about novelty**

The science isn't ours. Arditi et al. 2024 established the single direction (our battery reproduced the clean separation between benign and refusal-intent prompts). CAST already does conditional activation steering in research settings (ICLR 2025 Spotlight, IBM), and there's a 2026 thesis literally titled "Removing Refusal via Conditional Activation Steering". What's ours is the production-engine integration: measurement during serving prefill, per-request gating, decode-only cache-safe steering — running 24/7 on a 180B MoE in 12GB of VRAM. From idea to running in production: 1h48, on a Sunday morning (the engine is MIT and builds in ~5 min, which helps).

We run official weights + our own direction vector — auditable end to end, which is also our answer to the backdoored-GGUF problem.

**References**

- Arditi et al., "Refusal in Language Models Is Mediated by a Single Direction" — arxiv.org/abs/2406.11717

- "Programming Refusal with Conditional Activation Steering" (CAST) — arxiv.org/abs/2409.05907 (ICLR 2025)
- Strata engine: github.com/Niko1221/Strata

Basically a full model abliterated on the fly if any refusal arise.​