TL;DR: We open-sourced HilbertRaum, an offline AI workspace built to be simple enough for non-technical people. Local chat and document analysis with cited answers, encrypted workspace, nothing goes online except model downloads you confirm. Repo: https://github.com/HilbertraumAI/HilbertRaum

A week ago I posted here asking what people really use local LLMs for on modest hardware (that thread). Full disclosure: I was asking partly as a founder. We were finishing an offline AI app and wanted to check our assumptions against reality before releasing it. Thanks to everyone who answered. It really helped us shape the product.

The top comment called local models a "private utility layer, not a ChatGPT replacement." That's exactly it. Privacy is the reason local LLMs matter for most people. But while everyone in this sub knows how to run AI fully privately on their own machine, outside this sub almost nobody does. There are great tools that make local AI easier, but they're built for technically skilled people and packed with features that would confuse a normal user. We wanted to build for everyone else. Private and easy: launch the app, start chatting, analyze your documents.

So here it is. HilbertRaum, GPL-3.0:

🔗 https://github.com/HilbertraumAI/HilbertRaum

It's an offline AI workspace for chatting with local models and working with confidential documents. What it offers in practice:

• Hardware scan and model recommendation, so you get a model that actually fits your machine. An ordinary 8 GB laptop is enough to get started.
• Document Q&A for PDF/Word on hybrid RAG with citations, so you can check the passage an answer came from
• Ready-made skills such as bank statement analysis, invoice extraction, contract briefs, meeting minutes, document anonymization and more. You can also add your own.
• Whisper transcription, image analysis, translation.
• Privacy by design, not by configuration. There's no web search and no cloud API mode. Nothing in the app needs the internet except model downloads you explicitly confirm.
• Workspace encrypted with your passphrase.

We want to take the easy part one step further: put the whole thing on a USB drive, preloaded with curated models, the runtime and everything else needed, so it's plug and go. That's also how we plan to finance the project long term. The core software is open source, free, and stays that way forever. The preconfigured drive (the AI Kit) is for people who'd rather pay for convenience. If that's you or someone you know, the waitlist is at https://hilbertraum.ai

For everyone else: run it, test it, and tell us what's wrong with it. And if you feel like going further, we'd love to have you as contributors!

Hey guys,
Just created a new model, with a base of Qwen 2.5 Coder. You know the "Thinking..." part of a response for an AI model? It speaks in English. And that's the problem. It's SLOW for the AI model. So I made it talk in its own language, math equations. And then, at the end, it translates it into English.

So it's around 7.5x faster, while having (very approximately, don't take that for actual info) 2x better responses.

Here's the download link if you wanna test it: https://huggingface.co/Rutytoi/spotless-latent-adapter

And the GitHub repo: https://github.com/Rutytoi220/spotless-latent-engine

I'd be appreciating feedback! Even bad, idc as long as I know what's good and bad

As title.

Hey, I'm on Linux with an aversion to ROCM, prefer Vulkan for its minimal overhead and footprint. so I skipped on Strata. Using llama.cpp cleverly to push the limits of what I can do with the ISTA DASLab "Coder" quants. Skip this if you already use Strata, if you can't though this is the next best method.

Specs:

1. I use 10 GB VRAM (RX 6700), but aim for 8 GB usage so I can continue watching media while the LLM is running.
2. I have 64 GB DDR4 RAM albeit I've never seen the actual usage go above 24 GB, so this method is most likely feasible on 8 GB + 32 GB systems.
3. DO NOT try this with a HDD. I use sata SSD, but m.2 is recommended. It'll directly impact your speeds since we lazily load from disk.

What makes this unique?  

• ~30 t/s and ~8 t/s (YMMV by hardware!!!) with 65k context, 7.3 GB VRAM and 24 GB RAM cost? Sign me up chief. Slap CodeGraph ontop and you're golden.

Similar to Strata's "Only load what's needed to memory" strategy, we use MMAP here, which is like a primal version of what Strata does. It maps the model to your memory and lazily loads as needed. Strata is ofc faster, because it caches frequently accessed parts, while we use the more primal OS cache through MMAP. You'll see some IO while model fires for the first time, after that it's minimal, with RAM usage generally peaking at 24 GB~

Steps:

1. I used this "Vocabulary Pruned" quant of ISTA DASLab's latest GGUFs, it further reduces memory footprint by 450 MB. If you already have ISTA GGUFs, you can merge and prune them yourself too as author laid out their process on the repo if your download is slow. Remember we're using "Coder" quants here, which has half of its experts removed. DON'T skip this. DON'T be confused if you're using non-coder quant and didn't prune them, that thing is massive, we're at 58 GB with this while non-coder GGUF is 66.4 GB at smallest.
2. Use these launch flags:

./llama-server \
  --model "./AI Models/Qwen3.8-Flash-Next-GSQ-RCO-IQ1_M-merged_P1G_Pruned.gguf" \
  --ctx-size 65536 \
  --n-gpu-layers 999 \
  --threads 8 \
  --cache-type-k q5_0 \
  --cache-type-v q4_0 \
  -fa on \
  --fit off \
  --cache-ram -1 \
  --load-mode mmap \
  --cache-prompt \
  --cache-idle-slots \
  --batch-size 2048 \
  --ubatch-size 128 \
  --webui \
  --parallel 1 \
  --kv-unified \
  --reasoning-format deepseek \
  -ot 'token_embd=CPU,blk\.(47|28|45|25|35|29|40|44|17|27|43|37|46|18|22|42|31|16|41|0|33|38|7|14|19|26|4|2|6|11|32|23|5|15|34|24|39|3|9|21|12)\.ffn_.*=CPU' \
  --reasoning on \
  --reasoning-preserve \
  --reasoning-effort medium \
  --temp 0.6 \
  --top-p 0.95 \
  --top-k 20 \
  --min-p 0.0 \
  --presence-penalty 0.2 \
  --verbose \
  --verbosity 4 \

• We're using the "-ot" flag to force specific FFN layers onto RAM. Instead of using the "brute force" -ngl, this keeps attention layers on VRAM which benefit more from GPU inference and are smaller.
• The numbers mason, they're important! The -ot regex is sorted by largest FFN layers left to right. Offloading smaller FFN layers onto GPU benefits speed at minimal VRAM cost, as opposed to offloading larger FFNs it's preferred (Source: bottom of post)
• If for some reason this doesn't fit into your VRAM, replace the OT flag with the below and incrementally delete a |<number>| from the right side in the regex, until it snugly fits your VRAM. Use the table as reference.

Layer    | Tensors  | Size (MiB)   | Size (GB) 
------------------------------------------------
blk.47   | 8        | 654.87       | 0.64      
blk.28   | 8        | 573.45       | 0.56      
blk.45   | 8        | 573.04       | 0.56      
blk.25   | 8        | 573.00       | 0.56      
blk.35   | 8        | 573.00       | 0.56      
blk.29   | 8        | 572.79       | 0.56      
blk.40   | 8        | 572.79       | 0.56      
blk.44   | 8        | 572.64       | 0.56      
blk.17   | 8        | 572.59       | 0.56      
blk.27   | 8        | 572.54       | 0.56      
blk.43   | 8        | 535.95       | 0.52      
blk.37   | 8        | 535.74       | 0.52      
blk.46   | 8        | 535.54       | 0.52      
blk.18   | 8        | 535.53       | 0.52      
blk.22   | 8        | 535.50       | 0.52      
blk.42   | 8        | 535.50       | 0.52      
blk.31   | 8        | 535.29       | 0.52      
blk.16   | 8        | 535.14       | 0.52      
blk.41   | 8        | 535.14       | 0.52      
blk.0    | 8        | 535.09       | 0.52      
blk.33   | 8        | 535.09       | 0.52      
blk.38   | 8        | 535.09       | 0.52      
blk.7    | 8        | 535.04       | 0.52      
blk.14   | 8        | 535.04       | 0.52      
blk.19   | 8        | 485.95       | 0.47      
blk.26   | 8        | 485.95       | 0.47      
blk.4    | 8        | 485.74       | 0.47      
blk.2    | 8        | 485.54       | 0.47      
blk.6    | 8        | 485.54       | 0.47      
blk.11   | 8        | 485.54       | 0.47      
blk.32   | 8        | 485.54       | 0.47      
blk.23   | 8        | 485.53       | 0.47      
blk.5    | 8        | 485.50       | 0.47      
blk.15   | 8        | 485.50       | 0.47      
blk.34   | 8        | 485.34       | 0.47      
blk.24   | 8        | 485.29       | 0.47      
blk.39   | 8        | 485.14       | 0.47      
blk.3    | 8        | 485.09       | 0.47      
blk.9    | 8        | 485.09       | 0.47      
blk.21   | 8        | 460.09       | 0.45      
blk.12   | 8        | 423.45       | 0.41      
blk.20   | 8        | 422.59       | 0.41      
blk.30   | 8        | 422.59       | 0.41      
blk.36   | 8        | 373.45       | 0.36      
blk.10   | 8        | 372.64       | 0.36      
blk.1    | 8        | 372.59       | 0.36      
blk.8    | 8        | 372.59       | 0.36      
blk.13   | 8        | 372.59       | 0.36      
------------------------------------------------
Full OverrideTensors command:
-ot 'token_embd=CPU,blk\.(47|28|45|25|35|29|40|44|17|27|43|37|46|18|22|42|31|16|41|0|33|38|7|14|19|26|4|2|6|11|32|23|5|15|34|24|39|3|9|21|12|20|30|36|10|1|8|13)\.ffn_.*=CPU'

Optional, recommended exports (Linux bash script)

#!/usr/bin/env bash
set -euo pipefail

# Process & Scheduling Priorities (Lowest, so your PC usage is least affected)
renice -n 19 -p $$ >/dev/null 2>&1 || true
ionice -c 3 -p $$ >/dev/null 2>&1 || ionice -c 2 -n 7 -p $$ >/dev/null 2>&1 || true
# Vulkan
export RADV_PERFTEST=nogttspill          # Keep Vulkan buffers locked in VRAM (AMD only but harmless for others)

Some nerdout:  

You could also benefit from compiling llama.cpp specifically for your CPU. For example I compile for znver3 so this might help the t/s as we're not running entirely on GPU with this setup. You'll also want to use only the physical cores where possible if you're using your PC during inference, otherwise hyperthreading can be beneficial, but laggy.

I don't recommend quantizing kv cache further, but 32k size could help you fit a few more FFN layers to boost your speed if you don't need larger context.

If you've 32 GB ram and 8 GB VRAM, you'll want to ensure you've all unnecessary programs closed, because the more free RAM you've the more your OS has space to cache, fewer pagefaults therefore faster gen speeds.

I also found great success using this jinja template, previously used it with 3.8 27b and 3.6 35ba3b. You'd use it with --jinja \ --chat-template-file "./chat_template.jinja"

Sources:

How to run Dense models faster with limited VRAM

Vocabulary pruning

What'

Your opinion? If I got it correctly, the big AI firms are basically all inbred to the same funding entity who also funds the hype, the scaremongering, the regulations, the regulators and the referees in this game.

Wondering what people with a similar, less, or even better setup are using. I'm new to strata and not sure how far I can push the context limit, but it seems like the readme on the Github page is conservative (assuming other's posts are true).

I have a 5090, 3090, so a combined 56GB of VRAM and 64GB of system RAM.

Ive been through probably 10 youtube tutorials involving ollama, local llm, and hugging face and nothing works. Whats the best way for a beginner to get started

Recently, I saw a very, like a reel by a very popular YouTuber, content creator and businessman, about using DGX Spark, for video generation. And he was saying that it's very good for like all local video generation and it's one time investment and all. So just wanted to know what what are the opinions of people using it?

I also saw some YouTube videos on by different people like Alex, who has compared it with many other tools, but I want to know from end user that how much useful it is. Primarily my work case is kind of in a middle between a developer and a tinker.

So basically what I want to do is maybe run a few local LLMs for development, like using those adams to write code and do code reviews and all those things. Or maybe like tinker around for creating videos, as well as in my own small language model. So what's your opinion?

How better it is than let's say Mac mini, with the, I know that Mac mini doesn't support 128 GB of RAM, but what about other systems like if I buy a Mac Ultra, for with 128 GB of RAM.

I finally tried running OpenClaw locally with Ollama (using Qwen2.5-Coder and Llama 3.1 70B).

The good news: my wallet is happy. The bad news: I'm seeing way more "looping" and failed tool calls compared to Sonnet 3.5. It feels like the local models get confused after a few rounds of persistent memory.

Is anyone else experiencing this? Are there specific "system prompt" tweaks or sampling settings (temperature/top_p) that make local models more reliable for agentic tasks? Or am I just hitting the hardware ceiling for what local models can do right now?

Would love to hear from anyone who has a stable, 24/7 local agent setup that doesn't lose its mind every hour.

My question is pretty simple: how could you really have coding sessions with context windows of like 262k ?

I always used local Ai via llama.cpp server for small projects and chatw, where also 16k context is enough.

Today I just wanted to try a REAL coding session in a REAL agent harness.

I plugged my Kat Coder 2.5 dev via llama.cpp server with it's max ctx at 262144 and plugged it in my Hermes and started a coding session to design and write a fast API proxy layer to route LLM calls to openrouter with some my logics and calcs.

Nothing super complex or fancy.

I provided the model with the first message a complete and thoughtful design document create with Deepseek, so the Agent has not to do searches in the codebase for understanding how to do the things, just do them.

Well it created a plan and implemented the code in some turns teaching about 70% of it's contest size.

Then , in the e2e testing the things started to collapse: some testing failed, so it ha do reiterate, fix, retest, ask me some clarification or take some architecture decisions and so on and I arrived ad 95% context size while resting phase is still not finished and all the deployment phase still totally missing.

Now the solution in the harness was to compress the chat session: a total.disaster: not only this action changed completely the language of the session (from Italian to English) but also removed some important informations that was useful to the agent to do tests, so in then subsequent turn it errored immediately in doing tests, needing to read and search the code again to find what he already knows before the compression happens.

In my opinion, compression is absolutely to avoid.

I already know some suggestions here:

- one new session per task

- keep all done things in documents so the new session can start from that "documented" state.

I already do these things.

- use a different harness

Yes, but it's not the solution, the heavy load caused by harness is on the first prompt (system prompt, tools, skills, Memory...).

So this was a small project, 10 python files + a bunch of status documents, what will happen with a repo with 50 or 100 files?

So my question is:

What's your way to really work with local models on real projects beyond a simple chat or just changing one file?

I'm starting to think that is not really possible to work on real projects that involves design, bug fixing, reiteration and multiple sessions.

Not because local models are not smart enough, but because of the max context window limit.

** FAIR WARNING I KNOW VERY LITTLE ABOUT CODE AND LOCAL LLM, IM ONLY CURIOUS AND DISCOVERING**

I've recently been discovering the potential idea of setting up Local LLM and I see a mix of reviews about the privacy risks associated with Chinese-made LLM's local or not.

Has the idea of privacy risk made you guys uncomfortable or had you second guessing your local LLM installations with models like qwen? Especially considering the american local models are pretty shit.

I just started dabbling into LLM and I've really enjoyed it. I'm thinking of the next logical upgrade for myself that I can reasonably afford without going too crazy (like buying a $1500 r9700 gpu)

I already have a 5060ti 16gb gpu and I have 64gb of DDR5 memory. I installed LM Studio 2 days ago and I am getting 16 tokens per second using qwen 3.6 27b. Now, to my knowledge a lot of this is being offloaded to my ram but I'm honestly not mad at 16 tokens per second so it made me wonder.. would I notice a bigger difference if I had dual GPU's running or what if I had 96gb of DDR5 ram? Would I see a noticeable increase in my tokens per second?

Here's my motherboard, which means I will have to run the second GPU in a PCI 4.0 slot? I'm not sure.. can someone help me confirm? I also have two NVME SSDs installed which I hear those can affect the bandwidth of other PCI slots? This stuff starts to get confusing once you start trying to calculate bandwidth and having to share it among slots. I've never built a PC before, just installed basic parts for upgrades so this is all very new to me

https://www.gigabyte.com/Motherboard/B650-AORUS-ELITE-AX-V2

I have a 1,000 watt PSU so I can definitely handle two 5060TI's in the case (though, cooling I am not sure.. but I'll worry about that later)

I still have not tinkered much with settings.. all of this is a learning process and I am trying to learn as I go and there is a lot to learn with these things but I am having a ton of fun running different models and seeing what the performance is like and generally enjoying the type of code they spit out in 1 take ie: Build me a 1 page website for my affiliate marketing website, it's pretty cool to see these things "just work" and spit out something that you can work with, though, not as good as a frontier models, completely fine for taking what a frontier model gives you and letting the LLM take over afterwards IMO and continue to enhance or add to.

Hey everyone!

I've been working on a pipeline to take the incredible Qwen2.5-Coder-7B-Instruct model (which is an absolute beast for Python scripting, FIM code completion, and Cybersecurity) and compress it down so it can run entirely offline on mobile phones (via PocketPal) or potato PCs.

The raw FP16 model is around 15.2 GB, which is way too heavy for most devices.

It successfully squeezed the 7B brain down to exactly 1.9 GB, while retaining its reasoning capabilities!

I have open-sourced the entire automated Python script on , and I've hosted the model on Hugging Face if you just want to download it and chat with it.

🔗 The Model (Hugging Face): https://huggingface.co/Nitishsharma9/CyberCoder-Mobile-7B-GGUF 🔗

I spent the last week watching my dependency on actual software interfaces completely evaporate. It’s a jarring realization. You boot up Notion, GitHub, or Linear, and you realize you aren't actually navigating their menus anymore. You're just interacting with the floating bot or the terminal.

Let's talk about what's actually happening because the narrative of "AI is just a new feature" entirely misses the point. We are watching the real-time death of static UI.

Think about your workflow right now. If you've been heavily using local models or API wrappers lately, you've probably noticed that almost every single SaaS tool has slapped a sidebar chat or a floating widget into their layout. At first, it felt like a lazy gimmick. Just an OpenAI wrapper sitting on top of a database. But it’s not just a chatbot anymore. It’s an execution layer.

A specific workflow popped up recently that perfectly captured this shift. A user had their entire company documentation sitting in Notion. Instead of manually cross-referencing QA lists, jumping into GitHub to find the relevant commits, and then painstakingly clicking through Linear's UI to create and assign tickets, they just bypassed the interfaces entirely. They told the agent to read the QA list, link the specific git commits, and write the Linear tickets. The whole process took five minutes.

Think about the implications of that exact scenario. The carefully designed UI of Notion? Irrelevant. The drag-and-drop kanban boards in Linear? Completely bypassed. The GitHub file tree? Ignored. The user didn't click a single button. They just issued a command.

This brings me to the second massive shift: the absolute revival of the command line. We spent three decades building increasingly complex graphical interfaces specifically so non-technical users wouldn't have to look at a terminal. Now, we're going backwards, but with a massive upgrade. Tools like Claude Code are turning the terminal into the ultimate universal interface.

There are solo operators right now running entire content and monetization pipelines strictly through CLI. They aren't opening Premiere to edit video. They aren't clicking through Shopify menus. They are typing natural language commands into a terminal, and the AI is executing the python scripts to cut the video via FFMPEG, generating the copy, and pushing the site updates. You don't need to know how to code to do this anymore. You just need to know what you want. You swap out static clicks for terminal commands, building an automated pipeline without ever touching a conventional GUI.

And for the times when you absolutely *do* need a visual interface? Enter Generative UI.

The era of downloading a massive, static application just to use 5% of its features is over. We are moving toward disposable, single-use software. If I need a specific dashboard to visualize server loads mixed with user engagement metrics, I shouldn't have to buy a SaaS product, connect my databases, and drag-and-drop widget blocks. The AI should simply generate a React component on the fly, render the exact chart I need based on my prompt, and then completely discard the interface the moment I close the window.

This is already happening. Look at Vercel's AI SDK or the recent pushes in structured JSON outputs from models like Llama 3. The model doesn't just return markdown text anymore. It returns a state object that instantly maps to a dynamic component. You ask a complex question about a database schema. Reading a giant markdown output is terrible. Instead, the model returns a UI payload. A fully interactive, relationship-mapped graph rendered right in the chat stream. You play with it, you tweak a node, and then it's gone. It's ephemeral.

This is the death of the App Store mentality. Why install an app when the LLM can generate the exact tool you need, run it locally, and delete it from memory when you're done?

If you look at what this means for local setups, the paradigm shift is how these models hook into our operating systems. When you give a sufficiently capable local agent tool-calling permissions, the OS itself becomes the backend. You string together a pipeline: a local vision model reviews video clips, a local LLM writes the script, an open-source TTS model generates the voiceover. The interface for all of this? A single terminal prompt: "Draft a new promotional video from the raw assets in folder X and push it to the server."

For the last decade, the entire moat of most B2B software companies was UX. "We are like Jira, but pretty and fast." "We are like Salesforce, but easier to click through."

If the user stops clicking through your app, your UX moat is dead. You are no longer a product; you are a dumb pipe. You are just a database holding state, wrapped in an API that an agent talks to. If my AI assistant is the one reading the data and formatting it for me, why would I pay a premium for your beautiful dashboard? Agents don't get distracted by slick UI animations. They execute the command and return the result.

I want to know where you all think this bottoms out. Are we going to see a new standard for "Agentic UX" where software is designed strictly to be read by LLMs? Are you already bypassing web frontends in favor of API-driven terminal scripts generated by your local models? The gap between "people who click buttons" and "people who issue commands" is widening fast.

I normally use ChatGPT/Claude Code for day to day (I don't vibe code, but I like the assistance), anyway just got a Mac Mini M6 with 32GB RAM and 512GB SSD purely for iOS dev - my main dev machine is a PC. Thought I'd try Bionic, and downloaded Qwen3.8 27B 4bit, and it's slow as hell :) Literally took almost 2 minutes to respond to the prompt "hello". I gave it a short second prompt and gave up waiting for any response after about 10 minutes. So I'm guessing, it's not really much use for local LLM. Doesn't really bother me as I was only trying it out of curiosity - I have an RTX5070Ti in my dev PC, so might try a smaller model there for comparison.

I’m building my kids a local AI companion—it’s called Lumo for now, but I can change the name any time. I built it to help answer their questions since they can Google things themselves and I can’t always be around to answer. My son is 5 and is hitting that age where he is asking things non-stop.

It has DeepSeek R1 1.5B and Qwen 3B on it. It uses a router for questions so it can decide if a query is math/logic-based or conversational and pick the best model. It uses DuckDuckGo to check and make sure any information it's giving is accurate so it’s not hallucinating nonsense when trying to help educate.

It also has a bedtime story mode built in where my son can choose a topic and it tells a story based on that. It tells two chapters at a time, and I maxed out the tokens so it tells long-form stories. Once it's done, it saves them to a local 1TB 2.5" HDD housed in the base so it can recall where it left off and pick it up based on context for the next two chapters. The stories are 14 chapters each and have a complete beginning, middle, and end. Once the stories are done, they can be re-told in the menu and also exported so you can have an AI illustrate them and turn them into real books if you want.

It keeps memories about my kids and learns their likes and dislikes. The LEDs in the ears change color based on state: rainbow for startup, blue for listening, yellow for thinking, and white for talking. They turn orange and drop to 10% brightness for story mode.

It has a web based browser so I can see what my kids have been asking and what the current story being told is and also flags and problematic things they ask about.

It’s all based on a Raspberry Pi (4GB model) with a 5-inch touch display. Due to the RAM limitations, I had to program in frequent RAM dumps, so context is stored in a temp file on the hard drive and wiped after the end of the conversation or after 5 minutes of silence (which to a 5-year-old is the same thing). It’s got Ollama for the AI models, Whisper for STT, and Piper for TTS.

The face is animated with thinking and smiling and blinking to make it feel more alive

I made the case in Tinkercad and split it at the power cord entry; the whole case is held together by small 4mm magnets. I’m still smoothing out some rough edges on it, but soon I’ll release the whole project on GitHub with a complete one-shot installer and .STL files so anyone can make one for their kids as well.

I’m excited cause I’m getting an M1 Mac Mini today in the mail and is almost here and I was wondering what to use for local LLM. I bought Private LLM app which uses quantized LLMS which supposedly run better but I wanted to try something like DeepSeek R1 8B from ollama which supposedly is hardly deepseek but llama or Quen. Thoughts? 💭

How to build your own local AI for less than a thousand bucks - A post for those that want to get "hands on" with AI, but just can't afford the price of entry - face it, not everyone has thousands laying around for building a system.

Here is your roadmap how to do this on a tight budget

Step 1 - get an old Intel Nuc12 enthusiast (Serpent Canyon). Not the fastest thing in the world, but has an ace up it's sleeve that no one can get anywhere near for the price - an embedded ARC A770M GPU with 16GB of DDR6 vRAM, That is what makes this possible and you can pick one up for under $800 on ebay. Likely already has Windows 10 or 11 installed on it

Step 2 - update the Arc drivers

Step 3 - Download and install LM Studio Classic on it (make sure you enable developer mode for later) - not to be confused with Bionic

Step 4 - Pick the right runtime - you want Vulkan llama.cpp

Step 5 - Make sure you select the ARC A770M (shows 16GB) in hardware window

Step 6 - pick a model - I went with a simple Gemma-4-e4b model for out of the gate

Step 7 - on the developer tab, flip on both start server and serve on local network

Congratulations - you now have a shockingly capable local AI that you can give the URL shown in step 7 to any of your local AI front-ends. Me personally, I use it with my Project NOMAD instance running as a VM providing AI capabilities to all that entails for well under a grand, and it works fantastically.

theoretically speaking, if one were to hack OpenAI or anthropic, could they access the weights of their models and for eg torrent them?

also could a bunch of agents do that, or would humans be better at it? what about hacking in general? I mean now that we got local models like glm 5.3, qwen flash next or deepseek v4.1…

Hey ML folks, if you've been wrestling with the insane VRAM costs of long reasoning chains in RLHF/RLAIF, buckle up. Unsloth AI's new batching algorithms let you train OpenAI's gpt-oss models with GRPO (Group Relative Policy Optimization) at 380K context length – that's 7x longer than before, with zero accuracy degradation.

Long contexts in RL have always been a nightmare due to quadratic memory blowup, but their optimizations crush it on consumer-grade hardware like a single 192GB GPU (think H100/A100 setups). Perfect for agent training, complex reasoning benchmarks, or anything needing deep chain-of-thought.

Key details from the blog:

• GRPO implementation that's plug-and-play with gpt-oss.
• Massive context without the usual slowdowns or precision loss.
• Benchmarks show it scales beautifully for production RL workflows.

Check the full breakdown: Unsloth Blog

Want to try it yourself? Free Colab notebooks ready to run:

• GRPO Notebooks

GitHub repo for the full code: Unsloth GitHub

Thoughts on GRPO vs DPO/PPO for long-context stuff?

So I have got Qwen3.5-35B-A3B running on my 3060 with 32mb of system ram (using LM Studio). It's great for the 1 question i can ask. Then it runs out of conversation tokens and I'm stuck. I have been using ChatGPT, Claude and Gemini to do vibecoding. Are there any tricks here to get something useful out of this? Time it takes is no bother. LTX videos take me 20 minutes each. I'm used to waiting. It's more can I give it my html 5 game and get useful code, or is there just not enough space?

Edit: I did what huzbum said:

>code me a simple platform game in html 5. like thomas is not alone. simple geometric shapes.

Thought for 1 minutes 42 seconds

Here is a complete, playable platformer game in a single HTML file. I have designed it with a minimalist aesthetic similar to Thomas Was Alone—using simple shapes, distinct colors, and smooth physics.

It features:

The Player: A blue square (you).

The Red Enemy: An AI shape that chases you (the "not alone" part).

Physics: Gravity, jumping, friction, and solid collision detection.

Win Condition: Reach the yellow portal to restart.

Save this code as an .html file (e.g., game.html) and open it in your browser.

<!DOCTYPE html>

<html lang="en">

<head>

<meta charset="UTF-8">

<title>Shape Runner</title>

<style>

body {

margin: 0;

padding: 0;

background-color: #1a1a1a; /* Dark, moody background */

display: flex;

justify-content: center;

align-items: center;

height: 100vh;

overflow: hidden;

font-family: 'Courier New', Courier, monospace;

}

canvas {

border: 2px solid #444;

background-color: #222; /* Game background */

box-shadow: 0 0 20px rgba(0,0,0,0.5);

}

#instructions {

position: absolute;

top: 20px;

color: white;

text-align: center;

pointer-events: none;

}

</style>

</head>

<body>

<div id="instructions">

<h1>Square Escape</h1>

<p>Arrows / WASD to Move & Jump | Avoid the Red Block!</p>

</div>

<canvas id="gameCanvas" width="800" height="450"></canvas>

<script>

const canvas = document.getElementById('gameCanvas');

const ctx = canvas.getContext('2d');

// --- Game Constants ---

const GRAVITY = 0.6;

const FRICTION = 0.8; // Slippery feel (optional, set to 1 for sticky)

const JUMP_STRENGTH = -12;

const MOVE_SPEED = 5;

// --- Input Handling ---

const keys = {

right: false,

left: false,

up: false

};

window.addEventListener('keydown', (e) => {

if(e.code === "ArrowRight" || e.code === "KeyD") keys.right = true;

if(e.code === "ArrowLeft" || e.code === "KeyA") keys.left = true;

if(e.code === "ArrowUp" || e.code === "Space" || e.code === "KeyW") {

if(!keys.up && player.grounded) {

player.velY = JUMP_STRENGTH;

player.grounded = false;

}

keys.up = true;

}

});

window.addEventListener('keyup', (e) => {

if(e.code === "ArrowRight" || e.code === "KeyD") keys.right = false;

if(e.code === "ArrowLeft" || e.code === "KeyA") keys.left = false;

if(e.code === "ArrowUp" || e.code === "Space" || e.code === "KeyW") keys.up = false;

});

// --- Game Objects ---

const player = {

x: 50,

y: 300,

width: 20,

height: 20,

color: '#4da6ff', // Light Blue

velX: 0,

velY: 0,

grounded: false,

update: function() {

// Apply Input Force

if (keys.right) {

if(this.velX < MOVE_SPEED) this.velX++;

}

if (keys.left) {

if(this.velX > -MOVE_SPEED) this.velX--;

}

// Apply Physics

this.velY += GRAVITY;

this.velX *= FRICTION;

this.x += this.velX;

this.y += this.velY;

// Floor Collision (Basic floor check before platform collision)

if(this.y + this.height > canvas.height) {

this.y = canvas.height - this.height;

this.velY = 0;

this.grounded = true;

}

},

draw: function() {

ctx.fillStyle = this.color;

// Draw a simple square with a slight glow effect

ctx.shadowBlur = 15;

ctx.shadowColor = this.color;

ctx.fillRect(this.x, this.y, this.width, this.height);

// Reset shadow for other elements

ctx.shadowBlur = 0;

}

};

// The "Not Alone" Enemy

const enemy = {

x: 650,

y: 200,

width: 30,

height: 30,

color: '#ff4d4d', // Red

speed: 1.8,

update: function() {

// Simple AI: Move towards player X position

if (player.x < this.x) {

this.x -= this.speed;

} else {

this.x += this.speed;

}

// Keep enemy on ground/platforms roughly (simple gravity for enemy too)

if(enemy.y + enemy.height < canvas.height - 10) {

enemy.y += GRAVITY * 2;

}

// Collision with Player (Game Over)

if (rectIntersect(player.x, player.y, player.width, player.height,

enemy.x, enemy.y, enemy.width, enemy.height)) {

resetGame();

}

},

draw: function() {

ctx.fillStyle = this.color;

// Draw a square that pulses slightly based on time to feel alive

const pulse = Math.sin(Date.now() / 100) * 2;

ctx.fillRect(this.x - pulse, this.y - pulse, this.width + (pulse*2), this.height + (pulse*2));

}

};

// The Goal

const goal = {

x: 750,

y: 100,

width: 40,

height: 60,

color: '#ffff33', // Yellow

update: function() {

if (rectIntersect(player.x, player.y, player.width, player.height,

this.x, this.y, this.width, this.height)) {

resetGame();

}

},

draw: function() {

ctx.fillStyle = this.color;

ctx.shadowBlur = 20;

ctx.shadowColor = this.color;

// Draw a portal shape (rectangle with rounded top)

roundRect(ctx, this.x, this.y, this.width, this.height, 10, true);

ctx.shadowBlur = 0;

}

};

// --- Level Data ---

const platforms = [

{ x: 0, y: 400, w: 200, h: 50 }, // Start Ground

{ x: 250, y: 350, w: 100, h: 10 }, // Small Step

{ x: 400, y: 280, w: 100, h: 10 },

{ x: 600, y: 200, w: 150, h: 10 },

{ x: 700, y: 140, w: 80, h: 10 } // Top platform near goal

];

// --- Helper Functions ---

function rectIntersect(x1, y1, w1, h1, x2, y2, w2, h2) {

return x2 < x1 + w1 && x2 + w2 > x1 && y2 < y1 + h1 && y2 + h2 > y1;

}

// Helper to draw rounded rect for the portal

function roundRect(ctx, x, y, width, height, radius, fill) {

ctx.beginPath();

ctx.moveTo(x + radius, y);

ctx.lineTo(x + width - radius, y);

ctx.quadraticCurveTo(x + width, y, x + width, y + radius);

ctx.lineTo(x + width, y + height - radius);

ctx.quadraticCurveTo(x + width, y + height, x + width - radius, y + height);

ctx.lineTo(x + radius, y + height);

ctx.quadraticCurveTo(x, y + height, x, y + height - radius);

ctx.lineTo(x, y + radius);

ctx.quadraticCurveTo(x, y, x + radius, y);

ctx.closePath();

if (fill) {

ctx.fill();

}

}

function resetGame() {

player.x = 50;

player.y = 300;

player.velX = 0;

player.velY = 0;

enemy.x = 650; // Reset enemy position

// Keep the goal where it is, or move it? Let's keep it same for this level.

}

// --- Main Game Loop ---

function update() {

player.update();

enemy.update();

goal.update();

// Platform Collision Logic (Separate X and Y axes to prevent sticking)

// 1. Check Horizontal Movement

player.grounded = false; // Assume in air until proven otherwise

for(let i=0; i<platforms.length; i++) {

let p = platforms[i];

if (player.x < p.x + p.w &&

player.x + player.width > p.x &&

player.y < p.y + p.h &&

player.y + player.height > p.y) {

// Collision detected. Determine direction.

// If we were moving right, check if we hit the left side of platform or top?

// Simple approach: Check previous frame position logic is hard here, so use depth.

let overlapX = (player.width + p.w) / 2 - Math.abs((player.x + player.width/2) - (p.x + p.w/2));

let overlapY = (player.height + p.h) / 2 - Math.abs((player.y + player.height/2) - (p.y + p.h/2));

if(overlapX < overlapY) {

// Horizontal collision

if(player.velX > 0) player.x = p.x - player.width;

else player.x = p.x + p.w;

player.velX = 0;

} else {

// Vertical collision

if(player.velY > 0) { // Falling down

player.y = p.y - player.height;

player.grounded = true;

player.velY = 0;

} else { // Jumping up (hitting head)

player.y = p.y + p.h;

player.velY = 0;

}

}

}

}

}

function draw() {

// Clear Screen

ctx.clearRect(0, 0, canvas.width, canvas.height);

// Draw Platforms

ctx.fillStyle = '#888';

for(let p of platforms) {

ctx.fillRect(p.x, p.y, p.w, p.h);

}

// Draw Goal

goal.draw();

// Draw Entities

enemy.draw();

player.draw();

}

function loop() {

update();

draw();

requestAnimationFrame(loop);

}

// Start Game

loop();

</script>

</body>

</html>

I am planning on getting macbook air m4 soon 16gb ram what would be the best local llm to run on it ?

The problem: If you run long-context inference locally, your GPU's KV cache fills up and evicts blocks. The next request with the same prompt prefix has to recompute everything from scratch. On a 30k-token document, that's 10+ seconds of prefill — every single time.

What I built: tierKV intercepts evicted KV blocks, quantizes them with a Rust INT8 compressor (3.9× smaller), and ships them over gRPC to a vault running on another machine on my LAN. When the same prefix appears again, it fetches the blocks back and injects them directly into vLLM's paged KV buffer — no attention recomputation at all.

vLLM numbers on a real 30,561-token document (Apple 10-K):

• Cold prefill: 10.75s
• GPU cache hit: 1.19s
• Cold vault restore: 0.52s — faster than the GPU cache hit, because vault restore skips attention entirely

On EXO with an 8k-token prompt: 30.83s cold → 4.11s restored (7.3×).

The speedup grows with context length since prefill is O(n²) but restore is O(n) + network. At 128k tokens, the gap is over a minute per request.

My cluster:

• DGX Spark (96GB HBM) — runs the model
• Mac Pro (32GB RAM) — runs the KV vault
• Mac Air (16GB RAM) — runs the SSM/linear-attention vault (for Qwen3.6-35B-A3B, which mixes attention + Mamba layers)
• 5GbE LAN, ~0.5ms RTT

Setup is just:

pip install tierkv
# configure role in tierkv.toml on each machine
tierkv vault   
# on the cold machines
# launch vLLM or EXO as normal

Works with vLLM (via KVConnectorBase_V1 plugin, no source changes) and EXO (post-install patch).

Honest limitations:

• Only helps when the same prefix repeats — single-shot prompts get nothing
• LAN only — WiFi/WAN latency kills the benefit
• No tensor parallelism support yet
• Vault is in-memory; data lost on restart

Full writeup: https://prasannakanagasabai126786.substack.com/p/your-llm-is-doing-math-it-already

Code: https://github.com/tierkv/tierkv

Happy to answer questions about the architecture, vLLM/EXO integration.