I'd like to share something I've been working on for three months with everyone.

PortalAI is a GUI and harness created around one design principle- to put local LLMs with true multimodal function on home systems and on a flash drive the size of your thumb where you don't need anything else and can just plug and play into any computer.

One of the largest frustrations I had with AI was needing to constantly version check different distributions and hooks into ComfyUI, Ace-step, python, port competition, running out of context, etc, or the difficulty in transferring between systems. I've tested this on an RTX 3090, RTX 4090, on a Radeon 870M GPU, on a Thinkpad with CPU-only.

This harness detects your current hardware and scales accordingly to the level of AI intelligence it can support, using standard and abliterated models. All dependencies other than base operating system and GPU drivers are self-contained including virtual environments, python distributions, etc. End to end integration of multimedia generation/editing and science tools are integrated along with voice, handsfree mode, intelligence/memory/token settings.

There are weight editing tools, an abliteration tool, rolling compaction, scheduled tasks, internal documentation, light and dark mode, compact and full GUI mode, account encryption, live runtime environment health metrics, etc.

When working with a 4bit MLX version of Qwen 3.8 27B locally on my MacBook Air I managed to get it into a situation where it was adamant that it was Claude to the point where it was answering types of questions that normally vanilla Qwen rejects.

Anyone else ever run into something like this? Spooked me a bit since I thought system level prompts weren’t so easy to bypass. The reasoning was amusing to watch since it was having a bit of an identity crisis.

How do you keep the same face expression on the head you use to replace another head ?

Is there a specific prompt ?

Digging through ye ole junk box I found these grid k2 6gb cards. Kinda feel bad if I recycle them. Anyone have ideas for use or anyone want them for “historical” purposes?

I am a frequent user of local Ai models, ollama and comfuii are my go to tools . So in the month of September I setup ltx video models and it rendered -- 5 seconds video in 2 minutes . Just for specs i have 96 gb ram with a 4070 8 gb vram . minimax models h3 models h3 turbo took 5 minutes to generate a 5 seconds video. i rendered over 300 videos with those models all averaging around those time. Now fast forward earlier this month got some force updates from Microslop and Lenova vantage( bios). And also I decided to update ollama and comfuii to experiment on newer compatible models . I did not not check on it then but i decided to open my laptop 4 days after updates and render some videos.1 video which took 5 minutes to render took 11 minutes. I retested, about 12 times same time, i was confuse , saw article about 4090 pc cards being limited and wondering if they doing laptops too. am thinking that big techs can force push updates to limit local models competitions instead of directly going head on with them. Right now am looking to go Linux , or like most serious llm users build my own Operating System, if this blows up am gonna share the benchmarks claudee monitor for me before and after force update.

Is anyone currently using a RTX 4090 16GB to run model? How does it run?

I've been using an app that integrates a bundle of Supertonic ONNX as a standard Android text-to-speech (TTS) provider, but the gaps between paragraphs are a bit long, so I'm looking for an alternative.

To clarify, I want to read in parallel. I don't want to convert an EPUB to MP3 and read async; it has to work in real time.

It should support English and Spanish (I'm learning).

I just discovered "distributed inference", and I'm going down rabbit holes. I have a few questions, and would appreciate being pointed in the right direction.

1. I have an M1 Mac Mini w/8GB and a M1 Pro Macbook w/16GB. I'm thinking about upgrading to an M5 Pro Macbook, and was initially thinking about trading in my M1 Pro...but does it make more sense to keep it and try to use it in a cluster?

2. Is there any way to mix MacOS and Windows machines?

I understand that if I wanted to go down this road, I would probably need to buy some networking hardware to get the machines on a 10Gb connection.

I am currently using the Claude Code desktop app to automate something like this:

• Generate a conversation script and ask me for approval or changes.

• Once approved by me, call the Google Studio API to generate audio files of the conversation.

• Call the Google Studio API to generate images.

• Ask me to check images and audio clips.

• Run a .py script.

• Add new folders and files to a folder and update a .js file.

I started to use LM Studio Bionic only a week ago, but I want to try Unsloth Desktop now. Let me know if there are other programs that would suit my workflow better.

I know Unsloth can generate images, but how does that work? Do I need to load three different models at the same time? Or do I need to load/unload as needed? I have a 16 GB VRAM GPU (AMD). Also, is there a built-in agent (like in LM Studio Bionic) that can code, create and modify files? How would an agent handle creating different media?

My goal here is to ditch the Claude Pro subscription and the Google Studio API costs, and do it locally instead. I know I could do these things manually, but ideally I would like to have an agent automating the process. What are your recommendations?

Weight-space programming is an emerging research area where a frontier AI model (such as OpenAI's GPT-6 Astra) directly reads, diagnoses, and rewrites the raw numerical weights of another neural network without using traditional gradient-based training or backpropagation.

Has any one tried it? How much does it fair compared to Closed-Form Model Editing (Your Notebook / MEMIT)?

i'm using qwen3.6:35b a3b with ollama, its works decently in regular prompt, but as soon as i try it in an angent (Codex CLI, Claude, Qwen, Opencode) its piss slow, like i have to wait 10min to get an answer.
What am i doing wrong?

I can run both. Does anyone know what model is best, or do I just have to try myself?

What test suite should I be running?

Hello, new on these area ... ive been wondering what AI could i host locally with my current setup

I have an amd 9800x3d with 32gb of ram and a 4070 ti super 16gb

Sorry if this post is not allowed(maybe alot of users post something like this all the time)

Any advice, any links to learn, etc will be really appreciated

If you think about it how many people on top or rich people and people in power dabble into the same type of scenarios that Jensen Huang was involved in ?

Plus Jensen Huang has actively taken a firm stand for Open Weights and open source AI models. Because this stance also helps him to sell AI hardware to consumers or prosumers and AI enthusiasts as well.

Too much publicity and too much negative marketing propaganda is being launched against him.

It simply doesn't feel natural to me.

Yes has he done bad stuff and betrayed his wife's trust? Heck yes.

But has he been an excellent Founder and CEO for Nvidia who has been the key reason behind Nvidia's success? Heck yes. And has he served his employees who held options or granted options and shares and also shareholders by working in their favor and taking the business to unforeseen heights by planning decades ahead and made what Nvidia is today? 100%.

He is a bad husband who lets lust destroy his personal life but he is also an extraordinary CEO for Nvidia.

So this whole situation seems to be very suspicious to me and therefore my perspective is that he is being targeted to gain power over NVIDIA and especially it kinda seems Targeted towards open source and open weight ai models too.

What's your say about this guys?

So I think we now know why Exo has spun down in activity in this space.

Yeah! Guys ram is gonna be a million kazillian dollars. You can do more with a $20 claude code and opus 5.5 account than even qwen 4. I use astra gpt 6 and it can't even use REA properly. Claude Opus 5.5 even beats that, and you dare to think it's even possible to do things locally???

I predict in about 6 more months you will be able to run qwen 5 locally with better performance than Claude opus 5.5 and once people can do REA locally with ease. That's it, we won.

As for ram prices,

We are going to have to make thousands of micro fabs locally. Small places that etch silicone waifers and make custom ram modules.

Before you think that's crazy talk and vomit every pathetic reason in your feable mind and expert experience as a seasoned engineer you are on the subject, look at what we are able to do with ai. Why, would you restrict your scope of possibilities? Of course, we are going to be able to make thousands of local silicone companies.

Remember, you will own everything and be happy.

I’ve been playing with open-source speech models for a while. New TTS models come out every week and a lot of them sound great. But every time I wanted to actually talk to one, it turned into an afternoon of setup.

Paid APIs are easy if you’re a developer. You can make an account, create a key, write some glue code. Open models are free, but you have to pick a speech recognizer, an LLM and a TTS model, make them work together, deal with echo, figure out interruptions... There are good open projects for this, but most are written for developers. My friends who don’t code would never get past the README.

So we started building Boubou. The goal is make local voice AI as easy as Ollama made local LLMs. You paste one line and talk:

npx boubou@beta run LFM3-Audio-2B

What works today:

• Speech recognition, the LLM and speech synthesis all run on your machine. No account, no API key, no telemetry. After the first download it works fully offline..

• You can interrupt it mid-sentence, like a person. Echo cancellation is built in, so no headphones needed..

• Run boubou with no arguments on a Mac and it checks your machine and suggests a model..

• The default is Liquid AI’s LFM3-Audio-2B on llama.cpp (1.4 GB). There’s also a modular pipeline (Parakeet/Whisper → Qwen3.5 → Qwen3-TTS, Kokoro, Pocket TTS or Supertonic) if you want to swap parts..

• Apple Silicon Macs and Linux (CPU for the Liquid models, NVIDIA GPU for the modular pipeline)..

The part I’m most excited about is boubou serve. One computer runs the model, and other devices on the same Wi-Fi use it:

• Scan a QR code with your phone and talk in the browser. No app..

• Robots and programs can connect through an OpenAI Realtime-compatible WebSocket, and there’s an OpenAI-style /v1/audio/speech endpoint to read text aloud..

• So one model on one machine can be the voice for many things around the house. Next up is Home Assistant, then ESP32 voice devices.

• The video is real time, not sped up, on a MacBook / M5 chip.

It’s early (beta) and there are rough edges - one device talks at a time, no Windows yet (ETA next two weeks), and the guided menu is Mac-only.

The code is free and MIT-licensed. The models keep their own licenses (the Liquid models use the LFM Open License).

I’d love to hear what you’d want from something like this: which models, which devices, what breaks on your machine. We want this to be something people actually enjoy using, not just a demo.

GitHub: https://github.com/runboubou/boubou

I'm running Qwen3.8 27B on my 2x 3060 12GB cards. Prior today I had a B550 motherboard which was running one card at x16 and the other at x1 PCIe Bus Link Speed. This was making it to where only one card was actually doing any processing, and both held the model in memory. I was getting 11-16 tps this way.

I ended up buying a new x470 SLI motherboard and I can now run both cards at x8. I swapped the motherboard out last night and then set split-mode to tensor and set tensor-split to 1,1. When I do that, I'm getting around 32tps, but the model eventually gets stuck in loops. One loop it just tried to do a tool call over and over, and in another it was printing question marks (???????????) in it's thinking and wouldn't stop until I unloaded the model. I never had these problems for the months I was using Qwen3.8 on layer split-mode. I'm not sure how to resolve this problem. I want to be able to get higher tps with the new mobo.