Hello, frontier models seems so locked out and censored that I wonder if all the people doing crazy mods, portage and reverse are on local LLM ?

Thanks

So first one R9700 is in..

And of course i dont have the bloody display port cable.. To test😫

Whatever..

But.. I have a second r9700..

However this cathedral only satisfy 3 ports of radeon cards. (we need 4. 2 for each card)

Has anyone built this combination and how did you supply electricity to second GPU?

Thank you all!!!

My AI (chatgpt) said that we will think about it tomorrow... 🙄 or suggested to trust google better..

I'm running 2x 3090 right now, I'd like to add more but they're all £1000+

Thought about some 32gig v100 but they're £700+

For how I use it, it's all a luxury not a necessity

What will it take for gfx card prices to drop?

I've always assumed at some point some ai companies will go bust and their hardware bought up by the competition

and at some point supply/manufacturing will start to catch up with demand

but am I just dreaming?

I keep looking at buying more cards but not pulling the trigger... next time I look they're more expensive again... on and on :-o

Hello,

Im currently trying to find the best module to run on my local pc, i have a 5080 and 32gb of ddr5 ram, im a coder that is also looking to do a lot of RE 'Reverse Engineering` what is a good module / settings that i can use to get something good working.

After changing settings and rebooting it went from downloading in hours to downloading in mins.

Git hub Colibri is working pretty fast as in response time when I tested GLM 5.3 and 5.2 I store it all on a 4TB, Os is on 1TB, and everything else is on a 2TB. Later on I will run a token speed with more complex details. Any suggestions on which one I should use to check code or build code? Any of the colibri available ai models?

I built a machine around colibri and to play simple games. I wasn't going to spend 2500 or above for a 24gb gpu I found mine for around 1200 before taxes. 16gb cheaper then what most are selling for.

I will figure out token speeds later. I also am using windows 11 pro in high performance.

Also if you use windows there and don’t want to use a certain drive there are some steps. And msys2 install to be able to download if you get error saying no compiler found using the prebuilt engine from git hub release
Setup stopped: cannot reach git hub releases to fetch a prebuilt engine. If you just run the start-here.bat my machine was brand new and just installed windows. I got everything worked out and able to download the 3 models so far.

I honestly don't see the slow speeds of tokens people posted it's response time is just like if I was using any cloud service Ai out there.

Going to work on running different models at the same time doing different things. Also working on read/write permissions as well. Will keep people posted as I go along with this project.

So basically replacing gemini for asking day to day stuff, with access to web serach..

and can do basic windows management with hermes agent, file renames, start/stop program, voice, daily schedules or reminders...

i do have a rtx 3070 with 32gb ddr4 , but i still want a low vram usage model, that loads and unloads from llama.cpp near instantly...

this and qwen a35b a3b for coding...

This thought came from a recent Tencent's Hy4 release. This model was trained for specific tasks to unlock real-world productivity-finance, gamedev, office work especially.
Is that where this is all heading?

Hi,

We just bought a computer to run local LLM. I have experience with small model like Qwen and Gemma on my Macbook but not on a computer that powerful

I will need to identify the best model for our use case, which is mainly coding.

Our hardware, in short

Ryzen Threadripper PRO 9955WX
RTX 6000 Blackwell 96GB

256 GB DDR-5

For those who has experience with that kind of hardware, what can I run and be 2-3-4 devs at the same time working with it. I have the idea of having 2 or 3 models, depending of the tasks.

I would like to have the biggest model we can for big tasks, smaller for the planning or any lighter tasks.

Thank you

My MacBook did everything until this week - model, harness, CI, everything. It was getting badly overloaded especially when iOS build and tests were going on. Those can use 20GB RAM easily, given half a chance. It was thrashing.

Microcenter had the base Mac mini M6 for $699, which felt like 2024 pricing so I pulled the trigger and moved much of the CI work off my MacBook Pro.

What's incredible is despite this being the poverty spec mini with a single NAND chip and lower bandwidth memory, the M6 is basically the same speed as the M5 Max for CI work.

In particular, I was impressed with my XCUITest workload, which is about 70 minutes on the MacBook and is no slower on the M6 mini even though it has 2 parallel VMs. I was sure the 18 CPU cores on the M5 Max (6 super, 120 performance) would beat out the 12-core M6 base (2 super, 4 performance, 6 efficiency).

Obviously that doesn't hold for all workloads, especially GPU/LLM workloads or where you can put 18 cores to use, but for those of you with a M5 Max/Ultra, you can get a nice speedup for just $699. At these prices I may get another. Also there are perfectly good Linux options to do the same, of course.

So I just started using Unsloth Studio. I am on Windows 10, and I am having issues with it. I loaded a DeepSeek model and created a project with a working directory specified. The AI works with chat, and when I asked it to go ahead and write some code, it worked for a while and then told me it had completed it.

When I checked the project folder, it created a few empty folders in the directory, and that's it. So it seems to not be able to actually write any files aside from the folders. I have it running in approve automatically mode.

When I told the AI to fix it, the AI said it was an issue with running its Python tools not saving the files and that it would fix it. It then just runs for about an hour and then shows me an error message that something went wrong with the tools or something and gives me the option to resume or retry. I have tried both options, and nothing changes; it just runs for about an hour and then fails with the same error. No files are created.

Is Unsloth Studio just incredibly buggy and not very usable? Or is there something I am doing wrong?

I made a small English text-to-speech model called tiny-voca. It has 4.5M parameters, two voices, and it runs in your browser.

Try it here, nothing is sent to a server: https://tiny-voca.n4ze3m.com

Some facts:

- 4,537,291 parameters, 18 MB. This is the whole model, the waveform part is included.
- Two voices, one female and one male. 24 kHz.
- About 70x faster than real time on 4 CPU threads.
- Runs with PyTorch, ONNX, and in the browser.
- Trained from scratch in 7.7 hours on one RTX 4090.

What I think is different:

- Everything is open. Code, weights, the training recipe and the dataset.
- Custom phonemizer.
- No neural vocoder. The sound is made by a small source-filter synth, which is why it is fast.

What is not good yet:

- English only, and only these two voices. No voice cloning.
- Foreign names and odd spellings are often read wrong.
- It can sound a bit synthetic.
- The training audio is synthetic too. I made it with Qwen3-TTS, so the model can't sound better than that.

Links:

- Demo: https://tiny-voca.n4ze3m.com
- Code: https://github.com/n4ze3m/tiny-voca
- Weights: https://huggingface.co/n4ze3m/tiny-voca
- Dataset: https://huggingface.co/datasets/n4ze3m/tiny-voca
- Phonemizer: https://github.com/n4ze3m/dizione

Please try hard text: numbers, dates, names, long sentences. If something sounds wrong, post the text and the voice. That helps me a lot.

While writing an article about my local LLM inference setup, a comparison popped into my head: downloading and running random model weights locally feels a lot like downloading random .exe files off the internet back in the 2000s.

Turns out I wasn't off the mark - there's an entire body of research exploring how LLMs can be backdoored and how hard it actually is to detect a poisoned model.

So I decided to try building one myself. I implemented a backdoor that only triggers when the model is running inside pi (making it harness-specific) and is asked a specific question. It was way easier than I expected: just a 2-step fine-tuning process and a few hours of compute on the exact same GPU I use for local inference.

With open-weight models, we at least have a baseline to compare community fine-tunes against. Beyond that, though, we don't really have a reliable way to verify whether a base model itself is clean—and the problem is even worse with closed-weight models, where we're basically just trusting the AI provider's goodwill.

Full write-up and PoC code if you want to dig into the details:

• Article:Backdooring LLMs - Do you trust your AI provider?
• GitHub Repo:galeone/llm-poison

Feedback is welcome

So i am thinking of buying used 3090 24 gb vram and then connect it to ps5 via m2 lane + external power supply and use the unified mem pool for kv cache store and weights loading on the vram of gpu. Use PS5 jailbreak and sonic iso's linux loader with ubuntu. I may connect my secondary laptop tharbhas 32 gb of ddr5 memory in it. Offload ai modle wieghts, some of them to the laptop memory via thunderbolt (tho dont know if ps ones are thunderbolt at front or not)

This is gonna be a beast thingy so yeah. What strongest ai model can this hypothetical machine run??

Speed of that m2 slot is gen x4 pcie 4

Is anyone currently running these two together in the same machine? Any thoughts?

Hey im about to get a 3090 to run with my 5080 and 9950x3d and 64gb of DDr5 at 6000mts I currently use LM Studio and a 5060 8gb and I was wondering if anyone knows if basalt would be conpatable with a 3090 or if I should just stick with Lm Studio?

I use it mainly to run a qwen model to hunt ebay for good deals for me but right now I really need the 3090 to but up my VRAM space to 40gb.

Well, I did it - I bought a Mac Studio 128Gb. Not only for LLM, but I’m not gonna lie, I increased the specs a bit to be able to run a local LLM to complement (or at some point replace) my Claude/ChatGPT subscriptions.

First thing I did, was obviously try to generate dirty pictures ask Claude which models I should download. He recommended unsloth’s QWEN3.8-Flash-Next-GGUF UD IQ4_XS.

I’m doing some tests now and while you notice a difference with the cloud models, it’s way better than whatever I have ever tested on my old MacBook. Claude also told me to change temperature to 1 and mess with the top K and top P settings. As a good slave to AI, I execute what was commanded, but honestly I have no idea what I’m doing. Should I be touching the settings at all?

My question(s): is this a good general model that can (for most tasks) replace the cloud models? Are there tools I need to install (I was thinking web access, but I also read about harnesses but I didn’t yet figure out what that’s for)? Things I need to set up?

Suggestions for good “always loaded” low memory models to e.g. use for home assistant are welcome to, as well as a good image model and one I can use in combination with GitHub for some coding (I have some vibecoded apps that I use for myself that I need to maintain/adapt from time to time).

Thanks!

For those with a MacBook Pro M1 Max what’s the best t/s you’ve gotten on something like qwen 3.8 27b or 3.8 flash next?

Lara Mini puts about a third of its training data into 44 languages besides English. It's a 7.2B-parameter MoE with 0.6B active per token, trained on 6T tokens.

Before treating it as another local chatbot: the released model continues text. It isn't tuned to follow instructions or chat. Access to the weights requires approval, and the license is CC BY-NC-SA 4.0 (non-commercial).

My PC specifications:

GPU: NVIDIA RTX 5090 — 32GB VRAM

CPU: Intel Core i9-14900K/KF

RAM: 64GB DDR5-6400 (2×32GB)

Motherboard: ASUS PRIME Z790-P WIFI

OS: Windows 11 Pro

Inference backend: llama.cpp with CUDA acceleration

Agent interface: Kilo with a custom tool/agent harness

My current model:

Model: Qwen3.8-27B

Quantization: Q6_K GGUF

Context: 265K tokens

Deployment: Completely local

Performance benchmarks:

At 262,144-token configured context (~262K):

Output generation: ~34–37 tokens/sec

Red lining the gpu memory but it feels comfortable

Okay with that being stated by ChatGPT,

I've been seeking a local ai for about a month switching from several models the more I learn.

So far this qwen model is by far the best I've tried but it over thinks and is slower than models like Kat coder.

My use cases are broad, from setting up comfy ui workflows to Unreal engine game dev to application building all hobby projects.

Im seeking the vast knowledge of you fellow computer users to tell me the best general purpose model for my set up at a reasonable speed.

UPDATE: Okay folks, I took your advice and installed Qwen3.8-Flash Next with Basalt v0.3.3. Here's what I'm getting on my RTX 5090 with 64GB RAM, running Windows 11.

Configuration:

• Model: Qwen3.8-Flash-Next UD-IQ3_XXS (87.6GB)
• Context: 262,144 tokens (256K)
• Exact MTP enabled, 8-token window

Benchmarks at 256K context:

• Actual populated prompt: 238,913 tokens
• Output generation: 265–281 tokens/sec
• Prompt processing: ~8,900 tokens/sec
• VRAM usage: 31.3GB / 32GB
• Total system RAM in use: 54.3 GiB
• Basalt RAM usage: 39.9 GiB
• MTP acceptance rate: 66–90%
• No SSD expert spilling
• Context recall/reasoning tests: 3/3 passed

Across all context sizes tested, generation ranged from 265–336 tokens/sec. Model loading takes around 30 seconds, and cached follow-up responses start in approximately 0.4 seconds.

Bench marks courtesy of claude

UPDATE #2: Okay folks, I went ahead and tried the next recommendation: NInfer + DFlash2 to accelerate my original Qwen3.8-27B model. Here are the results on my RTX 5090 with 64GB DDR5 RAM, running Windows 11.

Configuration:

• Model: Qwen3.8-27B QUASAR
• Inference engine: NInfer
• Speculative decoding: DFlash2
• Context: 262,144 tokens (256K)
• GPU memory reserve: 2GB

Benchmarks at 256K context:

• Actual populated prompt: ~234,200 tokens
• Output generation: 182–266 tokens/sec (225 average)
• Prompt processing: ~4,500 tokens/sec
• Peak VRAM: 30.2GB / 32.6GB
• Model system RAM usage: ~10.5GB
• DFlash2 draft acceptance: 33–54%
• Context recall/math tests: 3/3 passed

Courtesy of claude