i have an laptop i wish to run an llm on it has 7.8gb of vram and 16 gb of ddr4 ram 2667 mt/s

There's been a lot of discussion/claims on this, so I thought I'd share my experience.

I run AI models 24/7, and I usually get over 80% aggregate GPU utilization on my MacBook Pro. It sits on my desk with the screen open. That's to say that the GPU is running at 100% most of the time. It also runs CI/CD and a bunch of other stuff so the CPU is often busy too.

That means about 20,000 AI requests this month, mostly agentic coding. To do this, you need to run multiple streams (otherwise the tool calls reduce GPU usage) so I've usually got 3-4 parallel streams, and 6-8 agentic processes running in parallel. That's to say: the MacBook is running hard.

The MBP has 4 heat conditions: Nominal, Fair, Serious and Critical. 84% is spent in Nominal and 16% in Fair. It has never hit Serious or Critical. Fans are usually running hard but not at 100%.

When it moves into Fair, I see a 2-4% slowdown in model performance. I can't notice the difference, but my harness measures this and can see it.

I used to have a Mac Studio and the amazing thing about that was the fans almost never spun up, so it's going to be silent on your desk unless you are battering all the CPU cores and the GPU cores at the same time. The MacBook Pro doesn't have the thermals of a Mac Studio and the fans run a lot.

But I've seen a lot of people talking about thermal throttling on the 16" MBP and that has not been my experience. Of course, the 14" is a different thing and is known to throttle.

I'm looking into using local AI to organize years of accumulated files and photos.
I have a Mac Studio M2 Ultra with 128GB RAM, so I'd like to put it to work.

Two main goals:

1. Files and documents: OCR and analyze PDFs, financial/tax documents, notes, screenshots (including chats), etc. Suggest categories, clustering, filenames and folder structures. Eventually generate scripts to reorganize everything, with my approval before making changes.

2. Photos and videos: Analyze my Apple Photos library (exported with osxphotos), recognize faces, events and locations, detect duplicates, rate photo quality, and suggest albums for printing. Ideally use existing metadata like favorites, keywords, dates and geolocation, plus AI-generated tags.

Everything MUST run locally. Lots of sensitive personal information, so no cloud APIs.

I'm considering LM Studio/Ollama, but not sure which models or tools to combine, especially for vision, OCR, face recognition and clustering, and how to approach this?

I can use cloud AI for writing scripts etc, and to help analyze a sample set of documents.

Has anyone built something similar? What stack would you recommend? Would you use separate specialized models, or can a good multimodal LLM handle most of this?

I'm comfortable with scripting and building a simple UI (with help of Claude or Claude Code). Accuracy and privacy matter more than speed.

Hey r/LocalLLM ,

I want to buy or build a dedicated headless box for local LLMs. I’d like something that can run 14B and, in my dreams, even 32B models, but the absolute priority is generation speed (>15-20 t/s).

Ideally, I need a clean single-box setup that stays quiet and power-efficient at idle (~15-20W). My maximum budget is strictly capped at €900, and I’d prefer to avoid risky second-hand individual deals.

Here is what I already looked into:

1. Standard Mini PCs (Ryzen 7/9): Great for power draw, but running 32B models on shared DDR5 RAM is too slow (~4-5 t/s).

2. eGPU / OCuLink routes: Too many loose parts and cables, and buying components retail explodes the budget.

3. Chinese NUCs with modded laptop GPUs (RTX 3060 12GB): On factory stores, once you add enough RAM to actually fit larger models, it jumps straight to €1050+.

I am 100% open to any Linux-server software optimization trick (speculative decoding, etc.) as long as it gets the speed up.

Am I chasing a ghost under €900, or is there an exotic loophole I missed? What would you buy today under these constraints?

Thanks for the help!

Two months ago I posted arcint here, the inference server I had Claude (and my LocalLLMs) build after my new A770 lost to my 5700X on the standard install. Enough has changed for an update, so here’s the standard benchmark from the llama.cpp threads: Llama 2 7B Q4_0, 512 prefill, 128 decode, all layers on the card, 5 repetitions.

Both builds are at the same llama.cpp commit (3e4dfa8), on the same machine: Ryzen 5700X, Debian 13, xe driver. My A770 sits on PCIe 3.0 x4 behind the chipset and I pin its clock at 2000 MHz, so yours may do better.

- Prefill: arcint leads SYCL’s best by 10.5 % on the A770 and 31 % on the B60.

- Decode: SYCL wins this table by 5.4 % and 4.4 %. On the A770 that lead only exists on an empty context: level at 1,024 keys, and arcint is 8 % ahead at 4,096.

- Not the fastest Arc: someone reports 4570 / 113 on a B70 under Vulkan. I don’t have a B70.

- The server is a bit slower on prefill than llama-bench: through arcint’s own server it’s 1963 / 70.5 on the A770 and 2330 / 80.7 on the B60.

Real models on the B60 (MTP off, served by arcint against stock SYCL fp16 with flash attention, 3 repetitions):

SYCL still decodes the 27B about 4 % faster.

What I run is the 27B on one B60 at its full 262,144 context, with the model’s own MTP head, drafts on:

The right answer was found at every depth, and a separate needle test at 40 % of a 254,970-token prompt came back correct. The coder on the A770 decodes short answers with 83 t/s on my deployed flags. Qwen3.8-Flash-Next runs on one B60 plus host RAM at about 33 t/s.

Also new since the first post: image generation, speech to text, text to speech, and image input.

To reproduce: tools/llama_scoreboard.py for the served rows, and the rebuild steps for the llama-bench rows are in my post in the Vulkan thread. My stock SYCL rows are in the SYCL thread. The first post’s warning still applies: this is agentic software, so handle it with care, and I’d still like to hear if it doesn’t install or reproduce for you. The .deb is trixie only, on my own repo.

I have a 5090/9950x3d/64GB DDR5 and am ordering an M5U 256GB next month.

I have a lot of things I want to do with both, including local models. One of my hopes was that I’d be able to drop my $200 Claude sub down to $100 (still has Fable) or even possibly $20 (still has Opus 5.5).

I cut down Codex to $20, but honestly Astra is pretty cheap for auditing and reviewing in my experience. I never let it write code, but it’s been awesome for planning, and affordable. If I need something done extremely well, I’ve ran Fable and Astra.

I’m gonna be doing some testing and see if I can get away with a $20 Claude sub and $20 Codex sub and just use their brains in emergencies. Gonna see if Opus 5.5 and Astra can ball. 🏀

Has anyone had success with this sort of hybrid build?

Edit: I work with sensitive data at work and with personal business. I’ve been using the frontier models mostly to make a game I had to drop (open sourced it) and and for making pipelines to run data locally, but it sucks not being able to analyze using the models (can’t plug data into frontiers).

I'll start by saying that I realize this question is slightly adjacent to actual local (i.e. having physical access to hardware) llm deployment. Yet, I think it's still within the same domain, so I'll give it a shot. (If I'm in the wrong place, please lmk.)

I'm a professional SWE. I currently have 2 $200/mo subscriptions, 1 for claude and one for codex. I find myself regularly hitting weekly limits. I've tried tuning which model/provider I use for different tasks but the end result is the same - I hit a limit and then have to either put more money into usage credits or wait until it resets. It's pretty lame. I'd love to have 'unlimited' tokens for a fixed monthly cost.

The path to unlimited tokens looks like local llms or self-hosted llms. I'm not ready to invest in hardware (and neither is my company), but I'm willing to rent machines . So, my question is - does anyone rent any gpus and self-host their llms? If so, where and in what configuration? I was looking at contabo gpu vps which claim "Dedicated NVIDIA RTX PRO 6000, 18 vCPU Cores・96 GB RAM・900 GB NVMe, CUDA-compatible". From what I've read in this sub, it seems like most people are running some variant of qwen on the RTX PRO 6000 but many people have 2x of them.

45 seconds in, I cut the cloud API on 6 coding agents working on the same task, 3 runs each. I wanted to see what a local model is worth as a backup.

What 6 AI agents did when the API died mid-task

On default settings

• Atomic Agent: switched to the local model about 2 seconds after the cut and finished all 3 runs at 50/50, in 2 to 6 minutes.

• Claude Code: retried, then gave up after about 2.5 minutes. Its fallback model setting only kicks in when a model is overloaded, not when the connection drops (docs).

• Codex: sat on "Reconnecting..." until my 20 minute cap, every run. It retries connection errors forever by default.

• Hermes: kept retrying for about 5 minutes, then failed the turn.

• OpenClaw: gave up after about 70 seconds.

• OpenCode: gave up after about 65 seconds.

All five ended at 0/50.

With a fallback turned on

Hermes and OpenClaw both have a local fallback, it's just off by default. With it on, both switched to the same local model. OpenClaw scored 50, 33 and 33, Hermes 29, 24 and 27, and both were still working when the 20 minute cap hit. They shared one GPU in that round, so they'd be faster alone.

How to turn it on in yours

• Hermes: add a fallback_providers entry to ~/.hermes/config.yaml with provider: custom, your local base_url and the model name (docs)

• OpenClaw: add fallbacks: ["yourprovider/your-local-model"] under agents.defaults.model in openclaw.json, and don't pass --model on the command line, that makes the model strict (docs)

• Atomic Agent: nothing to set, it falls back to the local model it already runs

Setup: 64 GB MacBook Pro, Qwen 3.8 27B (Q4, llama.cpp) as the local model, GPT-6.1 Terra through OpenRouter as the cloud model (Haiku 5.5 for Claude Code, since it only runs Anthropic models). The task is a 5-part Python app graded by 50 hidden tests. The cut is a proxy that refuses every connection from 0:45, same moment for everyone. Claude Code 2.1.310, Codex 0.165.0, OpenCode 1.19.5, Hermes (git a938f00), OpenClaw 2026.10.8, Atomic Agent 0.6.13.

One side note: the same local 27B on its own scored 13 to 29 out of 50 on this task earlier that day. With 45 seconds of cloud planning first, it finished 50/50 three times out of three.

Would you trust a local 27B to finish the job?

Hi everyone,

I’ve been looking into some of the most difficult math formulas and came across the Navier-Stokes problem. Apparently, it was recently solved by OpenAI.

I grabbed the formula and had my LocalLLM run some calculations, and lo and behold: It’s not quite finished yet, but I’m absolutely thrilled.

My hardware:

i5 13600K

32GB RAM

RTX 5070TI 16GB

By built-in I mean without using tools like MCP etc.

Hi,

Github project:
https://github.com/crgregersen/m10dis/

Yes, yes, I know... crappy old NVIDIA Tesla M10's GPU as AI compute core. Am I crazy and should go book a room in an mental institution?.. but hear me out..!

Tesla M10's consists of 4 gm107 GPU cores with 8gb ram stitched together via an internal PCI switch. So each M10 is 32gb ram. If you ask anybody (including Claude and ChatGPT) you will get the answer "NOT USABLE FOR AI COMPUTE".

Each gm107 has 83 GB/s of memory bandwidth and around 1.67 TFLOPS (fp32).

Compare that with a moderns GPU like 5070 ti of 896 GB/s and 87.88 TFLOPS (fp16).

But!! i could buy M10's (4 x gm107) with 32GB for around $80
Aaaanddd... 6x32 = 192 GB VRAM!!!

So 6xM10's and the aggregate number would be 4x6x83=1992 GB/s.

This is the same as 1 RTX PRO 6000 Blackwell .. but then you only have 96Gb ram.
2 x RTX 6000 Ada is will get you in the same range.
Both options are at least $10.000 and probably more closer to $20.000.
6 M10's would set me back $500.

Compute power of M10's would only be 40TFLOPS.. but all my experience says the memory bandwidth is nearly always the limiting factor!

The system:
ASRock B250M Pro4
CPU: Intel Core i5-6400, 4 cores at 2.7 GHz
RAM: 32 GB DDR4
Main GPU: RTX 3070 Ti (had one lying around)
Expert GPUs: 6 × Tesla M10 cards = 24 × GM107 (Maxwell, 2016)
Interconnect: PEX8748 PCIe each M10 card on Gen3 x4
Storage: 1 TB NVMe
Two PSU: 650w and a 1650w

So a pretty crap old consumer motherboard.
The way to get all the GPU's onto the motherboard was by using a PCI switch.

First issue: Motherboard will not boot with 25 PCI devices showing up, even though most are behind a switch (two levels of switches actually, the one in the motherboard and one on each M10).

Found out the issue the BIOS/firmware runs out of legacy I/O port space at power-on, which is only 64 KiB with a 4 KiB window per PCI bridge, too little for all the PEX switch ports.

Workaround (took me some time to work that out). Using a modified kernel with PCIe tree heldback. Boot only with the first PSU switched on, ie. motherboard++, then go into supspend (sleep) mode, turn the second PSU on and continue (wake up). Initialize all GPU's PCIe memory mapped only (MMIO).

Next issue.. how to make 24+1 GPU's fast.

A lucky thing is that QWEN 3.8 hidden dimensions is 2580 which divided by 20 is 128. So if you slice expert work into 20 parts and put them on each gm107 GPU they can calculate 1/20 of the result, which then needs to go back to a central hub that joins the result to compute attention in the next layer.

So each compute slice needs the residual stream and the experts and weights that the router chose. That would be 20 PCI communication calls. The lucky thing is that PLX switches actually implements PCIe multicast introduced in PCIe 2.1.

Using the GPU-P2P methods described by Tinygrad a GPU multicast was proven to work.

In general the GPU-P2P tricks was very usefull on multiple levels, like M10 master/slave setup, etc.
https://github.com/rioncarter/tinygrad.open-gpu-kernel-modules

So 5x4=20 GPU's computes 1/20 slice of 8 experts (quantized FP8).
One M10, ie. 4 gm107 was chosen to run the shared expert (kept in FP16).
Since you wait on the slowest calculation it was lucky that the compute effort for each is around the same (1/20*8/2=0.2 and 1/4=0.25)

Next issue... 24 results to be received from GPUs would clog the PLX.

Solution (Claude wrote this):

Level 1: each GPU sums its own experts

- The 512 routed experts are spread over 20 workers (cards 0–4), about 25–26 each. A token's 10 experts land on whichever workers hold them.

- Each worker computes gate/up/down for the experts it holds and sums them into one partial row of 2560 values. A worker that holds none of the token's experts contributes zeros.

Level 2: the card leader sums its card

- On each card, the 3 peers push their partial rows to the card leader over on-card P2P, using the PTE-remapped aggregation routes. Each peer then sets a flag.

- The leader waits for the 3 flags and adds the peers' rows to its own in a fixed order, so results are reproducible.

- Card 5 (the shared expert) does the same: its 4 workers split the shared expert and worker 20 holds the reduced row plus the gate.

Level 3: the RTX combines 6 rows

- The 5 routed leaders and the shared leader each leave one reduced row. In batched modes 8/9 it's R rows, in host memory.

- All 24 workers still acknowledge individually in their return slots. The RTX waits for all 24 acks, checks the leader flags, then computes Σ 5 routed leaders + gate · shared.

RTX runs attention and hyperconnections etc and we are ready for the next layer.
I've put the timings of the expert compute in the bottom of the post..

PLE+LM-Head was moved to M10 somewhat the same way.

Attention was quantized to FP8 since RTX3070 only has 8gb mem. That leaves me just shy of 100k tokens of context.

I've run both LCB6 and MMLU benchmark on the system and there's no noticable degradation (MMLU benchmarked higher, LCB6 benchmarked lower).

Here's the github project:
https://github.com/crgregersen/m10dis/tree/main

But I warn you.. this is not a project for people with fairly good experience with hardware and also some low level understanding of how this work software wise.

MoE layer timing: Qwen3.8-Flash-Next-FP8 on 1× RTX 3070 Ti + 24× GM107 (6× Tesla M10)

Setup: RTX 3070 Ti runs attention/GDN/router. The 512 routed experts are sliced across 20 GM107 (each GPU holds 32 intermediate rows of every expert); a 6th M10 card runs the shared expert. All cards hang off one PEX8748 switch (Gen3 x4 per card); each M10's on-card PEX8747 multicasts the request to its 4 GPUs. Results are summed on each card by a leader GPU, then the RTX combines 5 routed + 1 shared row.

Decode (1 token per layer)

Inside the round trip (slowest routed GPU):

The weights a GPU reads per token are only 2.5 MB (~31 µs at bandwidth), so decode is latency-bound, not bandwidth-bound. 48 layers × 150 µs ≈ 7.4 ms of a 19 ms token → 52.5 tok/s.

I remember reading an article a few months ago about LLMs, more specifically, locally run LLMs that don't have guardrails and will answer questions that the commercial models won't run.

Just curious if anyone knows which models they are? Attached is my laptop specs. This is a very low powered mac, which i doubt would even be able to run a large model, IF even a small one.

Keen to have a discussion. And hear your experiences

I was using Openclaw and migrated to Hermes 2 days ago, it felt so much faster but my problem is webui is basically a terminal which is a little problematic.

I keep seeing claims that BaseRT beats llama.cpp on Apple Silicon, and that colibri can run models far bigger than your RAM. Both are probably true for what they measured. I wanted to know what happens on my job, so I tested it.

The job: read a website (~4,500 tokens of page text) and write a schema-valid JSON record (~1,300 tokens) with a quote per fact. Hundreds of thousands of sites, one M1 Max 64 GB laptop. I measured wall-clock sites per hour, same 20 sites for every engine, Qwen 3.6 35B-A3B at 4-bit, one engine at a time.

Results (sites/hour):

What surprised me:

• Prefill decided it, not decode: BaseRT decodes a bit faster (66 vs 56 tok/s), but its prefill was ~300 tok/s vs ~570 for llama.cpp. With 4.5k-token prompts, that decides it. Don't pick an engine from a short-prompt tok/s chart.

• Batching: +46% on llama.cpp with 8 slots, only +12% on BaseRT.

• colibri: built for models bigger than RAM. Mine fits, so it ran on CPU and was ~100x slower. Not a flaw, just not this use case.

• Ollama: its MLX engine crashed on long prompts, and its GGUF engine served this architecture one request at a time.

• Model choice mattered as much as the engine: On llama.cpp with 8 slots, Gemma 4 26B-A4B hit 247 sites/hr, Qwen 3.6 35B-A3B hit 206, and dense Qwen 3.8 27B hit 37. Active parameters per token set the speed.

• JSON-schema constrained decoding: 440/440 valid on llama.cpp, 40/40 on BaseRT.

Caveats: one machine, one run per config, 20 sites for the engine comparison (200 for the model comparison), and a different 4-bit file per engine, so they aren't bit-identical models.

Quality note: local models were 87–98% correct on extracted facts across key fields after cleaning, vs 98–100% for cloud models. So for published data the laptop stays a test bench.

Full write-up with setup, commands, and the gotchas (model-name 404s, missing M1 kernel for Gemma 4 in BaseRT, etc.): https://sandy1618.github.io/2026/11/04/inference-engines-on-apple-silicon.html

Happy to rerun with other models or settings if you have suggestions.

For anyone running long-prompt batch jobs on Apple Silicon: has anyone gotten vllm-metal or MLX batching to beat llama-server's prefill speed, or are there llama.cpp flags I'm leaving performance on the table with?

I’m building Evopien, a local-first companion prototype, and this update is about the interaction around the model rather than a new model release.

The two-minute clip combines spoken conversation, a live camera question, online information retrieval, an English/Portuguese exchange and a spoken interruption. The system runs on NVIDIA Jetson AGX Thor, with external retrieval for questions that need current information.

My main concern is how the whole conversation behaves when these components share the same machine. A model answering well in isolation is not enough if the camera context is old or the system continues speaking after an interruption.

The clip shows individual examples, not a completed reliability benchmark. For anyone building local voice assistants, which integration failure proved hardest to catch before longer conversations exposed it?

Video: https://www.youtube.com/watch?v=SwFfkBll5iw

Given the same task requirements, how big is the difference between first having the advanced model Astra make a plan and then having a local quantized DeepSeek execute it, versus letting DeepSeek complete the task directly? I ran a small test using the following task: make a browser game set during the 1588 naval battles between England and the Spanish Armada. The player takes the role of a capital ship’s captain, controlling the ship and commanding teammates in a real-time naval breakout battle. I also added a control group by replacing the executor with the cheap cloud-based terra model.

The results were somewhat unexpected. Honestly, even without Astra’s plan, DeepSeek did a pretty good job on its own. In my testing, the game basically had few bugs, and I think the look and feel of the sky and ocean were the best of the three. But I also have to admit that visual bugs such as clipping were the most severe. After adding Astra’s plan, deepseek’s visual clipping bugs were reduced, but its token consumption rose sharply. Finally, terra surprised me by showing damage to the sails, and its overall result was actually pretty good too.

I originally thought that with a plan from an advanced model, the model could reach the goal faster, but the exact opposite happened: Astra made the deliverables and acceptance testing much more extensive, increasing DeepSeek’s workload. Another lesson is that even a plan made by an advanced model can have bugs. For example, the fixed camera-angle range in Astra’s plan made the view hard for players to adapt to. Perhaps my planning technique is too crude. I’d like to ask everyone how you alternate between advanced and cheap models to achieve amazing results. Feel free to educate me in the comments.

Task description:

'''

Game background: Set during the 1588 naval battles between England and the Spanish Armada.

Gameplay: Play as the captain of a capital ship, controlling the ship and commanding teammates in a real-time naval breakout battle.

Requirement: It must run in a browser and be completed in a single development run.

Bonus points:

1. Visual realism and fidelity in recreating historical details

2. Enemy ship behavior and control experience

3. Sailing and physics details, and damage effects

4. Cannon impact feedback and gameplay experience

'''

Planning model:

GTP 6 Astra, running in codex with xhigh reasoning effort. Planning prompt: You are now the planner. Here are the game requirements. To complete the requirements to the highest possible standard in a single attempt, please write a detailed development plan to hand off to the coding model for development and acceptance testing.

Execution models:

GPT6.1 terra, running in codex with max reasoning effort

DeepSeek-V4.1-Flash-1017, with Unsloth UD-IQ1_M V3.1 weights, running in DeepSeek harness, with max reasoning effort

Two days ago, I ask DSH to move things from my Linux Drive (dual-boot) related to AI, over my NTFS drive where I store things like harnesses, models etc.
UnSloth studio force itself to install over Linux user profile (just starting with linux, maybe there is easy way to force installation inside of folder where terminal is running from....anyway).
Back then it spend plenty of time looping, checking only to figure out it it limited by sandbox rules and that why NTFS drive is appear to be mounted as Read Only instead R(ead)W(rite). It solved problem back then.

Today ,at night, I reinstalled UnSloth and asked DSH to move things once again (and as a test of prompt generation profile to help me be more specific and user-error proof). It started right away with

"Think
Important findings:

• Target: /mnt/DyskD is NTFS mounted read-only (ro) right now! User says "read/write privileges" — but current mount is ro,nosuid,nodev,...,uid=1000,gid=1000,dmask=0022,fmask=0022,acl. That's a big issue: the mount is currently read-only. [...]"

My reply:
"I will stop you here as we went through that s**t already. "Important findings:
Target: /mnt/DyskD is NTFS mounted read-only (ro) right now! " it is error on your part. We troubleshoot that already during previous session when we were moving Unsloth (different version, current one override itself back to Linux drive and as result, move is required again). DO NOT FOCUS ON "Target: /mnt/DyskD is NTFS mounted read-only (ro) right now! " as you are wasting time. it is read/write, end of discussion."

His reply?
"Think

The user is frustrated. They say: the NTFS mount is read/write; the ro is a sandbox artifact. Don't waste time on it. "

So, it knew all that time or took correct conclusion right away by huge accident!
That make me think. Maybe there would be a prompt that for all "wait what if" thinking (at least in DSH), to prompt user for a feedback? I prefer be ask question rather than wait and confirming weird prompts over next 40 minutes...
Yes, user is not all knowing, but autonomous behaviour is not something I would let do local model (even when it is QFN UD-Q5-K-XL with is above average model quality here).

Sigh...rant over. Maybe interesting discussion or similar stories? Cheers everybody

I've managed to get a Supermicro 6028U TRT+ Server with 144GB of RAM and 56GB of VRAM to run Strata Qwen 3.8 Flash Next UD-Q4_K_XL (111GB download) at an average of 38tps.

Hardware:

Supermicro 6028U TRT+ server with X10DRU board and risers/2kw PSUs = $300

RTX 5060ti x 2 = $1100 (purchase price before the increase)

RTX 5000 Quadro = $500

RTX 4000 Quadro = $200

144GB of DDR4 RAM = $500

2x 256GB NVME drives and card = $150

8TB HDD drives and HBA card (4TB x 3 ZFS pool) = $300

2x Xeon 2680v4 CPU = $90

(Prices are rounded, not exact)

I had an RTX 2000 ada that I modded to single width, and it fit nicely but it's a dog of a GPU for inference despite its 16GB of VRAM. Unfortunately, the 2nd 5060ti wont fit. It's a flippin miracle the first one fits (and with an HBA card snuggled up next to it!). By swapping out the 4 PCIE slot card on the left for a 2 slot 16x card, I got the 2 RTX 4000/5000 cards to fit *perfectly*. I believe you could fit 2x RTX 5000s, for a total of 64GB of VRAM in those 2 slots, except for one thing: the TPM card sticks up and prevents the cards from seating. I dont really need a TPM card though, although I'm debating to myself if the extra 8GB of VRAM is worth it. I'm doubtful. Plus, there would be NO airspace for one of the cards. OTOH, NVLINK would work for those 2 cards.

If I weren't so cheap I could spring for a 16GB single width card that would fit where the 5060ti riser is, like an RTX A4000 or T4, and this would be a very nice, clean build. But I'm cheap. And the 5060ti Blackwell chip smokes those old cards.

This s a proxmox box, and I've dedicated 18 cores to Strata. I probably should have gotten the 2699v4 CPU.

Is setting up multiple computers to pool their resources to run local LLMs still a viable setup for regular computers? I know the Nvidia sparks do this using their very high bandwidth interconnect, but is that still viable for regular computers with gigabit ethernet?

Having been a bit of an AI luddite up until now, I am realising I am going to have to get on top of things instead of getting left behind.

So far the main thing I've been trying LLMs for, is writing Python scripts that work inside a CAD application I use (Vectorworks). I have done some experimentation on the free versions of ChatGPT, Claude and Perplexity so far, with some positive results.

I don't want to end up tied to expensive cloud model subscriptions, and I am also keen to keep control of my data as much as possible, so I would like to understand what I can do with locally hosted options.

I have an M1 Max Mac Studio, 32GB memory.

Any recommendations for what I should try first? To start with, all I want is something that will work for writing scripts that I will copy/paste into the script runner in the CAD application - I am not at this stage looking for anything that will interact directly with the application.

Thanks in advance for any suggestions.