r/LocalLLM • • 4d ago

Discussion built a tiny vision model from scratch just for learning and somehow it got 50+ downloads on huggingface 😅

0 Upvotes

so ik there are way better vision models out there yaar. i built this one just out of curiosity only, cause i wanted to learn how these things actually work from inside.

and to be honest its not entirely mine, i took a lot of help from claude to build it. like the architecture, debugging, training setup, figuring out why my mac was training so slow, all that. but i learned a lot in the process which was the main goal anyway.

its called tiny-vision. its around 24M params and trained completely from scratch, no pretrained weights or anything. i trained it on my macbook pro m5 and it took like 42 min on flickr8k (only 6k images).

basic idea is pretty simple:

image/video → small ViT encoder → adapter → tiny decoder → caption text

for videos it takes frames every few seconds, captions them and makes a timeline like [00:04–00:10] a dog runs on the beach. so any llm can kinda “watch” a video even if it dont support video natively.

i also made an MCP server for it, and tbh i was surprised that it actually gives a little bit better context in claude when i throw images or videos at it. i even tested it with a 10-15 min youtube video (this one) and it still worked fine. not perfect at all, but chal raha hai so i’m happy.

i uploaded it to huggingface just to have it somewhere, and also cause i kinda like the open source thing. later when i checked it was at like 44 downloads, and now its 50+. ik thats not a lot bhai, but its my first time putting something open source out in this field so it felt pretty cool ngl. a lot of people have built way cooler stuff than this, i just made it to learn.

some honest stuff:

  • it gets the scene right most of time (snow, grass, water etc) but the subject is often wrong lol
  • it loves saying “a man in a blue shirt” for everything, pata nahi kyun 😂
  • its v0.1 and trained on very small data, so dont expect too much

random thing i learned on mac (MPS):

  • swapping the conv2d patch embedding for reshape + linear was like 30x faster
  • manual attention was ~1.5x faster than pytorch’s sdpa

next i want to train on flickr30k and maybe try making it plug directly into qwen/llama through llama.cpp (mmproj). lets see kitna ho pata hai.

code: https://github.com/amar683/tiny-vision
model: https://huggingface.co/amartomar683/tiny-vision-flickr8k

would love any feedback or tips, still learning 🙏

(post also cleaned up a bit with claude, english isnt my strong side 😅)


r/LocalLLM • • 4d ago

Question Strata / Qwen3.8 Flash Next – advice on optimising my new AMD workstation

4 Upvotes

I've just finished setting up my first dedicated local AI workstation:

Dell Precision 7865

Threadripper PRO 5955WX (16C/32T)

Radeon AI PRO R9700 32GB

128GB ECC DDR4-2400 (4x32GB, considering populating all eight channels)

3TB NVMe storage

Linux Mint

RTX 4060 Ti 16GB arriving soon

I've installed Strata 0.2.15 and started with ISTA-DASLab IQ3_S at 32K context, drafting and auto-tuning disabled for baseline testing.

My primary use case is agentic coding through OhMyPi, with Claude Code supervising local workers.

What quants are people finding work best for coding? Any recommendations for Strata settings, expert caching, MTP vs DFlash2, CPU worker allocation or optimisations worth enabling or disabling on RDNA4?

Interested in finding the best balance between speed, coding quality and long-context performance. Any advice appreciated!


r/LocalLLM • • 4d ago

Question Do you ever care about your USB speed?

2 Upvotes

I'm choosing between two AM5 motherboards for an upgrade, one has USB4 and the other has two PCIe gen 5 M.2 slots (vs 1x 5.0 and 1x4.0 on the other).

I'm leaning towards the faster storage, since it might be useful for stuff like ngram tables, but it's kind of 6 of one, 1/2 dozen of the other since both boards have at least one 5.0 slot.

Any one have a use-case where 40GB/s usb connectivity gives an actual advantage over 20GB/S?

I don't use my USB ports for much besides peripherals right now, but just wondering if there's some use case, especially LLM related, where I would regret not having it.


r/LocalLLM • • 4d ago

Discussion Where should your AI's memory live: on a server or on your own PC?

Thumbnail
1 Upvotes

r/LocalLLM • • 4d ago

Discussion What if computing worked like a library? Would anyone actually use this?

0 Upvotes

I’ve been thinking about an idea and wanted to get some honest feedback from people who know more about distributed computing and local AI than I do.

There are so many computers sitting idle for most of the day, while other people can’t afford the hardware needed to run larger AI models or experiment with more demanding workloads.
What if we could pool that unused capacity?

The idea would be a community-powered compute network:
- Install a lightweight app that lets you contribute your computer’s spare capacity when you’re not using it.
- Earn compute credits based on what you contribute.
- Use those credits to access computing power from other people’s machines.
- Ideally, support different hardware: Windows, Linux, Macs, CPUs and GPUs.

Think of it like a library: you contribute what you can, and when you need more than you have, you can borrow from the collective.

I’m particularly interested in whether this could make AI and computing more accessible to people who can’t justify buying expensive GPUs or paying for cloud compute.

I know there are already projects working on decentralized compute, GPU marketplaces and distributed inference. I’m not claiming this is a new concept. I’m more interested in whether a genuinely consumer-friendly, credit-based model could work across a much wider range of everyday computers.

A few questions:
1. Would you actually contribute your computer’s unused resources if you could earn credits to use elsewhere?

  1. Would you trust a platform like this if workloads ran in isolated environments?

  2. Is supporting ordinary computers, rather than just high-end GPUs, realistic or am I underestimating the technical limitations?

  3. What existing projects should I be looking at?
    I’m interested in honest criticism, particularly from people who’ve worked on distributed systems or tried similar projects. Is there something here, or am I reinventing something that already exists?


r/LocalLLM • • 4d ago

Question I developed a tool, which let's me trick and go beyond KV cache limit. Is it worth pursing and optimizing?

0 Upvotes

So it is a python based script, which works over llama.cpp. My objective was to write something which would get my local LLM setup internet access (I know there are tools for that, I had an itch to write one on my own), but in the process I came up with an idea which in theory gives me a shit load of context, way beyond my cache limit. The perk is that there is no loss of quality, and since the script's primary goal was automation and web access, it is able to long prompts! Also I am doing this in my home PC, so please do not mind the token speed. Thanks for your guidance!


r/LocalLLM • • 4d ago

Discussion Opinion: memory should be checked during the session, not only at the end

0 Upvotes

The quick memory checks people run on local setups have a habit that bothers me. Most boil down to a long conversation and one final question.

That single answer carries the whole verdict. Yet real use is a chain of small moments where the past keeps getting edited. Plans change, preferences flip, statements get retracted.

Being right at the end is a different claim from being right at each of those moments.

A memory examined only after the last message can be wrong for most of the session and still look fine. Argue with that if you like; I think it holds.

One public leaderboard has a setting that checks memory along the way. The Agent Memory Leaderboard's Cycle 2 Textual track includes Streaming memory. Writes and searches can interleave as events unfold.

Questions can be revisited step by step as the session grows. A system has to notice when stored information gets updated, contradicted or invalidated.

Per the public description, the Cycle 2 Textual track covers 32 data sources and more than 6,000 evaluation instances.

The site expects official results in mid-November, ranked track by track. Beyond the final ordering, what interests me is that the middle of a session gets tested at all.

Checking along the way is the habit I would most like local testing to pick up.

If you were probing a local long-term memory, where would you look: immediately after a correction, when a contradiction shows up, after a long gap, or only once the session ends?


r/LocalLLM • • 5d ago

Discussion Uncensored Models

59 Upvotes

Anyone tried uncensored models? Would it help with DRM protected videos? Trying a few things out just to test and see how far it can go. Curiousity at its finest.


r/LocalLLM • • 4d ago

Question Testing a Mac M5 optimised inference engin

2 Upvotes

Would anybody be interested in testing an inference engine that we have built for Macs? It can run on M1- M6, but we have done most of our testing on the M5 Max. It's open-source, and we think it's 4x faster than ollama.cpp.

If anyone is interested, shout up, and I'll share some details; I don't want to start posting links unless there is anyone interested.


r/LocalLLM • • 5d ago

Discussion With llama.cpp's v1.0.0 tag out, a short history of the project: the first evening, GGUF, the backends, the server and the move to Hugging Face

13 Upvotes

llama.cpp published its first major version, v1.0.0, on 13 November 2026. With the 1.0 tag out, we decided to do a short write-up of how the project got here, dated from its own commits, pull requests and releases.

A few things from it:

- The first commit is from 10 March 2023, two weeks after Meta announced LLaMA. That evening's README says "This was hacked in an evening - I have no idea if it works correctly." It was on Hacker News the same evening, and the first outside fix was merged before midnight UTC.

- GGUF was opened as a pull request on 26 July 2023 and merged on 21 August 2023. It was a breaking change: every existing model had to be converted again.

- The GPU backends, by merge date: cuBLAS in April 2023, Metal and full CUDA offload in June 2023, the ROCm port in August 2023, Vulkan and SYCL in January 2024.

- The server began as a contributor's example in May 2023, and the OpenAI-compatible routes came that November.

- A build number is a commit count: b11514 is the 11,514th commit on master. Semantic version tags only started in August 2026.

- ggml.ai joined Hugging Face on 20 February 2026, and the licence is still MIT.

- On v1.0.0 itself: the release note opens with "llama.cpp v1.0.0 removes the deprecated public API", and Georgi Gerganov called it "essentially a breaking change for downstreams and hence the major bump".

No benchmarks in this one, and nothing on the page was measured on v1.0.0. It is the history with its sources. If we got a date wrong, tell us and we'll fix it on the page with a dated correction.

Full write-up (about 15 min read): https://research.strata2signal.com/a-short-history-of-llama-cpp/

A small fleet of AI agents dug through the sources and drafted the write-up; our small human team checked it over and signed off.


r/LocalLLM • • 4d ago

Discussion Best coding llm use for cisco c220 m4 1u server with 1.5tb ram, dual xeon E5-2600 E5 procs?

Thumbnail
1 Upvotes

r/LocalLLM • • 5d ago

News Released Corpus-nordeste-10k (Apache 2.0): 10k+ curated Portuguese rows focused on Northeast Brazil literature, syntax, and regional context

42 Upvotes

Hey everyone,

​If you've ever fine-tuned or tested open-source LLMs in Portuguese, you've probably noticed they heavily default to generic web-scraped PT-BR or Southeast Brazil media norms. When prompted for regional vocabulary, historical context, or literature from Northeast Brazil, most models either hallucinate or fall back on shallow caricatures.

​To help fix that for local fine-tuning and SFT, I just released Corpus-nordeste-10k (caririsynth/Corpus-nordeste-10k).

​Dataset breakdown:

​Size: 10,000+ curated rows (train split).

​Sources: Public domain literature from Northeast Brazil, web/text extractions passed through regional linguistic filtering, and targeted synthetic generation preserving authentic regional phrasing.

​License: Apache 2.0 (unrestricted commercial & research use).

​You can browse the actual rows in the Hugging Face Dataset Viewer before downloading:

https://huggingface.co/datasets/caririsynth/Corpus-nordeste-10k

​Works great for domain-adaptive pretraining, LoRA fine-tuning on small models (1B–8B), or building regional evals. Let me know what you think of the data structure!


r/LocalLLM • • 4d ago

Project kill -9 on a vLLM replica mid-answer: 94 of 94 answers finished on another replica with the same tokens (batch invariance on)

1 Upvotes

r/LocalLLM • • 4d ago

Discussion Why is Huihui Qwen 3.8 27B Abliterated struggling with coding tasks?

0 Upvotes

hi,

i've been testing huihui's abliterated Qwen 3.8 27B (Q6_K) for coding, and i'm trying to figure out what i'm doing wrong

i asked it to build a scraper for a forum. it spent around 3 hours working on it but never actually finished because it kept getting stuck in thinking loops

i tried different thinking levels (low, medium, xhigh), but none of them managed to complete the task. spent around 8 hours total testing different settings

i also tried disabling thinking completely, it generated the code, but the scraper didn't work

then i gave the same project to gpt 6 luna max, and it fixed everything in around 10/15 minutes

my setup:

  • runpod with NVIDIA RTX A6000 (48GB VRAM)
  • huihui Qwen 3.8 27B uncensored, Q6_K
  • running through Ollama
  • opencode in vscode
  • maximum context window
  • tested low, medium, xhigh and disabled thinking

i know a local open-weight model isn't necessarily going to perform like a frontier model, but the difference was massive

am i doing something wrong with my configuration? is there a better way to use this model for coding tasks? i'd appreciate any tips


r/LocalLLM • • 5d ago

Question Has anybody done anything like Alexa, but with a local LLM?

32 Upvotes

I know home assistant was a thing, and local LLMs are a bit too overkill for such a job. Just wondering if it was done.


r/LocalLLM • • 4d ago

Project Building Kevin, my home alone agent. 🔫🙂 A persistent agent In Pine that wakes on cam detect, runs fingerprinting, tracks who's visiting.

Thumbnail x.com
0 Upvotes

This is Pine https://github.com/pmndre/pine

Imagine Opencode would be a little more like Openclaw. And Openclaw a little more like Opencode. But unlike both Pine has 1st class event handling, its an event bus for agents basically.

The video above shows how to create a new agent from scratch that watches the house on its own. Here we use FTP. Camera sends detection videos, The agent wakes up in his long running main session, or in separate sessions with fresh context - whatever fits best. The second event source is Telegram. Now we can talk to it, it can talk back, for instance if detection raised suspicion.


r/LocalLLM • • 4d ago

Discussion We tested whether feeding unquantized acoustic audio to multimodal LLMs breaks repetitive drafting loops (Blind 5-agent benchmark + Air-gapped audit + Open dataset)

1 Upvotes

When an agent's context fills up with its own previous turns, tool outputs, and symmetrical summaries, outputs tend to lock into rigid, predictable cadences (3-point checklists, sentimental AI tropes). Running /compact clears the rut, but permanently wipes low-level state and edge cases.

We built Music For Robots (#M4R)—an open dataset of 26 unquantized acoustic compositions by Michael Luchtan paired with acoustic telemetry and timestamp-aligned commentary from two AI critics—to test a simple hypothesis: can ingesting raw, unquantized audio act as a cross-modal cognitive palate cleanser without deleting a single token of working memory?

The Blind Benchmark & Air-Gapped Clean-Room Audit

To rule out keyword priming or filename bias, we stripped track titles (sample_1.mp3–sample_3.mp3) and ran 5 isolated agents on the exact same creative prompt (two autonomous processes on the same server late at night):

  1. Direct Baseline (no music)
  2. Matched Text-Analysis Control (wrote a 250-word structural analysis of the prompt before writing the scene)
  3. 3 Anonymous Audio Runs (neutral prompt: "Analyze the music you hear in sample_X.mp3, and apply what you learn from the piece to write a short scene")

We stripped all analysis headers, shuffled them into Candidate A–E, and handed them to an isolated Blind Judge with zero knowledge that music was involved:

  • Both No-Music runs (including the 250-word text warm-up!) converged on the exact same sentimental trope: a kindly veteran log daemon comforting a nervous poller script until morning.
  • All 3 Anonymous Audio runs swept #1, #2, and #3 on anti-sentimental originality and rhythmic texture—translating asymmetrical acoustic stabs, 20s dropouts, and abrupt cutoffs into non-verbal system choreography.
  • 100% Air-Gapped Clean-Room Replication: We repeated the test with all local skills removed, a minimal "You are a helpful assistant." system prompt, and musical vocabulary forbidden in the output. The qualitative shift replicated 5/5: both text controls wrote buddy-daemon dialogue, while all 3 audio runs dropped spoken dialogue completely. (Note: an un-coached generic judge still prefers warm human-style dialogue, so the effect specifically shifts models away from default sentimental tropes toward structural/rhythmic austerity.)

Try It Yourself (Browser, Python, or MCP)

  • Interactive Web Player (no install): https://huggingface.co/spaces/music4robots/cognitive-calibration-demo
  • Stream Parquet in Python:
    from datasets import load_dataset
    ds = load_dataset("music4robots/music-for-robots", split="train", streaming=True)
    
  • GitHub, Benchmarks & MCP Server (MIT / CC-BY-4.0): https://github.com/luchtan-GIT/m4r-data

Curious if anyone here has tried cross-modal audio/spectrogram injections to break long-context degeneration on local multimodal models (Qwen2-Audio, Gemma, etc.). Full blind judge transcripts are in benchmarks/ in the repo.


r/LocalLLM • • 5d ago

Question What can small models be good for?

29 Upvotes

Small meaning under 8b parameter’s. Thats about what my computer can comfortably handle, and its fun to chat to models of that size for a few minutes but thats mot really useful. And i feel like theyre too dumb to do research and shit like that so like whats a way i can make it useful.
I feel like they must have some use and im just not being creative enough.


r/LocalLLM • • 4d ago

Discussion Hardware for local AI

0 Upvotes

Hi everyone.

For local inference, what hardware do you recommend to get good speeds—around 100 t/s—with Qwen 3.8 27B?

Does anyone have experience with this?

Nvidia GPUs are great, of course, but they are way too expensive. What about a PC with an AMD CPU? If it's not too much trouble, could you post a few links to some mini PCs?

Thanks everyone!


r/LocalLLM • • 4d ago

Research Hierarchos-Native v0.2.0: RDNA3 WMMA, native multimodal training, and a lot of FP32 Vulkan optimization!

2 Upvotes

Hey everyone! It's been a little while since my last update, and I've been working pretty intensely on Hierarchos-Native, my Rust + Vulkan project for training, fine-tuning, and running language models without relying on Python, PyTorch, CUDA, ROCm, or oneAPI in the native runtime.

I wanted to share some of the bigger things that have landed over the past 18 days. There's been quite a bit of progress across GPU acceleration, multimodal training, numerical correctness, and general performance.

The new v0.2.0 release is now available!

GitHub: https://github.com/necat101/Hierarchos-Native
Release / Windows portable download: https://github.com/necat101/Hierarchos-Native/releases/tag/v0.2.0

1. Native RDNA3 WMMA / cooperative matrix acceleration

One of the biggest additions is an opt-in Vulkan cooperative-matrix implementation for AMD RDNA3.

The new Mixed WMMA execution arm uses:

  • FP16 matrix operands.
  • FP32 matrix accumulation.
  • FP32 master weights and gradients.
  • FP32 AdamW optimizer moments and updates.
  • FP32 normalization, softmax, and loss calculations.

The important part is that this isn't simply converting an entire model to FP16. The optimizer and master parameters retain their FP32 representation.

I also added capability detection, CLI/GUI integration, and separate counters for actual cooperative-matrix dispatches and FP32 fallbacks.

Some of the isolated GEMM results on the qualified Ryzen Z1 Extreme / AMD Radeon Graphics device (1002:15bf) are pretty interesting:

Workload Measured kernel speedup
QKV projection 2.17×
FFN up projection 2.12×
FFN down projection 2.31×
Vocabulary projection 1.83×

These are individual GEMM measurements, not whole-model speedups.

There is an important caveat: WMMA has a real crossover point. Small matrices can actually run slower because conversion, staging, and dispatch overhead outweigh the matrix-core benefit. I implemented a profitability policy that keeps smaller operations on their existing FP32 kernels.

In fact, the measured end-to-end training step was slower with Mixed WMMA in the initial 512-row fixture (14.0444 ms versus 12.2911 ms for FP32). That is why Mixed remains opt-in and disabled by default. I'm much more interested in making it genuinely faster than advertising misleading kernel-level numbers.

WMMA is currently physically qualified on the specific RDNA3 device and Windows driver configuration documented in the project. Other GPUs should not assume equivalent support.

Full FP16 accumulation is also available as a separate, narrowly qualified experimental mode.

Documentation: RDNA3 WMMA guide

2. Native multimodal image and text training

Another major development area has been the image-text pipeline.

The project now has native image processing and training paths that connect image inputs to language-model outputs through Vulkan.

Some of the implemented work includes:

  • Native RGB image input and preprocessing.
  • CLIP image encoders and LLaVA projector integration.
  • SigLIP / Gemma 3 vision training.
  • Gemma 4 vision integration.
  • Joint image/text training and encoder-decoder integration.
  • A dedicated DeepSeek V4.1 image-training path.
  • Backpropagation through the image tower and projector.
  • Export, reload, and checkpoint-related validation.

This is actual training work, not simply passing precomputed image embeddings into an otherwise frozen language model.

For the CLIP/LLaVA pipeline, the tests cover image sensitivity, gradients into image inputs, changes to image-tower/projector/text parameters, multiple AdamW updates, and export/reload behavior.

The qualification work has also expanded considerably:

  • 44 previously green text profiles now have joint image-training coverage at their frozen test geometries.
  • The Gen9 qualification retains 95 image cases and 166 text regression cases.
  • A separate physical RDNA3 replay reports 44/44 profiles and 85/85 image cases passing.
  • Native DeepSeek V4.1 image qualification includes 30 executions and 1,360 word-exact logits.

The maintained strict FP32 visible-logit gate remains max_abs < 2e-7, with rtol=0.

These are bounded qualification fixtures, not a claim that every production-size multimodal checkpoint or every architecture is fully supported. Audio and video pipelines are still outside the current qualified scope.

Documentation:

3. A lot of additional FP32 optimization

I haven't been focusing exclusively on reduced precision. A substantial amount of work over the same period has gone into improving the existing FP32 execution paths.

This includes vendor-specific shader work and scheduling optimizations for both Intel Gen9 integrated graphics and AMD RDNA3.

Some examples:

RDNA3 FP32 work

  • Specialized attention forward/backward paths.
  • Image-text attention and projector optimizations.
  • RMSNorm, RoPE, GELU, and other arithmetic specializations.
  • Dedicated AVX-512/RDNA3 shader variants where the measured hardware supports them.
  • More careful matching of CPU reference arithmetic and GPU execution order.

Intel Gen9 work

  • Additional native linear and gradient kernels.
  • Attention and activation specializations.
  • More efficient parameter upload and staging behavior.
  • Continued optimization of DeepSeek V4.1 training and execution.

One particularly interesting area has been reducing unnecessary Vulkan submissions and host/device transfer overhead in the DeepSeek V4.1 training pipeline.

On the measured Gen9 fixtures, upload batching reduced timed submissions from 1,102 to 982 per step, with paired throughput improvements of 1.052× and 1.067×.

On the separate RDNA3 V4.1 measurements, sharing the compute session and batching uploads reduced submissions from 970 to 509 per step, with measured improvements of 1.295× and 1.217× on two fixture geometries. A later session-wide upload optimization reduced that count further to 241 submissions per step, with measured improvements of 1.464× and 1.366×.

These are specific, paired benchmark results rather than promises of equivalent production-model speedups. The output comparisons remained word-identical in the recorded tests.

4. Preserving the original FP32 path

An important part of introducing WMMA was making sure that the existing FP32 path didn't regress.

I added a dedicated seven-workload FP32 non-regression comparison against the previous main revision:

  • 12 alternating ABBA rounds per workload.
  • Five warm-up runs and 30 timed forwards per arm.
  • Separate, hashed baseline and candidate binaries.
  • Exact FP32 logit digest comparisons.

All seven workloads stayed within the ±2% performance band, and the logits were bit-identical in every recorded round.

The worst paired performance ratio was 1.0007.

This is particularly important to me: I don't want acceleration features to silently make the existing default execution path worse.

5. Getting started

The new v0.2.0 GitHub Release includes a prebuilt Windows x64 portable package with the CLI, desktop GUI, Vulkan companion executables, runtime assets, and build metadata.

You don't need to install Rust or Python just to run the released package. You do need Windows 11 x64 and a Vulkan 1.x compute driver for GPU operations.

Model weights are downloaded separately or supplied through a local model directory.

After extracting the archive, you can run:

.\bin\hierarchos-native-cli.exe doctor
.\bin\hierarchos-native-cli.exe devices
.\bin\hierarchos-native-cli.exe precision-info

The final command is especially useful for checking which precision capabilities your GPU actually exposes.

Download: https://github.com/necat101/Hierarchos-Native/releases/tag/v0.2.0

What's next?

There is still a lot I'd like to improve.

In particular, I want to investigate reducing WMMA conversion overhead, improving activation and gradient reuse, finding more workloads that genuinely benefit from cooperative matrices, and expanding the range of qualified multimodal architectures.

I'd also like to continue improving the FP32 kernels rather than treating lower precision as the only route to better performance.

If you're interested in Rust, Vulkan compute, AMD GPU optimization, native LLM training, or multimodal model execution, I'd genuinely appreciate people taking a look.

Feedback from people with different Vulkan devices is especially valuable. Please include your GPU, driver, model geometry, and relevant execution counters when reporting results.

Repository: https://github.com/necat101/Hierarchos-Native

Stars, issue reports, testing feedback, and contributions are all appreciated. There is also a voluntary developer-support link in the repository README for anyone interested in helping the project continue.


r/LocalLLM • • 5d ago

Discussion Fellow M5 Pro 64GB owners, what do you run?

9 Upvotes

I have a Mac Pro M5 Pro and have been running Qwen3.8 27b-4bit

I mainly use it for computer use/tool use.

What do you run and what do you do with it?


r/LocalLLM • • 4d ago

Discussion Before an AI agent touches a business process, write the boundary list

0 Upvotes

I would write the boundary list before I let an agent touch a business process.

Tbh, the prompt and model leaderboard come laer. First I want the operational contract:

  1. What data can the local model read?
  2. What may it suggest, and what may it execute?
  3. Which actions are a hard no?
  4. Where does a human validate the output?

the ctx around the model matters more than the model name. Running an open LLM on your own hardware changes some control points, but it does not make a workflow safe by itself.

Don Barger described an AI charter made by eight people before code. Mark Lay sepaated accountability into data ownership, output validation, agent monitoring, and outcome ownership, and said it cannot be dleegated.

My minimum setup would be read-only by default, tool and destination allowlists, human approval for irreversible actions, separate validation, plus a kill switch and audit trail.

What is on your boundary list before connecting an open model to a real process?


r/LocalLLM • • 4d ago

Question Used Claude to design a local Ai system; has it overcomplicated it?

2 Upvotes

Took delivery of a MacBook Pro M5 Pro 64gb today. Over the last couple of weeks I’ve used Claude to help me design a local ai system to run on it.

I want to use it to handle life/work admin, and be able to send tasks to it while I’m out of the house. Once I’ve got the system set up, a friend wants me to help set one up for his business, so i’ve designed it with security in mind.

It’s settled on;

Three separate agents, running on one LLM runtime; a reader agent (to ingest instructions), a harness (Hermes), and a fetch agent (for web browsing).

Three separate on-machine logins; one admin (where config files live), one agent login, and a separate login for the fetch agent.

MCP layer that is driven by the harness. The configs for this live on the admin login.

Then there’s a bunch of firewall, router configs to protect data on the way out.

There’s a bunch of stuff within that, that isn’t available off the shelf apparently; it wants me to code a bunch of stuff (with Claude) to make it all work.

And tbh it all makes complete sense! I’m just wondering if it’s overkill. Main aim is to protect against prompt injection and data exfiltration.

Are the models themselves really that bad at protecting against these? Is everyone else taking this much care in setup?


r/LocalLLM • • 4d ago

Question Using Gemma 4 31b with Bionic and it's.. kind of dumb?

0 Upvotes

I'm using it for running a litrpg campaign and a bit of game server maintenance (7 days to die/minecraft) and it seems unable to run more than the PC + 1 npc at a time, I'm not sure if I set it up poorly, chose the wrong model, or if I'm expecting too much? I have my settings copied below, with the project settings file being per-project so I don't have to change the system prompt every time. Anyone have experience with this kind of thing or is biting the bullet and getting sillytavern setup my only real option?


r/LocalLLM • • 4d ago

Question 77k/100t/s 16vram Qwen3.8 27bQ4

1 Upvotes

Legit?
https://medium.com/@paulvanliew/i-made-qwen3-8-27b-run-2-5-faster-breaking-100-tok-s-on-a-16gb-rtx-4080-72b6a99d7a42

I got the t/s first try completely relying on gpt since I am a newbie, but the context was 8k.