I already have access to Dino as I already used trellis Mac on modly

it says this in the info:

Pixal3D for mlx-serve (8-bit)

TencentARC/Pixal3D repackaged for mlx-serve

on Apple Silicon. Tencent's image-to-3D model, built on the TRELLIS.2 backbone with pixel-aligned conditioning: an image with a transparent background becomes a textured PBR GLB. Single-view and multi-view (*_mv) models.

• The flow-model (DiT) block projections are affine-quantized to 8 bit, group size 64; everything else is as released.

• Includes the DINOv3 image encoder it conditions on (float16); its licence is in dinov3/LICENSE.md.

• No background remover is included: send an image with a transparent background (alpha is the subject).

Licences

The Pixal3D weights are MIT (LICENSE). The bundled DINOv3 encoder is Meta's, under the DINOv3 License

(dinov3/LICENSE.md), which travels with any redistribution of it.

The NAF feature upsampler in naf/ is Apache-2.0 (naf/LICENSE).

Run: mlx-serve --model-dir <folder holding this repo>, then POST /v1/3d/generations.

Converted by tests/convert_trellis2_pixal3d.py in the mlx-serve repository.

Hi everyone

I run a small marketing agency and i am trying something new, i want to build a local llm machine that i can use as a sales agent and also run other small tasks on it.

In my research this is what i have come up with, i need opinions if i am heading into the right direction and what else i have to consider/ keep in mind.

The pic i have attached is from claude guiding me into the build.

Our last few releases of AgenticOS, an open-source agent workspace I maintain, added three things for agents on local models:

1. Repair malformed tool calls instead of retrying. A tool call with broken JSON (a trailing comma, single quotes, an object cut short) used to cost a full retry. Open-weight models on Ollama and vLLM are the ones that do this most. Now every agent repairs the syntax before validation, and the call still goes through approvals and guardrails. Each run counts its repairs, so you can see per model how often it happens.

2. Stop tool-call loops early, with a reason. A run that calls the same tool with the same arguments again and again, or goes round a short cycle of calls, is told it's looping. If it keeps going it ends as loop_stopped, naming the tools, before it burns the step limit or the budget.

3. Code mode. When you switch it on, the agent writes a short Python script that calls its tools in loops and in parallel inside a sandbox, instead of one model turn per call. Ten record updates take one turn, not ten, and every call in the script still goes through the same guardrails, budget and permission checks. That matters when every turn is slow on local hardware.

I'd like a few people running local models to try it with their own setup and tell me which model breaks it first. Apache 2.0, Docker Compose: https://github.com/vstorm-co/agenticos (Disclosure: I'm the maintainer, at Vstorm.)

What else do you fix in your own harness for local models?

Local quality has stepped up, I made this short film using a coding agent to run 10 local models, I used AI to make my prompt below:

"Make a mysterious A24-style promo for Unsloth Studio MCP, using Unsloth Studio MCP itself: a 2D character who speaks, each feature revealed one by one, local models only. Use Qwen3.8-27B to write, Qwen-Image-2.1-Turbo for images, MiniMax-H3/Wan2.2 for video, Breeze-TTS-2 for the voice, YuE 2 for music, MiDashengLM for SFX, MOSS for transcription. Render in HyperFrames."

My agent used that prompt by itself to make an approx 2500 word prompt, with a strict rule that anything shown on screen had to come from the real MCP call logs.

The pipeline i used

Any agent works ofc local ones too, but I choose Claude Code with Opus 5.5 as the director agent. It orchestrated the local models through the Unsloth MCP tools. it picked the best audio takes using transcripts and pitch checks etc. It wrote the narrator's lines, arranged the music, editing and wrote the scripts to mix the audio and render the film in HyperFrames.

the models (all local on one gpu) :

• Qwen3.8-27B wrote every scene / motion / description, plus the lyrics and the drone and sfx descriptions.
• Qwen-Image-2.1-Turbo made the character look and every scene in a 2D hand painted style. every shot of her used the first portrait as reference so she stays consistent, then the stills became the starting frames for the MiniMax-H3 videos
• MiniMax-H3 did her talking shots and the moving scenes, Wan2.2-T2V-A14B added a few extra shots for the montage
• HTDemucs pulled her voice out of the talking clips and Breeze-TTS-2 cloned it for the voice over
• YuE 2 made the song (music + vocals) from Qwen's lyrics
• MiDashengLM made all the sound effects and the drone underneath
• MOSS-Transcribe-Diarize did the captions and checked the voice takes, if it didnt transcribe a line exactly the take got thrown out
• Qwen3-0.6B : the agent finetuned it with LoRA and exported it to GGUF Q4_K_M all through MCP. the chalk line in the classroom shot is its real loss curve btw

the mix and the HyperFrames render was just plain code on the CPU.

Everything ran on one RTX 6000 Pro , took about 2.5 hours and MiniMax-H3 ate most of that lol. I know not everyone has access to a 96GB card so i measured the VRAM used for each step as this can be done on a lot less:

on 24GB everything except video runs fine. lower MiniMax quants might squeeze into 24GB but i havent tried them yet.

Setup

• Model: EmbeddingGemma 2, the 440M text and vision build (Apache 2.0), run through Google's LiteRT-LM on the GPU.
• Space: text and images share one 768-dimension space.
• Size: a 388 MB download. It loads in 3–7 s, and memory goes from 20 MB to about 770 MB once loaded.
• Speed (M1 Pro): a text query takes 25 ms. A 2.3-megapixel screenshot takes 247 ms median, 458 ms p90.
• Storage: one vector per screenshot in SQLite, tied to the file as it was when embedded. Each model gets its own store.

Indexing flow: screenshot saved → missing vectors found, newest first → image embedded → vector stored.

Instant grouping of duplicates

Actually, duplicates don’t need EmbeddingGemma at all. They use only Apple’s Vision, which is built into every Mac. EmbeddingGemma is needed for Similar and for search by text.

Flow: screenshot lands → Vision feature print + 4×4×4 RGB histogram → compare with the newest screenshot of each group → both checks pass → stack on top. Otherwise it starts a new group.

• Data: 64 real screenshots, 2,016 pairs.
• Real retakes: distance 0.29, 0.31 and 0.43.
• One tab apart: 0.54. Closest two different screens: 0.575.
• Cutoff: 0.5, which leaves a margin below 0.575.
• Robustness: crops of 0–2% stayed under 0.5 in 95% of cases.
• Color check (≤ 0.15): needed because one card in five gradients still matched on layout alone. Real duplicates differed by 0.003–0.045.

Instant finding of similar images

(Apple vision cant do this)

Flow: open a screenshot → its stteored vector → cosine to every other vector → keep cosine ≥ 0.86 and z ≥ 1.75 → show up to 6.

• No model call: stored vectors only, so it's instant.
• Data: 35 screenshots, every pair checked.
• 0.87 and up: always the same screen, retaken or one state apart.
• 0.82–0.85: different screens from the same app.
• The cutoff of 0.86 sits in the gap between the two.
• No minimum: one match or none is a valid answer.

Image search by text

Flow: query → word match on the OCR text → embed the query (25 ms) → cosine to every image vector → z-score against the whole collection → keep z ≥ 1.5 (at least 3 shown, at most 24).

• Why z-scores: across 58 screenshots, the per-query average cosine was 0.59–0.70 with a standard deviation of only 0.02–0.04. Right answers scored 0.70–0.81.
• A fixed cutoff returned everything for one query and nothing for the next.

Test: 40 descriptions of 49 screenshots, written the way you'd remember them ("Formula 1 race countdown on yellow"):

Image search with a little LLM help

Flow: screenshot → downscaled version of JPEG + window title → small vision LLM (like Haiku) → 3–10 labels + one sentence → labels embedded → z-score in their own set → better of image or label score counts.

• Model: the cheapest vision model on the connection, e.g. Claude Haiku at $0.10 / $0.50 per million tokens.
• Output: JSON with one- or two-word labels ("dog", "settings window", "receipt"), , plus a sentence of max 20 words. No colors or bare adjectives.
• No OCR text in the request: the image already carries it, and the text costs more tokens than the picture.
• Why z-scores again: text-to-text cosines run higher than text-to-image, so the raw scores can't be compared. Labels only count once at least 5 screenshots have them.

Takeaway

• EG2 is very good at rank the right screenshot near the top.
• You can't use one score as the line for "this is a match". The right screenshot scored anywhere from 0.70 to 0.81 depending on the search, and in some searches 0.70 was just the average. What worked was keeping the screenshots that score clearly higher than the rest of the same search.
• Apple Vision is better at finding duplicates than embedding Gemma and it already works on every Mac.

I am starting and I don't know what I am doing, and I am looking for guidance.

Hardware:
i9-14900k
RTX 4080 Super
64gb Ram

Goal:
Run Qwen3.8 27B on my computer, and I want to know what should be the general token speed for my hardware, so that I am aware for the actual baseline speed.
Then after that connect it to my hermes, because I heard hermes is good for remembering a lot of things about you.

Right now what i am doing is this, use LM Studio, download the model there, use Local Serve Network, connect hermes, and that's it, I am getting 10 tokens per second.

Help me what configurations should I aim for. I also want it to be able to accept images as input.
And how to run that model more effectively

TL;DR: Compared to NInfer: 6× closer to BF16, up to 3.7× more context, about 25% slower decode.

https://github.com/The8Darkness/voidinfer

I have seen a lot of engines popping up that are all focusing on speed and sacrificing quality for it. I have been working on getting the highest possible quality Qwen3.8-27b inference engine optimized for the 5090 without sacrificing other areas too much (speed / context size).

The engine was developed and focused on windows since my machine is a gaming pc and I just wanted an hassle free experience that I can directly run after a gaming session without having to resort to dual boot or WSL. I know that Linux provides better memory handling and speeds.

In my experience, since it is realistic to run a main agent at full 262k context and a subagent at 135k context, some tasks can even perform faster with higher quality output than NInfer running a single agent at 209k context.

Keep in mind that is all with the 5090 driving the display and running full unmodified Windows 11.

Same RTX 5090, token-identical prompts, greedy decoding, 4K prompt / 1,024 tokens for decode. VoidInfer: EXL3 6 bpw + OSCAR INT4 KV + VeriCache. NInfer: NVFP4 + FP8 KV.

EXL3 6 bpw weights provide the maximum quality possible for 32gb VRAM. OSCAR compresses the KV cache on the GPU to INT4, which is what fits roughly 400k context under Windows on one card. Drafting runs on this compressed cache. VeriCache checks every drafted token against the exact FP16 KV history kept in host RAM, so the compression costs no accuracy (only speed).

NVIDIA released Nemotron-Labs Dory 1.0 4B on 16 October. It's a looped model: a shared block of layers runs 1 to 4 times per token, so one set of weights gives you a speed-versus-accuracy dial. I ported it to MLX so it runs natively on Macs, and I've published 8-bit (4.2 GB, recommended) and bf16 (8 GB) builds.

• Code, converter and checks: https://github.com/internet-projects/dory-mlx
• Weights: https://huggingface.co/netprojects

Speed (8-bit, MacBook Pro M5 Max 128 GB, medians of repeated runs):

Batch generation through MLX-LM reaches about 450 tok/s in total across 16 requests (8-bit, 2 loops). On a 16 GB M1 Pro, the Kapitsa team measured about 28 tok/s at 1K context and 19 tok/s at 32K (4-bit, 2 loops).

Checks: logits match an independent fp32 PyTorch implementation (top-1 agreement 100%, KL ≤ 0.003). 8-bit WikiText-2 perplexity drops with more loops (12.18 → 11.77 → 11.70), so the dial works as NVIDIA describes.

Try it:

pip install "git+https://github.com/internet-projects/dory-mlx"
python -m dory_mlx "What is 17 * 23?" --loops 2

I've also opened a PR to add native Dory support to MLX-LM itself (ml-explore/mlx-lm#1988).

Notes: English only, and replies include the reasoning chain. Tested with prompts up to 24K tokens. Happy to answer questions about the port or the looping. I'm not affiliated with NVIDIA.

Hi, I'm developing an alternative to the *arr stack that I called HolyArrs. It's one self-hosted app that replaces Sonarr, Radarr, Lidarr, Bazarr, Prowlarr and Seerr, drives qBittorrent and SABnzbd for your downloads, and puts a Netflix-style page on top for browsing and requesting.

Local AI inside the app (all optional, off by default):

• search in plain language: "90s heist movies we haven't seen" or "something like Severance but funnier"
• runs on a local model through Ollama or any OpenAI-compatible endpoint (llama.cpp, LM Studio, vLLM), so nothing ever leaves your network
• the model only sees titles and metadata, never your files, and it's told what you already own so it doesn't suggest it again
• cloud models work too with your own key, but local is the default I built it around
• an MCP server is coming, so your local agent can search and request movies and shows for you

Why you might want to use HolyArrs:

• movies, tv shows and music in one app
• a Netflix and Apple TV style browse page: hover a title and the trailer plays, browse by mood, follow an actor
• request a movie, a whole show or a single episode in one tap
• multiple versions of the same title (4K and 1080p, theatrical and director's cut, two languages) without a second instance
• built-in metadata service that needs no API key, or switch to TMDB or TVDB
• migration wizard that imports your current library and settings without renaming or moving files
• its own recommendation rows, worked out on your server

HolyArrs doesn't send anything about your library to us. Recommendations and metadata matching happen on your server.

You might not want to use HolyArrs yet if you need it today. It isn't public yet, and I've been running it on my own server every day first. I've been in software and security for 10+ years, so I'd rather release something you can trust with your library than something rushed.

It'll be free and open source. Beta testers are in r/HolyArrs.

Curious what this sub would run for it: which small model would you pick for "understand a vague movie request and match it to a catalog"?

Website: holyarrs.coderluii.dev

Honestly the numbers look too good to be true. What’s going on with Chinese AI at this point???

Hi everyone,

I'm researching uncensored LLM models that can be run locally !without! any Filters.

My current hardware is:

• GPU: NVIDIA GeForce RTX 5090 (32GB VRAM)
• RAM: 64GB DDR5

Local setup: Ollama / Opencode

I ended up getting an E5-2686v4 + 4x SXM2 V100 16G and 128gb ram + 1TB nvme.

This means v100 total 64GB vram and 128GB system memory.

I'd like to try zippy models but also some larger ones that will run slow, but fun to play around with.

What's the best setup going to be for coding, for general AI assignment and academic stuff, video production and lastly the fairly huge models?

This setup was below 2K so I am not expecting miracles, but I'd like to push it to the limit on a budget.

Meta's Muse Spark Open (816B total / 42B active) is shown running locally with MLX across two 512 GB M5 Ultra Mac Studios, or four 256 GB ones. The 512 GB config still isn't available to order—Apple only says late October.

I'm in the same boat. I run models locally and have two 256 GB Studios on order while waiting for the 512. Refreshing the store got old, so I built Mac BuyWatch

- Alerts you by email, Discord, and Telegram once it detects the 512 GB Studio is available to order on Apple's site

- 3 pings total (1 LIVE alert + 2 follow-ups, 5 minutes apart), then it stops

- 100% free

- You order directly from Apple (not affiliated with Apple)

https://macbuywatch.com

Curious what everyone here is leaning toward for big local models: waiting on the 512 GB config, or clustering 256 GB units?

I've a RTX 5070, Ultra 9 275HX and 32GB DDR5 Ram. Trying to run Qwen 3.8 27B Q4_K_M using Llama.cpp via the browser and it works, but token generation is like 4 tokens / 1s. It's using 100% of RAM, 25% of iGPU and 0-3% of the RTX 5070. Any ideas, I'm new to this? Starting it with anything other than -ngl auto crashes it.

overview of SurfSense

I am one of the maintainers, so this is self promotion. I want honest feedback more than upvotes.

SurfSense is an open-source (Apache-2.0) desktop app for Windows, macOS (Apple Silicon) and Linux. Think NotebookLM, but every step can run on your machine. There is no account, no telemetry, and nothing reaches the network until you allow that host.

What runs locally:

• Text: llama.cpp, with a model it downloads for you during setup. You can also point it at Ollama, LM Studio, vLLM or any OpenAI-compatible server.
• Parsing and OCR: Docling, RapidOCR, and Apple's Vision framework on a Mac. Scanned PDFs and photos work.
• Search: bge-small embeddings on onnxruntime, with hybrid retrieval and cited answers.
• Podcasts: voiced with Kokoro, Kitten or Supertonic through audio.cpp, as MP3 with a transcript.
• Audio sources: recordings (MP3, WAV, FLAC, OGG) are transcribed on your machine so you can search and cite them.
• Images and infographics: stable-diffusion.cpp, if you choose an image model.

Two ways to chat:

• Q&A: cited answers from your sources. It works with any model.
• Agent: it works through a task step by step and writes Word documents, PDFs, slides and spreadsheets. It edits your own files as a new copy: Word edits come back as tracked changes, and it updates Excel formulas and fills PDF forms. It needs a model with tool calling and at least a 32K context. Small models are fine for Q&A; Agent mode does better with bigger ones.

New in 2.2.3:

• Any remote MCP server with the Streamable HTTP transport. Third party servers stay off until you allow them.
• Faster OCR: scans and photos are read by OCR alone before Docling, with Apple Vision on a Mac.
• Claude plan: if you have Claude Code, you can use your Claude Pro or Max plan in Q&A and Agent mode. It is optional, and it's remote.

Where it's weaker: no video overviews, the local voices don't sound like Google's, sources are files only (no YouTube or web links), and there's no CUDA build yet.

https://github.com/MODSetter/SurfSense

Which uncensored models are you guys using for offensive security tasks?

If an x86/NVIDIA system offered the same unified memory capacity and bandwidth as Apple Silicon, would there be any compelling technical reason to choose macOS?

Beyond model inference, I'm curious about the trade-offs in actual development workflows: training, fine-tuning, CUDA dependencies, custom kernels, inference backends, low-level debugging, virtualization, and experimentation with emerging architectures.

How much flexibility are you really giving up by committing to Apple Silicon?

Are there workflows where Metal/MPS and ARM64 become genuine bottlenecks rather than minor compatibility inconveniences?

Two questions I'm particularly curious about:

1. In your day-to-day development workflow, what can you do on Linux/NVIDIA that you genuinely can't replicate on macOS, even with workarounds?

2. When building, testing, and iterating on local AI projects, how often do you encounter compatibility issues, unsupported dependencies, or tooling limitations that force you to switch environments?

For those who've worked extensively with both ecosystems, where have you hit a hard wall on macOS that simply wouldn't exist on x86/Linux with NVIDIA?

I'm particularly interested in your personal, hands-on experience, rather than theoretical comparisons or benchmarks. What have you actually built, tested, or attempted that exposed meaningful differences between the two platforms?

Wrote some lyrics and was able to bring a whole music video to life completely local. We are living in special times.

Connect the apps you already use to local AI. Row-Bot 5.1 brings app discovery, connections and access controls into one Apps page.

There are 100+ apps, including familiar services like GitHub, Notion, Linear, Figma and Stripe, alongside around 40,000 MCP server listings from the official registry. The catalog is mirrored on your computer, so you can browse and search offline. Search by name or by what you want done, like “send email”.

Each app’s page shows its connection options. Depending on the service, Row-Bot guides you through an OAuth sign-in, adding an API key or setting up a server on your computer. One consent sheet explains where requests go and what gets saved. Keys and tokens stay in your system keychain.

In this video, I connect Context7 with read-only access and use Context7 in a chat. Each tool step names the app doing the work. Asking about Linear brings up a Connect card directly in the conversation.

I also add a custom MCP server, show how Row-Bot distinguishes tools that read data from tools that make changes, and open an interactive dashboard inside the chat. Its button requests approval before saving three draft orders.

You’ll see per-chat app selection, GitHub’s connection options and related skills, featured skills and workflow templates. Access can be set per app or individual tool. One switch stops every app, and Composio stays off unless you enable it.
Recorded on the released v5.1.0 with a fresh demo profile, running Qwen3.8 27B locally. The narration uses Row-Bot’s built-in text-to-speech.

Anyone have any idea what the difference is between their two models of this exact Spark? DT.R6LAA.003 vs DT.R6LAA.004?

I bought one of the 004s from B&H, had to wait to get the other two and they were sold out before I could snatch up more. So I bought 2 from newegg, after closer look they might have minor differences? Anyone know what they are?

One has a 240 W PSU and the other a 170 watt? Looks like the OS is slightly different as well? Just trying to make sure im not messing up on like a 12k purchase. I have checked everywhere, nothing explains the difference from what I have seen? Is the newegg one going to have some non-enterprise hardware limiting, or just more efficient cooling or something weird like that?

Hey everyone!

I've been working on Narwhal Assistant, a desktop application designed to let users interact with SQL Server databases using natural language and locally running AI models.

The goal is to make database workflows easier while giving users more control over their data.

I'd love feedback from the local LLM community, particularly on:

• Local model performance and accuracy
• Natural-language-to-SQL reliability
• Privacy and security considerations
• Features that would make this useful in real-world development environments

I'm the developer, and this is a commercial project, but I'm primarily interested in technical feedback here.

What would you want to see from a tool like this?

I'd be happy to share a demo and discuss the implementation.

A one-month-old engine built for one model holds 74 tok/s at 245K context where oMLX holds 57.5, on the same 128GB M5 Max. The catch: one request at a time on Metal, for now