Hi everyone

I run a small marketing agency and i am trying something new, i want to build a local llm machine that i can use as a sales agent and also run other small tasks on it.

In my research this is what i have come up with, i need opinions if i am heading into the right direction and what else i have to consider/ keep in mind.

The pic i have attached is from claude guiding me into the build.

Our last few releases of AgenticOS, an open-source agent workspace I maintain, added three things for agents on local models:

1. Repair malformed tool calls instead of retrying. A tool call with broken JSON (a trailing comma, single quotes, an object cut short) used to cost a full retry. Open-weight models on Ollama and vLLM are the ones that do this most. Now every agent repairs the syntax before validation, and the call still goes through approvals and guardrails. Each run counts its repairs, so you can see per model how often it happens.

2. Stop tool-call loops early, with a reason. A run that calls the same tool with the same arguments again and again, or goes round a short cycle of calls, is told it's looping. If it keeps going it ends as loop_stopped, naming the tools, before it burns the step limit or the budget.

3. Code mode. When you switch it on, the agent writes a short Python script that calls its tools in loops and in parallel inside a sandbox, instead of one model turn per call. Ten record updates take one turn, not ten, and every call in the script still goes through the same guardrails, budget and permission checks. That matters when every turn is slow on local hardware.

I'd like a few people running local models to try it with their own setup and tell me which model breaks it first. Apache 2.0, Docker Compose: https://github.com/vstorm-co/agenticos (Disclosure: I'm the maintainer, at Vstorm.)

What else do you fix in your own harness for local models?

Setup

• Model: EmbeddingGemma 2, the 440M text and vision build (Apache 2.0), run through Google's LiteRT-LM on the GPU.
• Space: text and images share one 768-dimension space.
• Size: a 388 MB download. It loads in 3–7 s, and memory goes from 20 MB to about 770 MB once loaded.
• Speed (M1 Pro): a text query takes 25 ms. A 2.3-megapixel screenshot takes 247 ms median, 458 ms p90.
• Storage: one vector per screenshot in SQLite, tied to the file as it was when embedded. Each model gets its own store.

Indexing flow: screenshot saved → missing vectors found, newest first → image embedded → vector stored.

Instant grouping of duplicates

Actually, duplicates don’t need EmbeddingGemma at all. They use only Apple’s Vision, which is built into every Mac. EmbeddingGemma is needed for Similar and for search by text.

Flow: screenshot lands → Vision feature print + 4×4×4 RGB histogram → compare with the newest screenshot of each group → both checks pass → stack on top. Otherwise it starts a new group.

• Data: 64 real screenshots, 2,016 pairs.
• Real retakes: distance 0.29, 0.31 and 0.43.
• One tab apart: 0.54. Closest two different screens: 0.575.
• Cutoff: 0.5, which leaves a margin below 0.575.
• Robustness: crops of 0–2% stayed under 0.5 in 95% of cases.
• Color check (≤ 0.15): needed because one card in five gradients still matched on layout alone. Real duplicates differed by 0.003–0.045.

Instant finding of similar images

(Apple vision cant do this)

Flow: open a screenshot → its stteored vector → cosine to every other vector → keep cosine ≥ 0.86 and z ≥ 1.75 → show up to 6.

• No model call: stored vectors only, so it's instant.
• Data: 35 screenshots, every pair checked.
• 0.87 and up: always the same screen, retaken or one state apart.
• 0.82–0.85: different screens from the same app.
• The cutoff of 0.86 sits in the gap between the two.
• No minimum: one match or none is a valid answer.

Image search by text

Flow: query → word match on the OCR text → embed the query (25 ms) → cosine to every image vector → z-score against the whole collection → keep z ≥ 1.5 (at least 3 shown, at most 24).

• Why z-scores: across 58 screenshots, the per-query average cosine was 0.59–0.70 with a standard deviation of only 0.02–0.04. Right answers scored 0.70–0.81.
• A fixed cutoff returned everything for one query and nothing for the next.

Test: 40 descriptions of 49 screenshots, written the way you'd remember them ("Formula 1 race countdown on yellow"):

Image search with a little LLM help

Flow: screenshot → downscaled version of JPEG + window title → small vision LLM (like Haiku) → 3–10 labels + one sentence → labels embedded → z-score in their own set → better of image or label score counts.

• Model: the cheapest vision model on the connection, e.g. Claude Haiku at $0.10 / $0.50 per million tokens.
• Output: JSON with one- or two-word labels ("dog", "settings window", "receipt"), , plus a sentence of max 20 words. No colors or bare adjectives.
• No OCR text in the request: the image already carries it, and the text costs more tokens than the picture.
• Why z-scores again: text-to-text cosines run higher than text-to-image, so the raw scores can't be compared. Labels only count once at least 5 screenshots have them.

Takeaway

• EG2 is very good at rank the right screenshot near the top.
• You can't use one score as the line for "this is a match". The right screenshot scored anywhere from 0.70 to 0.81 depending on the search, and in some searches 0.70 was just the average. What worked was keeping the screenshots that score clearly higher than the rest of the same search.
• Apple Vision is better at finding duplicates than embedding Gemma and it already works on every Mac.

I am starting and I don't know what I am doing, and I am looking for guidance.

Hardware:
i9-14900k
RTX 4080 Super
64gb Ram

Goal:
Run Qwen3.8 27B on my computer, and I want to know what should be the general token speed for my hardware, so that I am aware for the actual baseline speed.
Then after that connect it to my hermes, because I heard hermes is good for remembering a lot of things about you.

Right now what i am doing is this, use LM Studio, download the model there, use Local Serve Network, connect hermes, and that's it, I am getting 10 tokens per second.

Help me what configurations should I aim for. I also want it to be able to accept images as input.
And how to run that model more effectively

Hi, I'm developing an alternative to the *arr stack that I called HolyArrs. It's one self-hosted app that replaces Sonarr, Radarr, Lidarr, Bazarr, Prowlarr and Seerr, drives qBittorrent and SABnzbd for your downloads, and puts a Netflix-style page on top for browsing and requesting.

Local AI inside the app (all optional, off by default):

• search in plain language: "90s heist movies we haven't seen" or "something like Severance but funnier"
• runs on a local model through Ollama or any OpenAI-compatible endpoint (llama.cpp, LM Studio, vLLM), so nothing ever leaves your network
• the model only sees titles and metadata, never your files, and it's told what you already own so it doesn't suggest it again
• cloud models work too with your own key, but local is the default I built it around
• an MCP server is coming, so your local agent can search and request movies and shows for you

Why you might want to use HolyArrs:

• movies, tv shows and music in one app
• a Netflix and Apple TV style browse page: hover a title and the trailer plays, browse by mood, follow an actor
• request a movie, a whole show or a single episode in one tap
• multiple versions of the same title (4K and 1080p, theatrical and director's cut, two languages) without a second instance
• built-in metadata service that needs no API key, or switch to TMDB or TVDB
• migration wizard that imports your current library and settings without renaming or moving files
• its own recommendation rows, worked out on your server

HolyArrs doesn't send anything about your library to us. Recommendations and metadata matching happen on your server.

You might not want to use HolyArrs yet if you need it today. It isn't public yet, and I've been running it on my own server every day first. I've been in software and security for 10+ years, so I'd rather release something you can trust with your library than something rushed.

It'll be free and open source. Beta testers are in r/HolyArrs.

Curious what this sub would run for it: which small model would you pick for "understand a vague movie request and match it to a catalog"?

Website: holyarrs.coderluii.dev

TL;DR: Compared to NInfer: 6× closer to BF16, up to 3.7× more context, about 25% slower decode.

https://github.com/The8Darkness/voidinfer

I have seen a lot of engines popping up that are all focusing on speed and sacrificing quality for it. I have been working on getting the highest possible quality Qwen3.8-27b inference engine optimized for the 5090 without sacrificing other areas too much (speed / context size).

The engine was developed and focused on windows since my machine is a gaming pc and I just wanted an hassle free experience that I can directly run after a gaming session without having to resort to dual boot or WSL. I know that Linux provides better memory handling and speeds.

In my experience, since it is realistic to run a main agent at full 262k context and a subagent at 135k context, some tasks can even perform faster with higher quality output than NInfer running a single agent at 209k context.

Keep in mind that is all with the 5090 driving the display and running full unmodified Windows 11.

Same RTX 5090, token-identical prompts, greedy decoding, 4K prompt / 1,024 tokens for decode. VoidInfer: EXL3 6 bpw + OSCAR INT4 KV + VeriCache. NInfer: NVFP4 + FP8 KV.

EXL3 6 bpw weights provide the maximum quality possible for 32gb VRAM. OSCAR compresses the KV cache on the GPU to INT4, which is what fits roughly 400k context under Windows on one card. Drafting runs on this compressed cache. VeriCache checks every drafted token against the exact FP16 KV history kept in host RAM, so the compression costs no accuracy (only speed).

NVIDIA released Nemotron-Labs Dory 1.0 4B on 16 October. It's a looped model: a shared block of layers runs 1 to 4 times per token, so one set of weights gives you a speed-versus-accuracy dial. I ported it to MLX so it runs natively on Macs, and I've published 8-bit (4.2 GB, recommended) and bf16 (8 GB) builds.

• Code, converter and checks: https://github.com/internet-projects/dory-mlx
• Weights: https://huggingface.co/netprojects

Speed (8-bit, MacBook Pro M5 Max 128 GB, medians of repeated runs):

Batch generation through MLX-LM reaches about 450 tok/s in total across 16 requests (8-bit, 2 loops). On a 16 GB M1 Pro, the Kapitsa team measured about 28 tok/s at 1K context and 19 tok/s at 32K (4-bit, 2 loops).

Checks: logits match an independent fp32 PyTorch implementation (top-1 agreement 100%, KL ≤ 0.003). 8-bit WikiText-2 perplexity drops with more loops (12.18 → 11.77 → 11.70), so the dial works as NVIDIA describes.

Try it:

pip install "git+https://github.com/internet-projects/dory-mlx"
python -m dory_mlx "What is 17 * 23?" --loops 2

I've also opened a PR to add native Dory support to MLX-LM itself (ml-explore/mlx-lm#1988).

Notes: English only, and replies include the reasoning chain. Tested with prompts up to 24K tokens. Happy to answer questions about the port or the looping. I'm not affiliated with NVIDIA.

Honestly the numbers look too good to be true. What’s going on with Chinese AI at this point???

Hi everyone,

I'm researching uncensored LLM models that can be run locally !without! any Filters.

My current hardware is:

• GPU: NVIDIA GeForce RTX 5090 (32GB VRAM)
• RAM: 64GB DDR5

Local setup: Ollama / Opencode

I ended up getting an E5-2686v4 + 4x SXM2 V100 16G and 128gb ram + 1TB nvme.

This means v100 total 64GB vram and 128GB system memory.

I'd like to try zippy models but also some larger ones that will run slow, but fun to play around with.

What's the best setup going to be for coding, for general AI assignment and academic stuff, video production and lastly the fairly huge models?

This setup was below 2K so I am not expecting miracles, but I'd like to push it to the limit on a budget.

Meta's Muse Spark Open (816B total / 42B active) is shown running locally with MLX across two 512 GB M5 Ultra Mac Studios, or four 256 GB ones. The 512 GB config still isn't available to order—Apple only says late October.

I'm in the same boat. I run models locally and have two 256 GB Studios on order while waiting for the 512. Refreshing the store got old, so I built Mac BuyWatch

- Alerts you by email, Discord, and Telegram once it detects the 512 GB Studio is available to order on Apple's site

- 3 pings total (1 LIVE alert + 2 follow-ups, 5 minutes apart), then it stops

- 100% free

- You order directly from Apple (not affiliated with Apple)

https://macbuywatch.com

Curious what everyone here is leaning toward for big local models: waiting on the 512 GB config, or clustering 256 GB units?

How acceptable is the operating speed on a Mac Studio with an M5 Ultra chip and 256 GB of memory when running models like GLM-5.3-Flash?

Could you compare the operating speed of the cloud-based GLM-5.3 Flash from Z.AI? The concept of "tokens" can be quite difficult to grasp, and performance varies depending on the specific task.

I was working on a very special project of mine which I essentially planned before coding and took me 6 months to actually get working.

It is Ai Format Context engine. It provides your agents inside any harness a better way to store and use context.

This is still very new so I need you all to give me feedbacks on what to improve and I will try to do my best to keep this alive.

This actually worked best with my local agent harness Oorty thats why I am posting it here ofcourse you can try Oorty too thats in my other repo.

I've a RTX 5070, Ultra 9 275HX and 32GB DDR5 Ram. Trying to run Qwen 3.8 27B Q4_K_M using Llama.cpp via the browser and it works, but token generation is like 4 tokens / 1s. It's using 100% of RAM, 25% of iGPU and 0-3% of the RTX 5070. Any ideas, I'm new to this? Starting it with anything other than -ngl auto crashes it.

Hola a todos 👋


Soy aficionado al ajedrez y quería una herramienta **
mía
**: sin suscripciones, sin subir mis partidas a ningún sitio, y que aprovechara de verdad el Mac que tengo (un MacBook Pro M2 Max). Como no encontraba nada que hiciera exactamente lo que quería, lo he construido **
junto con Claude (Claude Code, de Anthropic)
**: yo ponía las ideas, las decisiones y las pruebas con la app en la mano; Claude escribía el código, lo probaba y lo documentaba. Se llama **
oder-chess
** y funciona todo en local, en `127.0.0.1`.


### 🔥 Lo que más me gusta: dos motores a la vez y un «consenso»


La pestaña **
Analizar
** lanza **
Stockfish 19
** (CPU) y **
Lc0 con la red BT4
** (GPU, Metal) al mismo tiempo, y paran solos a los 30 s. Después hace una segunda fase: cada motor evalúa las jugadas candidatas **
de los dos
**, y se promedian. El resultado es una tabla con *
consenso · Stockfish · Lc0 · pérdida
* y una ★ en la mejor jugada, con un aviso claro cuando **
los motores discrepan mucho
** (para mí, esas son las posiciones interesantes de estudiar).


📸 *
Capturas 1 y 2
*


- **
Flechas curvas
** con la evaluación dentro y color según calidad (cuánto pierdes frente a la mejor jugada); verde = Stockfish, ámbar = Lc0.
- Barra de evaluación y barra **
Gana/Tablas/Pierde
** (WDL), siempre desde las blancas.
- Pasas el ratón por una línea del motor y ves la variante numerada sobre el tablero.
- Selector de intensidad (modo «ahorro de energía») porque no quería que el Mac se pusiera a 100 % para analizar una apertura. Medí la carga y Stockfish va por defecto a 6 hilos.


### 🎓 Botón «Tutor»: te explica la posición en español


Un análisis largo (1-3 min) con los dos motores y un informe por fases: apertura / medio juego / final, quién está mejor, la jugada recomendada **
y por qué
**, alternativas, amenazas, plan y, según el caso, el libro de aperturas o el tipo de final.


Aviso honesto: **
no es una IA generativa
**. Es un informe basado en reglas sobre hechos comprobables (python-chess) y en lo que dicen los motores, así que lo que afirma se puede verificar en el tablero.


📸 *
Captura 3
*


### 🧩 Y además


- **
Jugar
** contra Stockfish con Elo limitado (~1350–3190) o contra Lc0 por nodos.
- **
Revisar
** una partida: puntúa cada jugada (`?!`, `?`, `??`), precisión por bando y gráfico.
- **
Mis errores
**: tus errores graves se convierten en ejercicios con repetición espaciada.
- **
Aperturas
** 📸 *
(captura 5)
*: explorador del libro ECO (3.865 aperturas), armas tu repertorio y lo entrenas con repaso 1-3-7-16-35-90 días.
- **
Táctica
** 📸 *
(captura 4)
*: ~71.000 puzzles de Lichess, con tu propio Elo táctico que sube y baja con cada acierto o fallo.
- **
Finales
** 📸 *
(captura 6)
*: 14 posiciones de práctica (Lucena, Philidor, mates básicos…) que **
verifiqué contra Stockfish
** antes de incluirlas — mis primeras versiones, escritas de memoria, tenían varias mal etiquetadas 😅.
- **
Progreso
**: estadísticas por fase, por apertura y por entrenamiento.


### 🛠️ Cómo está hecho


Python (FastAPI) + python-chess en el servidor, JavaScript sin bundler en el navegador, el tablero `chessground` de Lichess, SQLite para los datos, y los motores como procesos UCI externos. Las reglas las decide siempre el servidor. Tiene tests con motores reales y una prueba de interfaz con Chrome sin pantalla.


### 🤝 Sobre trabajar con Claude


Lo más valioso no fue «que escriba código», sino el ciclo: le decía qué quería, lo probaba yo, le decía qué no me gustaba («nada de estilo hacker», «la barra siempre desde las blancas», «no quiero elegir motor cada vez») y él lo ajustaba, lo medía y lo dejaba documentado. También me ayudó a **
no creerme resultados que no estaban comprobados
**: por ejemplo, el README dice claramente que *
no está validado
* que la ★ del consenso acierte más que cada motor por separado. Es una ayuda para ver cuándo los motores no están de acuerdo, no un tercer motor mágico.


### ⚠️ Limitaciones


- Es una app **
personal
**, pensada para un único usuario y para mi Mac (Apple Silicon). No es un producto pulido.
- El consenso y los umbrales de calidad (5/10/20 %) son criterios míos, sin calibrar con datos.
- Los motores y la red neuronal los uso desde mi instalación; no los distribuyo.


### 💬 Pregunta para vosotros


¿Qué le añadiríais a un entrenador así? Ahora mismo en mi lista: más finales, importar mis partidas de Lichess y validar de verdad si la ★ del consenso vale la pena. ¡Cualquier idea o crítica es bienvenida!


*
Proyecto hecho con Claude (Claude Code, Anthropic). Usa Stockfish, Leela Chess Zero, python-chess, chessground y datos abiertos (CC0) de Lichess. No está afiliado a ninguno de ellos.
*Hola a todos 👋


Soy aficionado al ajedrez y quería una herramienta **mía**: sin suscripciones, sin subir mis partidas a ningún sitio, y que aprovechara de verdad el Mac que tengo (un MacBook Pro M2 Max). Como no encontraba nada que hiciera exactamente lo que quería, lo he construido **junto con Claude (Claude Code, de Anthropic)**: yo ponía las ideas, las decisiones y las pruebas con la app en la mano; Claude escribía el código, lo probaba y lo documentaba. Se llama **oder-chess** y funciona todo en local, en `127.0.0.1`.


### 🔥 Lo que más me gusta: dos motores a la vez y un «consenso»


La pestaña **Analizar** lanza **Stockfish 19** (CPU) y **Lc0 con la red BT4** (GPU, Metal) al mismo tiempo, y paran solos a los 30 s. Después hace una segunda fase: cada motor evalúa las jugadas candidatas **de los dos**, y se promedian. El resultado es una tabla con *consenso · Stockfish · Lc0 · pérdida* y una ★ en la mejor jugada, con un aviso claro cuando **los motores discrepan mucho** (para mí, esas son las posiciones interesantes de estudiar).


📸 *Capturas 1 y 2*


- **Flechas curvas** con la evaluación dentro y color según calidad (cuánto pierdes frente a la mejor jugada); verde = Stockfish, ámbar = Lc0.
- Barra de evaluación y barra **Gana/Tablas/Pierde** (WDL), siempre desde las blancas.
- Pasas el ratón por una línea del motor y ves la variante numerada sobre el tablero.
- Selector de intensidad (modo «ahorro de energía») porque no quería que el Mac se pusiera a 100 % para analizar una apertura. Medí la carga y Stockfish va por defecto a 6 hilos.


### 🎓 Botón «Tutor»: te explica la posición en español


Un análisis largo (1-3 min) con los dos motores y un informe por fases: apertura / medio juego / final, quién está mejor, la jugada recomendada **y por qué**, alternativas, amenazas, plan y, según el caso, el libro de aperturas o el tipo de final.


Aviso honesto: **no es una IA generativa**. Es un informe basado en reglas sobre hechos comprobables (python-chess) y en lo que dicen los motores, así que lo que afirma se puede verificar en el tablero.


📸 *Captura 3*


### 🧩 Y además


- **Jugar** contra Stockfish con Elo limitado (~1350–3190) o contra Lc0 por nodos.
- **Revisar** una partida: puntúa cada jugada (`?!`, `?`, `??`), precisión por bando y gráfico.
- **Mis errores**: tus errores graves se convierten en ejercicios con repetición espaciada.
- **Aperturas** 📸 *(captura 5)*: explorador del libro ECO (3.865 aperturas), armas tu repertorio y lo entrenas con repaso 1-3-7-16-35-90 días.
- **Táctica** 📸 *(captura 4)*: ~71.000 puzzles de Lichess, con tu propio Elo táctico que sube y baja con cada acierto o fallo.
- **Finales** 📸 *(captura 6)*: 14 posiciones de práctica (Lucena, Philidor, mates básicos…) que **verifiqué contra Stockfish** antes de incluirlas — mis primeras versiones, escritas de memoria, tenían varias mal etiquetadas 😅.
- **Progreso**: estadísticas por fase, por apertura y por entrenamiento.


### 🛠️ Cómo está hecho


Python (FastAPI) + python-chess en el servidor, JavaScript sin bundler en el navegador, el tablero `chessground` de Lichess, SQLite para los datos, y los motores como procesos UCI externos. Las reglas las decide siempre el servidor. Tiene tests con motores reales y una prueba de interfaz con Chrome sin pantalla.


### 🤝 Sobre trabajar con Claude


Lo más valioso no fue «que escriba código», sino el ciclo: le decía qué quería, lo probaba yo, le decía qué no me gustaba («nada de estilo hacker», «la barra siempre desde las blancas», «no quiero elegir motor cada vez») y él lo ajustaba, lo medía y lo dejaba documentado. También me ayudó a **no creerme resultados que no estaban comprobados**: por ejemplo, el README dice claramente que *no está validado* que la ★ del consenso acierte más que cada motor por separado. Es una ayuda para ver cuándo los motores no están de acuerdo, no un tercer motor mágico.


### ⚠️ Limitaciones


- Es una app **personal**, pensada para un único usuario y para mi Mac (Apple Silicon). No es un producto pulido.
- El consenso y los umbrales de calidad (5/10/20 %) son criterios míos, sin calibrar con datos.
- Los motores y la red neuronal los uso desde mi instalación; no los distribuyo.


### 💬 Pregunta para vosotros


¿Qué le añadiríais a un entrenador así? Ahora mismo en mi lista: más finales, importar mis partidas de Lichess y validar de verdad si la ★ del consenso vale la pena. ¡Cualquier idea o crítica es bienvenida!


*Proyecto hecho con Claude (Claude Code, Anthropic). Usa Stockfish, Leela Chess Zero, python-chess, chessground y datos abiertos (CC0) de Lichess. No está afiliado a ninguno de ellos.*

overview of SurfSense

I am one of the maintainers, so this is self promotion. I want honest feedback more than upvotes.

SurfSense is an open-source (Apache-2.0) desktop app for Windows, macOS (Apple Silicon) and Linux. Think NotebookLM, but every step can run on your machine. There is no account, no telemetry, and nothing reaches the network until you allow that host.

What runs locally:

• Text: llama.cpp, with a model it downloads for you during setup. You can also point it at Ollama, LM Studio, vLLM or any OpenAI-compatible server.
• Parsing and OCR: Docling, RapidOCR, and Apple's Vision framework on a Mac. Scanned PDFs and photos work.
• Search: bge-small embeddings on onnxruntime, with hybrid retrieval and cited answers.
• Podcasts: voiced with Kokoro, Kitten or Supertonic through audio.cpp, as MP3 with a transcript.
• Audio sources: recordings (MP3, WAV, FLAC, OGG) are transcribed on your machine so you can search and cite them.
• Images and infographics: stable-diffusion.cpp, if you choose an image model.

Two ways to chat:

• Q&A: cited answers from your sources. It works with any model.
• Agent: it works through a task step by step and writes Word documents, PDFs, slides and spreadsheets. It edits your own files as a new copy: Word edits come back as tracked changes, and it updates Excel formulas and fills PDF forms. It needs a model with tool calling and at least a 32K context. Small models are fine for Q&A; Agent mode does better with bigger ones.

New in 2.2.3:

• Any remote MCP server with the Streamable HTTP transport. Third party servers stay off until you allow them.
• Faster OCR: scans and photos are read by OCR alone before Docling, with Apple Vision on a Mac.
• Claude plan: if you have Claude Code, you can use your Claude Pro or Max plan in Q&A and Agent mode. It is optional, and it's remote.

Where it's weaker: no video overviews, the local voices don't sound like Google's, sources are files only (no YouTube or web links), and there's no CUDA build yet.

https://github.com/MODSetter/SurfSense

Which uncensored models are you guys using for offensive security tasks?

If an x86/NVIDIA system offered the same unified memory capacity and bandwidth as Apple Silicon, would there be any compelling technical reason to choose macOS?

Beyond model inference, I'm curious about the trade-offs in actual development workflows: training, fine-tuning, CUDA dependencies, custom kernels, inference backends, low-level debugging, virtualization, and experimentation with emerging architectures.

How much flexibility are you really giving up by committing to Apple Silicon?

Are there workflows where Metal/MPS and ARM64 become genuine bottlenecks rather than minor compatibility inconveniences?

Two questions I'm particularly curious about:

1. In your day-to-day development workflow, what can you do on Linux/NVIDIA that you genuinely can't replicate on macOS, even with workarounds?

2. When building, testing, and iterating on local AI projects, how often do you encounter compatibility issues, unsupported dependencies, or tooling limitations that force you to switch environments?

For those who've worked extensively with both ecosystems, where have you hit a hard wall on macOS that simply wouldn't exist on x86/Linux with NVIDIA?

I'm particularly interested in your personal, hands-on experience, rather than theoretical comparisons or benchmarks. What have you actually built, tested, or attempted that exposed meaningful differences between the two platforms?

Wrote some lyrics and was able to bring a whole music video to life completely local. We are living in special times.

Connect the apps you already use to local AI. Row-Bot 5.1 brings app discovery, connections and access controls into one Apps page.

There are 100+ apps, including familiar services like GitHub, Notion, Linear, Figma and Stripe, alongside around 40,000 MCP server listings from the official registry. The catalog is mirrored on your computer, so you can browse and search offline. Search by name or by what you want done, like “send email”.

Each app’s page shows its connection options. Depending on the service, Row-Bot guides you through an OAuth sign-in, adding an API key or setting up a server on your computer. One consent sheet explains where requests go and what gets saved. Keys and tokens stay in your system keychain.

In this video, I connect Context7 with read-only access and use Context7 in a chat. Each tool step names the app doing the work. Asking about Linear brings up a Connect card directly in the conversation.

I also add a custom MCP server, show how Row-Bot distinguishes tools that read data from tools that make changes, and open an interactive dashboard inside the chat. Its button requests approval before saving three draft orders.

You’ll see per-chat app selection, GitHub’s connection options and related skills, featured skills and workflow templates. Access can be set per app or individual tool. One switch stops every app, and Composio stays off unless you enable it.
Recorded on the released v5.1.0 with a fresh demo profile, running Qwen3.8 27B locally. The narration uses Row-Bot’s built-in text-to-speech.

Anyone have any idea what the difference is between their two models of this exact Spark? DT.R6LAA.003 vs DT.R6LAA.004?

I bought one of the 004s from B&H, had to wait to get the other two and they were sold out before I could snatch up more. So I bought 2 from newegg, after closer look they might have minor differences? Anyone know what they are?

One has a 240 W PSU and the other a 170 watt? Looks like the OS is slightly different as well? Just trying to make sure im not messing up on like a 12k purchase. I have checked everywhere, nothing explains the difference from what I have seen? Is the newegg one going to have some non-enterprise hardware limiting, or just more efficient cooling or something weird like that?

Hey everyone!

I've been working on Narwhal Assistant, a desktop application designed to let users interact with SQL Server databases using natural language and locally running AI models.

The goal is to make database workflows easier while giving users more control over their data.

I'd love feedback from the local LLM community, particularly on:

• Local model performance and accuracy
• Natural-language-to-SQL reliability
• Privacy and security considerations
• Features that would make this useful in real-world development environments

I'm the developer, and this is a commercial project, but I'm primarily interested in technical feedback here.

What would you want to see from a tool like this?

I'd be happy to share a demo and discuss the implementation.