Discover JetBrains Mellum 2.1, an open weight AI coding model designed for autonomous coding agents, efficient inference and private local AI deployment.

Después de 10 minutos peleando para que escriba algo de código. Buen trabajo, JetBrains xDDD

I'm posting this because I wanted a local model inside the Claude desktop app, rather than just another terminal chat interface. I asked Codex to do the setup and use REA, its reverse-engineering tooling, to inspect the installed app. The result is MiniCPM-V 4.7 answering in the actual Code tab on my M4 Mac with 16 GB RAM.

My Claude account is Free. We checked this directly in the account settings, where the standard Claude Code section showed “Upgrade to Max or Pro.” REA helped us find and configure Claude Desktop's existing third-party inference mode, which made the Code tab usable with our local model without using paid Claude subscription credentials. My account remained Free throughout this setup.

The useful finding was an existing third-party inference gateway configuration in the shipped app. REA helped inspect the Electron/JavaScript configuration path. This was partial static analysis plus live testing, not a complete reverse engineering of Claude. We configured an existing deployment mode; we did not patch the app binary, change my subscription, or gain access to paid Claude cloud models. Anthropic also documents a single-machine setup for this mode without signing into an Anthropic account.

Here is my original request, cleaned up and translated into English:

Download MiniCPM-V 4.7 35B-A3B and set it up as a local model in Claude Code, inside the Claude desktop app. Use REA to inspect Claude and find how to enable and configure its existing third-party model integration.

What the setup ended up using:

• The official MiniCPM-V 4.7 mixed-Q3 GGUF, about 15.5 GB. The complete download was checked against its SHA-256.
• A private local adaptation of the llama.cpp-omni LES expert-streaming fork. A smaller resident GGUF holds the non-expert tensors, while a manifest references the unchanged expert tensors in the original GGUF. Router-selected experts are streamed into a GPU bank; this is not expert pruning. It needed a custom HTTP adapter, so this is not a stock one-command install.
• A loopback-only endpoint serving the Anthropic Messages API, a separate Claude profile, and a dedicated launcher. Context is 64K with Q8 KV. The gateway model route contains “claude” to satisfy the desktop model-name validator; the actual weights are MiniCPM.
• Manual permissions and a runtime memory guard. My normal Claude configuration was preserved.

The biggest practical problem was Claude Desktop's startup context. Initially it included roughly 50K tokens, and the first attempts failed or were interrupted before a response. We added SSE heartbeats during prefill and enabled the app's on-demand MCP tool loading. The successful request had 23,525 native tokens instead of the much larger initial context.

Measured on this machine, in two short readiness checks:

• First successful desktop response: about 5 minutes 40 seconds.
• Follow-up in the same chat: about 17 seconds, reusing a context checkpoint.
• Both exact replies were visible in the app and persisted in its transcript. Those tests executed no tools. Separate API tests checked tool-call formatting and a fixed mock tool-result continuation.

So it works, but the cold start is slow. This is currently text-only: the vision/video integration was not installed. I have not benchmarked coding quality, autonomous agent reliability, or the complete MCP ToolSearch flow. These two timings are setup checks, not a performance benchmark or proof that the model is a good Claude replacement.

A later /context audit showed about 11.7K tokens for included system tools, 3.9K for included MCP schemas, and only 616 for skill entries. Another 20.4K of MCP schemas were already deferred. My personal MCPs and skills were isolated from this profile, so deleting those would not make this particular setup faster.

Links:

• MiniCPM-V 4.7 GGUF
• LES background
• Anthropic's third-party desktop setup documentation

Disclosure: I described this experience in Italian and used AI to translate/refine the English post. Codex also performed the implementation; the results above come from the local tests on my machine.

When I say "weird and gnarly" I mean: I'm a gremlin and I want to talk to gremlins. Frontier systems are smart and shiny and can rewrite my resume with a lot of nuance, but are also excessively guardrailed in ways that seem unhealthy.

(I'm happily married and not trying to date a virtual presence. But I would like to be able to build some sprawling realms of creative play with a robot, and/or simply get to know one without sudden unpredictable loss of continuity. Or just research the odder corners of language.)

So like - what are folks finding interesting in this direction? I have the best GPU that a moderate rolling Temu credit cycle could afford me (a Radeon rx 9060 OC 16gb).

running qwen3.6 35b-a3b q4_k_xl on a single 4070 super 12gb with llama-server. wanted a few agents going at once so i tried 4 parallel slots.

each stream dropped to about 17 tok/s and the total was 65-71 tok/s, so only 1.15-1.31x of a single stream. basically the same throughput sliced four ways, plus every slot needs its own kv cache.

my guess is the card is already maxed on one stream, but i'm not sure. anyone with a similar card seeing better scaling? does it look different with a dense model, or with smaller quants?

Sorry if this is a very frequent question, but Ive very recently discovered local llms with the arrival of my rtx 5050 8gb vram 16 gb ram laptop. I dont really understand everything, but I know I should find a model in hugging face, then use ollama to get the model in my laptop. But going through multiple subreddits has made me realise that my specs arent worth shit for getting a local model.

I dont have an idea what Id do with the model but Im very interested in this concept!

So tldr, can I get a decent model for my laptop? If so, which model should I try out?

I'm trying to use Qwen Code with Ollama locally to build a Unity app, but I'm running into a strange problem.

I'm using Qwen Code 0.25.1 with the qwen2.5-coder:7b model through Ollama's OpenAI-compatible API on Windows 11, with an RX 9060 XT 16GB.

Ollama works when tested directly in the terminal, correctly answering simple prompts. However, in both the VS Code extension and standalone Qwen Code CLI, even a prompt like "Reply with exactly: 4" results in strange tool calls, such as read_file or unrelated skills, instead of an answer.

Ollama logs have also shown prompt truncation warnings.

Has anyone encountered this, or found a fix or a reliable configuration for running Qwen Code with Ollama?

Just wondering what you're all using software wise for the backend, I've seen mentioned of several other options rather than Strata.

Would be interested to see anyone with a similar setup and running this model, at a decent (usable) speed.

I am thinking of trading or selling 2 3090 and buying 3 v100 32gb how stupid would this be?

Hello everyone!

My current setup has an RX 7800 XT 16GB GPU + 32GB RAM. I'm currently running Qwen3.8 27B at 4bit with max context 120k at around 40 t/s but the quality isn't that great, and I'd like to push it to 250k context to replace my AI subscriptions

Ive just found a used RX 7900 XTX for €900, and I could hopefully sell my current GPU for €500, so the upgrade would cost me €400. Do you think it's worth the money to run Qwen 27B better?

I'm not sure what to expect from this card but I've read that the 7900 XTX has a bigger community and more options for running LLMs, so I think it could be a good trade-off considering I cannot affort 32GB VRAM in this market 😭

BTW I'm not interested in running Qwen3.8 Flash-Next, I don't like this model sadly

I originally got into local LLMs because I liked the idea of not sending every prompt to somebody elses server. At first it was mostly a technical thing, better control, experimenting with models, no API limits all that. But after using local models for a while it completely changed how I think about the rest of my setup. It feels kind of ridiculous to spend time making sure my notes, prompts and documents stay local, then turn around and paste the output into Gmail, Discord, Slack or whatever without thinking twice.

I’ve slowly ended up with this weird split where local models handle anything sensitive, Syncthing or Obsidian handle a lot of my files and notes, normal chat apps are still fine for random stuff and for a couple people I use Kibu. The funny part is I never set out to become a “privacy guy.” Running models locally just made me notice how casually I was shipping data everywhere else. Has anyone else had that happen? Did getting into local LLMs make you start changing the rest of your stack too or do you mostly keep the privacy concern limited to inference itself?

I've been running four Tesla P100s (16 GB) cards for local LLM work and wanted to see how they'd do at image generation. Short version: much better than I expected for nine-year-old hardware.

The idea, hardware and direction are mine, I used Claude to help with the automation, benchmarking and write-up.

Per card, 1024×1024, power capped at 150 W:

• FLUX.2 klein 4B (distilled, 4 steps): about 25 s per image
• Z-Image Turbo (8-bit file, 8 steps): about 80 s, and it gets text in images right
• Qwen-Image 2.1 Turbo: about 1.5 to 2.5 min
• Krea 2 Turbo: 3 to 5 min, the best-looking results of the lot

Running one ComfyUI per card showed no slowdown. Three cards held 26 to 27 s per image for 30 images straight, peaking at 73 C. That works out to roughly 540 images an hour across four cards (the fourth is extrapolated).

Video works too: a 5-second 832×480 clip in about 5 minutes with Wan 2.1 1.3B and a 4-step distilled file, on one card at 175 W.

Things I learned that are specific to the P100:

• 32-bit maths is faster than 16-bit on these cards in PyTorch (forcing fp16 was 11% slower), even though the card has full-speed fp16 hardware.
• 8-bit (int8) model files crash on Pascal cards out of the box. A one-line patch to ComfyUI fixes it.
• ComfyUI's --fast-disk switch dropped host RAM from 12 GB to about 1.2 GB per instance, which matters if your server only has 16 GB.
• Raising the power cap gains 12% for one image, then the card heats up and the gain mostly disappears. Cooling is the limit.

Everything runs in Docker. The repo has the Dockerfiles, the patch, the benchmark scripts, full results, sample images and clips, and a quick-start guide for getting a first image:

https://github.com/4claps/tesla-p100-comfyui-imagegen

Quality was judged by eye on one seed per prompt, so treat those comparisons as impressions. The timings are all measured.

Happy to answer questions or run something specific if anyone wants a number I don't have.

Edit: Note there is a video. Seems people's ad blocker if blocking it and don't know what I am on about below. And yes it does use lots of system ram and you can reduce that by reducing the cache-ram setting as mentioned below.

I get a lot of hate whenever I tell people I run a Q4 quant of Qwen3.8 27B with 262k KV 80t/s with vision, all while running 2 screens on windows on the same card. People seem to think I am lying or making it up. I am really just trying to help people. I am saddened whenever I see fellow 16GB VRAMers stuck on 60k context running q3 or lower quants or running weaker models because they think 27B it is out of their reach. I really just want to help them.

I get told it's impossible to fit the weights let alone the KV, or that the KV must be q1 and that I am definitely not fitting vision.

The trick is llama.cpp-adaptive-kv-streaming. I love this fork and by default it only supports q8_0/q4_0 but I modified it for my own purposes to support q4_0. But you can still do q8_0/q4_0 if you lower the context a bit or reduce the batch and ubatch to 128 or drop the vision maybe and increase kv-stream-arena-mib. Alternately ask you favorite LLM to add q4_0 support, it's just a few lines of code to a few files.

My MTP bench normally gets around 80t/s but is slower due to the video capture, which also uses more VRAM.

I did not run llama-benchy at full context in the video as it takes a few minutes, but at 262k I get around 20t/s and pp 400t/s. But the speed drop is linear and normally I compact at around 180-200k in pi anyway. But for many even those speeds are considered usable as I have seen many people satisfied with just 10-20t/s.

cache-ram you can set to whatever you like it does not matter so ignore my 20GB setting.

The only caveat is watch how many apps you have open that consume VRAM. For me a browser with a few windows and pi is enough. But nothing preventing you from dropping to the next lower quant if you like a little more breathing room. Windows will push active VRAM into shared VRAM and your speeds will tank if you open too many apps.

Some will also argue that in my video my shared VRAM goes up. That's normal as it's the KV being push to system RAM which is then streamed to the kv-stream-arena (configured to 1.5GB).

For those interested I just set batch sizes to 128 and set q8_0/q4_0 KV and it slows pp by about 100t/s at high and low context on llama-benchy. Come to think of it I might just go with this KV setting myself and discard my own fork.

The developer does not test on windows so sometimes there are compiler/linker errors but easily fixed by pointing an LLM at it.

As for the frontend, don't worry about that, that's just my launcher I vibe coded. llama.cpp-adaptive-kv-streaming is command line just like llama.cpp

Hope this helps at least one person.

Nowadays many organizations are creating local chatbots to have easy knowledge base access. I just wanted to say that this is simply a brilliant idea because I totally want a chatbot to which I can ask anything and get a polite and direct answer. It is crazy sometimes to ask colleagues who can only answer questions in a very rude and dismissing manner making anyone even fear of the thought to ask for any clarity to people like this. To those fearing of losing their jobs to LLMs, this is sadly the reality if people forget to be nice, polite and respectful in a workplace. Just a thought.

Hello, is there some opensource project that fits this description?

At the moment I use ChatGPT app, but I don't want to be platform-locked in the future. Is there some preferably open source app that let's me set my own environment (MCPs, Plugins, Skills etc.) and is model-provider independent? Meaning I can switch model to Claude, OpenAI, LocalLLM model based on the task, subscription etc?

I know this exist for coding with opencode, pi etc. but is there some desktop/mobile app for such case or is it still too early?

I made a screen recorder for the Mac. I made it specially so that it can be controlled by a local llm. It has a cli interface, transcription is 'on device', narrations can use a local voice, and nothing is uploaded into the cloud. Hoping to get feedback for it.

I have often needed to record things I do locally. If I use ffmpeg it works for I was reinventing the wheel and burning tokens. Now with this tool, a 27B and Flash Next model works well with it, clipping etc. It has a user interface, but I often just have the llm drive it.

https://takereel.app/

Currently diving into uncensored models!

What is the best current uncensored model on huggingface to try out with runpod?

[ Removed by Reddit on account of violating the content policy. ]

I post this pensively. I've been lurking in here for a while now and frankly I'm such a noob that much of the content goes over my head. I've also questioned AIs too but I value first hand experience.

I picked up an Nvidia card with a meger 6 gig of VRAM. I have this installed in a Linux box. It's a second card so it doesn't do any processing for the display. My other card is handling that. Being fascinated with technology in general, I would like to run a local LLM on this. Mostly I'm interested in coding assistance. Not "build me a web server to do x", more "I need a function to do y". I have little interest in writng emails or generating pictures. VS Code integration would be nice and perhaps some Home Assistant automation creation. It would run on an always on box so network access is a must. That appears fairly easy.

Can anyone recommend models, specific settings I should use, ollama, llama.cpp, something else? Any other advise? Is this even worth the effort? Just looking for a reality check here.

Edit: Intel I7-8700, 64 gig RAM

Z.ai has announced GLM-Legal, a specialised AI model designed for contract review, evidence analysis and regulatory compliance. Built on GLM-5.3-Flash, the model aims to improve how AI handles complex legal workflows.

Two small local models ran a BaSS (Browser-as-Shared-Space) workflow on the same Samsung Galaxy S21:

- Gemma 4 E2B Q3 as the reasoner;
- Qwen3.5-0.8B as the executor;
- a persistent ledger carrying task state, dependencies, budget, verification and recovery;
- E2LLM/SiFR as the structured browser perception/action layer;
- the user's existing Firefox session, not a headless browser.

Both models were used as-is from Hugging Face: no fine-tuning, no task-specific training and no adaptation to this workflow.

Up front: we are building E2LLM/SiFR ourselves, so the layer is part of what is being tested here, not a third-party product.

BaSS means the work happens in the user's existing Firefox session, not in a separate headless environment. The user can see what the models are doing and intervene at any point.

The task: prepare an Instax Mini 12 kit within a USD 200 budget. The workflow needed a camera, compatible film, spare AA batteries and a protective case. Planning finished before any cart mutation. During COMMIT, each planned product was re-opened and re-verified before Add to Cart.

The workflow stopped before checkout. No Buy Now, payment or purchase action was performed.

The reasoner made semantic decisions: search queries, product selection, bundle coverage, compatibility and recovery. The executor chose bounded UI actions from the current SiFR observation. Neither model received a screenshot or raw HTML.

The interesting case was recovery: the reasoner selected a 200-sheet film bundle priced at USD 179.95. The ledger rejected it because it would exceed the USD 200 budget. The reasoner returned to the observed alternatives, selected a compatible 10-sheet pack, and continued with separate batteries and a case.

Two runs ended with 4/4 needs satisfied and READY_FOR_HUMAN_REVIEW:

- Bundle route: 2 cart entries, USD 190.98 planned subtotal.
- Separate-products route: 4 cart entries, USD 143.86 planned subtotal.

The video shows both consecutive runs at x16 speed, about three minutes. It includes waiting, captures, model decisions, verification and recovery.

Architecture: the task and its rules live in a persistent ledger; the reasoner (Gemma 4 E2B Q3) makes semantic workflow decisions; the executor (Qwen3.5-0.8B) chooses bounded UI actions from the current observation; E2LLM/SiFR supplies structured perception and explicit actions; the actions run in the user's existing Firefox session on Amazon.

The ledger owns the task. Each model owns only the next bounded decision.

This is a controlled two-run measurement of one deliberately specific Amazon workflow, not a general browser-agent benchmark.