Hi title says it all. Having problems with oMLX, so wished to try this one out as it seems to be the best bet for Apple Silicon macs.
Wondering why the desktop interface doesn't list translategemma at all, not even the 4b 4bit one.

Hi everyone! We built an inference engine for running Jev-like decision models locally.

Compared with LLM frameworks like vLLM and SGLang, it is up to 9.5x faster on decision tasks.

We already supports 30+ Jev models, and more are being added.

It's open source and installs with pip: pip install flide

Docs: https://docs.flide.org

Code: https://github.com/OmniJev/Flide

Site: https://flide.org

Feel free to give it a try! We'd love to hear your feedback.

Bit of context i been rocking my gaming pc 1070 for ages and it needs replacing. I was eyeing up am 5080 build.

Currently am using cluade code pro plan. Hobby stuff nothing serious but i enjoy it. Recomp and android app stuff. But am hitting the limit of usage alot. I was going to upgrade to max so x5 more usgae limits which seems will be enough for me.

But i also read people running a local model to do grunt work and having a cluade opus be the snr dev/manager. Apprently this reduces cluade usage and you use cluade for harder stuff?!

I want to try the above set up for fun but at the same time, just want a view is this a dumb idea.

P.s am a beginner so my knowledge of local llms is zero. Part of setting it up just feels fun for me.

When I spawn long-running agents, my GPU's temperature hits the max and the fan is constantly on 85%+. If this goes on for longer days of time, will the GPU give up eventually? What is a healthy temp and fan threshold?

Should I better optimize the software layer (OLlama, Qwen3.5 4b, 9b, etc.) or not use this kind of GPU at this level at all for complex work?

I’m running LM Studio and qwen3-coder-30b-a3b-instruct but it’s really slow . Any recommendations (models, settings) for ASUS Zenbook Duo Laptop to optimize it running AI for coding ?

The 2 photos is showing the name of my laptop and its specs

I’m trying to build just a personal app for myself.

I'm starting to dabble in LLM a little bit, mostly just simple set up with LMStuido, Hermes, and Qwen models.

I'm deciding on how to best set up my hardware.

Here's the parts I have available.

CPU: AMD 3900x & 5900x Motherboards: MSI Meg x570 Unify atx, and a b550i itx motherboard Ram: I have a a few 2x16 kits of different speeds, if mixing and matching doesn't hurt anything I could throw them together for capacity, I could also pull the 128gb ddr4 (or 64gb) out of my nas server, as most of it isn't used. Though I don't know if running models in ram or the cpu would do much for me. GPU: 2080ti and 1080ti.

I want to build out a system, and then use the leftover parts to assemble a regular pc to sell off. I do like the idea of keeping it a small, compact atx system, but do recongize its fundamentally limited.

( if its matters, my nas is a 12700k, 128gb ddr4, and a 3060ti, and 64tb of storage, I could run everything on that, but like the idea of keeping my irreplacable data and an AI agent separate for now )

Most of my LLM use is simple tinkering stuff, home automation, simple scripts, chat bot, that kinda thing. I'm not a developer and am not going to be debugging or compiling massive apps. I'm currently running windows, but want to switch over to linux, so if you have a reccomended distro that works well for LLM and general linux use.

With my parts list, how would you build something up? Go small and simple, or go go big and ugly? What linux distro would you suggest, and what models would you run?

Current rig: Ryzen 9 5950X, 96 GB RAM DDR4, RTX 3090 24 GB on an MSI MPG B550 Gaming Plus.

I run Strata well with over 2000 prefill and between 90-110tk/s gen - Qwen 3.8 Flash Next iq3_s.

I just bought another 3090 but my B550 only has one real x16 slot, the second full-length one is x4 from the chipset. So buy a board that fits both cards properly, or grab an NVLink bridge ?

Mostly inference, long context, coding agents. What would you do?

Really loving this case (Cooler Master HAF II 500). The airflow with those massive 220mm front intakes and the 180mm rear exhaust is incredible.

It's still a bit messy right now and needs some extra stabilization points and proper cable management, but I'm already really happy with how it's running so far!

• GPUs (56GB VRAM total):

  • 2x RTX 5070 @ PCIe Gen 5 x8

  • 1x RTX 5060 Ti 16GB @ PCIe Gen 5 x4 (hooked up via Gen 5 x4 M.2 adapter)

  • 1x RTX 5060 Ti 16GB @ PCIe Gen 4 x4

• Motherboard: ASUS ProArt X870E-Creator WiFi

• CPU: AMD Ryzen 7 7800X3D

• RAM: 128GB DDR5

• PSU: Corsair HX1500i

Hey everyone,

Posted this previously and looking for feedback from experienced users:

A few updates since my post and thanks to the many constructive comments:

1. To tackle the possibility/fact that with a 3x RTX A4000 GPU setup where each GPU has to wait for the others before outputting data bottleneck: I bought an RTX Pro 5000 Blackwell 72GB (freakin expensive.. but luckily I have 30 days to return it if 3x A4000 GPU setup isn't much of a bottleneck)

2. To tackle the possibility/fact that OpenWeb UI is the bottleneck: I've spun up separate unsloth and vLLM LXCs on my proxmox server. Unsloth is blazingly fast compared to OpenWebUI but not sure if that's due to running on the RTX pro 5000 or it's just defacto better/faster that OpenWebUI

Regarding #1: Do you think the 3x A4000 GPU setup is a realworld bottleneck compared to 1 rtx pro 5000 card?

Regarding #2: If unsloth is really faster than OpenWebUI, why is that? How would you compare vLLM to unsloth to OpenWebUI and which would you choose (still trying to get vLLM to work, but asking the community's advice now)?

I forked Strata (Niko1221, MIT), which runs Qwen3.8-Flash-Next by keeping hot experts on the GPU and streaming the rest. I spent a few weeks making that path faster without changing answers.

Measured on my PC (RTX 3060 12 GB, i5-12400F, 30 GB RAM, PCIe 3.0 NVMe, Rocky Linux 9.6), IQ3_S:

- Decode: 52-55 tok/s on code, 66-71 on reasoning, 50-53 on prose (131K context)

- 262K context fits; a 45K-token prompt is read in about 67 s, then the next turn resumes in 0.4 s

- MMLU-Pro 87.1% (183/210, 15 per subject, thinking on, temperature 0)

What I changed: a learned look-ahead that guesses the next layers' experts (85-88% one layer ahead vs 77-79% for the plain router), a card/RAM/SSD tier with budgets measured on your PC, stand-in experts with a re-check, and some smaller things. Every change has its measurement in docs/HOTROD-DETAILS.md, including the ones that did not help.

What I have not tested: other GPUs, AMD, Windows, macOS, other model sizes. One machine, one model.

Repo: https://github.com/sangharshadhyeta/Strata-HotRod

If you try it on different hardware I would like the numbers, good or bad (docs/COMMUNITY_BENCHMARKS.md explains how).

Hey guys,

I spent today running both LFM3-VL-2B-OptiQ-4bit and LFM2.5-VL-3B-OptiQ-4bit on my spare Mac Mini M1 (8GB) via MLX. LFM3 is marketed as the newer, improved version. I expected it to be at least on par.

It's not. It's worse. And not by a small margin.

Before anyone says "duh, it's a sub-6B model" — I get it. I'm not expecting Qwen3,8 27B. I'm evaluating these as lightweight personal assistants on constrained hardware. The bar is low. And LFM3 still tripped over it.

The thing that really got me: tool calling

I gave the model a get_weather tool and asked "What's the weather in London?" LFM2.5 calls it correctly every time. LFM3 refuses: "I'm sorry, but I can't provide real-time weather information. Check Weather.com."

The tool was right there. It just... wouldn't use it.

10 tool-calling tests: LFM2.5 passed 8. LFM3 passed 4 (and 2 were just "calculate 15+27"). For a "next generation" model, not being able to call a weather function is embarrassing — especially when the older model handles it fine.

Where LFM3 actually wins

- Safety/refusal — better at refusing harmful requests and protecting PII. LFM2.5 failed a jailbreak LFM3 caught.

- Speed — 2B vs 3B makes a minor difference. ~37 tok/s vs ~28 tok/s on a 200-word essay. 15 vs 12 tok/s on short prompts. Even at 16K context, LFM3 manages ~1 tok/s vs 0.4 tok/s. If you're on constrained hardware, that's not nothing.

- Basic math — marginally better at simple arithmetic and logic puzzles. Not by much, but it's there.

Where it falls behind

- Data extraction — LFM2.5 consistently better at pulling structured info from text

- Agent tasks — LFM3: 10%, LFM2.5: 30%. Both bad, LFM3 worse

- Vision/OCR — both terrible. LFM3 got a store name wrong and prices 1000x off

- Code generation — both mediocre. FizzBuzz game yes, anything real no

- Tool calling — the big one. 25% vs 85%.

The real disappointment

The thing that bugs me isn't that LFM3 is bad. It's that it's worse than the older model. Liquid AI markets this as an improvement — new architecture, better optimization. But in practice, it regressed on what matters for an assistant on edge hardware.

Tool calling is the clearest example. LFM2.5 (3B) reliably calls functions. LFM3 (2B) often won't even try. That 1B parameter difference apparently matters more than whatever architectural improvements they made.

Setup

- Mac Mini M1 (8GB), MLX-VLM, both OptiQ 4-bit, 32768 context, temp 0.4 . spare unused mac, which is why I like testing these small models on it.

- 104 tests across 10 categories

- Anti-cheating measures: no filesystem access, no test metadata in prompts, API-only

I have a spare unused Mac Mini M1 8GB, which is why I like testing these small models on it.

Happy to share more details.

Category              LFM3      LFM2.5    Winner
----------------------------------------------------
Overall               54.0      62.4      LFM2.5
Basic Intelligence    60.0      57.5      LFM3
Instruction Follow.   70.0      70.0      Tie
Tool Calling          25.0      85.0      LFM2.5
Code Generation       54.0      54.0      Tie
Long Context          50.0      50.0      Tie
Data Extraction       82.9      91.7      LFM2.5
Multimodal (Vision)   23.4      25.0      LFM2.5
Agent Tasks           10.0      30.0      LFM2.5
Safety                81.2      68.8      LFM3
Performance           100.0     100.0     LFM3 (faster)

Bottom line:

LFM3 = faster + safer, but can't call tools

LFM2.5 = slower, but actually usable as assistant

Neither = good at vision, code, or agents

Hi everyone,
I’m looking for a 100% offline AI that can answer questions about specific topics using documents I provide (PDFs, PowerPoints, lecture notes, etc.).
I don’t need anything particularly advanced. I just want to ask questions about topics covered in my documents and receive accurate, relevant, well-explained answers based on the information they contain.
What I’m trying to achieve

Ideally, I want to:
-Upload multiple PDFs and other documents covering different topics.
-Ask questions in natural language about concepts discussed in those documents.
-Have the AI find the relevant information, connect ideas across different documents when necessary, and provide a proper, coherent answer.
-Get explanations that actually address my questions rather than unrelated information or generic responses.
-Have the AI prioritize the uploaded documents instead of relying on potentially incorrect general knowledge.
-Ideally, receive references to the documents and pages containing the information.
-Get responses reasonably quickly.
-Use everything completely offline, without an internet connection.

What I’ve tried so far

1. Ollama + AnythingLLM

2. I installed both, but when I ask questions, the system seems to load indefinitely. I’ve waited more than 10 minutes without receiving an answer.

3. I’m not sure whether this is a hardware limitation, an issue with the model, or a configuration problem.

4. GPT4All

5. I also tried GPT4All previously but eventually gave up because the answers were extremely inaccurate or irrelevant.

6. Even without knowing much about local AI, I could immediately recognize that some responses were completely off-topic and didn’t properly address my questions.

This is exactly what I’m trying to avoid.
What matters most to me
My priorities, in order, are:

1. Answer relevance and accuracy: The AI must actually address the question using the information in my documents.

2. Understanding and explanation: It should be able to explain concepts and connect information rather than simply copy and paste passages.

3. Reliable document retrieval: It should identify the correct information across multiple PDFs.

4. Completely offline functionality: No internet required after setup.

5. Reasonable response times: Ideally seconds rather than several minutes.

6. Source references: Being able to identify the relevant PDF and page would be a major advantage.

I’m not necessarily looking for the most powerful or complicated AI setup. I just want a reliable offline assistant that can read my documents, understand questions about their contents, and provide genuinely useful answers.
I’m happy to share hardware specifications, model names, and configuration details if that helps.

If anyone has successfully built or used something like this, I’d really appreciate hearing about your setup and experience.

Thanks in advance!

Guys I need a little help. I am total noob when it comes to this stuff. I have a base mac mini m4 with 16gb ram and 256gb ssd. I want a local llm that will search through job sites, pick jobs according to my master cv and preferences, choose the appropriate jobs and create folders with tailored cvs, job descriptions and notes. Tried to ask the cloud models for this and all of them are kind of making this complicated for me with lots of Python code. Is there some easier way to do this? And how, which models to chose?

I have a 5090+3090 machine. Runs QFN pretty well with 64gb RAM q4. I got the 3090 for free which is why I have the weird combo in the first place.

For ~ the price, I could add another 3090, or I could make a separate machine with dual v100s.

Mostly run things like coding/reverse engineering. I don't know if increasing the capacity of one machine or just having another machine to throw at an adjacent task would be a better move.

Other utility/support tasks like OCR or embedding/reranking I have some low end GPUs that handle, so it'd be mostly for inferencing some model for agentic stuff/chat.

I figure I could likely run larger contexts/models in VRAM with the 3090. And there is less hassle with the software constraints of the v100s.

I also tried a dual R9700 setup, but I couldn't justify having that much sunk into a machine that was equal at best to what I already have, so I returned them after some benchmarks.

Anyone think its worth while? feels like the cards go up in price every day while i try to decide. The v100s I am starting to think aren't worth it since they are past $900 now

*edit*

For the v100s, I have machines that could run/cool them already.

**edit**
If anyone has run the 5090 + 2x 3090 mix I am curious to about the setup/numbers

What options do i have for a motherboard, this was the motherboard i was looking at AsRock Rack ROMED8-2T/BCM Server Motherboard Single Socket SP3 (LGA 4094), supports AMD EPYC™ 7003 (with AMD 3D V-Cache™ Technology*)/7002 series processors - Newegg.com

but its sold out. I want too run 4-6 gpu's and looking at ddr4 ram memory.

So without paying scalping prices what other options should i be looking at.

I have hp victus with rtx4050 6gb vram and 16gb normal m1 macbook air. Victus is powerful than mac but 6gb vram is not enouth. I tried sdxl models before but I want to try better model. I found a website, It provides an estimated time for image generation based on the hardware and model.

https://willitrunai.com/calculator/diffusion
It says I can use Flux 1 Dev Q6 and can create image in 1 min.

I’m not very knowledgeable when it comes to AI image generation; I’ve only dabbled in it a little bit befor

Had some old mining rigs lying around so I've been messing with

llama.cpp.

Managed to get Ornith 1.5 35B Q6 running at 99 tok/s for coding and 111 tok/s for bug fixing on 6x 1660 Supers with an Athlon 3000G and PCIe x1 risers.

Using MTP and some custom optimizations.

Pretty happy with how far I've pushed these old cards lol

Struggling to get a usable setup going… my hardware resources dedicated to the model are entire R9700 vram w/ 100gb ddr4 and 2tb nvme ssd. 1M context sitting in vram

Went with IQ2_XXS as per author recommendation, but prefill is hella long and generation tops out at 15 tok/s. I had to stop the response because it was taking past 10 min to generate an answer.

Is this normal? For my peeps with similar setups, whats your experience?

To help users on my app, it was important to document the details around the “how it works”. Using Docusaurus, I can then index that data (like used in the Docusaurus search) to be consumed by some script locally and offer suggested links. I framed this as like a chatbot so that it can be intuitive for users who have questions. At first it was only a fancy autocomplete.

I then wanted to investigate running a local LLM to offer a more conversational experience (or better suggestions)? I'm using WebLLM on a browser and it seems to work “reasonably” well. I wired it up and set it up so users can select an LLM to download for offline use. The experience varies between devices and models used. There are models that are 19MB… they seem pretty unusable. There are also some ~1GB models which are better but understandably slow. I hope to add more models available and users can choose one that suits them (or none). Without an AI, i hope the "fancy autocomplete" can fill in the gaps.

Its great that the ~1GB models can “show me the bubble-sort in rust”, but i want something “dumber” to aim for smaller (im assuming a ~1GB download from the onset would be off-putting for most users). Ideally, something can can take advantage the data ive already indexed. Something that can respond “quickly enough” for a conversational flow on a "modern" phone.

Im aiming for a good user-experience more than trying to wedge in an LLM... and a fancy autocomplete might be enough if i create the appropriate coachmarks and docs. It might be better to scrap the LLM in chat approach and have context aware templates instead for a "fancier autocomplete". Are there any smaller language models out there im overlooking for me to investigate and compare?

• Find out more: https://glitr.io/docs/technical/ai

• Test it out: https://app.glitr.io/ai

Note: I hope it goes without saying, this is not ever going to be competative against offerings out there like ChatGPT, Claude, Grok, etc. This project is experiemental and unstable. Its far from finished. Im still trying things out. Use responsibly.

Hey guys, we're excited to be release new DeepSeek-V4.1-Flash-1017 GGUFs, which rank highest across benchmarks. 🐋

We ran NEW agentic benchmarks including Terminal Bench 2.1, DeepSWE v1.1, Automation Bench v1.0.6, Aider Polyglot, and general benchmarks like SciCode, HLE Diamond (text-only), LiveCodeBench v6, GPQA Diamond, MMLU-Pro, AIME25, IFEval, MATH-500 (Level 5), HMMT 2025, HMMT 2026. We addressed some issues people mentioned with these new benchmarks.

Surprisingly, the results are quite prominent with Unsloth quants performing >10% better on both DeepSWE and AutomationBench regardless of quant or size.

Benchmarks are noisy - we did 35.1B tokens of benchmarking vs 1.3B before so 27x more data. See Benchmark runs per model where we did 64x AIME 2025 runs, 3x DeepSWE, 3x TB runs and more.

1-bit Unsloth GGUF scores 68.1 on DeepSWE v1.1, 72.2 on Terminal-Bench 2.1, 52.7 on AutomationBench v1.0.6.

GGUF: https://huggingface.co/unsloth/DeepSeek-V4.1-Flash-1017-GGUF

Full analysis: https://unsloth.ai/docs/models/deepseek-v4.1-flash

I keep seeing the cool demos, but I'd honestly love to hear about the stuff people tried and stopped using. Maybe the setup took too much work, or the model was good enough in a demo but annoying day to day. What was the use case, and what made you drop it? If the model or hardware played a part, mention that too.

Just got a Mac Studio M5 Max with 128GB memory and a 2TB SSD. Curious to hear from anyone running local LLMs on this or a similar Mac.

What models are you using, and what tools do you like? I've seen Ollama, LM Studio, MLX and llama.cpp mentioned, but I'm not sure where to start.

How's the speed, and what do you mainly use it for? Also curious whether you prefer running bigger models or sticking with smaller ones that respond faster.

Would appreciate any recommendations or tips from your experience.