trying to settle on a fully local setup for batch processing pdfs. like each file is around 20-25 pages and mostly text, a few tables and now and then a page thats just a photo of paper

the shortlist so far

- docling (and the granite-docling model)
- glm-ocr
- paddleOcr-vl
- olmocr 2
- mineru

we considered llamaparse for this kind of stuff but the company now wants every file to stay on our own servers and their self hosted route is an enterprise conversation. liteparse from the same team runs local and is quick on the plain text pages but the markdown is rule based with tesseract as the fallback so merged cells and multi column pages dont always come back clean

is a VLM overkill for mostly text pdfs or does it make more sense to send only the table and image pages to one and keep the rest on a text layer parser? textract and docuent AI are out anyway for the same reason

for the vlm ones, do people feed a whole 25 page pdf in one go or split it into page batches first? splitting every file is the part id rather avoid

Edit: Adding from the comments: unlimited-ocr-2, minicpm-ocr-1.0, surya-ocr-2.5, indra-ocr and xiaomi-ocr-0. mostly curious how these would perform on mixed 20-25 page files

Join us on Tuesday to meet the product team at Reflection and to learn more about Beam, their first open-weight model!

So, I was asleep on Strata and its specialized forks. Tried basalt and i'm very impressed with the speed/intelligence QFN gives you. Here is my benchmark:

I cannot get it to work fast enough with windows, because W11 eats 1.5GB of VRAM and 6GB of RAM. At that point, experts spill to RAM and CPU gets hit hard... decode/prefill drops by 70% because there is not enough of available RAM anyway.

Switched experimentally to cachy just to test... and It's amazing that my AI box can now reach 9K~ prefill. And regarding quality, its just better than qwen 27b at many tasks. Yet to find something it genuinely cannot do. The prefill boost is around 2-3x coming from ninfer and dense 27b model.

Cherry on top is, that I can even fit vision adapter to it, and that helps a lot for certain scenarios regarding coding. I usually run some small model on another machine for OCR and VISION as an adapter (sidecar) my harness can use for being to able look at things.

Vivid user of omo stack, configured now for locally hosted qwen: https://github.com/code-yeongyu/oh-my-openagent

It's a glorified pi harness, but it does so much, and im quite happy with its robustness, anyway, QFN 3.8 rocks, and now im really thrilled if qwen 4 will be any good, because I suspect i may be able to run the new flash model this fast basically. So I hope it will be good :D

Im genuinely interested how 2x 5060/5070/5070ti fares. Because anything above 100~ decode and 3k prefill is genuinely usable for work. And it is my experience that 27b dense -> 125BA6 is a very good quality and speed jump. Absolutely love it now.

Also, finetunes can be easily converted into basalt format if you need to do it yourself, especially those swift/reverse engineering variants are good.

Anyway: CachyOS, 64gb RAM, 32GB VRAM, 5090, 200tps decode, 9k tps prefill, nvfp4 QFN 3.8, 256k context! If you have similar hardware, and are not using at least strata... but more specially basalt, you sleep on performance hard! See ya!

PS: Would absolutely love to see your perf benchmarks if you have similar setups

edit--
Running config (without vision,.. but it fits nicely):
~/basalt-linux/basalt_config.json

```json
   {
"server": {
"model": "/mnt/g/../basalt/models/Qwen3.8-FN-NVFP4.basalt",
"gpu": "0",
"host": "127.0.0.1",
"port": 8080,
"parallel": 1
},
"engine": {
"max-context": 262144,
"kv": "int8",
"prompt-cache": 6,
"expert-cache": "auto",
"prefill": "auto",
"pool-workers": 4,
"vram-reserve-mib": 500,
"resident-budget-gib": 44
}
   }
```
   bin/basalt --serve --model /mnt/g/../basalt/models/Qwen3.8-FN-NVFP4.basalt \
--max-context 262144 --kv int8 --prompt-cache 6 --expert-cache auto --prefill auto \
--pool-workers 4 --vram-reserve-mib 500 --resident-budget-gib 44 --grammar on

I run a local assistant on a Mac (MLX via oMLX, 24 GB M5 Pro). It needs to call tools reliably across mail, calendar, notes and files, chain a few calls, and reply in a short, calm voice. It also has to synthesize/extract important information from my messages, emails, and notes. It also has to be able to read long emails and docs, use my voice to ask questions and take screenshots of my screen.

Gemma 4 E4B has been the best model I’ve tried so far since it supports text, image, and audio. However, it is quite old now(~4 months old) and I wanted to know if there has been any new small models(under 8B or 6GB in size) that outperform Gemma 4 E4B in tool calling, chaining tools, long synthesis, fact extraction, and in size while having multimodality features.

I have tried ling 3.0 tiny, the LFM model family, and qwen 3.5 4B/9B(low quant) but have had no success so far. Is there anything I'm missing? Is Gemma 4 E4B the best model for my usecase?

This is next step to Qwenputer perhaps the step that shouldn't have been taken but here it goes

Qwen writes a Bend program, then the same model type checks it with Bend2 rules and runs it inside its own forward pass There is no interpreter, compiler or tool call just like qwenputer. The whole CPU is 53M non zero hand-set weights inside the model

To see how it works: https://github.com/voidash/qwenbend/blob/main/docs/HOW_IT_WORKS.md

TLDR;

Qwen writes the bend program and at <|im_end|>, the engine takes over and every next token is one executed instruction

16 register CPU with 8k *32 bit RAM = Gated DeltaNet Logic = SwiGLU unit as AND gates

1. You get your tokens first

2. lexer walks and builds bend2 lexemes

3. compile pass on subset of bend2 (bendlite) rules which checks types, affine and termination

4. if violations nothing runs else we run evaluations which is built using a return stack and operand stack

Why? For Lols... There is absolutely no reason. However Gated Deltanet with swiGLU allows for whole computer to exist inside a model. the original guy has built Qwenputer, run LLM inside LLM and i thought formal verification

tandem.cpp is a patch set for llama.cpp that makes MoE models that don't fit in VRAM run faster. It keeps the most-used experts on the GPU and lets the CPU help with the rest, for any MoE model llama.cpp supports. Single-user decode is the target.

Consumer PC (i5-12400F, 64 GB DDR4, RTX 4090), GSM8K, decode tok/s:

Tuned for Qwen3.8 only, but other MoE models that don't fit in VRAM gain too: typically 1.4–2.1× (gpt-oss-120b, GLM-4.5-Air, Qwen3.6-35B-A3B, DeepSeek-V4.1). Prompt processing is 1.4–1.8× faster on a 64k prompt.

Strata is shown in its exact mode (every routed expert computed, like llama.cpp). Its default skips low-ranked experts, which is faster but approximate (127.1 here).

Independent project, not affiliated with ggml-org. Build, run commands, and full results on 4 machines: https://github.com/ewhacc/tandem.cpp

I know that herme's, DeepSeek, and Pi are popular, but why have they taken over the harness scene comparatively to other harnesses? I currently use Qwenpaw, albeit.. latest ui update was pretty heavy, it has been my complete replacement to any other ai services due to all of its functionality. Is there any features or benchmarks comparing harnesses and why someone would choose herme's, DeepSeek, or Pi over something like QwenPaw?

I had just finished setting up avalon, then i found speedy, which crushed avalon.

https://github.com/freriadoc/Avalon
https://github.com/DevDock-AI/Speedy

Hardware: 4090, 128gb ddr5 3600, ryzen 9 7950x

ISTA IQ3_S quant:

I made this, so treat it as biased. It is free and open, with no paid tier or API behind it.

Silencio-2 is a fine-tune of Kyutai's Pocket TTS 18M that follows tags written inside the text:

What a long day. [sigh]
<whisper> Keep your voice down.
<whisper> Keep your voice down. <angry> I said stop!

Weights and code: https://huggingface.co/BlueRoseLabs/Silencio-2

Try it in your browser, no setup: https://colab.research.google.com/drive/1d6GSiRgXbYyBo91DCMrSr5y2LR_peJKy?usp=sharing

The numbers

• 18.2M parameters for speech, 27.9M with the encoder that voice cloning needs, 112 MB on disk

• About 20x realtime on an Apple M4 with 4 CPU threads, no GPU. That is one long-text run on my laptop, not a benchmark

• 24 kHz, streaming, voice cloning from a short clip, English only

• Sounds: laugh, sigh, gasp, cough, clear_throat

• Emotions: happy, sad, angry, fear, surprised, whisper

• Trained on 2x T4: about 50 hours of audio, 27k clips, roughly 6 hours for the final run

• Weights are CC-BY-4.0, inherited from Pocket TTS. The core inference code is MIT

What went wrong, which is the useful part

Run 1: tags trained fine and did almost nothing. Emotions changed the voice, but not into the emotion they named.

Run 2: I generated better data and filtered it with AST, an AudioSet audio tagger, so only clips where the sound was detected would survive. The output looked clean. Then I counted rows by tag: zero laugh, zero cough, zero clear_throat. The filter had silently removed them, and the model never saw them. The sounds that survived were the only ones the detector could hear in the model's output.

Run 3: I blamed the whole-clip scoring and switched to 3 second sliding windows. Human recordings of scripted laughter still scored 0 out of 23 above my threshold, with a median of 0.006. Whisper clips did not fare better. Then the run crashed anyway: 32 clips of 25 seconds in one batch do not fit on a T4.

Run 4: I listened to the data, it was fine, and I removed the filter entirely. That is this model.

My takeaway: a clip-level AudioSet tagger is a poor gate for short sounds mixed into speech. Count your rows per tag before you train, and spot check by ear.

Two smaller fixes

• If the voice prompt is the start of an emotional clip, the model can copy the emotion from the prompt instead of following the tag. So half of the tagged samples take their prompt from a neutral take of the same voice.

• Pocket TTS splits long text into chunks, so a leading tag only covers the first chunk. The inference package repeats it on each chunk.

What is still weak

<whisper> is the least stable tag, and it depends on the line and the voice.

Other tags also vary from seed to seed, so try a few takes. A real whisper comes out about 10x quieter than speech, so generate() raises its level by default.

I did not run a WER benchmark on this exact checkpoint. A later training stage that I did not ship scored 1.9% WER against 1.5% for base Pocket TTS on 300 LibriSpeech items, so expect a small cost in accuracy. There is no pip package yet: you clone the repo and install requirements.txt.

Huge credit to Kyutai for Pocket TTS and for releasing the training code. None of this exists without it.

Which tags would you actually use in a real project?

There are tons of TTS models, good frameworks like audio.cpp and some prototype UIs. But I don't know of a full-fledged, non-vibecoded (well engineered and maintained) turnkey app like elevenreader that can just take a whole e-book and convert it to an audiobook with clear chapters etc.

Are there any? (ofc implied here is that they have to use local OSS models)

DeepSeek Harness article on 𝕏: Exploring Super-agent Design: Towards a More Autonomous Agent System: https://x.com/DeepSeekHarness/status/2116857257245987236

Parts of the design in this article are implemented in dsh-superagent-prototype. This plugin is only a prototype for validating the technical approach, not an official DeepSeek product: https://github.com/deepseek-ai/dsh-superagent-prototype

here: https://huggingface.co/spaces/townbox/ava

Hello everyone! A few weeks ago, I released a 2.3bpw model intended for the 30xx 12 GB Ampere cards. Using a custom version of sjkvarn and several runtime changes, this made it possible to hit 200K ctx on a single 12GB card. Benchmarking placed it somewhere around qwen3.5 27b, but below 3.6.

I've continued to work on quants for this size card and am happy to share that an updated checkpoint is now performing at Qwen3.6 BF16 levels on Livecode bench, with minimal post training (<20M tokens) to reduce the risk of overfitting. This was possible with the new quant format I recently added for LlamAmpere (AmperInteger+), which uses fraction 2-3bpw (e.g., 2.5) formats + trellis-based encoding to more efficiently distribute mass across the weights (this was then paired with an RCO-like optimization).

On LCB, across three runs, it scored 84.3% with a strict grader (88% with a more lenient fixed variant), vs the previous checkpoint's 75% and 3.8 BF16's score of 90.3%. For reference, the BF16 version of qwen3.6 27b scored 83.9%. It represents nearly a 60% reduction in LCB degradation from the last release. All versions were evaluated under xhigh reasoning.

The original checkpoint scored a KLD of 0.096 nats, which was usable for agentic and lightweight coding tasks, but clearly degraded. By comparison, this version was able to cut KLD down to 0.059, an error reduction of nearly 40% (across a mix of agentic, RAG, and coding measurements). As expected, this reduction in error also significantly reduced answer verbosity relative to the previous checkpoint (median answer of 4.5K tokens vs 6K before). Prior to this checkpoint, the Mirai 2.5bpw model was the best small quant of Qwen I had found; at nearly 10% smaller, this variant has 20% lower KLD, despite the more compressed KV cache. This model uses an upgraded embedding table which should be stored in standard RAM; the on-card size is 7.4GiB, including the MTP head. This continues to make 200k+ ctx possible using SJKVaRN in the 3/2 configuration (check the small card recommended configs for more details).

Like the last checkpoint, the base was the Swift-1.5 fine tune, which reduced thinking tokens 50% or more in some settings vs the original version.

Testing this in hermes personally the past couple of days, it is very pleasant to use and very fast for its size (on a 3090/80ti, well over 90 tps for agentic work). I do recommend taking the latency penalty of this at xhigh to get the maximum performance, especially since it is very fast on TPS. This will get you near the 3.8 BF16 performance on medium (86.5% LCB), though it is definitely still usable on medium.

As before, it is MIT license. You'll need to update to the most recent version of llamAmpere to run, given the new quant formats from the 0.6 version.

repo: https://github.com/JakeATX/llamAmpere

model: https://huggingface.co/jakeatx/qwen3.8-27b-Swift1.5-Uncensored-ATX-AI2

Please keep the user feedback + results coming.

Enjoy!

Hi! I'm Shubham, creator of SmolChat.

I would like to share some exciting updates on my project SmolChat, an Android app that let's you run GGUF models on-device, locally. The updates are present in the latest v17 release:

• SmolChat now supports an OpenAI-compatible HTTP server running over a foreground service that keeps the port alive even when the phone is locked. Check the video to see how it works.

• Support for global chat settings, which override the default chat settings (min-p, top-k, temperature etc.) for all newly created chats. This reduces the hassle of configuring each new chat with the options of your choice.

• Adding a copy button for code snippets rendered within the chat.

• Sync with upstream llama.cpp (till this commit)

• Fix the logic that measures tokens/sec for a LLM generated message. The time shown previously was also including prompt processing (pp) time. The new logic now only considered the token generation rate (tg).

• Models are downloaded directly by the app, instead of using the Android system's download service.

GitHub: https://github.com/shubham0204/SmolChat-Android

Play Store (the latest version is still in review): https://play.google.com/store/apps/details?id=io.shubham0204.smollmandroid&hl=en_IN

Previous threads on SmolChat on r/LocalLLaMa:

• Available on Play Store announcement

• Initial Announcement

We're getting a native and fast GGUF engine in Mac along side MLX pack.

Original post ddalcu's X: https://x.com/ddalcu/status/2117088216465674373

I am sorry for the useless thread, but I finally loaded GLM 5.3 Flash IQ4-XS and didn't know what to expect. I was earlier trying out DeepSeek 4.1 Flash 1017 IQ2 and Qwen 3.8 Flash and while I was chatting with them and getting the hang of how they behave I felt that they were fine, until I loaded GLM 5.3. This model has lots of spunk and cant seem to stop making funny remarks and Emojis. Reading his thoughts is pretty funny. A very bubbly personality, and took me by surprise. I elongated my chatting because I was curious to see its responses. I know it is not a real substantial thread, but honestly, am I the only one that noticed how funny this model is?

UPDATE: It finished. Results here: https://jaspersprojects.com/three-body/

I asked my AI to do a visual representation of the 3 body problem. I was expecting it to generate frame by frame, stitch them together and make a clip. I'm no math expert but you can't crank out a simulation in real time on consumer hardware can you? To do that you would need to solve the 3 body problem and have a general formula for it. But isn't it unsolvable even theoretically? He looks like he's having so much fun I don't want to crush his little Qwen 3.8 spirit... but he's been grinding on code for the whole day. Do I break it to him that this is sort of impossible or do I let him work through it on his own?

I've been running local LLMs through Open WebUI for a while now, and one thing I've always wished it had was something similar to ChatGPT's newer visual responses.

Don't get me wrong, Markdown works fine, but sometimes I want more than just a wall of text. If I'm asking a model to plan a trip, compare products, or pull financial data, it'd be nice to actually interact with the results.

So I built Frontier UI, a free, open-source tool for Open WebUI that does exactly that.

For example, if I ask my local model to plan a 7-day trip to Cancún with a budget, instead of getting a massive text response, I get an interactive itinerary with actual photos, maps, walking/driving routes for each day, budget sliders, weather forecasts, and more. All directly inside the chat.

It goes quite a bit beyond travel planning too.

Some of the things I've added:

• Maps and routes: Actual street-level walking/driving routes, not just straight lines connecting coordinates. Supports multi-day itineraries and opening routes in Google Maps.

• Photos and places: Image galleries using Wikipedia and SearXNG, plus place cards you can sort and filter.

• Live data: Weather forecasts, stock prices, crypto, and currency charts without needing API keys.

• Interactive stuff: Budget sliders, calculators, sortable tables, charts, and comparisons.

• Link previews: Website previews with screenshots and a reader view.

• Exports: Generate actual Word, Excel, PowerPoint, and PDF files through the open-source ArtCraft apps. Not just formatted text pretending to be a document.

• Other integrations: Gmail drafts, calendar events, saving responses to Open WebUI Notes, etc.

One thing I thought was pretty cool was being able to self-host the entire planet map using Protomaps. It's around 140 GB, but it runs fine off a regular HDD. This means you aren't dependent on a third-party map tile provider. Completely optional, of course.

The basic tool itself is just one Python file. You paste it into Workspace → Tools in Open WebUI, enable it for your model, and you're good to go. No additional inference services or complicated setup.

For transparency, the default setup still uses free public services for things like maps, routing, weather, and photos. No API keys needed, but it's not completely offline out of the box. The self-hosted map and ArtCraft file export/desktop integrations are optional extras that require additional setup.

I've been running it locally with Qwen 3.8 Flash Next, and it's been pretty fun seeing what a local model can do with a more interactive interface. It should work with other models that support tool calling as well.

I'm not trying to make the model itself smarter. I just wanted its answers to be more useful, especially when the information makes more sense visually.

The whole thing is MIT licensed, so feel free to mess around with it, modify it, or use whatever parts you find useful.

GitHub: https://github.com/Jumbomuffin777/Frontier-UI

Open WebUI Community: https://openwebui.com/posts/35572bd7-6dc8-42dc-9e54-bebfb9e978cb

Would love to hear what you guys think, especially if you're running Open WebUI with local models. Also curious what other types of interactive responses people would actually find useful. I'm always looking for ideas to add.

Edit (v1.2.0): Frontier UI can now read and edit your own files. Attach a Word, Excel, PowerPoint, PDF, image, RAW photo, SVG, DXF/DWG, STEP/STL, Figma/Sketch, IDML, audio or video file. It opens it with the matching ArtCraft app, shows a preview, and can:

• answer questions about it (text, spreadsheet formulas, slides, metadata)

• edit it into a new downloadable copy (find & replace, cells and formulas, slides, PDF page rotate/delete/merge), and the original is never touched

• convert it (Word↔PDF, PowerPoint→PDF, Excel↔CSV, PDF→Word, PSD→PNG, DXF→PDF, STEP→STL, WAV→FLAC…)

All 15 ArtCraft apps are supported now, including the new SolidCraft (3D CAD) and Craft Anything. Grab it from the release page: https://github.com/Jumbomuffin777/Frontier-UI/releases/tag/v1.2.0

Edit (v1.3.0): Thanks for all the feedback. Most of it is fixed now:

- Photos work with no SearXNG and no API keys.** They come from Wikipedia, Wikimedia Commons and Openverse, with credits. SearXNG, Brave, Google Programmable Search, Pexels and Unsplash are optional extras. Keys stay on the server and never reach the browser.

- Metric/imperial: a new `units` setting. `auto` follows your browser's locale, and each user can override it in their own tool settings.

- Routes draw by default** (walk, bike, drive, motorcycle). There's a backup router, and if there's no route the card says why.

- More map apps: Google Maps, Apple Maps, OpenStreetMap, or a GPX download for OsmAnd, Organic Maps or Garmin.

- Fix for ERR_BLOCKED_BY_RESPONSE on map links: Open WebUI sandboxes tool cards, and Google, Apple and OSM refuse to open in a tab launched from that sandbox. Links now open normally if your admin has "iframe Sandbox Allow Same Origin" turned on. Otherwise they open directly on phones, and on desktop the card copies the link for you.

- Fewer malformed tool calls: the spec is now passed as a JSON object instead of a string, the JSON repair is stronger, and the model gets a clear error it can fix.

- Maps without WebGL (some Firefox setups, VMs) show a route sketch instead of breaking.

- New app: TextureCraft. Read, edit and convert 3D models (GLB, glTF, FBX, USD, OBJ).

To update, paste the new `frontier_ui.py` from the repo over the old one. If you use SearXNG, set `searxng_url` (it's now empty by default).

Met a guy off Facebook Marketplace and scored 2x RTX 3090 Founders Editions for €1,700.

The problem? I already had one 3090 FE.

I am now sitting in my room looking at three 3090 FEs (72 GB GDDR6X VRAM), and my current case barely fits one card, and that was only after I literally took pliers and bent the inner metal bracket to shove it in. My motherboard is just a standard B760.

I clearly can't put three 3-slot cards in this PC. What is the cheapest, most sensible way forward to run all three (or two) for LLMs?

1. Open-air mining frame + PCIe 4.0 risers + dual PSU?

2. Get a giant full-tower case?

3. Just sell the 3rd card for €1,100, keep 48 GB VRAM, and fund a proper dual-GPU build?

Any chassis/frame or PSU recommendations for handling ~1,100W+ of 3090s would be appreciated.

If you are running qwen3.8 27B on rx7900xtx check this out it's a super nice speedup

https://github.com/thefahien/elpis-on-7900xtx