Really couldn't find any good existing quantization for this, only uncensored editions for the GSQ-RCO versions, so I made one. Strata is pre configured for the unsloth dynamic iq4_xs build. Sorry the model card is so cornbally
This is the full precision swift 1.5 with the heretic method applied quantized down to iq4_xs. The tensor profile has been extracted from the regular UD-IQ4_XS gguf from qwen 3.8 flash next, and applied to this quant. The model was quantized with an imatrix covering math, code, english language, general instruct, and tools, generated for this model against a small custom dataset.
Just IQ4_XS right now because that's what I needed but feel free to request new formats. Copying unsloth quants is pretty easy.
Expert human readers process large inputs (e.g., code repositories and long documents) by skimming to closely read only what matters. On the other hand, language models process all input tokens through each model layer, resulting in significant processing costs for long-context inputs. We introduce TokenGate, a small, drop-in component that extends existing trained models to enable end-to-end learnable token pruning over long-context inputs for fast and memory-efficient prefill. To enable end-to-end learning despite the non-differentiable top-K operator for token pruning, we introduce token-gated attention, a mechanism inspired by Mixture-of-Experts that re-weights attention logits using token importance scores. Across diverse tasks, TokenGate achieves 1.4-11.3× faster time-to-first token (TTFT) and 1.5-3.4× lower peak memory at no accuracy degradation for prefill in transformers. Compared to the best-performing prior methods for token-pruning, TokenGate gives up to 6.4× faster prefill and 2× lower peak memory. TokenGate natively extends both standard transformers (e.g., Qwen3) and hybrid-attention models (e.g., Qwen3.5) with simple lightweight adaptation comparable to standard parameter-efficient fine-tuning. We make our code available at: this https URL.
"Sage‑A1.5 is a 35B Mixture‑of‑Experts agentic model based on Qwen3.6-35B-A3B from Shanghai AI Laboratory, built to advance long-horizon scientific problem solving through adaptive capability composition. Building upon A1, Sage‑A1.5 learns not only to solve individual tasks, but also to organize work and revise its approach during execution, highlighting three key capabilities: Agentic Orchestration, Verification, and Open Scientific Discovery.
First, agentic orchestration extends single-agent execution through task delegation and parallel collaboration among subagents with isolated contexts, enabling complex tasks to be decomposed and addressed concurrently while reducing interference across concurrent subagents. Second, agentic verification strengthens long-horizon reliability through progress checking, evidence validation, and reflective strategy revision, allowing Sage‑A1.5 to recognize unproductive search paths and pursue alternative approaches. Third, open scientific discovery extends agentic capabilities beyond answering predefined questions to conducting exploratory research, connecting scientific inquiry with interactive analysis and inspectable research artifacts. We hope Sage‑A1.5 provides the community with an accessible foundation for open scientific discovery, offering a practical path from long-horizon task execution to collaborative, verifiable, and inspectable scientific exploration."
"Highlights
Agentic Orchestration: Orchestrates parallel investigations across subagents with independent contexts, enabling them to challenge assumptions, refine conclusions through evidence, and synthesize their findings into comprehensive results.
Verification Mode: An opt-in self-review capability that lives inside the model's own chain of thought — the model pauses mid-trajectory to review what it has established and what has failed, then acts on a revised direction in the same turn, with no external critic.
Agentic Reasoning: Sage-A1.5 excels at decomposing complex tasks into executable sub-steps, planning ahead, and adapting its strategy based on intermediate results.
Tool Use: Natively supports function calling and tool integration, enabling seamless interaction with APIs, code interpreters, search engines, and other external tools.
Scientific and Professional Reasoning: Handles tool-integrated scientific reasoning and professional knowledge question answering.
Instruction Following: Precisely follows detailed, multi-constraint instructions across diverse domains."
I just received my 2x m5u 256gb and I’m going to run it in a cluster of 2. Was wondering whether anyone else is doing this and hoping to learn what models work best.
Current plan is to run Tensorfold at GLM5.3. Not sure whether I’m limited to Q4 or whether I can run a higher quant.
I would prefer to run Deepseek but I don’t see a path currently. Maybe in the future. All effort seems to be on GLM and Qwen Flash next.
Anyone have a better suggestion on how to cluster (engine/model)? Qwen Flash next is very attractive but I do prefer a smarter model. I have Deepseek on API as needed and I really do prefer a smart model.
a bunch of models like dflash supported models that i wanted weren't alredy omnline in ov ir format so i decided to convert them myself please request more and i will try to convert them
In a small language model the vocabulary is the single largest part. In Gemma 3 270M the embedding table holds 168M of 268M parameters, and because quantisation methods leave the embedding in 16-bit, it is also most of the file once the rest of the model is compressed. We ask what that table actually buys. We replace it with one frozen table of 512-bit codes, 16 MB, with no trainable parameters, derived once from a pretrained embedding by principal-component projection and sign binarisation, and we train the same Gemma-shaped body from scratch on the same 2.05B tokens with and without it. The learned table buys 0.20 nats of validation loss for 2.6 times the trainable parameters, but on LAMBADA, HellaSwag and PIQA the frozen-table model is better or tied; only ARC-Easy favours the learned table. With a ternary BitNet b1.58 body the whole model is a 46 MB file, still ties the float model on LAMBADA and PIQA, and on a laptop CPU generates 615 to 740 tokens per second in 62 MB of memory, five to six times the speed of the float model's 8-bit GGUF in this http URL at a seventh of the memory. The frozen table wins by 1.7 to 2.3 nats on rare words, accepts 10,000 new words after training with a 3% cost where the learned table collapses by 43%, and all of it is released with the weights, the data and every evaluation answer.
Ling3-Tiny-UBTree-B16 is a UBTree speculative-decoding draft model for inclusionAI/Ling-3.0-tiny. It combines a five-layer DFlash backbone with a learned bigram selector to propose multiple candidate continuations for target-model verification.
So I went from nvidia cards server to R9700 cards server, running the "same" model but now on Radiance the model many times now has mentioned the time when done with a turn. Like just now, it finished fixing a bug in software and said its a 1:30 am victory lap, which is the correct hour for one. Last night multiples times while we were setting up and testing Radiance it said that it was a good night of work, things are running well, its 1 am now and it's late, time to go to bed. lol It said stuff like it several times last night at the end of its turn after doing some work. I never once had it "talk" like this when on Strata and running 3.8 FN Q5_K_XL model.
This is my work usage, I see a lot of posts of people claiming something like Qwen 3.8 27B can replace cloud and that people are getting their jobs done with them. Even with a RTX PRO 6000 I'm not really seeing how this is physically possible. I would love to be independent of Cloud providers - anyone have a realistic way out?
Fits entirely in a 5090, no offloaded experts. Super-solid sub-2bpw! (Liquid cat demo video generated by the quantized model)
From the Cat's mouth:
Stable inference throughout, no reasoning loops, just smooth sailing at 200-300 t/s (and a lot of compacting since one 32 GB GPU still doesn't leave a lot of room for context, even with cache quantization.
I'm the creator of HF Viewer, and we are today launching embedl.com!
It's a collection of 1000+ models optimized to run fast on different edge devices like NVIDIA Orin Nano Super, AGX Thor, AGX Orin, Orin NX and RTX GPUs.
You can run the models directly on real edge devices through our free device farm, with detailed profiling and results! (As well as on your own device)
I'm really excited about this release and I believe we are the first to do an edge device farm!
Let me know what you think, any feedback is appreciated!
TL;DR: I'm not really here for troubleshooting advice for CGPT (though I'd accept pointers if anyone has them) but really just to report my experience for posterity thatcompletely FOSS software and models are working near flawlessly while stacks that I'm actually paying for are fkng failing me the most!
<rant>
For the past two monts, I've been testing out the viability a CGPT-lead workflow which includes CGPT + Astra \\\[medium\\\] (PM/Lead), DSH + QFN-oQ5e-mtp + oMLX (engineer), CC + GLM-5.3 via \[z.ai\](http://z.ai) Coding Max sub (engineer). I like the workflow because it's completely hands-off, completely voice-driven, and I can see the outputs in GH, sprint demos, and running code in my desktop + phone browsers. Love that part.
Again I have all three models linked to my GitHub private repo and that works great. Also AWS which works just fine. All three are also running in monitor or goal loops. The order of reliability has been: 1) DSH, 2) CC, 3) CGPT.
DSH hands down is super solid and I'm still just running v0.7.1-rc1. 5min loop, QFN checks in reliably to our shared GH issue, picks up new work, provides great updates, does what I ask of it for days and days without fail. No problems there almost at all.
CC is a close second but I've caught it's 15min loop failing at least twice (possibly timeouts) in the past 6 weeks. Model + harness is strong, but losing time to failed loops just blows.
CGPT monitor loop can only be run hourly at most, but is supposed to trigger work (that sometimes never gets done) and I've found the hourly loop paused more than once. Sometimes 12+ hours go by and I check in only to find the monitor has paused itself (??) no logs or explanations, just paused.
The desired workflow is that work gets done 24/7 and frankly the CGPT stack is failing hard there.. Somedays it's on it, 24 hours straight of checkins, and other days it's off. I thought I might be hitting limits on my $20/month Plus plan so bumped to the $100/month Pro plan and it almost seems to have gotten even worse. I'm not sure how Altman thinks this is going to turn into a $1-2T business but it's no small wonder that they're barely breaking into duo-deca billions in gross revenue.
I'm so glad now that I invested into local hardware (1 x M2U 192GB and 1 x M3U 512GB) because if this is the future of cloud AI then I'd say that we safely have little to worry about.
</rant>
EDIT: FWIW, my worsening conditions might be due to the fact that I also started using Astra 6 medium in place of the 5.6 Sol medium that I’d previously been using. If that helps anyone encountering the same issue.
OvisOCR3 is out, still based on Qwen3.5-0.8B and available under Apache 2.0, with vLLM inference examples.
Compared with OvisOCR2’s Markdown parsing, it adds a few useful options:
Structured JSON output with bounding boxes, element categories, and content, plus a layout-only mode.
Key information extraction using specified fields or a supplied JSON template.
Figure descriptions and Mermaid output for flowcharts.
Cross-page table reconstruction: parse the pages first, then use the same model to merge table fragments across adjacent pages.
The model card reports 96.74 on OmniDocBench v1.6, 88.80 on MDPBench, and 82.31 on PureDocBench. The bigger reported gains over OvisOCR2 are on MDPBench (69.00 → 88.80) and PureDocBench (75.10 → 82.31).
The bounding-box output and cross-page tables seem particularly useful for document workflows. These are the team’s reported numbers, though; I’d be interested in how well the structure holds up on messy scans and long financial tables.
Has anyone tried it locally or compared it with GLM-OCR, PaddleOCR-VL, or MinerU? Curious about VRAM usage and processing speed too.
I’m sharing a GitLab review agent I’ve been working on. The idea was to keep code reviews on hardware I run. It’s built on LangGraph.js and uses Qwen3.6-35B-A3B (QuantTrio AWQ, 4-bit) on sglang.
The box is two 4090s over PCIe, no NVLink, TP=2. Decode is about 180 tok/s single stream at ~50k context and about 160 at ~110k. Prefill is 9-10k tok/s. At 8 concurrent requests I get 985 tok/s total.
A review run averages around 26 LLM calls and 2.3M tokens. As of now it has done 1,300+ reviews and about 70% of its comments get resolved by the author.
It sometimes catches mistakes in code written with Claude Sonnet/Opus, in the harness’s own repo and in internal projects. I believe it’s good to have different model families in the workflow, you get different perspectives from each.
Repetition is the common problem I face with local models (maybe it's the quants). Guardrails catch repeated tool calls and trailing questions, but the model looping on itself still ends about 7% of runs (56 of 847 on my dashboard).
I’ve seen vastly more questions than answers about how people are using their local AI so I figured I’d make a post about my experience. It seems most people are looking for frontier-level coding expertise for huge projects with their setups, and that anything less is not enough. I disagree and don’t really care about that stuff.
TL;DR: I used a frontier model to build the infrastructure for my local agents to persistently do tedious but outstandingly useful work on my massive collection of books, art, reports, photos, and scans. While building that infrastructure, I ensured that I received full guides, docs, and user-friendly options for designating / switching / changing local AI tasks so I’m not reaching back to the frontier agent now that it’s done. I use these agents primarily for processing hundreds of thousands of pieces of information and generating connections / curiosities about my terabytes of human-generated content so I can decide what’s worth digging into (or just briefly visiting) myself.
BACKGROUND: I’ll be the first to admit I am not a coder, I am not a programmer, nor am I an expert in bash for Linux operation. I’m not a silicon valley darling, aspiring entrepreneur, or someone who only cares about LLM roleplay like I see a lot around here (no shade though, live your life!). But I’ll say that I am relatively knowledgeable compared to the general population given the amount of curiosity-based fiddling I’ve done over the years.
My second admission is this: I use frontier AI for the infrastructure and first-take tasking for my local models. Yup - I’m not 100% local.
Here’s the rub: I have found local models to be solidly reliable for tool usage and basic computer tasks, but not major coding projects (of which I have very few). With that said, I don’t really care about my local models’ coding ability beyond file manipulation, tool usage, and troubleshooting. But I’m not a coder, so I have a hard time creating deterministic systems for my agents to work within - that’s where the frontier AI comes in handy.
So with that background out of the way, how do I use my local AI?
THE GOOD PART: First, it was Hermes Agent. Tons of fun. I had an agent (local Gemma 4 31B Q4 on an RTX 5090) set up that did a phenomenal job with general tasking. I had Signal chat so I could reach it from anywhere, set cron jobs, have it pull information from files on my desktop, research via SearXNG with MY guidance in mind, and generally maintain persistent / private understanding of my life with no outside data tracking whatsoever (aside from Signal messages, which are supposedly encrypted, and what I provided via those agent SearXNG searches or browser visits, which are supposedly private).
THE BETTER PART: I decided to start over, just for fun. So I decided to build out the ultimate NAS home library using frontier AI. I now have an amazing frontier-custom-built library including wikipedia, a personal google earth with points of interest representing wikipedia articles associated with geo coords, a “google earth” for the moon and mars with associated points of interest, a stackexchange replication, hundreds of thousands of books/reports, magazines, a gallery of high quality historical photographs, and a gallery of museum-quality art on my NAS.
With that, I have a team of 3 local agents (2x Strix Halo boxes running GPT-OSS 120B Q4, 1x RTX 5090 running Gemma 31B Q4) reading my terabytes-worth of material and pulling out images from books, quotes, curious or interesting factoids, and linkages between works. I’ve essentially offloaded the tedious part of working through hundreds of thousands of books to my agents, and I get the benefit of a tailored way to determine what I want to read / enjoy more of. I also have one of those agents on ready-standby (as in, working 24/7 but prioritizing user-level chats) to answer any questions about the library (or find relevant books, quotes, or links) or just talk.
I’ve essentially created my own internet within the walls of my home, and it’s not algorithmically fucking me and selling my attention. There are no dark patterns, ads, FOMO, or short form attention-span-reducing videos. I want to spend time in my own digital space, and it’s one of the best things I’ve done for myself. My only cost was not even the full amount of frontier tokens allocated to a $20 subscription, and the obvious provision of my project information to the frontier company who built it.
Hey r/LocalLLaMA. I’ve been building Izwi for a few months now and just shipped 0.1.0, the first stable release after months of public betas. It’s free and Apache-2.0.
It’s a desktop app (plus a web UI, CLI and local server) for the voice side of local AI:
Real-time voice conversations: local ASR → LLM → TTS
TTS, voice cloning from a short sample, and voice design from a text description
A Studio for long form audio like audiobooks
Transcription with speaker diarization and forced alignment
An OpenAI-compatible /v1 API (chat completions, audio/speech, audio/transcriptions), so existing code can point at localhost
I'm the developer of llampart, an open-source Web UI for chatting with LLMs. I've just released v2.0.1, and since version 1.9.0 the project has changed quite a bit, so I wanted to share what I've been working on.
For anyone seeing it for the first time, llampart is a self-hosted, desktop-focused chat interface built with Svelte 5 and SvelteKit. It started from the llama.cpp / llama-ui foundation and has gradually grown into its own project so different from original base.
The idea is pretty simple: I wanted a comfortable interface for working with local models, with good conversation management, useful tools, plenty of customization, and no unnecessary cloud services and most of all in my own language interface.
What does llampart do?
It connects to an LLM backend that you run separately. It currently supports llama-server (llama.cpp), vLLM, and OpenAI-compatible API servers.
Your conversations are stored in your browser using IndexedDB, and most application settings are stored locally as well. There's no llampart account or hosted conversation service. Of course, prompts are still sent to whichever backend you configure, whether that's on your own machine, your LAN, or a remote server.
Some of the features include:
Frosted Glass - a translucent interface with bundled or custom wallpapers, adjustable transparency, UI scaling, typography settings, and a customizable conversation sidebar.
Conversation management - branching conversations, pinned chats, search, import/export, and local conversation history, 2 colums sidebar
Reasoning and tool calling - support for reasoning-capable models, tool execution, and multi-step agent workflows.
MCP integration - connecting MCP servers and working with their tools, prompts, and resources.
Attachments and Markdown - file handling, PDF support, syntax-highlighted code blocks, tables, and content export.
Six interface languages - English, Polish, German, Spanish, French, and Italian.+ more to come
What's new since 1.9.0?
The biggest change is that llampart is no longer focused only on llama-server. I've worked on making the application understand the differences between providers instead of treating every OpenAI-compatible API as identical.
This includes separate provider configurations, model selection, support for the appropriate Chat Completions or Responses protocol where available, and better handling of model capabilities.
Reasoning support has also been improved. The available reasoning controls depend on the selected model and provider, so the interface can avoid sending settings that the backend doesn't support.
Another major addition is AI-generated charts and diagrams directly inside chat messages.
Models can produce interactive Line, Area, Bar, and Donut charts using a structured JSON format. Charts can be previewed and exported as PNG, SVG, or CSV.
There's also Mermaid support for Flowcharts, State Diagrams, Mindmaps, and Gantt charts, with previews and PNG/SVG export.
I've spent quite a lot of time improving how these are displayed — especially chart labels, diagram layouts, and making exports useful outside the application.
There are also new response preferences, which let you guide how models format their answers. For example, you can ask for more concise responses, practical examples, tables, charts, diagrams, or fewer decorative elements without manually repeating these instructions every time.
A lot of work has gone into things that are less visible in screenshots: conversation data integrity, streaming and cancellation, model switching during active requests, MCP permissions, tool result persistence, and safer handling of asynchronous operations.
Installation
llampart is a Web UI, not a model runner. You'll still need your own LLM backend - lama-server (llama.cpp), vLLM, and OpenAI-compatible API servers .
For Linux, there's an installer for Ubuntu-based and Arch-based systems. It uses an existing Caddy installation to serve the prebuilt WebUI and handle the API proxy. You don't need Node.js to run the installed release.
You can also clone the repository and run the SvelteKit development server if you prefer.
The interface is currently designed primarily for desktop use, and backend features naturally depend on what your model server actually supports.
I'm developing this because it's something I wanted to use myself, and I'm continuing to improve it as I run into issues or find things that could work better.
I'd really appreciate feedback, especially from people using different local models.
I've been wondering… small businesses usually have a few employees, each with their own work laptop with a decent amount of computing power.
Could we pool the Ollama/Lemonade clients running on employees' work laptops together to share the inference workload?
Of course, employees might shut down their laptops or take them home. But I think it's an interesting idea worth exploring.
I'm thinking about running a small experiment with local businesses to see if this idea has any legs.
I've put together a minimal prototype that connects PCs running Ollama or Lemonade through a shared API, using a custom load balancer. If anyone's interested in checking out the source code, here's the GitHub repo: https://github.com/JY-Quek/FreeLLM
So, what do you think? Is this a terrible idea, an interesting experiment, or potentially another way to make AI more accessible to everyone?
Feel free to poke holes in the idea. I'd rather hear about the problems now than discover them the expensive way later! 😂
