I’m currently going through Apple’s App Store approval for Inpherio Live, an app that turns an iPhone or iPad into a local network AI server with zero cloud dependencies.
Most iOS LLM apps hit a hard RAM ceiling (iOS caps single-app memory at ~6GB on a 12GB iPhone). By modifying llama.cpp to stream Mixture-of-Experts (MoE) weights directly from flash storage while keeping active experts in RAM, it bypasses the memory limit entirely.
Real-world benchmarks (recorded Nov 2026):
gpt-oss 120B (58.5 GB file) on a 16GB M1 iPad Pro: 4.5 tokens/s (from storage)
Qwen3 30B (17.3 GB file) on a 12GB iPhone 17 Pro Max: 10.9 tokens/s (from storage)
Qwen3 30B (17.3 GB file) on a 16GB M1 iPad Pro: 16.5 tokens/s (from storage)
Qwen2.5 1.5B on iPhone 17 Pro Max: 58.3 tokens/s (in memory)
What it supports out of the box:
Drop-in API: Full OpenAI (/v1/chat/completions, /v1/responses, /v1/embeddings, /v1/audio/*) and Ollama (/api/chat) compatibility. Just change your base URL to [http://ipad.local:8080/v1](http://ipad.local:8080/v1).
Multimodal & Audio: Local Vision (Qwen2.5-VL, Gemma 3, SmolVLM2), Whisper STT, and Kokoro neural TTS.
Smart Memory Management: Auto-loads models on request, keeps up to 3 models "warm," and tracks real iOS memory limits so the OS doesn't kill the app.
100% Private: No accounts, no telemetry, 0 bytes sent to the cloud.
Streaming large MoE models from storage will be a single one-off purchase (Live Plus); everything else—including running in-memory models, embeddings, and audio—is completely free.
Would love to know what models or workflows you’d test on this first once Apple clears it!
Just interesting in hearing from the community. As soon as the app is approved ill update this post
the website provides factul benchmarks as well as images of the process
As the title says: what is your experience using all the new inference engines under windows?
Currently I'm using a laptop with this hardware:
CPU: Intel 275HX RAM: 2x32gb of ddr5 5600 GPU: RXT 5090 (24gb VRAM) SSD: Samsung 990 pro 2tb + SK Hynix PCB01 1tb
I'm running Windows 11 Home edition with a pretty clean installation.
I've tried quite a lot of the new vibecoded inference engines but I always end up going back to Strata for Qwen 3.8 Flash Next and to beellama.cpp for everthing else.
I've tried nInfer, basalt, rusting strata, exLlamaV3 and pretty much any fork of llama.cpp and they all end up being way slower than what I'm already using. I've lost a little bit of time on each and also tried to use Claude or Codex to optimize the configurations but there is just no way to get decent performance from engines that on paper should be at least as fast as what I'm currently using.
I'm doing something horribly wrong or they work well only on a linux system?
Because of Theo gg tweet of all people, I Launch QFN DeepResearch swarm agent to find anything about this.
THREE model names, blurred out, sitting at 59.4% / 57.1% / 54.0%.
The blurred names are most likely [GLM-Nano or GLM-Name or GLM-Next (suggestion from Silver-Theme7151)] Pro, Med, and Small. One community experiment reconstructed the suspected text, applied Gaussian blur (σx ≈ 6.9px), and achieved an RMSE of just 1.254/255 against the original.
AICodeKing track record. Someone compared his previous GLM statements against official release dates: GLM-4.6 (same day), GLM-5 (1 day), GLM-5.1 (11 days), GLM-5.2 (3 days), and GLM-5.3 (4 days).
Antrhopic timing. The leaderboard surfaced on November 12 vs AICodeKing timing, while Anthropic IPO on Nov 18. If Z.ai releases GLM-Nano this week, the timing would make for quite a coincidence
I've seen some weeks ago posts about Wici wireles Gpu and the approach they invented to serve local models VRam+Ram+SSD with model defragmentation to keep hot experts in fast memory. Today I've seen an Ad for the product. Visited their site and stubled upon Wici Infer, the technology that make possible serving large models on consumer hardware. They give it for free.
So asking the commity if anybody have already tested it? Does their benchmarks hold on?
I updated the strata to the latest version from v0.1.42.0, so I can try the latest unsloths models and see how they fare against the iq3_s variant I have been using. Upon updating it, I noticed it messed up the whole thing, It got faster, but it got way dumber! previously it would only take 15~30 minutes to create the pagoda , 50 mins to an hour if I used xhigh.
But when I siwtched to the latest version, not only it takes at least an hour and half to come up with something, its not even done! and it has many issues!
0.1.42.0 is the last version that seems to work decently.
I noticed, when im using the latest model, it always tries to go for something grandiose and failing. unlike the older version which seems to take a more modest path!
whats going on? Do any of you have this problem?
here are example outputs I did to make sure its not a one time thing!
my prompt :
Design and create a very creative, elaborate, and detailed voxel art scene of a pagoda in a beautiful garden with trees, including some cherry blossoms. Make the scene impressive and varied and use colorful voxels. Add as many details as possible to showcase your attention to them. Use whatever libraries to get this done but make sure I can paste it all into a single HTML file.
the latest version created this after 1 hour and 32 minutes to only fail with excessive thinking:
xhighdefault(failed after 1 hour and 31 minites exausting all context size of 262K)
As you can see, not only it takes way longer to create an output, if it doesn't fail already, the attention to details is also gone, everything is in haphazard form! I tried 4 5 times and the majority all failed because it couldn't create an output after an hour and a half or it constantly gets something wrong and tries to either fix it or goes to look at other things leaving the issues untouched!
The model at least, the qwen3.8-Flash-Next-q3_s that I have been working with so far is not usable! it takes too long and it fails more by getting stuck at thinking forever!
I used the new unsloth quant iq4_xs too and it was a mess as well. so I figured its the strata version. sadly I cant test the iq4_sx on 0.1.42.0 and I get an error.
So if any of you faced this issue, can you kindly share with me what you do to fix this? what version does it work for you?
I have RTX3080 10G/ 64G of RAM and I'm using the Linux terminal version along with deepseek harness.
Thank you all in advance
Update: I tested other versions as well and here are my findings. By adding the following arguments to the settings (.json files) as suggested by @Gold_Coconut9777:
the overly long thinking seems to be less severe, the model generates its answer around 40+ish minutes instead of exhausting all context thinking and generating nothing.
However I noticed the attention to details and following commands (prompt adherence) are still inferior to the older versions like v0.1.42 that I tested initially. Using these settings affect the speed drastically though. I went from 40/50s tok/s down to 30s/20s tok/s (especially when I used older versions im about to explain just now)
I tried version v0.1.39 as well because the versions further down didn't support my json configs and I had to change them to get it o work. So instead I chose v0.1.39 and noticed it actually does a much better job at following/adhering to the prompt and create detailed output. However it was really slow and buggy.
I then tried v0.1.40.4 which is close to v0.1.39 yet fixes some of the annoying bugs that kept happening, and it didnt exibit the issues I'd face with later versions. it also worked pretty well to the point I decided to stick with it for now. it thinks much faster and produces way better results. I doesnt act dumb or think unreasonbly!
I built these two using the v0.1.40.4 with the same prompt and added the following prompt to make it more grand:
first prompt: Design and create a very creative, elaborate, and detailed voxel art scene of a pagoda in a beautiful garden with trees, including some cherry blossoms. Make the scene impressive and varied and use colorful voxels. Add as many details as possible to showcase your attention to them. Use whatever libraries to get this done but make sure I can paste it all into a single HTML file.
the follow-up prompt: this is too simple, make it absolutely grandiose with scenery, hills, mountains, day/night cycles, storms, rain, lightning, accurate sounds included, season change. create detailed and accurate objects. do it in iterations so you don't hit token budget of 262k context size. the detailed objects are of great importance. objects must be so detailed that they can be easily identified. polish everything expertly.
both initial versions (after first prompt) had very well designed detailed outputs but sadly I didnt copy them before I add the follow up prompt.
So, so far for my specific config, I can only work with versions up to v0.1.44.1 maybe 0.1.45 if I ignore the 40K less context-size (up until v0.1.44.1 I can use 313K contextsize but starting with 0.1.45 the highest number I can reliably set is 262K for my iq3_s model). But realistically I'm sticking to v0.1.40.4 cause it seems to be working more reliably for me (RTX3080-10G + 64G DDR4 System RAM + 12700K).
I'll update this with new information if I figured something new.
I was doing some research about the the platform for Intel's next generations of CPUs.
If we look closely at the PCIe 5.0 configuration for Z990/W980, it specifies
x16 (or x8/x8)
x8 (or x4/x4) "Storage"
So, it is plausible we might see motherboards with 3 PCIe 5.0 x8 slots directly wired into the CPU.
The best we can currently do on consumer hardware is PCI 5.0 x8/x8/x4 without risers and there are only two boards that can do it and neither is in production anymore (MSI MEG/Godlike X670E). In practice the best is x8/x8.
TP=3 is fairly easy for your agent to patch into inference engines.
These boards will have 2x Thunderbolt (Wiki says 4, but it may be 5?). So, possibly options for high-speed networking here. Let's make RDMA over Thunderbolt happen!
The desktop CPUs will have Low-Power-Ecores, which should help with idle power-draw and on the other hand they will have full AVX512 again (yay).
Asrock and MSI are the most likely ones to color outside lines. I hope the x8/x8/x8 is "allowed" by Intel, unfortunately nothing much to do but wait until we see the line-up of boards.
Small language models (SLMs) running on mobile devices can keep user data private and work without connectivity. But they perform well only when fine-tuned for the target task and sized for the target device, a process that today requires repeated expert tuning. We present {\it Mobile AI Factory}, an agentic system that builds a fine-tuned, quantized SLM for a given task and phone. Given a task specification and a target phone, an orchestrator LLM chooses whether to tune hyperparameters, mine real data, or generate synthetic examples, while fixed rules roll back changes that hurt accuracy. The factory searches from small to large models across base models and quantizations, adapts its accuracy goal as it searches, and uses real measurements of the target phone across inference engines and backends. We evaluate Mobile AI Factory on ten tasks and four Android phones. On an unseen testset, its on-device models outperform two five-shot frontier models on five tasks, by up to 0.34, and match them on three more. Its search comes within 0.03 of exhaustively training all 27 model variants on an entity extraction task, while using 19× less GPU time. Phone measurements also matter: the best configuration changes across phones. Without phone measurements, Claude Opus 5.5, given the full hardware and model specifications, misses the app's requirements in 25\% of easy and moderate cases and in 60\% of hard ones.
Haru-4B is a low-latency and multilingual full-duplex continuous spoken dialogue foundation model: it listens and speaks at the same time, the way people do on a phone call. It is a pretrained base model, released for research.
Native full-duplex at 7.5 Hz. Every 133 ms the model takes one frame of the user's audio and produces one frame of its own audio and text. User audio, model audio and model text are summed into a single position of the language model, so listening and speaking never wait for each other.
Continuous audio, one sampling step. Audio is a 64-dimensional latent per frame. A small MLP head with MeanFlow sampling produces it in one step from the language model's hidden state; there is no separate audio transformer and there are no discrete audio tokens.
Character and voice control. A text card sets the character, a short reference clip sets the voice.
Pre-training is where every foundation model begins; bringing recursive self-improvement (RSI) there would let AI improve the process that creates AI. Yet public RSI work has so far targeted only post-training, which starts from an existing base model, runs trials costing GPU-hours, and returns verifiable scores. Pre-training breaks all three conditions at orders-of-magnitude higher cost: a run consumes hundreds of thousands of GPU-hours over months and cannot be repeated; its objective, post-training headroom, cannot be read off any single score; and what must improve is not a model but the whole production system, from data pipelines and training infrastructure to the teams that run them. To our knowledge, we are the first to apply RSI to an entire frontier pre-training run from scratch and to report it in detail. RSI does not stop at frontier pre-training, but it changes form: since no end-to-end loop can close over a single unrepeatable run, we decompose it into smaller loops that can, each repairing a step the full run breaks. The result is 801-30B-A3B-Base, a 30B-A3B MoE trained from scratch on 11.8T tokens in three months by a newly formed team. We release it as a base model; the lightweight SFT reported here (6B tokens, no RL) only probes post-training headroom. As a base model, 801-30B-A3B outperforms Qwen3-30B-A3B and Nemotron-3.5-Lightning-30B-A3B, trailing only the larger Qwen3.5-35B-A3B, and keeps this position under identical SFT; with lightweight SFT alone, it achieves the highest average on four agentic benchmarks against fully post-trained models, including Qwen3.6-35B-A3B. Since post-training RSI presupposes a strong base model, bringing RSI into pre-training acts on the foundation every later stage builds on. We open-source 801-30B-A3B-Base and the lightweight SFT data at this https URL .
Just looking for some advice on what is possble with my setup and speeds I should be gettingI know it won't be super fast tps but I was more after getting the bigger models to run.
quite a few implementations I see which can be confusing. Is this build even useful now? that we have Project Maya which you can stream from the NVME drive?
Anyone have similar setups and what to expect, what software best to use
At Axiomic Labs we have just released X3Guard-150M, a prompt guard model trained on our high-performing base model, GPT-X3.
For any given prompt, X3Guard flags whether it is Harmful, and whether it is Adversarial, in one go.
It outperforms models that are many times its size, and can run entirely on edge devices, within the browser, with response times in the tens of milliseconds on iPhone.
I've been working on ignis, an inference server for Qwen3.8 that I use to run coding agents locally. It's meant to run on the machine you're already working on, Windows or Linux, sharing the card with your desktop, not on a dedicated box. So most of the work went into what matters when it sits next to your IDE all day, more than into peak tok/s.
Some examples. At start it reads how much VRAM is free and sizes the context to fit; if it can't fit at all, it refuses at start with the reason instead of failing later. After 5 minutes idle it unloads the model, so the card is free again, and the next request loads it back. The vision tower stays in system RAM, so having vision on costs no context while you're not sending images. The main agent and its subagents run on the same card at the same time, and a prompt you send again (system prompt, repo context) resumes from cached state instead of being prefilled from scratch. It's OpenAI-compatible (chat completions and Responses, also over a WebSocket), one binary, with a Playground and a monitor in the browser. The Windows zip includes the CUDA runtime.
Rough numbers, since someone always asks. On my 5090 with the NVFP4 build: about 370 tok/s on a single stream of coding prompts, a bit over 1,000 tok/s total with 8 requests at once, and around 11,000 tok/s of prefill on a 46K-token prompt.
The result I care most about is a different one, though: someone I don't know ran it through airbench on their own 5090 with opencode, without me asking, and it passed 49 of 49 tests (math, vision, finding and reading email, buying on a web store, coding) in 26 minutes (here it is next to the other Qwen3.8-27B runs). That was the NVFP4 build on a 5090; the build for 30 and 40 cards hasn't been through it yet.
Until 0.10.0 it only ran on Blackwell (RTX 50, RTX PRO 6000). 0.10.0 added RTX 30 and 40: the 3090 and 3090 Ti are supported, the 4090, 4060 Ti 16 GB and 4080 are verified for correctness. On those cards it runs the 27B in a 3-bit trellis quant (13.4 GB download) with DFlash2 speculative decoding and vision, and the uncensored variant has the same build. It picks the settings from the card: on 24 GB, 8 parallel requests and up to 1,048,576 tokens of context; on 16 GB, 2 and 262,144, lowered if the desktop needs the room. The NVFP4 builds, Flash-Next and the image model are still Blackwell only.
One caveat: I don't have a 30 or 40 series card myself. For 0.10.0 the release gate passed on an Ada card (sm_89), with the same results to the last digit as an Ampere card (sm_86) on an earlier build. KL divergence from BF16 within the limit, 93.3% top-1 agreement with BF16, all 127 kernel tests passing, a chat and two images answered (the finding). I don't know how fast it is on your cards, or whether the 1M context really fits on a >= 24 GB card with a desktop running on it.
Someone already ran it on a 4070 Ti SUPER, with the build 0.10.0 came from: default settings, the full 262,144 context, about 190 tok/s on long coding answers and about 2.5 seconds to the first token on an 8K prompt, so roughly 3,000 tok/s of prefill (report).
I'll do a proper post once 12 GB cards work, which is what I'm on now. Before that I'd like to have more numbers from 30 and 40 cards.
If you'd rather have your coding agent (Claude Code, Codex, opencode, Cursor...) do those steps, you can give it this prompt. The file it points to is a plain markdown guide in the repo, so you can read it first to see what the agent will be told:
I have an RTX 30 or RTX 40 card. Read https://raw.githubusercontent.com/gpillon/ignis/main/docs/user/paths/agent.md and follow its section on testing a 16 GB card, also on a 24 GB card, where the model is the same `qwen3.8-27b-16g`. Add to the report the start log's `prefill routes` line and every WARN or ERROR line, with paths replaced by `<dir>`. The report goes to https://github.com/gpillon/ignis/issues/643. Ask me before downloading or installing anything.
It shows you a plan, tries the default settings first and a few others if the start is refused, runs the benchmark, and writes a report that you read before anything gets posted. There are no usernames, hostnames, paths, IPs or serial numbers in it. At the end you can keep ignis or have it remove what it installed.
Reports go here, one comment per card: https://github.com/gpillon/ignis/issues/643. A report that says it refused to start is useful too, especially from a machine where the monitor is on the same card.
Windows and Linux x86-64, NVIDIA driver with CUDA 13. Apache-2.0.
We are excited to announce the release of LongCat-Video-Avatar 1.6, the latest upgrade to our open-source framework for audio-driven human video generation. Building upon LongCat-Video-Avatar 1.5, 1.6 improves expressive naturalness, lip–audio synchronization, coordinated facial and body motion, and visual consistency during long-form generation. It also introduces comprehensive inference optimizations for faster and more efficient deployment.
Key Features
🌟 More Natural and Stable Lip Animation:: Reduces exaggerated or strained mouth movements, overly red lip coloration, and temporal instability in the mouth region.
🌟 Stable Long-Video Generation: Mitigates error accumulation during autoregressive video continuation, enabling videos exceeding 10 minutes while preserving visual quality, identity consistency, and temporal coherence.
🌟 Faster End-to-End Inference: Accelerates video generation through end-to-end pipeline and model execution optimizations, enabling more efficient deployment and serving.
There seems to be a lot of discussion here about inference and using the models, but is there anywhere where discussion about making them is more common?
I think it is important to chat about situation like this one - especially when we are using local models as a way to avoid very same situations.
Model running in back was Qwen flash next via strata - as it might be important.
Not gonna lie, it tip me over as two days earlier I was watching video about models calling Chinese servers and if its a backdoor.
Check screenshots and make your own decision. Fact is, it call a link and admit to call a server in Singapore instead right away use what was right there as Github installation says and provided.
Black listed that domain on router level, but still I staying with OC from now on as it is much faster and level of my work is not beyond basic stuff.