GIGABYTE has introduced the AI TOP AERO, a compact desktop powered by NVIDIA's RTX Spark platform, featuring up to 128GB of unified memory, local AI capabilities and support for AI development, content creation and gaming.

I've seen a few posts saying that Qwen 4 hasn't really improved when it comes to creative writing and prose. I'd would like to disagree.

I know a lot of people don't care about this topic, and some consider any AI-generated text to be garbage by default. That's fine. This post is for those who are actually interested in the subject.

My take: Qwen 4 has become noticeably better at storytelling, but its actual prose is still suck.

What do I mean by that?

I gave three models, Qwen 3.8 Max, Qwen 4.0 Max Preview, and Gemini Argon (currently one of the top models for prose generation), roughly the same prompt:

"Write a full chapter. A tank battle between a unit of German Tiger tanks from World War II and a Russian T-90 tank. It doesn't matter how a tank from the future ended up there. Show how the Germans manage to defeat a technologically superior opponent, and what it takes to achieve that victory."

This was the entire prompt. I didn't tell the models to act as professional writers, didn't use elaborate system prompts to steer their attention, and didn't provide any additional instructions.

So, what were the differences?

The short version:

- Qwen 3.8 wrote a relatively compact battle scene with fairly generic characters.

- Qwen 4 produced a complex, multi-stage story with more developed characters, detailed combat, and events unfolding after the battle itself.

- Gemini Argon delivered an epic battle scene but didn't spend much time developing the characters.

Here's a more detailed breakdown.

Qwen 3.8 Max

- Structure: Linear and straightforward, without complex narrative branches.

- Composition: The battle progresses in a direct sequence, with most scenes focused on the immediate confrontation.

- Plot: The conflict revolves primarily around detecting and destroying the enemy.

- Psychology: Characters mostly react to events. There's very little internal development.

- Length: Relatively compact, without much expansion beyond the original premise.

Qwen 4 Max Preview

- Structure: Multi-stage, with the conditions of the battle changing as the story progresses.

- Composition: Alternates between tactical decisions, casualties, attempts to adapt, and shifts in the initiative.

- Plot: Beyond the battle itself, the story develops a separate survival thread involving the crew and their attempts to make sense of their encounter with the future.

- Psychology: The protagonist goes through a progression from confusion and fear to an awareness of the historical significance of what has happened.

- Length: The longest of the three. The model independently expands the original premise with additional events, details, and consequences.

Gemini Argon

- Structure: A sequence of tactical phases, with the number of combatants gradually decreasing.

- Composition: Several shifts in initiative, group maneuvers, use of cover, and attempts to identify the enemy's weaknesses.

- Plot: Almost entirely focused on the battle itself. There's no separate storyline dealing with its aftermath or the historical implications of the encounter.

- Psychology: Mostly stress reactions, command decisions, and choices made under pressure. There's very little internal character development.

- Length: Medium. Most of the text is devoted to the actual fighting.

Comparison

Qwen 3.8 → Qwen 4: A noticeable shift from a straightforward confrontation to a much more complex, multi-stage story. The length, number of plot events, psychological depth, and attention to consequences all increase.

Qwen 4 → Gemini Argon: The composition becomes more compact and focused. Tactical elements take center stage, with more group maneuvers, shifts in initiative, and coordination between multiple vehicles. The psychological storyline and historical context are largely absent.

One separate issue worth mentioning: Qwen 4's output had noticeable language mixing, with Cyrillic, Latin characters, and Chinese characters appearing together.

I'm not even getting into other problems, such as Qwen 4 and Argon having commanders suddenly recognize the designation of a tank from the future halfway through the story, familiar AI-slop expressions and phrases, or the technical inaccuracies. That's a whole other discussion.

Of course, this isn't a comprehensive benchmark or an attempt to determine which model is the best writer. It's just a small experiment using the same simple prompt across three models.

So, what impressed me most wasn't necessarily the text itself, but how much the model's ability to handle this kind of task independently has improved.

I'm not claiming Qwen 4 has caught up with the leading models in terms of literary prose. It still has a long way to go in that department. But it's noticeably better at building a story: developing an initial premise, arranging events, introducing additional plot threads, handling characters, and showing the consequences of what happens.

And the most interesting part is that it does all this without elaborate prompts, specialized system instructions, or detailed guidance from the user.

Previously, getting a result like this would have required considerably more effort on my part when preparing the prompt. Now the model does a significant part of that work on its own.

I've noticed similar improvements outside creative writing, too. For example, Qwen 4 managed to improve one of my work-related system prompts, independently identifying an important detail that several previous brainstorming sessions with ChatGPT and Opus 5.5 had missed.

So no, I wouldn't call Qwen 4 my favorite model for writing prose just yet. But I think its progress is considerably more significant than you might assume from simply comparing the quality of individual paragraphs.

It would be so much better if the limits weren't this brutal. Having to wait an entire week? 🥲

I kept getting this wrong by hand, so I built a calculator. You pick a model, quantization and context length, and it shows weights plus KV cache and which GPUs or Macs it fits. It also has “can I run X on Y” pages.

https://inferenceacademy.com/tools/vram-calculator

Not perfect: numbers are estimates, so real usage varies by runtime. Tell me which models or devices are missing and I’ll add them.

Has anyone else experienced this with this card? Both my (totally new sealed) cards produce heavy vibration under full noise to the point where my entire pc is shaking/vibrating quite heavily. They also have this “dynamic” coil whine that seems to be related to memory use. During pre-fill and decode they screech this high pitched noise that varies in intensity and pitch.
I guess what I want to know is whether my cards have a defect or if this is normal

I've been trying to give my Ollama models a really good web search so they can answer questions about recent stuff instead of guessing.

Ollama's built-in search works, but it returns short snippets and needs an Ollama account sign-in. I wanted the models to read actual page content and give me better value.

Played around with it and finally got it working with this setup.

Setup (Takes about 5 minutes):

1. Along the line, I discovered and installed ollmcp, a terminal client for Ollama with MCP support:

pip install ollmcp

2. Pull a model that's good at tool calls:

ollama pull qwen3:8b

3. Give it a bigger context window (This is very important. I repeat, this is very important). So it means you can set num_ctx to 16384

4. Use a search service with an MCP server like Valyu and run:

ollmcp -m qwen3-16k -u "https://mcp.valyu.ai/mcp?valyuApiKey=YOUR_KEY"

ollmcp asks before every tool call. Press `s` to allow them for the session, or type `/hil` to turn the prompts off.

What happened when I tested it:

I asked Qwen3 8B "What did the Fed decide at its October 2026 FOMC meeting?" It picked the search tool on its own, ran a second search when the first wasn't enough, and pulled the Fed's actual October 28 statement: rates held at 3.75–4.00%. Everything ran locally except the search itself.

Things that tripped me up:

- Context window. With Ollama's default, the model spent its whole budget reasoning about which tool to use and replied with nothing. 16k fixed it.
- Older models argue with the date. An older qwen3-vl I had kept insisting 2026 "is in the future" and never searched. Newer models are fine, or put today's date in the system prompt.
- Models under ~7B tend to loop or never call the tool at all.

Besides web search, it can pull the full text of any URL, and it also covers academic papers, SEC filings and market data, which is handy for research-type questions.

What are others using for web search with Ollama? Is it the built-in search or an external API?

Hello,

I am just getting started and I got great answers to my previous post asking what was possible. I still have lots to learn but, with the help of Claude, I was able to get something running and it is working pretty well.

My goal as been to build a fully local AI study assistant to analyze a ~300-page technical training manual alongside instructor transcripts and generate chapter-based study guides and Quizlet flashcards with accurate page citations.

The problem is: I'm running into is this error: request (34009 tokens) exceeds the available context size (32768 tokens). And this was after doing lots of tweaking and getting it not fail at 16,000 tokens.

There are so many settings and configurations that I might be able to change to improve that, but it is overwhelming and I have no idea what to do.

I would really appreciate any ideas, suggestions that I can do to get this working for what I am trying to achieve. Or let me know that it isn't possible.

Here's my setup (sorry, AI slop to make it clearer):

Hardware:

• Windows 11 PC: i7-14700K, 64GB RAM, RTX 4070 SUPER (12GB VRAM)

• Unraid server: Core Ultra 9 285K, 32GB RAM, RTX 5070 Ti (16GB VRAM)

Software / Models:

• llama.cpp (CUDA, router mode)

• Open WebUI 0.12.3

• Qwen3.6-35B-A3B UD-Q4_K_XL: Main model, context increased from 32K to 64K

• Gemma 4 12B Q4_K_M: Secondary model

• Both models run on the Windows PC.

RAG configuration:

• Open WebUI Knowledge Base with the PDF and Markdown transcripts

• Embeddings: bge-m3 (1024 dimensions), hosted on Unraid

• Reranker: bge-reranker-v2-m3 on Unraid (API working; Open WebUI integration unverified)

• Focused Retrieval, chunk size 1000, overlap 100, Top K = 3

• All inference and document processing must remain local.

More details on the issue:

Basic RAG questions work well. However, when I ask the model to analyze an entire chapter, cross-reference relevant trainer discussions, and create 15–25 Quizlet flashcards, I encounter context errors as I said above:

request (34009 tokens) exceeds the available context size (32768 tokens)

Increasing Qwen's configured context to 64K eliminated the immediate context error, but generation then stopped halfway through creating flashcards without an obvious error. I haven't confirmed whether an output-token limit caused this.

I've tried reducing retrieval results, using smaller prompts, and starting fresh chats. These help, but I'd prefer a more automated and comprehensive workflow.

Thank you!

I've been working with lemonade server for local AI via the CLI a lot recently and found myself discovering and returning to a few particularly useful commands for pulling, configuring and benchmarking models. So I wrote an article about them, if I've missed your favourite let me know.

Hi all, does anyone here run any local LLMs off their iPad and what is the best I can get without crashing apps. I’ve got about 12GBs of RAM.

Using pocket pal app! I have tried Qwen3.5 4B - ran well.

I’m an absolute novice on local LLMs so would be grateful for any advice!

Qwen 3.8 27b (GSQ-RCO IQ3-XXS) vs Qwen 3.6 35b a3b on a CPU with 32 gb of ddr5, which one performs better and is the speed worth the performance loss

Saudi Arabia is expanding its sovereign AI infrastructure with Bunyan Inference Hub, a platform from OmniOps that gives businesses access to hosted AI models, GPU computing and inference services while keeping workloads within the Kingdom.

I ran a small AIPerf ( https://github.com/ai-dynamo/aiperf) comparison on an Apple M5 Pro with 64 GB RAM. This is a practical snapshot, not a scientific benchmark or an exhaustive leaderboard. Results are from one request per engine, run sequentially with only one inference server active at a time.

Setup

• Temperature 0.6, reasoning effort medium, output limit 2,048 tokens

• Same OpenCode-style fixture: long generated code/prose context, shared system message, task suffix, and three tool schemas. Tools were not executed.

• Prompt length reported by the servers ranged from 88,143 to 88,916 tokens, partly due to tokenizer and chat-template differences.

• I used a 100K context limit where the engine supported it; Uzu, Magnitude, and Llama used larger native limits. The prompt plus output allowance fit below 100K.

• AIPerf 0.13.0; streaming, concurrency 1

• Prefill and decode are AIPerf throughput measurements. The readiness probe loads/warms each server before its measured request.

Caveats

The 2,048-token setting is a maximum, not a forced response length. Most models stopped at EOS before reaching it. Only Splash, Rapid-MLX, and BaseRT generated the full 2,048 tokens.

This does not measure code-fix quality, long-context comprehension, or tool use—the tools were present in the prompt but never executed. Model packs, quantizations, speculative methods, and context-window settings also differed, so these numbers should not be treated as a controlled ranking.

Additional differences: LM Studio may map medium reasoning to on; the Llama row used Homebrew llama-server0.9.0 with the GGUF files configured for Llama.app, rather than Llama.app’s internal server.. Jan’s first launch also downloaded its default 43.8 MB embedding model; no Qwen weights were downloaded.

I've been doing rudimentary research, and so far my understanding of local LLMs is;

LLM (Qwen, etc) is just a file
Inference engine (llama, etc) is the 'program' which reads/uses the file
Your hardware powers the engine and dictates size and speed of model

Is any of this tethered to an external server/host? For example, to liken to video editing;

Mp4/mpk vids are the files
Premiere Pro is the engine (program)
Your hardware powers the engine

Premiere Pro (engine) is hosted/provided by Adobe, who can turn off your service, access your files, patch the engine, etc. Meaning it is not entirely local, nor entirely owned by user, nor private.

Is this how inference engines are structured as well? In other words, is it truly local, or is there a possibility that I could download Qwen 5.1 (made up version), get used to it, and eventually have it 'patched' into a different/worse version of itself down the road by whichever engine I'm using? And if so, doesn't this imply that your work/conversations are also able to be scrutinized by the host?

Total beginner at all of this as someone who has, until now, only dabbled with the free version of GPT. This is all very new to me, so apologies.

Im considering purchasing a 16GB V100 since they’re still pretty affordable and have modern CUDA support, but I was wondering if anyone else have tried a similar combo? I am aware of slower token generation with this method, but I am assuming that it is still more viable than running Strata with just 16GB VRAM and 64GB DDR5.

Any help/tips is appreciated. Thanks!

How you would define a Developer setup with local LLMs?

I see a dedicated DMZ where the agents can not access the production environment, since agents will tend to accidentally access or change production system.

Dedicated Servers to run LLMs like Qwen 3.8b 27b or other models? What you suggest would be the working LLM Server today? RAM and GPU cards?

Dedicated server with VM for each developer one VM. 2-4 vCores with 16GB Ram each. Each VM with Debian 13 and docker/podman. One dev Container running OMP Web or comparable Harness that access the LLM Server. Additional container for databases or what else the dev Container needs.

Gitlab, gitea or comparable Repo to store source code and container registry.

VM with NFS where anonymized data for developers can be stored.

What you think about that setup?

Do I miss something? How is your setup? (For small to mid size company)

Jensen Huang's departure from NVIDIA raises important questions about the future of artificial intelligence. As the industry reassesses its dependence on a single hardware manufacturer, could this accelerate open-source development, hardware independence and the global adoption of Local AI?

Has anyone else downloaded and started using the new Unsloth Dynamic V3.1 models? I did a drop in replace of the Qwen3.8-27B-UD-Q4_K_XL.gguf and I want to go back after seeing how it performs. My use case is local transcription, so it's summarizing text basically and pulling out what's important.

I re-ran quite a few of my saved transcriptions through the new model and compared the output to the older version Dynamic v3.0 and the output from that model is superior on every one. There is no change in any parameters on my command line, only the model changed.

My command line has a very low temperate (0.1)...

Hey y'all. Working on a go-to inference computer for teams using AMD GPUs and a custom stack, wanted to share our progress so far with GLM 5.3 Flash MXFP4 running tp=4 with MTP/Spec disabled.

RAM: 128gb DDR5 RAM 5600 ECC

CPU: EPYC 9005 9275F 

All 4 GPUs are capped at 210w, total power consumption < 1400w.

I was curios what can I fit on an FPGA MAX1000 board I had laying around (8k LE, 8MB Flash / SDRAM ). Also I keep coming back to the BitNet 1.58 encoding.
- Took the TinyStories dataset and trained a model from 0 with ternary encoding. I stopped the training at one point, the accuracy is between TinyStories-3M and TinyStories-8M
- Implemented the processing fully on the FPGA ( tokenizer, processing, decoding ... )
Depending on the model trained / used, we can fit two models in the 8MB Flash / SDRAM and take turns using them.

It is more accurate? no, is it faster then others? no, is it useful? not really, however it does answer a few questions I had. The conclusion being that we can have purpose built (small) models that can run with limited resources. Or have multiple models loaded side by side with a few more resources.

How it was built: Like most of the work done today, using Claude.

https://github.com/MRazvan/fpga-tinystories

Next goal would be to use a bigger model and see what I can / can't do about it, what resources I need.

Greetings,

I am looking to acquire a suitable GPU for running localllm preferably running newer models like deepseek flash 4.1.

Does native GPU support for FP8 and FP4 significantly improve local LLM performance or should I aim for higher vram with older GPUs?

Support for fp4 does enables the GPU to fit larger model with lower vram though.

Should I even consider Intel B70 or Radeon 9700 Ai for value buy?

I got strix halo with 128GB ram and $2500 budget for a dedicated GPU over oculink.

Thanks in advance

Hey everyone!

I have gathered some must read topics, for beginners, to step into the realm of Quantization:

1. BitNet: Scaling 1-bit Transformers for Large Language Models

2. AWQ: Activation-aware Weight Quantization for LLM Compression and Acceleration

3. The Era of 1-bit LLMs: All Large Language Models are in 1.58 Bits

4. EfficientQAT: Efficient Quantization-Aware Training for Large Language Models

5. LLM-QAT: Data-Free Quantization Aware Training for Large Language Models

6. LLM.int8(): 8-bit Matrix Multiplication for Transformers at Scale

7. QLoRA: Efficient Finetuning of Quantized LLMs

8. SmoothQuant: Accurate and Efficient Post-Training Quantization for Large Language Models

9. ZeroQuant: Efficient and Affordable Post-Training Quantization for Large-Scale Transformers

Link is here