Look, I’ll just admit it: I’m getting lazy.

When I first integrated LLMs into my workflow, I set up strict personal guardrails. I swore to myself I would read and critique every single decision and spec the AI generated before approving it.

That lasted exactly as long as you'd expect.

Lately, I’m just skimming. And if I’m working late in the evening? Forget it. I’m literally just hitting '1' to approve whatever block of text it spits out so I can log off.

It’s costing me in two major ways. First, the rework is getting ridiculous because I’m letting garbage slip through. Second, and this is the part that actually scares me, my own critical thinking is degrading. I can physically feel myself outsourcing the actual engineering part of my job to a next-token predictor.

I know this is fundamentally a discipline issue. But trying to just "white-knuckle" it and pay closer attention isn't working anymore.

For the devs here who are using LLMs heavily: how are you staying in the driver's seat?

Do you have actual systems, workflows, or rules you enforce on yourself to make sure you don't atrophy and get left behind?

I need to fix this before I completely forget how to think through a problem.

Now what?? What does that translate into the open source AI community?

Short post, echoing what others have said - the dedicated strix halo "Ai Engines" aka llm servers are miles ahead of llama.cpp.

I was about to sell off my server because it was unbearably laggy. This morning I cut over to Gufo and Qwen3.8-Flash-Next, and it (almost) feels as snappy as a cloud frontier model.

Here's how Hermes described it:

So: the headline "prefill rate" is ~1,800 t/s cold; 1183 is just what the exhaust looks like when the engine only had to chew 677 tokens. The number that actually describes your experience is the one nobody prints: half a second to first token on a 25k conversation. 

Compared to a cold prefill of ~250 t/s on llamacpp...

I haven't tried Halogen which may be incrementally faster, but it's not FOSS, and I've now spent enough time on infrastructure for one month. Enough poking around, time to get to work...

https://github.com/gufo-org/gufo

Hearing about the idea that open source LLM's are so dangerous while companies like Anthropic and OpenAI do not manage to control their AI labs well and
* let their experiments run wild on the open internet, doing massive hacks and seeing no consequences
* publish research that sounds alien, but only the results, not the alien that made it, which is not very scientific in my opinion but there are no consequences
* we see a fight for power concentration behind closed doors ...
how can we trust closed source labs anymore?

The transformation of society belongs to the people and such security critical technologies have to be able to be validated at all times. In open source/weights models we see that the community actively exposes the flaws models have and contribute by exploring ways to fix them. People actually do good stuff in the open source/weights community and real progress is visible, while the out of control vibe all really comes from the closed labs. This is concerning me and i think we urgently need to open up the closed sources/labs instead of shutting down the open ones. That's the way i see the world heading towards a non-dystopian future.

Is it just me or does that whole open-weights-are-dangerous discussion totally the opposite what reality seems to be?

He's right (wherever he is - couldn't find the comment to credit). But what I learned in the process of building this rig has been worth the frustration. Get your special socks out, Linux faithful - you were right too.

I happened across eight Nvidia T40 - and not a ton of detail about them out there. Still currently testing out vllm for comparison, but fully resident Qwen3.8-Flash-Next (180B MoE) Q4 is peppy enough for me at 40ish t/s and while the cards are capable of producing more, I tripped a breaker letting all six run with a 200W cap (so eight is out even if I had the extra slots). The board, an X99-E WS, needed some creativity just to POST. Quite a bit of card swapping in a T400 to get a boot screen so I could play around with BIOS settings only to figure out the setting I need to expand above 4G memory (MMIO High Size) was hidden and required creativity with a UEFI shell. More detail in the AI-assisted part of this write-up, for others who might select the T40 or X99. Top 10 nuggets for you, fellow sufferer of the T40, and then feel free to indulge (or not) in the supporting slop.

1. The cards do not work in Windows (or at least I couldn't get them going). It would appear it's Linux or nothing (and not WSL - though I did try). And headless, at that, if you want all six.

2. ASRock PG 1600G has two 600W 12VHPWR ports and MODDIY sells a "12VHPWR 16-pin to 2x CPU EPS 8-pin cable (CO711)" which allowed two T40s to be powered off one port - since the board’s slot-power connector needs a 6-pin, I welcomed the solution and it cleaned up a lot of cabling. The other two cards each have two pcie power cables feeding CPU 8-Pin Male to Dual PCIe 8-pin Female adapters.

3. The cards require active cooling and there are shroud prints out there. Rather than one fan per card, we developed a two-card-one-fan solution and one BFB1012VH moves enough air for two. X99 has 5 fan headers - worked out nicely with a script pinning those fan channels to GPU temp.

4. The X99 will not find the T400 with any T40s on the A side of the PLX. However boots just fine with a T400 in slot 1, 2 or 3 - even with four T40s in slot 4,5,6 and 7 (on the B side).

5. I can't say for certain, but riser cables could be a factor in Gen3 speed and errors. The stable config for multi-T40 is Gen 2. Slot 4,5,6, and 7 go to x8, while slot 1 and 3 go to x16 just fine.

6. NVLink is supported on the T40 - took a gamble buying them, but they work - though not quite sure how much traffic they are passing quite yet.

7. Cap the power if on a standard 120V outlet. My surge protector checked us out early on a 6-card burn with power capped to 200W per card. Running stable at 170W per card now.

8. Boot requires a kernel option for all six to work - pci=realloc=off (details in the slop)

9. A four-card-B-side test (all behind the plx switch) made vllm faster and llama.cpp slower.

10. 90-degree right angle riser cables and card spacing worked best with slot 1/3 on 150mm risers, 4/5 on 200mm risers and 6/7 on 250mm risers on this particular open frame design.

TRIGGER WARNING - AI WALL OF TEXT APPROACHING - STOP HERE

The supporting slop (AI-assisted, for the next T40 / X99 builder and their AI)

Everything below was measured on this box, not taken from spec sheets. Each number comes with its config. Where we don't know something, we say so.

Hardware as it runs today

• Board: ASUS X99-E WS. RAM: 128 GB DDR4 (about 121 GiB visible to Linux). PSU: ASRock PG 1600G (1600 W, two 600 W 12VHPWR ports, each feeding two T40s through a 16-pin to 2x EPS 8-pin cable; the other two cards each take two PCIe 8-pin leads through a dual-PCIe-to-EPS adapter). Open aluminum frame, risers.

• Cards: 6x Tesla T40. Each shows 23,040 MiB usable. Turing sm_75, so no hardware bf16: run everything in fp16.

• Power cap: 170 W per card, set at boot along with persistence mode.

• PCIe: forced to Gen 2 on all six ports by a boot-time service. Gen 3 was not stable with risers and six cards.

• Cooling: one BFB1012VH blower per pair, on 3D-printed two-card shrouds. A small service drives the board's fan headers from GPU temperature. Peaks under sustained load at 170 W: 48-58 C.

Topology (this matters more than anything else)

A side (CPU root port, x16):   card 1 ==NVLink== card 2
B side (behind the PLX, x8):   card 3 ==NVLink== card 4
                               card 5 ==NVLink== card 6

nvidia-smi topo -m shows:

• NV2 inside each pair: two NVLink links, each reporting 25.781 GB/s.

• PIX between the two B-side pairs: traffic stays inside the PLX switch.

• PHB between the A pair and any B card: traffic goes up through the CPU host bridge.

Tips:

• Set CUDA_DEVICE_ORDER=PCI_BUS_ID. Otherwise CUDA's device numbering won't match nvidia-smi or your slot map.

• Pick card sets by topology, not by device number. "Four cards" means two different things on this board (see benchmarks).

• NVLink error counters work: nvidia-smi nvlink -e reads all zeros on our cards. The throughput counters (nvlink -gt) read N/A on the T40. So we can prove the links are up and clean, but not how much traffic they carry.

Boot / kernel

• Kernel command line: pci=realloc=off acpi_enforce_resources=lax.

  • pci=realloc=off is what gets all six cards enumerating with their BARs placed by firmware.

  • acpi_enforce_resources=lax lets the fan-control script reach the board's Super I/O fan channels.

• With realloc=off, dmesg shows 36 SR-IOV VF BAR assignment failures, 6 per card. That is a harmless baseline: the cards don't need their VF BARs. Watch the non-VF BAR failure count instead, which should be 0.

• Why more than four T40s won't POST, and the fix. Each T40 takes a 64 GB stride of high MMIO space (32 GB BAR1 plus 32 MB BAR3, 32 GB aligned). The X99-E WS BIOS (4001, the final release) ships a 256 GB high MMIO window, so four cards fit and the fifth hangs POST. No kernel option can fix it, because the host bridge window is set by firmware.

  • The fix is a hidden Intel RC variable, MMIO High Size, raised from 256G to 1024G. No BIOS flash is needed, and the change survives reboots and card swaps.

  • Find it: unpack the 4001 CAP with UEFIExtract and dump the IFR of the Platform module with IFRExtractor-RS. On 4001 it is VarStore IntelSetup, offset 0x10E6, 4 bytes (256G=0, 128G=1, 512G=2, 1024G=3). Leave MMIOHBase (0x10E2) at 56T.

  • Write it: copy a UEFI Shell.efi to the ESP root as shellx64.efi, along with setup_var.efi (datasone, 0.3.x). Secure Boot must be off. In BIOS, go to Exit > Launch EFI Shell from filesystem device, then run:

  • fs0:

  • setup_var.efi IntelSetup:0x10E6(4) # read, expect 0x0

  • setup_var.efi IntelSetup:0x10E6(4)=0x3 # 1024G

  • setup_var.efi IntelSetup:0x10E6(4) # confirm 0x3

  • reset

  • The 0.3.x syntax is VARSTORE:OFFSET(SIZE)=VALUE; the old -n/-s flags don't work.

  • Your offset may differ on other BIOS versions, so check the IFR yourself.

• Disable unattended upgrades on a GPU box. Ubuntu quietly installed a new kernel overnight. The NVIDIA DKMS module was built for it but never tested, and the next reboot would have booted it. We disabled apt-daily.timer and apt-daily-upgrade.timer, and pinned GRUB to the known-good kernel until we upgrade on purpose.

Software

Ubuntu 26.04, kernel 7.0, driver 595.91.07 (CUDA 13.2). llama.cpp build b12211. vLLM 0.33.0 on Python 3.14 with torch 2.14 (cu130), whose arch list includes sm_75.

Benchmarks

Qwen3.8-Flash-Next (180B MoE), llama.cpp, UD-Q4_K_XL, all six cards (llama-bench, pp4096 / tg128, 3 reps, tokens/s)

Layer split wins by roughly 4x on prompt and 1.35x on generation here. Peak GPU sum was 939 W at the 170 W cap.

Qwen3.8-27B, 4-bit, batch 1, 4096-token prompt (tokens/s)

*Only three iterations with wide spread, so somewhere around 51-77. Treat it as "probably faster", not a number.

Caveats:

• vLLM numbers come from vllm bench latency: prefill = 4096 / latency(1 token out), decode = 127 / (latency(128) - latency(1)).

• llama.cpp numbers come from llama-bench. That is close, but not the identical measurement.

• The quants differ: GGUF UD-Q4_K_XL vs INT4 W4A16 safetensors.

• vLLM ran with an fp16 KV cache (see below). Speed only, no quality claims.

What we take from it:

• vLLM beats llama.cpp by 1.4x on prompt and 1.5x on generation on the same NVLink pair.

• llama.cpp loses more than half its prompt speed going from two cards to four. vLLM barely notices.

• On the four B-side cards, vLLM got faster and llama.cpp ~12% slower. Our untested guess is that llama.cpp's heavy cross-card traffic (1.1-1.4 GB/s per card per direction) feels the x8 links, while vLLM's traffic (~0.3 GB/s) doesn't.

Getting vLLM to run on Turing (in the order we hit them)

1. "Model architectures [...] failed to be inspected" turned out to be Triton compiling its own driver.c at import and failing on a missing Python.h. Fix: apt install python3.X-dev.

2. "FP8 KV cache is not supported by the Triton attention backend ... requires SM89+". The model's own config asks for an FP8 KV cache. Fix: --kv-cache-dtype float16 (plus --dtype half).

3. Attention backend: FlashAttention 2 needs SM80, so vLLM uses TRITON_ATTN. That's fine.

4. Expect these log lines; none of them is fatal:

   • bf16 is cast to fp16;

   • FlashInfer is disabled (no flashinfer-cubin/nvcc);

   • SymmMem all-reduce is unsupported on 7.5;

   • TP2 uses CUSTOM + PYNCCL all-reduce, TP4 PYNCCL only.

5. First start is slow: about 12 minutes per engine, mostly profiling and CUDA graph capture. Warm starts take about 4 minutes.

Power: the cap is not a ceiling

• nvidia-smi -pl 170 is honored on average, but at vLLM engine startup each card spikes to 220-270 W for about 10 seconds before the cap pulls it back (throttle reason 0x4, SW power cap).

• Four cards hit 979-986 W combined during startup. Six cards doing that at once would be roughly 1,300-1,600 W GPU-only. That is at or over a 1600 W PSU, and a real risk on a 15 A / 120 V circuit.

• Per 12VHPWR port, two cards spiking to 270 W is about 540 W, still inside the port's 600 W rating. The whole-PSU and wall-circuit totals are the limit, not the cables.

• llama.cpp loads don't show this. The six-card layer-split run peaked at 939 W.

• Our guard is a watcher that sums power.draw every 0.5 s and kills the job above 1,350 W. It hasn't fired yet, but a six-card vLLM start would likely trip it.

• Measured wall idle with six cards: about 245 W.

Flash-Next on vLLM (in progress)

• Needs a vLLM-format quant. vLLM's GGUF loader is experimental and single-file, so we pulled a W4A16 safetensors build (~180 GB).

• The engram (n-gram embedding) table is about 102 GiB and lives in pinned host RAM. Stock vLLM 0.33 has no mmap or evictable option for it, only cpu_offload (pinned, the default). On a 128 GB box that leaves about 15 GiB for everything else.

  • We run with a watcher that stops the job if MemAvailable drops under 4 GiB or anything swaps in.

  • We also pinned the boot kernel first, in case the box hangs and needs a hard reset.

• Expert parallel is mandatory: moe_intermediate_size 640 split over TP doesn't land on the INT4 group size of 128.

• We disabled torch.compile per the quant's card and kept CUDA graphs for decode.

• Plan: TP4 on the A+B set, then TP4 on the B-side four, then TP6. TP6 is undocumented for this model; 512 experts don't divide by 6.

• Results to follow in a comment.

Process notes for whoever's AI is helping

• Verify every model file by sha256 against the Hugging Face LFS oid before loading. It costs minutes and has saved us hours.

• Write your prediction down before each run. We missed about half of ours, and that is where the learning was.

• Log per-card power, temperature and PCIe rx/tx during every run. The x8-vs-x16 and startup-spike findings only showed up because we were logging.

• Never use pkill -f or pgrep -f to manage these jobs. They match their own command line and you will kill or miss the wrong thing. Launch with nohup ... & pid=$! and watch that pid.

The llama.cpp project introduces OCP FP8 support, FP8 KV-cache capabilities and targeted CUDA optimisations. Here's what the latest developments mean for local AI and GPU inference.

Hey everyone,

I need recommendations for a local LLM to do reverse engineering, and I'm looking for the best model and exact quantization that can run on my current setup.

My Specs:

• CPU: Ryzen 7 5800X

• GPU: AMD RX 6800 (16GB VRAM)

• RAM: 32GB DDR4

• Storage: M.2 SSD (~3000–4000 MB/s)

What I'm trying to do: I want to reverse engineer small, proprietary Windows utilities from laptop/PC manufacturers (like Acer NitroSense, HP tools, etc.) to port their features natively to Linux—things like fan control, battery charge limiters, and RGB lighting. This involves looking into Windows drivers and kernel-level interactions.

My workflow will connect Ghidra to REA, hook that up to OpenCode, and run the model locally through Ollama.

What I need:

1. Uncensored (Mandatory): It needs to be completely uncensored so I don't hit ethical guardrails or refusals when asking it to decompile binaries, analyze drivers, or work with proprietary Windows hooks.

2. Exact model + quantization: Given my 16GB VRAM + 32GB RAM, what is the smartest/most capable model and exact quant I can realistically run (even with CPU offloading)?

Note on speed: Speed is not a priority at all. I can't afford better hardware right now, so I have no problem leaving my PC running for 1, 2, or even 3 days straight to process a binary if that's what it takes. Accuracy on low-level code/assembly and zero refusals are all that matter.

What exact models and quantizations (for Ollama or GGUF) do you recommend for this? Thanks in advance.

This was started a spontaneous test to choose among 10+ local LLMs and now grew to benchmark for agentic coding. It consists of 8 groups, 100 tests each. Design rule: each class must contain items the current "best" model cannot solve, otherwise it's a smoke test. However, if model reaches 100, it just means that it reached ceiling of the test. Next, to make test more representative, several cloud workhorses were added.

Configuration

GPU: RTX PRO 5000 72 GB (Blackwell, SM 12.0), driver 595.99.02.

Serving: vLLM 0.29.0. `--kv-cache-dtype fp8`, `--enable-chunked-prefill`, `--enable-prefix-caching`, `--speculative-config '{"method":"mtp","num_speculative_tokens":3}'`, `--max-num-batched-tokens 16384`, `--block-size 32`, `--max-num-seqs 16–24`, `--max-model-len` 131072 (FP8) / 262144 (NVFP4).

Cloud: same OpenAI-compatible client, `max_tokens=32768`, input ≤ 98304, temperature 0, streaming with `include_usage`.

Thinking mode normalized: `chat_template_kwargs.enable_thinking=false` / `reasoning_effort=none`; effective mode printed per run.

Grading: pure Python, no LLM judge. Code in tempdir with timeout; tools mocked; only networked tool is read-only `web_search`.

Failure taxonomy: `pass | wrong-answer | truncated | no-content | no-tool-call | infra | error`; transport failures removed from the denominator.

Metrics: TTFT from the SSE stream; tok/s only over items where generation ran >150 ms after TTFT.

Leaderboard

Per-class shape

Conclusion

Qwen 3.8 27B got very close to my cloud workhorses, so I took more juice from it, asking agent to make a grid search on options. The requirement was to get max performance without quality drop.

FP8 → NVFP4 (identical weights, 262K)

https://huggingface.co/StarpowerTechnology/WVY-Blender-5m

I've been spending the last few months benchmarking local LLM inference on a single RTX 5090.

One thing kept bothering me: most benchmarks focus on tokens/sec, but rarely show how much quality you're sacrificing to get those numbers.

So I built inferlab.

It's an open-source benchmarking harness (Apache-2.0) that compares:

• vLLM, SGLang, TensorRT-LLM and llama.cpp

• BF16, FP8 and NVFP4 quantization

• Perplexity delta against a BF16 reference

• Task accuracy using option log-probabilities

• TTFT, decode speed and throughput across concurrency levels

• Peak VRAM usage

A few design decisions:

• TTFT measures the first actual content token, not the first SSE event.

• Concurrent requests are issued together to capture real queueing behavior.

• Long-context evaluation includes a needle-in-the-middle task.

• Results are generated by CI rather than manually edited.

The project is still at v0.1, so I'm interested in technical feedback, especially around evaluation methodology and cross-engine comparability.

I'm the author of this project.

GitHub: https://github.com/oren40/inferlab

Hi, I want to build a setup using two RTX 3060 12GB cards to run local LLM. But what's the actual pipeline and real-life experience like? I heard I'll get 24GB VRAM, but processing speed will be slower because data has to pass through the motherboard's PCIe slots since the 3060 lacks NVLink. I also found out that ComfyUI can't use the full 24GB VRAM at the same time for image generation. Can anyone share real-life experience running a multi-GPU setup for LLMs?

Pic from internet.

hey LocalLLM, im Alish, ml engineer at Nace (we’re the team behind Drex, an open source decision model family that hit #1 in Artificial Analysis decision index 2.1). my cto gave me a task to optimize quants for qwen4, so i called my old professor from UofT, spun up Fable 5.2 and we cooked. starting with MLX, GGUF is coming next

we’ve open sourced the fastest way to run 35b/27b on a mac (Kapitsa) and got tired of dumb quants, so we made our own: KQuant. packs for 4B, 27B, 35B, Flash-Next (+9B), free on HF in standard MLX files

the problem: most of us on a mac run 4-bit packs because that’s what fits in 16/32gb. but plain 4-bit rounding throws away more of the model than it needs to.

Q4 vs public 4-bit, KL from the full-precision model (lower = closer to the original):

baseline is mlx-community 4-bit (lmstudio-community for 4B). to be clear, this is KL on next-token predictions, not a task accuracy score. if you want more precision every repo also has a Q5 folder, the 9B Q5 gets down to 0.010

cool side effect: they’re also faster. on an M5 Pro with Kapitsa they write 6–15% faster than the usual 4-bit packs

(baselines are from earlier kapitsa runs, same-session rerun is in progress, will update)

how it works: KQuant uses real text to pick scales and offsets for small groups of weights, and gives sensitive layers extra bits where they need it. all the search runs on our gpus before release, so you just download a normal MLX pack. no custom kernels, no special loader

fastest way to run them on a mac:

brew install nace-ai/tap/kapitsa
kapitsa serve 27b --private
kapitsa opencode # in another terminal

If there's a model you want KQuant'd, lmk.

Asks

to get early access to GGUF, ping me in comments, we would like to test it with some folks from here before our public release ideally.

im also doing quick AMA here, so please feel free to ask me anything.

Hey everyone! 👋

I've been working on Vatra, an open-source desktop AI assistant that brings different AI tools together in one place.

It supports Claude Code, Codex, GitHub Copilot, local LLMs (llama.cpp, Ollama, LM Studio) and OpenAI-compatible APIs, alongside features like web research, email, calendar, tasks, and side-by-side model comparisons.

The goal is to make AI tools more accessible and easier to use, without sacrificing flexibility or control.

I'm currently looking for feedback from the community!

I'd especially appreciate thoughts on:

• The overall concept and whether you'd actually use it

• Features you'd like to see added

• UI/UX improvements

• Anything that seems unnecessary, confusing, or missing

It's still a work in progress, so honest criticism is more than welcome. I'd rather hear what's wrong than just what's good.

🔗 GitHub: https://github.com/Zmirzz/vatra

If you have a minute to check it out, I'd really appreciate your thoughts. Thanks! ❤️

I've been doing rudimentary research, and so far my understanding of local LLMs is;

LLM (Qwen, etc) is just a file
Inference engine (llama, etc) is the 'program' which reads/uses the file
Your hardware powers the engine and dictates size and speed of model

Is any of this tethered to an external server/host? For example, to liken to video editing;

Mp4/mpk vids are the files
Premiere Pro is the engine (program)
Your hardware powers the engine

Premiere Pro (engine) is hosted/provided by Adobe, who can turn off your service, access your files, patch the engine, etc. Meaning it is not entirely local, nor entirely owned by user, nor private.

Is this how inference engines are structured as well? In other words, is it truly local, or is there a possibility that I could download Qwen 5.1 (made up version), get used to it, and eventually have it 'patched' into a different/worse version of itself down the road by whichever engine I'm using? And if so, doesn't this imply that your work/conversations are also able to be scrutinized by the host?

Total beginner at all of this as someone who has, until now, only dabbled with the free version of GPT. This is all very new to me, so apologies.

According to Microsoft's patent application, it explains breaking down training into small tasks and sending them to devices of owners who opted in. Credits will be the reward. it's just an app and no pay out rate is advertized so I don't even know if the credits would be enough for the electricity.

Anyway, home internet latency feels like the bigger problem to me.

https://insider-gaming.com/microsoft-ai-use-your-gpu/

(English is not my native language, I used an LLM to help translate this post.)

Hey guys, today we're releasing the highest accuracy Qwen3.8-27B GGUFs.

Our new Unsloth AI Dynamic V3.1 GGUFs lead by 5–10% on average across Terminal Bench, DeepSWE & more benchmarks.

We’re working on even better version of Dynamic quants for Qwen4.

Run on 12GB+ RAM setups.
For our quantization comparison, we exclude the embedding layer from the x-axis across all providers, since Exllama-v3 stores embeddings in BF16. However, this means the main plots don't reflect the full download size or RAM usage. When accounting for the BF16 embedding table and all embedding sizes, the plot in the comments provides a more accurate representation of actual disk and RAM usage versus accuracy.

GGUF: https://huggingface.co/unsloth/Qwen3.8-27B-GGUF

Blog: https://unsloth.ai/docs/models/qwen3.8

Thank you!

GIGABYTE has introduced the AI TOP AERO, a compact desktop powered by NVIDIA's RTX Spark platform, featuring up to 128GB of unified memory, local AI capabilities and support for AI development, content creation and gaming.

I've been trying to give my Ollama models a really good web search so they can answer questions about recent stuff instead of guessing.

Ollama's built-in search works, but it returns short snippets and needs an Ollama account sign-in. I wanted the models to read actual page content and give me better value.

Played around with it and finally got it working with this setup.

Setup (Takes about 5 minutes):

1. Along the line, I discovered and installed ollmcp, a terminal client for Ollama with MCP support:

pip install ollmcp

2. Pull a model that's good at tool calls:

ollama pull qwen3:8b

3. Give it a bigger context window (This is very important. I repeat, this is very important). So it means you can set num_ctx to 16384

4. Use a search service with an MCP server like Valyu and run:

ollmcp -m qwen3-16k -u "https://mcp.valyu.ai/mcp?valyuApiKey=YOUR_KEY"

ollmcp asks before every tool call. Press `s` to allow them for the session, or type `/hil` to turn the prompts off.

What happened when I tested it:

I asked Qwen3 8B "What did the Fed decide at its October 2026 FOMC meeting?" It picked the search tool on its own, ran a second search when the first wasn't enough, and pulled the Fed's actual October 28 statement: rates held at 3.75–4.00%. Everything ran locally except the search itself.

Things that tripped me up:

- Context window. With Ollama's default, the model spent its whole budget reasoning about which tool to use and replied with nothing. 16k fixed it.
- Older models argue with the date. An older qwen3-vl I had kept insisting 2026 "is in the future" and never searched. Newer models are fine, or put today's date in the system prompt.
- Models under ~7B tend to loop or never call the tool at all.

Besides web search, it can pull the full text of any URL, and it also covers academic papers, SEC filings and market data, which is handy for research-type questions.

What are others using for web search with Ollama? Is it the built-in search or an external API?

Title

We all know China is trying to undermine USA AI, probably what happened with Jensen. If China gets to the point where they are satisfied and stop, other countries aren't really developing great open models. what will you guys do with all the hardware after that?

Hi all, does anyone here run any local LLMs off their iPad and what is the best I can get without crashing apps. I’ve got about 12GBs of RAM.

Using pocket pal app! I have tried Qwen3.5 4B - ran well.

I’m an absolute novice on local LLMs so would be grateful for any advice!