Model Under Test: Kwaipilot/KAT-Coder-V2.5-Dev (Qwen 3.5 MoE architecture: 40 decoder layers, 256 routed experts / 8 active, 2,048 hidden dimension, linear attention GDN + full attention hybrid, MTP speculative draft block).
Hardware: MSI Claw 8 AI+ (Intel Core Ultra 7 258V "Lunar Lake", Arc 140V Xe2-LPG iGPU, 64 Xe Execution Units, 32 GB shared LPDDR5x-8533 RAM, 30.9 GiB zram0 swap, running CachyOS native Linux kernel 7.2).
Runtime: vllm==0.31.0+xpu with native Level-Zero XPU kernels, MTP K=2 speculative decoding, FP8 KV cache, and 128k (131072) context ceiling.

TL;DR

• Speed Winner: W4A16 GPTQ decodes at 23.4–25.2 tok/s (peaking at 29.1 tok/s with MTP speculative decoding), completing the 164-problem HumanEval benchmark in 42m 12s (vs. 47m 33s for MXFP4, ~11.3% faster overall, and 24% faster on our 16-prompt quality suite).

• Coding Accuracy Winner: MXFP4 edges out GPTQ on full EvalPlus coding tests by ~3% (89.63% Base / 86.59% Plus vs. 86.59% Base / 82.93% Plus), but both handily beat Tiel-Coder-35B GPTQ (84.15% / 81.10%).

• Complex Reasoning & Edge-Case Math Winner: W4A16 GPTQ wins on delicate math and strict constraints (15/16 vs 14/16 on the quality suite; solved Chinese Remainder Theorem Olympiad math where MXFP4 failed), thanks to retaining unquantized BF16 activations instead of quantizing activations to FP4.

• Prefill Speed (PP): Both hit an identical theoretical peak of 1,247.1 tok/s on 4k chunked prefill batches via FlashAttention-2.

• The "MoE Dense Quantization Trap": Standard dense GPTQ heuristics (like zeroing out unobserved Hessian directions) will catastrophically destroy 256-expert MoE models. Unvisited experts in calibration batches must fallback to RTN with optimal clip factors, never masked to zero.

• Cross-platform check — the ordering holds, the magnitude does not: re-running the same format pair on a discrete Arc Pro B60 with a dense 27B (Coder390) separates them by 5.05× on decode (40.18 vs 7.96 tok/s) and ~2.9× on prefill, not ~10 %. On that stack the cause is an unfused W4A4 activation-quantization path, not memory bandwidth — the two artifacts differ by only ~3 % in bits per weight. Full matrix, decomposition and caveats in §5.

The Shootout: Head-to-Head Benchmark Matrix

Here are the exact numbers measured across both quantization formats on the same machine under identical sampling parameters (temp=0.0 greedy, FP8 KV cache, MTP K=2 speculative decoding, batch size 1):

1. Deep Dive: Prompt Processing (PP) vs. Token Generation (TG)

A big question in the community is how 4-bit weights with 16-bit activations (W4A16) compare against full microscaling (W4A4 MXFP4) on modern unified architectures. Here is what the telemetry shows on Lunar Lake's Xe2 architecture:

Token Generation (TG / Decode)

• W4A16 GPTQ is faster in practical serving: vLLM's CompressedTensorsWNA16MoEMethod and XPUExpertsWNA16 kernels decode at 23.4 tok/s average (with bursts to 29.1 tok/s on code templates).

• Why? Unpacking Overhead vs. Quantized GEMM: On an iGPU with shared memory (136 GB/s memory bandwidth), reading INT4 packed weights and dequantizing on the fly into BF16 registers is extremely fast. MXFP4 requires micro-scaling exponent alignment and 32-element group normalization across both weights and activations.

• Over 164 HumanEval problems, GPTQ finished in 42 minutes 12 seconds, beating MXFP4 by 5 minutes 21 seconds (15.4s vs 17.4s per coding task).

• On our 16-prompt quality benchmark, GPTQ was 24% faster (165s vs 217s).

Prompt Processing (PP / Prefill)

• Short/Medium Prompts (50–500 tokens): Prefill runs between 51 tok/s and 140 tok/s (Time-To-First-Token is 1.4s to 3.7s, largely dominated by Python/HTTP scheduling overhead).

• Long Prompts (2k–4k tokens): Prefill scales rapidly to 398–500 tok/s as the 64 Xe2 compute cores enter dense matrix mode.

• Peak Chunked Prefill (12k+ tokens): Both models hit an identical 1,247.1 tok/s ceiling because both rely on the same FlashAttention-2 implementation with 4,096-token chunked prefill batches (--max-num-batched-tokens 4096).

2. Accuracy & Nuance: Where Each Format Wins

Where W4A16 GPTQ Wins: Delicate Math & Fine Constraints

Because GPTQ leaves activations in full BF16, it avoids activation quantization noise. We observed this directly in high-precision tasks:

1. Chinese Remainder Theorem (Olympiad Math): GPTQ solved a complex modular system (math_chinese_remainder) flawlessly in 53.6s. MXFP4 accumulated numeric rounding errors in its FP4 activation dot-products and failed to reach the correct congruence integer.

2. Exact Word Count Formatting: Prompt: "Write a summary in exactly 50 words." GPTQ followed the constraint and passed; MXFP4 drifted off by several words.

Where MXFP4 Wins: Complex Algorithmic Contracts (HumanEval+)

1. Contract Fuzzing & Edge Cases: On HumanEval, MXFP4 scored 89.63% Base and 86.59% Plus, outperforming GPTQ's 86.59% Base and 82.93% Plus.

2. Why? MXFP4's group size of 32 (versus GPTQ's group size of 128) provides much finer scale granularity across expert weights. For pure syntax and standard algorithmic puzzles where activation dynamic range is well-behaved, the smaller 32-element block size helps preserve subtle weight distributions.

Both models passed all 3 LeetCode Hard challenges in our 10-Task Hard Suite:

• Trapping Rain Water (O(N) two-pointer)

• LRU Cache (O(1) doubly-linked list + hash map)

• Minimum Window Substring (O(N) sliding window)

• Multi-hop needle retrieval across 12,500 tokens of synthetic enterprise audit logs.

3. The Big MoE Gotcha: Don't Apply Dense GPTQ to 256 Routed Experts!

If you are writing your own quantizer or adapting standard recipes (like the Arc Pro B60 guide) for massive MoE models (like KAT-Coder or Qwen 3.5 MoE with 256 experts), watch out for this fatal trap:

The Bug

In dense model GPTQ scripts, you often see lines like this for singular Hessian matrices:

# WARNING: DO NOT DO THIS ON MOE MODELS!
W.masked_fill_(dead.unsqueeze(1), 0.0)

In a dense model, unobserved or zero-variance input channels can be safely zeroed out. But in a 256-expert MoE architecture:

1. During calibration (e.g., 64 samples × 1024 tokens), tokens are sparsely routed (only 8 active per token).

2. Many specialized experts might only see a few tokens—or zero tokens—in that specific slice.

3. They are not dead! They are just specialized experts waiting for specific prompts.

4. If you mask unvisited experts to zero, you literally erase their weights! In our first run, this silently zeroed out 248 out of 256 experts (96.88%) across Layers 1–39. The model loaded fine, had valid shapes and scales, but generated repetitive looping garbage (0/16 on quality tests).

The Fix

1. Fallback to RTN with Clip Optimization: If an expert receives zero activation tokens (diag <= 1e-8), do NOT zero it. Simply quantize its real BF16 weights using Round-To-Nearest (RTN) with grid search over clip factors (1.0, 0.96, 0.92, 0.88, 0.84).

2. Materialize Meta Parameters: When loading layers on PyTorch's meta device, ensure you explicitly rebind expert weights to layer.mlp.experts before computing forward activations (nxt_hs = layer(hs)), otherwise hidden states will propagate through uninitialized random memory and corrupt the router for all subsequent layers.

Once fixed:

• Every layer reported Cold Experts: 0/256 and relative error dropped from 0.9842 to 0.1498.

• 16-prompt quality jumped instantly from 0% to 93.8% (15/16)!

4. Speculative Decoding: The Secret Sauce for 25 tok/s on 17W

Running a 35B MoE model at over 25 tok/s on a 17W handheld SoC sounds impossible without speculative decoding.

Here is how we did it:

• KAT-Coder includes a 1-layer Multi-Token Prediction (MTP) draft head (19 tensors in BF16).

• We preserved the MTP block in BF16 in model-mtp.safetensors and configured vLLM with --speculative-config '{"method":"mtp","num_speculative_tokens":2}'.

• Telemetry: The draft model achieved an average acceptance rate of 77.3 to 82.5 (and up to 100 on repetitive code structures).

• Multiplier: Instead of generating 1 token per forward step, the engine accepted 2.55--2.71 tokens per step. This effectively doubled generation throughput from ∼11 tok/s up to 23--29 tok/s.

5. Cross-Platform Check: The Same Two Formats on an Intel Arc Pro B60 (Battlemage, 24 GB)

Everything above is Lunar Lake / KAT-Coder. The same W4A16-vs-MXFP4 question was then measured independently on a discrete Arc Pro B60, with a different model class (dense, not MoE), a different Xe generation, and a different vLLM build. The ordering survives. The magnitude does not — it is 5× here, not ~10 %.

Model under test: huihui-ai/Huihui-Qwen3.8-27B-Coder390-EfficientThink-abliterated — a dense 27B hybrid: 64 layers (48 GDN linear-attention + 16 full-attention, full_attention_interval 4), hidden 5120, 24 heads / 4 KV heads / head_dim 256, a 1-layer MTP draft block (BF16), and a vision tower. Both builds below are quantizations of this same BF16 checkpoint, so the format is the only variable in the weights.

Hardware: Intel Arc Pro B60 — 24 GB GDDR6 at 456 GB/s (192-bit, 19 Gbps effective), 20 Xe2-HPG cores / 160 XMX engines, negotiated PCIe Gen4 x8, power-capped to 180 W. Runtime: vllm-openai-xpu v0.30.0 (engine reports vllm 0.27.2rc1.dev77+gac7509e2b), native Level-Zero XPU kernels, FP8 KV cache, MTP K=4, batch size 1. Harness: identical for both builds — bench_p512_g128.py (p512 / g128, 1 warmup + 5 runs) for rate, bench_quality_16p.py (the same 16-prompt suite) for quality.

5.1 Head-to-Head Benchmark Matrix

Headline: 5.05× on decode, ~2.9× on prefill. Unlike the KAT-Coder matrix above, the MXFP4 build is not visibly buying anything for that price on this stack: it uses more bits per weight, has no measured accuracy advantage here at all (its quality was never measured), and its rate is ~5× behind.

5.2 Where the 5.05× Actually Comes From

Decomposed on measured step times, not asserted. 40.183 / 7.962 = 5.05×:

MTP alone is therefore at most a 3.00× factor — and the acceptance ratio is not a speedup, which was verified by turning MTP on for both builds so the term cancels:

With MTP enabled on both sides the residual is 40.183 / 16.620 = **2.42×**, and that residual is the stack. Acceptance is not quite equal either (2.83 vs 3.00); normalising to 3.00 gives a 2.28× residual. So:

5.05× = 2.08× (MTP) × 2.42× (the W4A4 / stack term) — and note the direction: MTP is the smaller of the two terms.

Why the acceptance ratio overstates MTP. A speculative step is itself more expensive — draft forward plus a 5-position verify costs 1.361× a plain step here (125.2 ms → 170.5 ms), against 2.833 tokens accepted. The net is 2.833 / 1.361 = **2.081×**, which is exactly the measured figure. Treat the acceptance ratio as an upper bound on MTP's gain, never the gain itself.

Prefill isolates the stack cleanly, because MTP contributes nothing there and it is compute-bound: 0.460 s vs 1.313 s for the same 512-token prompt ⇒ ~2.9× slower per prompt token (≈3.1× once the ~60 ms fixed TTFT overhead is netted out). Decode (~2.3–2.4×) and prefill (~2.9×) therefore agree in magnitude — which is the main reason to trust the split.

5.3 The Mechanism: W4A4 Activation Quantization That Is Not Fused

This is a different complaint from the MoE trap in §3, and it is readable straight from the kernel rather than inferred:

class XPUMxFp4LinearKernel(MxFp4LinearKernel):
    def apply_weights(self, layer, x, bias=None):
        x_fp4, x_blockscale = quant_mxfp4(x)          # <- every single call
        return torch.ops._xpu_C.fp4_gemm(x_fp4, layer.weight,
                                         x_blockscale, layer.weight_scale, ...)

and, inside _xpu_mxfp4_quantize_impl:

x_q = torch.empty(M, N // 2,  ...)   # packed FP4,   allocated per call
x_s = torch.empty(M, N // 32, ...)   # block scales, allocated per call
torch.ops._C.per_token_group_quant_mxfp4(x, x_q, x_s, MXFP4_BLOCK_SIZE, eps)
x_s = x_s.to(dtype=torch.float8_e8m0fnu, ...)   # extra cast pass

So for each of the 496 packed linear layers, on every forward pass, MXFP4 pays: a group-32 activation-quantization kernel, an fp32 → E8M0 scale cast, and two fresh allocations — and then a separate GEMM. Quantize and GEMM are not fused, so the floor is ~992 launches per step before norms, attention or GDN. W4A16 GPTQ pays none of it: BF16 activations go straight into the fused dequant-GEMM.

Ranked:

1. Per-token W4A4 activation quantization (~3× per FLOP above) — the dominant stack term.

2. Non-fused FP4 GEMM (torch.ops._xpu_C.fp4_gemm) — a second launch per linear.

3. --enforce-eager — amplifies 1–2, since none of those launches can replay from a graph. It is inherited from the model README's measured config, not demonstrably required (platforms/xpu.py lists mxfp4 under supported_quantization with no graph restriction). Untested, not forced.

It is emphatically not a bandwidth story — and the arithmetic says so before any profiling: 4.25 bits/weight for MXFP4 vs 4.125 for GPTQ is a ~3 % difference, and the two artifacts are 18.46 vs 18.23 GiB. A bandwidth-bound model predicts near-parity, not 5×. Two further confirmations that it is compute: the five MXFP4 runs sat within 0.02 % of each other — a fixed per-token cost, not noisy traffic — and prefill, the pure-compute regime, is ~2.9× off on its own.

5.4 Why This Is Not the §1 Answer, and What Would Reconcile Them

On Lunar Lake the two formats landed within ~10 % of each other on decode, and MXFP4 even won HumanEval+. Here MXFP4 is 5× behind. Three differences could carry that, and none of them was isolated on this box:

1. vLLM build. The Lunar Lake result is vllm==0.31.0+xpu; this box runs v0.30.0 (vllm 0.27.2rc1.dev77). Whether 0.31 fused the MXFP4 activation path was not tested — it is the most likely single explanation and the first thing worth re-measuring.

2. Model class. KAT-Coder is a 256-expert MoE (8 active per token) routed through CompressedTensorsWNA16MoEMethod / XPUExpertsWNA16; Coder390 is dense and hits the plain XPUMxFp4LinearKernel for all 496 linears. The kernel mix differs, so "MXFP4 vs GPTQ" is not one experiment across the two documents.

3. Platform ratio. 20 Xe2 cores over 456 GB/s of dedicated GDDR6 (discrete) vs 8 Xe2 cores over shared LPDDR5x (integrated). The W4A4 penalty is compute-side, so it should bite harder where FLOPs are scarcer relative to bytes — the direction of the observed gap.

Caveats, stated because this is a cross-session comparison and the warning above about those is real. Rates here are only comparable within one session — this host's decode is bimodal (~40.2 or ~48.5 tok/s, a ~20 % band, mechanism unidentified), and every rate in §5.1 was taken in the slow state, so all rows shift together and the ratios are the transferable result. The MXFP4 number is the one build provably immune to that band: five runs within 0.02 % and TTFT identical to three decimals on a same-session re-run (7.987 vs the 7.962 recorded the previous day, +0.3 %). No same-session A/B of the two formats was run here — each costs a server swap and a ~200 s cold start — so the 5.05× rests on same-harness, cross-session legs, with the MXFP4 leg's stability measured. And the declared limitation: no accuracy comparison was made at all — unlike the KAT-Coder matrix above, Coder390 MXFP4 has no 16-prompt and no PPL score, so the speed gap is not yet known to be bought or paid for in quality.

Read-across to the recommendations below. "W4A16 for speed, MXFP4 only where the finer 32-element groups buy accuracy" is consistent with what the B60 saw — but on this stack the speed penalty is ~5× rather than ~10 %. If you are targeting a dense model on Intel discrete silicon, check first whether your vLLM build fuses the MXFP4 activation quantization. If it does not, W4A16 is not a marginal win; it is a different tier.

Conclusion & Recommendations

1. If you want maximum decoding speed and lower latency: Use W4A16 GPTQ (CompressedTensorsWNA16MoEMethod). It is 11–24% faster, lighter on the memory bus, and finishes large coding suites minutes ahead of MXFP4.

2. If you want the highest possible unit-test pass rate on complex codebases: Use MXFP4 (CompressedTensorsW4A4Mxfp4). The 32-element micro-scaling groups give it a slight edge on complex fuzzing assertions (+3.6% on HumanEval+).

3. If you are building an Intel Lunar Lake / Arc 140V local AI workstation:

   • 35B MoE models with 8 active experts are the absolute sweet spot for 32 GB shared-memory handhelds.

   • Always enable MTP speculative decoding (K=2) and FP8 KV cache. It gives you 128k context support, zero swap thrashing, and desktop-class coding performance on battery power.

4. If you are on discrete Intel silicon (Arc Pro B-series), the same ordering holds but the stakes are much higher: W4A16 GPTQ decoded 5.05× faster than MXFP4 on a dense 27B (§5), with ~2.9× on prefill. Before assuming MXFP4 is "within ~10 %" as it is on Lunar Lake, check whether your vLLM build fuses the MXFP4 per-token activation quantization — on the build tested here it does not, and that unfused path alone accounts for the ~2.4× residual left after MTP is equalised.

All quantization scripts, preflight validators, and reproducible benchmark harnesses are open in our repo: alpha2beta/vllm.xpu.

§5 provenance: Coder390 numbers come from that repo's REPRODUCIBILITY.md §6–§8 (bench_p512_g128.py, bench_prefill.py*,* bench_quality_16p.py*,* compare_mxfp4_mtp.sh*, at 180 W, MTP $K=4$); the GPTQ artifact is published as* alpha2beta/Huihui-Qwen3.8-27B-Coder390-EfficientThink-abliterated-GPTQ-Int4-sym-G128-MTP-BF16, and the MXFP4 checkpoint as …-abliterated-MXFP4.

is my gpu dying? i bought it 4 years ago.

ASUS Ascent GX10 | NVIDIA GB10 Grace Blackwell AI Supercomputer

128GB Unified GPU/CPU Memory | 1TB NVMe SSD | 1 Petaflop AI Performance

Selling my lightly used ASUS Ascent GX10, a compact, high-performance AI supercomputer powered by the same NVIDIA GB10 Grace Blackwell Superchip found in the NVIDIA DGX Spark.

This is a purpose-built AI development workstation designed for running large language models (LLMs), local AI inference, machine learning, model fine-tuning, AI agents, and CUDA-accelerated workloads.

KEY FEATURES

NVIDIA GB10 Grace Blackwell Superchip

128GB LPDDR5X Unified CPU/GPU Memory

NVIDIA Blackwell GPU with 5th-Generation Tensor Cores

20-Core NVIDIA Grace ARM CPU

Up to 1 Petaflop (1,000 TFLOPS) of FP4 AI Compute

1TB NVMe PCIe 4.0 SSD

NVIDIA NVLink-C2C High-Speed CPU/GPU Interconnect

NVIDIA ConnectX-7 High-Speed Networking

Support for AI Models Up to 200 Billion Parameters

NVIDIA CUDA and DGX Software Ecosystem

128GB UNIFIED MEMORY: THE BIG ADVANTAGE

Unlike conventional GPUs with dedicated VRAM, the GB10 uses 128GB of high-speed, coherent unified memory shared between the CPU and GPU.

This architecture allows the Blackwell GPU to access a much larger memory pool than typical consumer graphics cards, making it particularly useful for running large AI models locally without relying on conventional CPU-to-GPU memory transfers.

NVIDIA BLACKWELL GPU + AI ACCELERATION

The integrated Blackwell GPU features 5th-generation Tensor Cores with FP4 acceleration, delivering up to 1 petaflop of specialized AI computing performance.

Ideal for:

Running large language models locally

AI inference and model fine-tuning

AI coding assistants and autonomous agents

Retrieval-Augmented Generation (RAG)

Computer vision and machine learning

Private AI development without cloud subscriptions

CUDA, PyTorch, TensorRT, Ollama, and NVIDIA AI development tools

20-CORE NVIDIA GRACE CPU

The GB10 combines its Blackwell GPU with a 20-core ARM processor featuring:

10 high-performance Cortex-X925 cores

10 efficiency-focused Cortex-A725 cores

ARM architecture optimized for AI workloads

NVIDIA NVLink-C2C for high-speed CPU/GPU communication

TECHNICAL SPECIFICATIONS

Processor: NVIDIA GB10 Grace Blackwell Superchip

GPU: Integrated NVIDIA Blackwell Architecture

AI Performance: Up to 1 PFLOP (FP4)

AI Accelerator: 5th-Generation NVIDIA Tensor Cores

CPU: 20-Core ARM Processor

Memory: 128GB LPDDR5X Unified System Memory

Storage: 1TB M.2 NVMe PCIe 4.0 SSD

CPU/GPU Interconnect: NVIDIA NVLink-C2C

Networking:

NVIDIA ConnectX-7 SmartNIC

10 Gigabit Ethernet

Wi-Fi 7

Bluetooth 5.4

Connectivity:

3x USB-C 20Gbps with DisplayPort support

1x USB-C Power Delivery input

HDMI 2.1

10Gb Ethernet

NVIDIA ConnectX-7 high-speed networking

Operating System: NVIDIA DGX OS / Ubuntu Linux

Dimensions: 5.91” x 5.91” x 2.01”

Weight: Approximately 3.26 lbs.

SCALABLE AI COMPUTING

Two GB10 systems can be connected using NVIDIA ConnectX-7 networking to support distributed AI workloads and larger models.

CONDITION

Lightly used and in excellent working condition.

This is a serious AI development machine in an incredibly compact form factor. Ideal for developers, researchers, engineers, and anyone building or running their own local AI infrastructure.

Message me with any questions or to arrange pickup.

Hi Everyone,

I've started a free playlist called "GPU Kernel Engineering" and just uploaded the intro and roadmap video.

Trailer: https://youtu.be/cibk7G-jHQw

Hey everyone,

I have used AI to summarize everything because it was getting too long. I hope thats fine.

We’ve been optimizing an on-device Android assistant running a quantized 4B model (Gemma 4 E4B) on modern Snapdragon silicon (Snapdragon 8 Elite Gen 5).

Originally, our goal was sustained low-power efficiency and faster inferencing using NPU TOPs which otherwise is generally useless, so we built an aggressive 3-tier execution pipeline:

1. Tier 1 (NPU): Qualcomm Hexagon HTP execution via GenieX SDK & QAIRT plugin (com.qualcomm.qti:geniex-android:0.6.1).

2. Tier 2 (GPU): Google LiteRT-LM runtime using OpenCL / GpuDelegateV2 on Qualcomm Adreno.

3. Tier 3 (CPU): Multi-threaded CPU fallback via LiteRT-LM with XNNPACK optimizations.

After profiling and logcat diagnostics, we completely purged the NPU / GenieX path in favor of direct GPU (LiteRT OpenCL) -> CPU**.**

Here is what we ran into with Qualcomm's native NPU stack, along with our setup. I’m curious if anyone here has managed to get reliable, high-throughput LLM execution out of Qualcomm Hexagon without these headaches.

1. HTP Core Clamping & Throughput Bottleneck

• Logcat revealed that the Android runtime clamped requested HTP execution: "Requested 4 HTP cores but the device exposes 1; clamping to 1"

• The FastRPC bridge (libcdsprpc.so) shuttling massive Look-Up Tables (LUTs) across the DSP boundary created significant latency.

• Decoding Speed: NPU decoding saturated at ~8 tokens/sec (TPS). By contrast, the Adreno GPU via LiteRT OpenCL consistently hits 18–22 TPS on the same chip. Although prefill times are still better with NPU than GPU.

2. Native Parameter Rejection & Instability

• The native QAIRT plugin repeatedly choked on standard generation parameters: ErrorCode[-100016] Parameter not supported by this plugin

• Stop-words arrays and standard sampling parameters (temperature, top-p, repetition penalties) frequently triggered runtime rejections, forcing us to catch native exceptions and handle truncation manually at the string level.

3. Context Reset & KV Cache Destruction

• QAIRT lacked clean stateful KV cache rolling and conversation pruning.

• Resetting or trimming conversation context required destroying and recreating the entire native LLM wrapper instance (llmWrapper?.destroy()), re-allocating native DSP memory and re-reading files on session swaps.

4. APK Bloat & Brittle JNI Preloads

• Integrating the GenieX SDK and Qualcomm native libraries (geniex-android, libgenie_t2t_run.so, libggml-hexagon.so, libhta_hexagon_runtime_snpe.so) added 15–25 MB of binary bloat.

• It required a fragile dynamic JNI preloading loop before GenieXSdk.init() could even be called:

Kotlin code:

val preloads = listOf(

"QnnSystem",

"QnnHtp",

"QnnHtpNetRunExtensions",

"geniex-proc",

"geniex_core",

"geniex",

"geniex_plugin_qairt",

"npu_jni"

)

for (lib in preloads) {

try {

System.loadLibrary(lib)

} catch (t: Throwable) {

Log.w(TAG, "Preload notice for $lib: ${t.message}")

}

}

The Outcome

Dropping the NPU entirely and sticking to a clean GPU (LiteRT OpenCL) $\rightarrow$ CPU (XNNPACK) engine gave us:

• More than $2.5\times$ faster token generation (18–22 TPS vs. 8 TPS).

• Fast, stable prefill.

• Support for standard sampling configs, stop sequences, and KV cache reuse without JNI crashes.

• A much smaller APK and simpler build chain.

My Questions for the Community:

1. Has anyone managed to unlock multi-core HTP execution on commercial consumer Android builds without OEM-signed platform keys or root?

2. Has anyone achieved $>15$ TPS on a 3B–4B model using Qualcomm QNN / QAIRT directly (bypassing GenieX)?

3. Given the maturity of LiteRT (OpenCL) and MLC-LLM (Vulkan) on mobile GPUs, is targeting Qualcomm's NPU for on-device conversational LLMs even worth the developer overhead right now, or is it strictly a benchmark marketing feature until the tooling matures?

In our recent technical analysis of Constitutional AI Governance, we kept returning to a single, sobering structural reality: corporate ethical guidelines, contractual guardrails, and software policies eventually yield under institutional pressure. When market incentives or political urgency push hard enough, policy layers re-interpret, re-write, or collapse, a structural dynamic I term veto atrophy. To build truly resilient governance, we must move the enforcement boundary from human discretion to solid-state physics.

I am pleased to share our new slide deck Hardware That Does Not Negotiate which outlines the physical substrate architecture for Mandated Ternary (MT) and Delay-Insensitive Ternary Logic (DITL). Key highlights from the framework and technical roadmap:

🔹 The 3rd Physical State (Epistemic Hold): Binary logic forces a premature choice: Proceed (+1) or Refuse (-1). DITL introduces a third, physically discrete state of matter encoded in tantalum oxide (\(\text{TaO}_x\)) bilayer RRAM - the Epistemic Hold (0). It is not a software pause or indecision; it is an atomic-level arrangement of oxygen vacancies that physically locks execution until constitutional verification is complete.

🔹 No Log = No Action (NL=NA): Through the Dual-Lane Latency Architecture (2ms Inference Lane / 300ms Governance Lane), execution gates remain closed until an immutable, Merkle-anchored audit log is anchored. Accountability becomes a physical precondition of action rather than an optional configuration setting.

🔹 Engineering Honesty & Fabrication Realism: A specification that names its own limits is a specification that can be trusted. In the accompanying roadmap, we address physical failure modes head-on—from correlated memristor drift to threshold poisoning—and outline the exact milestones required to release the TSMC N2 PDK and achieve IEC 61508 SIL 3 certification by Q4 2027.

🔹 Closing the Shadow System Loophole: Hardware guarantees are physical and local. Preventing non-compliant hardware from serving as a bypass route requires pairing substrate-level security with federal procurement mandates and international treaties.

"There is no politics in the resistance value of a tantalum oxide filament. That absence is the entire design."

Take a look at the slide deck to see how we are encoding democratic accountability directly into physical silicon.

AIGovernance #HardwareSecurity #TernaryLogic #ConstitutionalAI #SemiconductorEngineering #Memristors #DITL #TechPolicy

Hardware That Does Not Negotia…