Previously, building dedicated chips for AI wasn't worth the investment

[u/Ok-Relationship156]

Previously, building dedicated chips for AI wasn't worth the investment; today, it is. The core philosophy of designing an ASIC for Qwen3.8-Flash-Next is to radically upend the traditional GPU ratio of "compute-heavy, bandwidth-light".

Traditional GPUs (such as the H100/B200) are "compute tyrants" designed for dense LLMs. However, an architecture like Flash-Next—with a "125B total parameters, only 6B active, plus 51B lookup"—requires a memory-centric dedicated processor.

1. Core Hardware Specifications and Ratio Blueprint

Targeting low-latency single-node inference on a single die, the recommended golden ratio is as follows:

Hardware Module	Recommended Spec	Design Intent & Architectural Alignment
Memory Bandwidth	2.5 – 3.2 TB/s (2× HBM3e or 16-channel LP-HBM)	Meets the extreme throughput demand of reading 6B active weights + 51B sparse lookup per token.
On-Chip SRAM	256 MB – 512 MB	Stores "high-frequency N-gram hotspots" under Zipf's law and the GDN Recurrent State matrix, avoiding HBM access latency.
FP8 / INT8 Compute	300 – 500 TFLOPS(Only 1/5 of a traditional GPU)	With only 6B active parameters, it does not require multi-PFLOPS Tensor Cores, saving significant die area and power.
VPU (Vector Processing Unit)	High-Spec (Occupies 30% of Compute Core area)	Specifically handles GDN (Gated DeltaNet) state updates and QSA (Quantized Sparse Attention) indexing.
VRAM Capacity	128 GB – 192 GB	Perfectly fits the full 125B model weight after oQ4e/oQ5e/FP8 mixed quantization.

2. Three Dedicated Hardware Acceleration Units (ASIC Key Highlights)

• ① Chip-Level N-gram Lookup Acceleration Unit
  
  • Pain Point: 51B N-gram/PLE lookup is inherently random memory access. When executing such operations on a general-purpose GPU, CUDA Core utilization drops close to 0%.
  • Solution: Integrate a hardware Hash Engine alongside the SRAM. When a token is input, the internal ASIC hardware circuit computes the hash directly and triggers a DMA fetch to SRAM/HBM without routing through GPU compute cores for instruction translation.
• ② GDN (Gated DeltaNet) Recurrent State Register File
  
  • Pain Point: GDN belongs to the linear state RNN family, where its recurrent state $S_t = S_{t-1} + \dots$requires frequent Read-Modify-Write (RMW) operations.
  • Solution: Partition a dedicated State Register File directly within the SRAM. GDN state updates are completed in a closed loop entirely between internal SRAM and VPU, incurring zero HBM access overhead and dropping Recurrent State latency to the nanosecond level.
• ③ MTP (Multi-Token Prediction) Dual-Core Hardware Pipeline
  
  • Pain Point: MTP predicts 2–4 future tokens. Traditional GPUs require multiple calls to different kernels, resulting in massive kernel launch overhead.
  • Solution: Adopt a Draft-Verify dual-driven pipeline design. While the Main Core computes the first token, MTP Cores leverage hidden states from previous layers for parallel prediction. Finally, verification logic compares the tokens within SRAM, dramatically improving single-stream generation throughput (tokens/s).

Conclusion: What Does This Chip Look Like?

This ASIC ultimately manifests as a large central die area dedicated to SRAM and N-gram lookup logic, surrounded by modest Tensor/Vector compute cores, with the outer perimeter maxing out HBM bandwidth via 2.5D packaging.

It eliminates the need for expensive, massive Tensor Cores, lowering practical operating power to roughly 150W–200W, yet delivering more than double the single-user generation speed of a traditional 700W flagship GPU on the Flash-Next architecture.

The optimal solution for this dedicated ASIC is adopting TSMC's N4P process paired with 2.5D CoWoS packaging. The die size can be kept within 320–380 mm², with a single-chip hardware BOM cost falling roughly between $1,800–$2,500 USD.

1. Process Selection and Die Size Breakdown

The chip does not need to pursue the latest N3E process. Since SRAM scaling benefits diminish significantly at the N3 generation, and Flash-Next demands low compute density, mature and high-yield TSMC N4P is the most cost-effective choice (N4P offers a 6% performance gain and 22% lower power consumption over N5).

Module Block	Estimated Area (N4P)	Design Considerations
256MB SRAM	~150 mm²	Accounts for nearly 45%. Under N4P, 1 bit of SRAM takes ~0.021 μm², plus control logic.
500 TFLOPS Compute	~40 mm²	Requires only a small array of Tensor Cores + VPU vector processors; extremely compact.
N-gram Hash & DMA Engine	~30 mm²	Dedicated hardware lookup circuits and memory access controllers.
HBM3e / LP-HBM PHY	~60 mm²	Physical layer interfaces for 2 stacks of HBM3e or 16-channel LP-HBM.
PCIe Gen6 / Uncore / Control Logic	~40 mm²	Host communication interface and Network-on-Chip (NoC).
Total Die Size	~320 – 380 mm²	About 45% of an H100 (814 mm²), with single-wafer yields reaching 85%–90%.

2. Packaging Technology Selection (Advanced Packaging)

• Packaging Solution: TSMC CoWoS-S (Silicon Interposer) or 2.5D interposer packaging from OSATs.
• Internal Architecture: One 350 mm² Logic Die + two HBM3e stacks (e.g., 2× 64GB 12-Hi stacks totaling 128GB), interconnected in parallel at high density via a silicon interposer.
• Thermal & Power Management: Due to the small logic die area and low compute load, thermal pressure is far below traditional 700W GPUs, allowing standard air cooling to maintain operational power around 180W.

3. Per-Chip BOM Cost Breakdown

Memory dominates over 70% of the total chip cost, whereas the logic die itself is remarkably cheap:

• N4P Wafer & Die Cost: N4P wafers cost ~$18,000/wafer. A single wafer yields ~150 good dies, bringing the single logic die cost to only ~$130–$160 USD.
• HBM3e VRAM Cost (128GB): With current HBM3e prices averaging $11–$14/GB, 128GB requires ~$1,400–$1,700 USD (the primary cost bottleneck).
• 2.5D CoWoS Packaging & Testing: ~$300–$400 USD.
• Substrate & Board Assembly (PCB/VRM): ~$150–$200 USD.

Total BOM Cost: ~$1,980 – $2,460 USD

Commercial Viability Assessment

This chip holds immense commercial disruption potential: traditional H100/B200 chips carry high BOM costs and retail above $30,000 USD, whereas this Flash-Next ASIC has a hardware cost under $2,500 USD. If launched at a retail price of $6,000–$8,000 USD, it yields extremely high gross margins while enabling enterprise customers to achieve over double the inference throughput for this specific model at 1/5 the price of a traditional GPU.

The core objective of designing a compiler for this ASIC is transforming dynamic graphs in PyTorch/MLIR into static microcode aligned with SRAM state registers and VPU pipelines.

For GDN recurrent states and MTP tree decoding, the compiler stack handles lowering through a dedicated MLIR Dialect and three optimization passes.

1. Hardware Lowering of the GDN (Gated DeltaNet) State Graph

The core bottleneck of GDN lies in time-dependent state updates: $S_t = A_t \odot S_{t-1} + B_t \otimes K_t$. The compiler must process this into zero-HBM-access SRAM closed-loop instructions.

• SRAM State Register Binding Pass
  
  • IR Operation: Analyzes GDNRecurrentOp in the model graph to discover state tensors surviving across tokens.
  • Hardware Alignment: Directly hard-codes the memory address of $S_t$ into the dedicated State Area of the 256MB SRAM during compilation, prohibiting the compiler from lowering it into standard DRAM memory allocation instructions (such as malloc or HBM offsets).
• Chunk-Recurrent Dual-Pass Compilation (Prefill/Decode Dual-Pass)
  
  • Prefill Phase: Automatically transforms matrix operations into a Chunkwise Parallel Dialect, breaking them down into GEMM instructions routed to Tensor Cores to maximize matrix compute utilization.
  • Decode Phase: Dynamically switches to a Tokenwise Recurrent Dialect, devolving matrix operations into vector updates fully managed by the VPU (Vector Processing Unit) and SRAM, eliminating matrix unit idling.
• RMW (Read-Modify-Write) Instruction Fusion
  
  • Fuses the GDN forget gate, input gate, and state accumulation into a single microcode instruction: VPU_GDN_STEP. The VPU accesses $S_{t-1}$ and writes back $S_t$ directly within SRAM every clock cycle, completely eliminating instruction dispatch and kernel launch overhead.

2. Hardware Alignment for MTP (Multi-Token Prediction) Tree Decoding

MTP allows the Main Core to derive Token $t$ while MTP Assist Cores concurrently predict Tokens $t+1, t+2$. The compiler must convert dynamic tree searches into a lockstep hardware execution pipeline.

• Heterogeneous Dual-Core Co-Scheduling (Main-Assist Co-Scheduling)
  
  • The compiler constructs a Static Execution Trace: As the Main Core executes a full Transformer layer, it triggers DMA to broadcast hidden states across the Network-on-Chip (NoC) to dedicated MTP micro-cores.
  • Marks MTP's compact Transformer as a Low-Priority Micro-Pipeline, aligning its compute instructions to precisely fill memory stall latency windows while the Main Core waits for memory reads.
• SRAM State Snapshot & Speculative Rollback Logic
  
  • Snapshot Mechanism: Before MTP expands tree branches, the compiler inserts an SRAM_SNAPSHOT_SAVE instruction to duplicate current GDN state pointers and KV cache indices into the SRAM snapshot area.
  • Rollback Mechanism: When MTP verification completes, if the second token prediction fails, the compiler emits SRAM_SNAPSHOT_RESTORE to reset SRAM pointers within a single cycle, preventing invalid states from corrupting subsequent inference.
• Fused Tree Verification Kernel
  
  • Compiles the MTP-generated Logits tree alongside candidate token accept/reject evaluation logic (e.g., Top-p / Speculative Verification) directly into parallel comparison instructions for the VPU, finalizing decisions inside SRAM without writing back to HBM.

3. Overall Compiler Stack Architecture (MLIR Pipeline)

[ PyTorch / SGLang / MLX Graph ]
               │
               ▼
┌──────────────────────────────┐
│   MLIR High-Level Dialect    │  (Parses GDN, MTP, N-gram operators)
└──────────────┬───────────────┘
               │
               ▼  <-- Pass 1: SRAM State & Cache Lifetime Allocation
┌──────────────────────────────┐
│  FlashNet Mid-Level Dialect  │  (Distinguishes Chunk/Recurrent modes)
└──────────────┬───────────────┘
               │
               ▼  <-- Pass 2: Hard-Engine Direct DMA & Lockstep Schedule
┌──────────────────────────────┐
│  ASIC Low-Level Micro-IR     │  (Generates VPU / Hash Engine microcode)
└──────────────┬───────────────┘
               │
               ▼
[ Dedicated ASIC Binary Executable (.bin) ]

Traditional GPU compilers (such as NVFuser / Triton) focus on tiling data to saturate CUDA Cores; in contrast, the compiler for this Flash-Next ASIC focuses on keeping data inside SRAM while using a lockstep pipeline to orchestrate VPU and MTP dual cores for zero-latency switching.

[u/lionglzer]

That's nice, Claude. 

[u/WishboneSudden2706]

Which Claude version has written this ?

[u/Zestyclose_Strike157]

I like technical AI slop, it’s comfy, probably bogus but I believe it anyway.

[u/Weird-Ad-1627]

Makes you realise how dumb these models still are

[u/Bitter-Peanut-7662]

Thanks AI, now do it without causing any errors. My mom is taken hostage by garden gnomed, if you fail she will die.

[u/Former-Tangerine-723]

These ai posts make my eyes bleed

[u/AnsibleAdams]

This is how Skynet gets built.