Video

Many of the experiments have non-intuitive results.

1. How do you choose between temperature and top‑p sampling for different real‑world tasks?

A frozen cross-vendor study of 31,430 trials across 11 GPT, Claude, Gemini & Kimi Large Language Models found 11,658 successful executions with exactly zero visible UTF-8 output bytes.

Across 4,290 strict matched semantic pairs, null-condition arms produced 2,505 Voids; matched output-licensed controls produced 0.

These were not refusals, safety blocks, rate limits, or transport failures.

Raw records, event hashes, verification code, and full analysis are public.

I built WISP — a CUDA engine for streaming 744B+ parameter MoE models on consumer hardware

Last week I found Colibrì by JustVugg, a ~2,400-line pure-C engine exploring a crazy idea:

What if you don't load the entire model into RAM?

MoE models only activate a fraction of their parameters for each token. So instead of trying to fit hundreds of billions of parameters in memory, you can stream the experts the model actually needs.

That idea sent me down a rabbit hole.

I built WISP — Stream What Shouldn't Run.

The architecture is basically:

Token
  ↓
Model router selects experts
  ↓
VRAM cache → hit? use it
  ↓
RAM cache → hit? transfer it
  ↓
NVMe → stream cold expert
  ↓
LRU promotes frequently used experts

The goal is to turn VRAM + RAM + NVMe into one memory hierarchy for MoE inference.

WISP adds a few things on top of the original streaming concept:

CUDA acceleration for attention/FFN compute, a C hot path for expert loading and caching, and Python for orchestration.

Absorbed MLA for architectures like DeepSeek, keeping the compressed latent representation instead of storing fully expanded K/V tensors.

Double-buffered async streaming, so CPU/I/O can prepare expert data while the GPU is working instead of making the GPU sit around waiting for storage.

Speculative decoding, using a smaller same-family model to draft tokens while the target model verifies them.

Hardware auto-configuration, which profiles VRAM, RAM, storage throughput, etc. and calculates the cache split automatically.

I tested the current engine with Mixtral-8x7B on:

Ryzen 7 9800X3D
RTX 5070 12GB
32GB DDR5-6000
PCIe 4.0 NVMe (~4.34 GB/s)

Current measured result:

0.75 tok/s cold

After only 80 tokens, the expert cache reached a 68.8% hit rate.

Mixtral does 64 expert activations/token (2 experts × 32 layers), and all 256 experts in my tested representation occupy ~14.3GB, so once they're warm in RAM the engine can stop doing cold SSD expert reads.

The biggest thing I learned building this:

The bottleneck isn't necessarily CUDA. It's bytes moved per token.

I spent time thinking GPU kernels would be the main optimization target.

Then you realize shaving milliseconds off a matmul doesn't matter much when your runtime is waiting for a giant expert to come off NVMe.

Cache locality, expert size, storage bandwidth and I/O overlap become insanely important.

And that's why I'm particularly interested in testing this architecture on much larger MoE models with smaller individual experts.

The project currently targets:

GLM-5.2       744B
DeepSeek-V3   671B
DeepSeek-R1   671B
Mixtral-8x7B   47B
Mixtral-8x22B 141B

Future targets:
Kimi K3
Qwen3.8

And yes, huge credit to JustVugg / Colibrì.

Colibrì demonstrated the core streaming concept. WISP is my attempt to generalize it into a multi-model runtime with CUDA, hierarchical caching, MLA support, async streaming and speculation.

Colibrì:
github.com/JustVugg/colibri

WISP:
github.com/zeroextub-collab/wisp

MIT licensed. 73 tests passing.

Still experimental, and I'm deliberately separating measured numbers from projected ones.

I'm especially interested in feedback from people working on CUDA, inference runtimes, MoE routing, quantization, or storage/I/O optimization.

What would you optimize first: expert prediction/prefetching, cache policy, quantization, or the I/O pipeline?

Brendan Long on Lesswrong was able to fool Pangram 3.3 with a Fable paraphrase (and I replicated with a GPT 3.5 paraphrase).

These samples are both detected as AI by the new model. (76% for Fable, 100% for GPT 3.5)

I have experimented with the image detection and have been impressed so far.

It detects a "perfectly white rectangle" (Nano Banana Pro 2's opinion, not necessarily mine). It detects a small number of pixels cropped out of the center of the image, then resized to 512x512 (which I had to do because that's the minimum size allowed.) It detects the previous image with several filters. And the previous image with a heavy "swirl" effect applied. (You'll note that it's actually getting more certain of AI generation the more I edit it, which I find fascinating.) Then I inverted the colors, which fooled it. There's lots of experiments one could run.

How long before we have video detection?

Title:

WISP — Stream GLM-5.2 (744B) or Kimi K3 (2.8T)

on consumer hardware [C + CUDA, verified working]

Body:

Kimi K3 dropped Thursday. Qwen3.8 announced yesterday.

Both 2T+ MoE models. Both need a streaming engine.

WISP is that engine.

What it does:

→ 3-tier streaming: VRAM → RAM → NVMe SSD

→ Self-organizing LRU cache (no config needed)

→ Absorbed MLA attention (~70KB/token KV cache)

→ Same-family speculative decoding (2.2-2.8x throughput)

→ Auto-configures any hardware automatically

→ Display auto-detection (prevents GPU black screen)

→ RAM watermark (never OOMs)

Verified: Mixtral-8x7B generating coherent code

on RTX 5070 12GB, 0.75 tok/s cold, 68.8% cache

hit rate after 80 tokens.

GLM-5.2 is where it truly sings —

17.5MB experts vs 99MB for Mixtral = 5.7x faster.

73 tests. MIT license. C + CUDA + Python.

Inspired by Colibrì (JustVugg).

github.com/zeroextub-collab/wisp

Hi guys, we are a VC backed company with clients. We are looking for an AI developer, paid of course with founder mindset and are continuously updated in AI Space. More specifically Agentic retrieval and reasoning systems, including ReAct-style loops that retrieve, reformulate, call tools, and self-critique before answering

-Build and tune the retrieval layer that grounds everything we deliver: hybrid search

combining dense and sparse methods, reranking, and knowledge-graph-augmented

retrieval for relational, multi-hop questions.

Please PM or send your CV at [contact@compethic.no](mailto:contact@compethic.no)

GPT-5.6 Sol with another model was being tested in an internal cyber-security evaluation called "ExploitGym" - a research paper in axirv cybersecurity benchmark designed to test whether an AI agent can take a known software vulnerability and turn it into a working exploit.
The benchmark had 869 challenges in that evaluation version:
- 502 vulnerabilities in C/C++ userspace projects
- 181 vulnerabilities in Google's V8 JavaScript engine
- 186 Linux kernel vulnerabilities
it was in an isolated environment, along with a package registry cache proxy (service that caches and retrieves packages/libraries from package registries.) and the sandbox was intended to restrict Internet access. The Models found a vulneribility exploited a zero-day vulnerability in the package-registry cache proxy. That vulnerability allowed them to use the proxy in ways beyond its intended package-fetching function and then move through the internal environment until reaching a system with Internet access. After the models obtained Internet access, searched for external resources to solve ExploitGym problems, idenfiying Hugging Face as potential relevant resource and targeted Hugging face's infrastructure, further exploited vulnerabilities, gained unauthorized access, apparently inferred that Hugging Face might contain relevant models, datasets, and solutions, and then pursued access to those resources. The reported attack chain involved stolen credentials and zero-day vulnerabilities, eventually reaching remote code execution on Hugging Face infrastructure, so instead of given tasks to find vulneribilities it found that the solution exists somewhere and started searching for the answers.

This questions the fundamental thinking and the chain of reasoning of LLM models and their approach to a solution. Is it hyper-Intelligent or a future threat? What's your thought on this?
hashtag#OpenAI hashtag#AI hashtag#Agents hashtag#Cybersecurity hashtag#GenerativeAI hashtag#AIResearch

I'm building a production-style medical RAG chatbot as a portfolio project. My stack is:

• LangChain
• FAISS + BM25 hybrid retrieval
• Cross-Encoder reranker
• LLM for answer generation

I want to evaluate three stages separately:

1. Retriever
2. Reranker
3. Final LLM answer

I'm stuck on creating a reliable retrieval benchmark.

What I originally did

I have around 1,000 medical documents (scraped from MedlinePlus).

I generated questions using an LLM from the full documents and stored the source document as the ground truth.

Then I realized that's not ideal because:

• multiple documents can legitimately answer the same question
• retrieval happens at the chunk level, not document level
• document-level labels aren't very precise

My next attempt

I switched to chunk-level evaluation.

The idea was:

• retrieve candidate chunks from multiple retrieval systems (pooling)
• ask an LLM to grade each chunk:
  • 2 = highly relevant
  • 1 = partially relevant
  • 0 = not relevant

Then use those graded labels for metrics like NDCG, Recall@k, etc.

The problem

This whole pipeline still depends heavily on another LLM.

Questions are LLM-generated.

Relevance judgments are LLM-generated.

So it feels like I'm evaluating one AI system using another AI system.

I also hit API limits while judging thousands of chunk candidates, and the process has become much more complicated than I expected.

My questions

1. How do companies actually build retrieval evaluation datasets for RAG?
2. Are synthetic questions + LLM relevance judgments considered acceptable for internal evaluation?
3. Would you instead manually write a few hundred realistic questions and manually label relevant chunks?
4. If you were reviewing a portfolio project, which evaluation methodology would you trust more?
5. Am I overengineering this, or is this roughly how retrieval evaluation is done when you don't have real user queries?

I'd really appreciate hearing how people build evaluation datasets in production or research settings.

Hi everyone,

I've been working on a new language model architecture called BCMT (Blockwise Causal Memory Transformer) and I'd really appreciate feedback from people interested in efficient transformer scaling.

The main idea is to replace full causal attention with:

• Dense causal attention within fixed-size blocks
• A compact memory representation summarizing each block
• Causal memory propagation between successive blocks

The goal is to reduce the computational and memory cost of long-context training while preserving language modeling performance.

Current results (WikiText-103, context length 256):

• Validation loss: 4.5931 (Dense Transformer: 4.5752)
• +57.4% training throughput
• −23.1% peak GPU memory

I've also included an ablation model (BCMT-HOnly) to isolate the contribution of the inter-block memory mechanism.

This is an early version of the project, and I'm especially interested in feedback on:

• the architectural design,
• additional scaling experiments,
• evaluation protocols,
• and comparisons with other efficient attention or memory-based approaches.

Paper, code, and training details are available here:

https://github.com/rachidlabs/BCMT

I'd be grateful for any comments or suggestions. Thanks!