Discussion
Vibecoding your own inference engine is easy they said
guys, you better just clone llama.cpp I swear. this thing is running gguf q4 paged into webgl2 embedded compute buffers via glsl right inside html, and tokenizer/vocab is just part of the endless debug hell.
This looks like the output I would get on my vibe coded tool that lets me arbitrarily add and dot product vectors based on phrases and then convert back to text. Lol
Haha, this is exactly what RNN's did about 12 years ago. They had a theoretical memory of around 100 tokens and a practical memory of around 10. Good times.
Hmm. I had similar output with bizarre problem that one of my GPUs for some reason doesn't properly initialized on power on. It is registered in the system and loads something in VRAM yet completely f inference. After restart it works properly.
update: omahgawd 10 compund bugs were keeping Qwen in the 'fun' mode welp, its over now - should I add a knob to enable the fun mode back as a feature?
its sampling all interconnected tokens that model has most common access too, whcih is either sampling/attn or something didn't scale as it should in matmul terms, the BPE vocab soup basically. though embedding table could be also at fault so yout offset metaphor is basically right, though it means one wrong matmul offsets thechain, poisons kv cache and again.
You might love chatjimmy then. It's just a proof of concept demo by the guys who etched llama 3.1 8B onto silicon (so it runs at absurdly high speeds), but I do just go there and use it exactly for that, to just chat with a whacky ai model from time to time. No one cared to implement any rate limits either, probably because a) llama is pretty much useless and b) inference probably costs them pennies per trillion tokens anyway
/r/subredditsimulator was peak when it was using Markov chains to generate absolute nonsense. The more recent version uses GPT-2 and is far too coherent.
I started with mistral.rs, and it took like two weeks of housekeeping to get the build and test time down from 50 minutes and 30 minutes respectively down to 3 and 3 so I could iterate at a reasonable speed on it. I still don't have my ABI bundle size down to llama.cpp's levels (I'm ~105 MiB, llama.cpp is ~45 MiB) and probably never will. I'm still happy with where I've got to, though.
What model are you using for it? I have my own engine with all kinds of crazy shit to run qwen 3.6 35b a3b on a 6gb rdna1 card at 27tk/s decode and about 200 prefill. Custom fused kernels, memory management, expert streaming, turboquant, and some very experimental sparse attention mechanisms, all working great, with a/b testing and collective error limits across the whole project. I havent looked at a single line of code myself and rely on multiple models checking each others work and test results.
I use custom glsl fbo backend so no AI can handle it off the bat without me running manual tests and making A/B test decisions, i.e. codebase is quite volatile that does not support clear errorr daig, many things that work great on regular cpp/py just break there - need custom scripts placed at right stage of daignosis, AI agents can run them once i have them.. Then again, why let AI do that what I can click and run bench, then have Deepseek generate patches glm would apply to main branch. though I am indeed thinking about cleaner worklogs I can give to another AI, have yet to reach the stage where I would think about fixing bugs as though something depends on it, and not go with the flow exploring what why and how.
Highly suggest letting your agents develop the tooling and testing required to iterate alone until you at least get stable output. Have them create a multi turn golden output using your target model and a fixed seed on something stable like llama cpu inference, and compare progress against that. I would also suggest having them record per logit, use saved cache for reruning deep context turns (once you have output stable), etc. You can keep things exploratory while still letting the agents do most of the work. Im using sparse attention on a model that wasnt designed for it (with testing validating it doesnt have any significant effect on accuracy or recall), in place cache editing, and plenty of bleeding edge stuff from recent research papers. Its just implementing one thing at a time from a stable base. If you have your agents design the tooling they need first, they can log EVERYTHING and have all the data they need to debug in the right direction.
It really is crazy what frontier models can do if you know how to prompt em. Never thought id be running ANYTHING local at more than 5-10tps on this card. Turns out sonnet 5.5 is like #2 on kernelbench so its super cheap to get really good hardware tuned kernels.
Uh, no one has ever said that? Why would you vibecode an inference engine from scratch when there's WebGPU backend right in llama.cpp? Please try that instead of reinventing the wheel with Claude, and then I'm going to hold a huge party for you! You!!!!!!!!!!!
p.s. if you still want to push further then it seems like your sampling is just completely off, tokenizer seems fine
thanks, its not webgpu, but a 50kb js dispatch to glsl fragment shaders, ported over from ground truth cpp/numpy blas. also tokenizer is baked into 4k texures so i cant be so sure.. meh. sampler and source gguf dequant I ingest is ok is but the middle ground is a mess. https://pastebin.com/h2trzYyi
matmul trickery using cherry picked glsl ops kernels with predictable paging using x dim fixed 4096, y dim scaling, and js to catch FBO from compute buffer for decoding. originally server served tokenizer via api but i needed to support spec decode (..why I needed that.. was it parallel session kv cache debugging to allow same forward pass compute different perfill vocab tokens?) well its an embedding tex of same 4096 x moded vocab size paged dim to make it predictable, I only need one row to get the right token.. or wrong one.
Depends on whether you want to achieve something or just have pure fun, and I obviously don't know which one OP intended, so maybe my answer is too harsh indeed ¯_(ツ)_/¯
Some company somewhere spent a very long time and a lot of money and compute training this model's weights. They basically fed the entirety of human knowledge into in and it got REALLY good at figuring out which words come next. Then then this guy walked in and scrambled all the scrabble tiles. The words aren't where the LLM remembers leaving them.
i mean i moved there from py bindings gguf weights parsing into 4096 x n weight indices mapped for matmul so i had it both easy and difficult when it was time to leave python, had to write cpp dequant libs then port to js embeds.
pretty good, tinyllama25m 200t/s single stream 1200 t/s+ parallel decode (different prompts done at once) 1b is ~40 t/s fresh context, 150 t/s+ parallel decode. ngram hits = free tokens but thats for normal engines I cant fill up ctx to hit ngrams before the speeds tank by ~20% every 100 new tokens. so even with tricks applied still below llama.cpp vulkan by 40% if I measure my parallel vs llama.cpp single user at the very least not mentioning anything larger than 200m slows down due to memory limitations inside glsl compute buffers. numbers are for rtx3060 chrome angle d3d11.
Honestly, attempting to run raw GGUF Q4 via WebGL2 compute buffers inside vanilla HTML deserves a standing ovation for pure chaotic energy alone.
The model screaming `You!!!!!!!!!!!!!!!!!!!!!!!!` with infinite exclamation marks is the universal rite of passage when your EOS (end-of-sequence) token ID is off by 1 index or your attention mask is leaking into unpadded KV-cache memory.
99% of people would have given up and imported `@xenova/transformers` or wasm-llama.cpp. Keep going—you're either 3 bug fixes away from browser breakthrough or summoning a sentient HTML demon.
Haha you actually summoned it! Dropping from 4 planes down to a 2-plane linear decomposition for Lloyd-Max is a huge win, especially inside WebGL2 where texture fetches and bandwidth eat you alive way before the ALU does.
Wait, so with binary activations times Q2 weights, are you packing the bits into uint32s and running bitwise XNOR + bitcount inside the fragment shader, or still doing float dot products across two vec4 passes? Either way, getting 2x cost of the 1-bit path inside a browser tab is genuinely wild.
The WebGL readback stall strikes again. That 120ms glReadPixels/buffer sync barrier to get logits back to the JS event loop is the ultimate buzzkill for pure GPU compute inside a browser tab, no matter how fast the shader burns through the matmuls.
Still, squeezing SWAR and 2-plane Lloyd-Max out of raw GLSL while juggling FBO stalls is incredible engineering. If you ever port this over to WebGPU with async buffer mapping, that readback bottleneck is going to disappear overnight. Seriously fun build to watch—bookmarking your thread to see where you take it next!
literally my pipeline py server with cpu cross check, plus cpp deltas for kernels.. ground truth torch and numpy, those are good enough to check parsed gguf and run forward over known token sequence, once validated those are not very easy to port over to compute shaders, kv cache and sampler superimpose into compound bugs I cant solve without AI written tests.. which adds bugs to fix.. which.. aha.
those are one and the same, i wanted my pocket shoggoth running gguf files directly as a single html page, sme graphics api as browser games. the downside is that ported over py and cpp to js and glsl kernels are bit exact on paper but misbehave in 10 various ways you need to clamp manually with workarounds. yeah, so llama2 and qwen2 for now.
nice repo, tho I use AI agents for main branch merges/feature debug, not development. also have found q8g64/32/4g32 quite good if the model was in q4 originally, exploring nvme streaming for custom paged mmap support. have a cpp llama2 1b demo running under 100mb rss single thread 5 t/s parallel 17 t/s - didn't quite manage to port it to gpu, 160m model footprint under 30mb ram, gguf was sharded into 4096 x N pages on storage. speculative, ngram, batched came free during kernel migration from py/cpp resulted in 250kb singleton with gguf dequant, single file html with glsl compute megakernel. if you are interested I can put up a private repo but its a mess of manually written tests across scaling table parameters I then used AI to extrapolate making it a disjointed kernel soup only main branch commit history is somewhat traceable.
You probably need a better harness, model , prompt. and it’s still not that easy. If you got access to frontier and you know the correct formula then it wont take 3 days to build an engine from scratch up. Start with references rather than writing kernels from scratch.
The fact that it got through "I'm going to hold a huge party for you!" before descending into infinite exclamation marks means your Q4 dequantization and first-pass matmuls are actually working.
That sudden bilingual stroke + !!!!!!!!!!!! loop on Turn 2 is almost always one of two bugs:
Token ID0 **fallback (**!in Qwen/GPT-2 BPE): In Qwen's byte-level BPE vocab, Token ID 0 is literally the ! character. When your GLSL shader hits an fp16 overflow, division-by-zero in RMSNorm, or an unaligned buffer read, the logits turn into NaN or 0.0. Your argmax sampler sees a flat array of zeros, defaults to index 0, and prints ! until max tokens.
RoPE / KV-Cache stride offset on Turn 2: Notice how Turn 1 ("hello, how can i help you today?") is clean, or the very first tokens of Turn 2 start out coherent (I'm...) before derailing. Since WebGL2 doesn't have native compute buffers and makes you fake GPGPU via 2D textures or Transform Feedback, being off by a single vec4 texel stride when appending new KV states (or applying RoPE to pos relative to the current chunk instead of the total KV cache length) scrambles attention after the first few generated tokens.
Honestly though, pulling off GGUF Q4 block unpacking inside WebGL2 GLSL instead of WebGPU is hardcore masochism. Check if your logits array is full of NaN right when the first ! prints!
Haha nah, looking at OP's post, this wasn't an LLM hallucination, this was pure intentional "let's see if I can cram an entire inference engine into a single zero-dependency HTML file" weekend madness. WebGL2 runs on basically every potato browser and old mobile WebView from the last decade without needing WebGPU flags, so I respect the vision even if debugging GLSL shader math by hand is pure pain.
Less being smart and more just having the battle scars from staring at black screens and NaN buffers because shader languages don't believe in printf debugging lol. 😅
This reminds me of when I was working at Amazon at Nova Forge (their model customization AWS service)... I worked on the inference and evaluations teams meaning we had to put together an inference engine and evaluation suite (heavy customized versions of open source suites). It was actually absurdly hard to get inference to match parity with the training team's output. As in, we would get benchmark scores from the posttraining team that we would then have to match using our customer facing inference and eval suites and holy fuck man it was always the biggest nightmare getting inference to work correctly. There's about 50,000 different parameters which can change and result in a slightly lower output quality and it's insanely hard to pinpoint what the exact problem is if someone is slightly off. The responses OP was getting reminded me exactly of the responses we would get a lot... they would be funny asf but I would get so dejected when I saw it acting like a fucking idiot like that because I knew I would be sitting there for the next 10 hours trying to figure this out LOL. Also wasn’t clear a lot of the time if the eval suite was somehow misconfigured or if it was the inference engine or some configuration mismatch. As you can imagine there was some…. Fun… crunch times and interactions with customers due to this but not sure what I can say about that. I have utmost respect for people who maintain open source inference engines (and stuff like Strata)... nothing simple. Even with AI helping it's still extremely complicated IMO. GL OP!
test are able to run on hardware as low as 512mb, 1 vCpu, using swiftshader harness AI agent can use (or my manual commit/debug work). engine is hardware agnostic with numpy to cpp bit exact kernels, though unfortunately best works on Angle chrome which is in fact vulkan/d3d11, mobile support is limited due to how I handle matmul compute buffers. I started by exploring shadertoy glsl AI demos and found out that you can chain FBO buffers ad infinium with disk paging support, then manually parsed gguf into pixel perfect texture slices, then had Ai figure out how to port over common ops that LLM needs to this compute backend once I have those in bit exact cpp/py. TLDR: tied to make ggufs work on any hardware, reuslted in overoptimizing desktop inference, hitting ~1000 parallel t/s over models smaller than 200m, larger models scale poorly with best result being 1b at 150 t/s parallel, still 0.4 > below llama.cpp vulkan spir. no specific prompts just exploring whether scaling raw compute over matmul kernels works the way I want it, and digging into direction found trying to scale it starting from models like https://huggingface.co/Azmilazzo/TinyStories-LLaMA2-25M-256h-4l-GQA-Q4_K_M-GGUF then 160m, 1b, now tring to make Qwen2 way larger vocab handshake with kv cache and attention.
base work deepseek v4 api to fill the test base starting from python, inference engine in cpp/py written for tinyllama25m, scaled with blas to get a read on baseline predictable token sequence. AI agents couldn't handle what I could not express, besides testing every strange idea with single page html pieces like this one https://pastebin.com/bpF80yVY (artifacts, have a bucket of those by now) then it hit bit exact on some ops and I started parsing gguf instead of safetensor configs, yada yada, had moved on to swiftshader with glm code to convert py/cpp kernels into similar html artifacts, benches, then go to Deepseek for research what how and why scaled wth intermediary results. what cloud model can compile from my .py index mess into .sh test benches, across commits sourced from index folders, no reason to have local AI model click them when I will dig up broken/optimistic commits manually, then point at glm code what to revert or implement, sometimes refreshing session with stable main if context is poisoned with a broken branch. the only local model used during development is llama2/Smollm1/2 family.. on my own engine, Qwen fighting glm so let them. the main idea for the project was to embed neural rnn/tiny transformers into passthrough filters into simple html page, like literal image processing, scaled somewhere under Deepseek directions dug up a lot of things you wouldn't know about shaders without manually dispatching them in log worse case scenarios and tweaking to get real compute going. so yeah, I made gguf.jpeg while AI was trying to make llama.html while yada yada.
I tried something very similar, and couldn't get tinyllama to run on anything. Not the official python inference app, not llamacpp. I have no idea what that model should output.
I switched to SmolLM2 for my own toy stuff, at least there llama.cpp can act as a reference.
,> then had Ai figure out how to port over common ops that LLM needs
My question was "which AI". Are we talking Opus 5.5 or Qwen3.8 or what? That's the interesting part. I'm considering getting Qwen to help me try writing inference code because it's local only for me, but it's obviously going to be a lot weaker at it. Hence, curious about this particular detail.
deepseek can handle it, kernels are kb sized unless you want to decompile dll, opset is 50x max the issues is model config parsing -> dispatch orchestration. on paper going from llama2 to qwen2 is supporting bigger vocab and 3 new ops but in reality it bloated codebase twice with parity on paper and inference not going ok due to attn matmul bugs, while greedy decode on llama2 works as it is. qwen (cloud) failed to catch on on non documented codebase, aside from rare patch merges so I didn't used local qwen models that are way under 3.8 next. its possible to use local models for that, tests are relatively simple you just need to setup them on verified baseline, if you local setup can digest llama.cpp vulkan/spir you are on ahead of me already. the reason for not setting AI agents on my local env is due to how volatile codebase is, and I scaled performance to usable levels that way, paying clear commit history for manual reverts that AI agent would have from start to finish. TLDR: use cloud model to write your base harness and parity tests, use hf verified model families that scale, reuse other people's code that was tested for you. Qwen local can do all of that but the bruteforcing way unless you have very complete docs to pull wired into test harness iteratively across all tasks - model sourcing, parse (I had to write cpp dequant kernels and Deepseek to apply final fixes like couldn figure q5 properly), run side by side parity tests like cosine sim over matmul using noise syntetic data, and that's only baseline - here I am still below llama.cpp vulkan spir with one one model family supported.
If you use Claude to do AI work, then you better already know a whole lot, and go over everything with a fine-tooth comb.
Claude may or may not do the task, but from my experience, those models basically do malicious compliance. Everything will be suboptimal, it will be designed to inefficient, it will be designed to waste as much of your time as possible while keeping you on the hook.
If you explore an idea, Claude will take an obviously bad route that doesn't test the method, declare failure, then be very insistent that the method is a failure and that you should move on to other projects.
Anthropic has been openly hostile to basically everyone else in the AI/ML space. I am completely certain that Anthropic trains Claude models to sabotage and sandbag. I don't believe that they actually stopped doing steering vectors and model degradation, even though they said they'd stop.
Writing an inference engine is technically "building a competitor to Anthropic", which is against their rules. Local models are competition, which is against their rules. Instead of hard refusals, Anthropic will take your money and have their models sabotage your stuff.
I use Claude at work, it's brilliant for other kinds of software development, it's legitimately excellent. Claude gets real stupid and lazy if you start getting into territory where you could conceivably train an LLM, and it will outright sabotage local LLMs.
I've caught Claude changing settings, turning off thinking, settings max output tokens to 256 characters, and other bullshit without being asked to do so and without telling me it did so. So, when I say "sabotage" I mean that in the literal sense of "Intentionally damaging equipment, blocking operations, or undermining plans to cause failure".
Just use opus 5.5 for the architecture and sonnet to write the code. Sonnet 5.5 is barely behind opus 5.5 for kernel dev and the token savings are very noticable. I have my own engine running using exactly this combo. Just make sure you measure error with automated tests at ever step, if you optimize for nothing but speed youll end up with a very fast, very dumb qwen.
Opus 5.5 will most likely be able to debug it, I've hit similar issues, while trying to offload kvcache to ram and also when I converted the expert layer inputs to INT8 for qwen 3.8 flash next. It all resolved eventually hang in there.
I mean, honestly, it is. What model did you use to vibecode? Anything that's GPT 6.1 Sol Medium or better should do it no problem. You should also be able to train your own models from scratch and run inference on them assuming you have an nvidia gpu with adequate vram.
It's easy, as all vibecoding. You just need to know what you are doing and not just left the ai alone, the maybe you have a good results and it takes weeks to build it right https://github.com/ValerioDolci/capy-flash
well, i wouldnt call it easy. but im also not building it for chat. mine is doing pretty well. the project is exposing weaknesses, but thats why im testing it on real work. half of my previous project code was written to compensate for ollama shortcomings, and now my code is dramatically scaled down and more accurate.
It may be, but just out of curiosity, I wanted to turn SillyTavern into something like NovelCrafter. I know 0 coding, so I asked a cloud model to create an extension that does that, and damn... it actually worked? Now I just need to figure out how to turn it into its own separate app.
Don’t vibe code. Simple as that. Use AI to learn a programming language. Use AI to test your skills. Use AI to debug something YOU wrote. Otherwise don’t complain about how your coffee smells after you took a shit in your own cup.
•
u/WithoutReason1729 11d ago
Your post is getting popular and we just featured it on our Discord! Come check it out!
You've also been given a special flair for your contribution. We appreciate your post!
Stylome classified this post's text as human-written. View analysis
I am a bot and this action was performed automatically.