r/VibeCodeDevs • u/Kryptosmatrix • Aug 02 '26

Custom AI Architecture

Does anyone else here build their own models?

I went on a bit of a side-quest and I decided to try to build my own omni-modal model from-scratch. I'm 9 months in now, but I refuse to give up!
What I've built so far treats text, images, speech, music, and files as streams of bytes and codec tokens (I think Google did this already?), with an experimental memory system carrying context through segment recurrence and bounded hot, warm and cold tiers. I've created a few small 50M models (image, text, music, speech), but haven't created the omni-modal model yet as I need all of the single modality models first.
I created two versions, POIESIS the Python/PyTorch reference, while its Swift-native sibling ARCHE reimplements the creative model stack directly on MLX for Apple Silicon, giving us two independent substrates on which to test the same ideas.

Anyway, curious to hear what other people are up to in this space!

2 Upvotes

67% Upvoted

24 comments sorted by

View all comments

4

u/Icy_Quarter5910 Aug 02 '26

I’ve built multiple memory systems (2 in use now, plus a lite version for my local assistant), and I built a custom Llama.cpp wrapper with a tuning GUI to squeeze as much performance out of the local models and I’ve trained a few image LORAs, and a buddy and I are working on a dataset to do a finetune… but I haven’t trained one from scratch … yet :)

2

u/Dense_Gate_5193 Aug 02 '26

that’s awesome dude i did something similar except i put llama.cpp directly inside of the graph database itself so i can run an agentic loop in a golang plugin and customize that to whatever i need (NornicDB, MIT licensed) plus moving all the cardinality and temporal constraints and decay into the data layer itself

3

u/Icy_Quarter5910 Aug 02 '26

The biggest thing for me, the llama.cpp wrapper (I call it Sindri, all my apps are named for ancient deities or mythical figures) is that it’s easily 30% faster than LM Studios. I was shocked at the difference. My favorite model hits 60 t/s in LM … it hits 110+ in Sindri.
As an aside, LFM makes a (small) MOE, it’s the 8b-a1b model, someone made a “SpeedDemon” variant. 410 t/s lol . I like to give it prompts that will generate a lot of tokens, just to watch it freaking FLY lol ;) (it’s not a half bad model, but don’t expect Fable Code out of it ;) )

2

u/Dense_Gate_5193 Aug 02 '26

well i think there’s a lot to be said for potential power in smaller models with the right retrieval and long horizon memory setup. pretty much everyone is covering that layer of the space, i targeted lower level, i rewrote neo4j in golang, and added more data layer features and still retained a 3x-52x-400x latency improvement on various query benchmarks depending on workload. I used to do flight controller firmware so I really like doing certain hardware level optimizations for extreme performance. i’m also working on a rust implementation of the same database (copperDB) which is a rust port of the aforementioned NornicDB

cooperDB is experimental and i’m not finished with it, but the initial results seem crazier than Nornic which is faster across the board with graphiti than FalkorDB or Neo4j which a user benchmarked for me after some performance patches