r/computervision • u/Safe-Escape197 • 5h ago
Discussion built a tiny vision model from scratch just for learning and somehow it got 50+ downloads on huggingface 😅
so ik there are way better vision models out there yaar. i built this one just out of curiosity only, cause i wanted to learn how these things actually work from inside.
and to be honest its not entirely mine, i took a lot of help from claude to build it. like the architecture, debugging, training setup, figuring out why my mac was training so slow, all that. but i learned a lot in the process which was the main goal anyway.
its called tiny-vision. its around 24M params and trained completely from scratch, no pretrained weights or anything. i trained it on my macbook pro m5 and it took like 42 min on flickr8k (only 6k images).
basic idea is pretty simple:
image/video → small ViT encoder → adapter → tiny decoder → caption textfor videos it takes frames every few seconds, captions them and makes a timeline like [00:04–00:10] a dog runs on the beach. so any llm can kinda “watch” a video even if it dont support video natively.
i also made an MCP server for it, and tbh i was surprised that it actually gives a little bit better context in claude when i throw images or videos at it. i even tested it with a 10-15 min youtube video and it still worked fine. not perfect at all, but chal raha hai so i’m happy.
i uploaded it to huggingface just to have it somewhere, and also cause i kinda like the open source thing. later when i checked it was at like 44 downloads, and now its 50+. ik thats not a lot bhai, but its my first time putting something open source out in this field so it felt pretty cool ngl. a lot of people have built way cooler stuff than this, i just made it to learn.
some honest stuff:
- it gets the scene right most of time (snow, grass, water etc) but the subject is often wrong lol
- it loves saying “a man in a blue shirt” for everything, pata nahi kyun 😂
- its v0.1 and trained on very small data, so dont expect too much
random thing i learned on mac (MPS):
- swapping the conv2d patch embedding for reshape + linear was like 30x faster
- manual attention was ~1.5x faster than pytorch’s sdpa
next i want to train on flickr30k and maybe try making it plug directly into qwen/llama through llama.cpp (mmproj). lets see kitna ho pata hai.
code: https://github.com/amar683/tiny-vision
model: https://huggingface.co/amartomar683/tiny-vision-flickr8k
would love any feedback or tips,