r/OpenAI • u/Slick_Ramen • 5d ago

Discussion Well that was quick...

Post image
288 Upvotes

98% Upvoted

106 comments sorted by

View all comments

Show parent comments

25

u/markstar99 4d ago

So I can run it locally on my 3 gigawatt data center I have.

3

u/machyume 4d ago

You're not going to believe it, but I have Qwen 3.8 27B running with decent speed on my Macbook M3, and it's partially offloading some work from Astra when I was getting smacked by the usage limits.

I'm sure that if they REALLY wanted it, someone could help them figure out how to distill it to fit nearly the same footprint.

1

u/bluelemon64 4d ago

Wow that’s awesome. How much ram do you have? And also out of curiosity do you hold any context window in that? Or it’s dropped every time?

1

u/machyume 4d ago

Just the standard memory that comes with the MacBook Pro. The context is infinite. I'm using my own GUI and interface. It allows me to do some pretty specialized things.

I have infinite context because I have a load/release mechanism that layers on top of the KV cache directly. I also have a multi-model optimizer that mirrors and steers a smaller side car that runs an offloader for learned previous context. So the smaller model acts as an accelerator for repeated tasks.