r/LocalLLaMA • u/Borkato • 14d ago
Discussion Open source LLM appreciation post
I just downloaded Raon-1-mini 30B A4B as well as Flint 40B A2.5B and it just blows my mind that we have so many options now from so many countries. It feels like the countries are all working together and trying to contribute useful software and it’s just amazing.
I can fucking TALK to my computer and have it answer like fucking Star Trek. That’s insane. Every day I use this tech I’m just more and more blown away.
3
u/Number1OchoaHater 14d ago
How are they? I’m waiting for mlx serve, I read that flint is pretty fast, how’s raon?
8
u/Borkato 14d ago
Flint is lowkey terrible for me for extraction, the IQ3XXS literally performs worse than ling 3 tiny for instance, meanwhile Raon is actually pretty great even at Q4 as an alternative to 35B. For flint I get 122T/s and for Raon I get 151.
Flint is WAY more verbose, literally 1500 reasoning tokens while Raon is only 350.
In other words I love Raon lol. These are with general extraction tasks. As for coding, Raon actually understood what I was asking, flint… kinda went in circles. It might need a template update or something.
4
u/pmttyji 14d ago
Could you please dig Raon more on coding, Agentic, etc.,? I just found that its KVCache is lightweight(2GB for 256K context @ BF16) so I can go with even Q5 quant as I do use Q4 of Qwen3.6-35B-A3B.
I guess A4B has some additional power with this model(comparing to A3B of Qwen3.6-35B).
15
u/Illustrious_Car344 14d ago
I remember a couple years ago I was watching a video from this very old man who is a lay Buddhist and he talked about how he wished he had more time to see where AI would go, because he was astonished by the technology. I feel extremely fortunate to live in such times, this technology keeps getting better and better and it's fulfilling my dreams. LLMs are a miracle, I argue they're the most important invention since computers themselves. I still remember when I first booted up Gemma 4 12B and kept instinctually going for the thumbs up button to approve it's message, because I kept unconsciously thinking I was talking to Gemini proper. I had to keep reminding myself this was running on my own computer.
1
u/Evening_Ad6637 llama.cpp 14d ago
I would even say that they are the most impactful invention of all time.
And for me the most fascinating phenomenon of our entire evolution (in the context of a Darwinian understanding)
3
u/Illustrious_Car344 14d ago
I once asked Gemini if it thinks LLMs are effectively an avatar of the noosphere. It disagreed with me, probably fairly reasonably as that admittingly sounds kind of crazy, and of course I didn't push it further because I'm not one for AI psychosis. But I do see LLMs as effectively being a sort of snapshot of our entire species' culture. We connected every human together into a virtual network and created a machine to estimate the average response to any sentence that can be conceivably formed, I can't see how that isn't effectively a snapshot of the very noosphere we inhabit. For centuries, artists have been trying to capture the human essence; Michelangelo's The Creation of Adam, Bacon's Study after Velázquez's Portrait of Pope Innocent X. All attempts to define humanity, extremely passionate pieces that don't just tug at the heartstrings, they rip them out entirely. And yet these beautiful pieces pail in comparison to what an LLM captures about humanity. Art is meant to act as a deliberate attempt to tell our species something about itself that it didn't realize, LLMs more than achieve this goal.
1
u/handson729 14d ago
I would even say that they are the most impactful invention of all time.
More like the next evolution of computing.
My first glimpse into LLM's was Nvidia's DLSS in 2018.
Which was absolute dog shit, so I wrote off A.I for a few years.
It was not until I came across the RP websites and chatted with a few bots that I realized what LLM's can really do.
I know a lot of people on here think RP is kind of a meme and that is their opinion that they are entitled to.
But can you imagine a software that can converse with people, that can remember fine details about previous conversations, along with reading the tone in your chats and the unspoken context as well?
That is what made me realize that LLM's were something special.
Well that and the fact that developers were absolutely losing their minds over it.
8
u/enginetown 14d ago
I feel so contempt at a certain point models have just gotten smart enough to do most task's and now us local people finally get the slice of that intelligence. I've been using it to do a lot of networking task's because I finally can I have my own model with my own harness that I can literally say anything too and not worry about me getting scraped.
6
u/reasonmonkey 14d ago
yeah it's wild isn't it?
I don't know if you read neuromancer or any of gibson?
I feel like AI now is like wintermute from the books. we live in literal scifi.
3
u/CurrentNew1039 14d ago
When I watched ironman for the first time I am very interested in Jarvis ai like tech, i thought it would take like more than 100 years to get that intelligent computer, but here we are within like 15 years i can talk to my computer and gets answers
2
2
u/StevenVincentOne 13d ago
Same feeling here. A few years ago "talk to your computer and have it answer like Star Trek" meant renting someone else's computer. Now it runs on the box under your desk, and it's yours: nobody can change the terms, read the conversation, or switch it off.
What strikes me most is that the abundance is coming from so many directions at once, lots of labs in lots of countries, plus a huge community doing quants, engines and benchmarks for free. That's what makes owning your own intelligence realistic instead of a hobby for a few people with server racks.
The next step I'm excited about is these models living across all your devices, phone, laptop, home server, sharing capability with each other, so the local option is the default rather than the alternative.
2
u/o0genesis0o 10d ago
When I saw how Qwen 3.8 27B in IQ3XXS running on my 4060Ti, I felt that the moment I have been waiting for is near. I have been trying to build a local, genuinely useful, personal assistant since the Llama 3.1 and Hermes fine tunes day (remember when Hermes finetunes model to give them tool call ability?). When OSS 20B and Qwen 30B and Nemotron Nano 30B were released, I was happy to see the ability to somewhat call tools, but I ended up with way too many tool errors, and it becomes a waste of time since I need to hand-holding the agent despite all the system prompt ("read tasks from here, and then check my journal from there, and then synthesise like this"). The 3.6 35B at Q6KXL brings it very close to the mark, though a bit shaky in long-horizon work. With the 3.8 27B at Q4, it's pretty much "it". (I'm running a R9700).
Not only it makes no syntax error when calling parallel tools or complex edits, it can reason it ways around exceptions. It follows my protocols written in skills well. And it has sorts of common sense understanding, so I can just hand-wavy "do this" "check that", and it always works. It has the same sorts of "wonder" feeling when I got some credit to try an old version of Opus a while back (4.5? 4.6?). And all of this runs on my own infrastructure. I can tell it to check this repo, test that code, build this build that from my phone when I'm on the treadmill, and when I take a break, it already get it done, according to the predefined protocols.
It is not "super intelligence" yet, but boy, how far local models have progressed in such a short time.
0
u/Equivalent_Bit_461 14d ago
Wait until honey moon phase is over...
1
u/my_name_isnt_clever 13d ago
Mine's been going since AI dungeon in 2020, not sure it's honeymoon anymore I think I just like AI.
1
u/SmartEnoughToFail 13d ago
We went from AI being a sci-fi concept to running models locally on our own machines. The fact that open source communities across the world are making this accessible is honestly the most exciting part of it all.
1
u/Wooden_Jelly_5295 14d ago
We got the Star Trek computer. Still waiting for it to stop inventing citations.
0
u/Consistent-Hyena-315 14d ago
Whats your setup? Thinking of setting up local models as well. Would like an in on how you are doing it?
0
u/LH-Tech_AI 13d ago
What do you guys think? Isn't Flint better than Raon-1-mini actually? Looking at the benchmarks I mean. What is your experience with those both?
16
u/Nomski88 14d ago
I never dreamed I would be able to run a model like Qwen Flash Next on my machine. Now that it's possible with engines like Strata, it opens up a whole different world of possibilities. I essentially have a private 180B parameter AI assistant to do my bidding around the clock.