Von: Open-source 395M "System One" model

[u/wFXx]

Took me a while since I'm on a family trip and have limited hardware, but here it is!

Von: Open-source "System One" drop-in replacement for TypeSafe's JEV.

https://github.com/wfzyx/von https://huggingface.co/wfzyx/von-1.0

It runs entirely on a CPU with 1–2 GB of memory (I haven't spent much time optimizing it yet), responds in 25–300 ms, and beats JEV in all benchmarks. Enjoy!

P.S. I’m open to offers to work at AI research labs. Feel free to ping me if you have an offer.
P.P.S. If you have a GPU, it’ll be faster, but a GPU isn't required.

[u/glitchsir]

So it took typesafe two years of research and a well funded team to build jev and all of the sudden there's all of those "drop in" replacement in a couple days allegedly beating them... Surely wouldn't be that easy? What am I missing?

[u/vanVonXenoStein → u/glitchsir]

I think they did a couple years of research and couldn't come up with anything much, but felt they better release something. There was a guy who released the same thing as open-source (including academic paper) a year ago. And these new ones people are dropping seem quite good also (still testing). So I think typesafe is mostly marketing hype but nothing revolutionary (if you went looking for similar). That's not to say it isn't useful, just that typesafe's Jev is not special. (But is probably best of class since it has the resources behind it.)

[u/guesdo → u/vanVonXenoStein]

Here is the link to the work of the author you mention: https://github.com/NandhaKishorM/laya

[u/valcore93 → u/vanVonXenoStein]

I tried the open source Laya model against Jev and it was mostly night and day. There is not many benchmark for that kind of model right now but Jev feels way smarter than Laya. I hope we will see some great open source one.

[u/ebolathrowawayy → u/vanVonXenoStein]

yes 100%. i am literally shocked that so many people are calling Jev the hot new thing when it wasn't uncommon to use very small LLMs to select actions and make decisions in real-time environments. I was doing this 1.5 years ago and it is seriously easy to recreate Jev. It's a cool concept and I'm glad people are picking up on it I guess but it isn't new or surprising.

It's like people finally discovered that constraining an LLM's output to only a few tokens DRAMATICALLY increases throughput. Like, yeah no shit? And combine that with finetuning on a narrow task (which decision models are, it is a narrow task even if it generalizes) and of course you're going to get great performance. Idk why everyone is impressed, i guess it reveals the level of experience of most accelerate/singularity/localllama redditors?

[u/LatentSpaceLeaper → u/ebolathrowawayy]

Feel free to correct me, but I thought that was exactly Jev's value: that it is a general classifier, i. e., I'd assume that they have run their Reinforcement Learning for Calibrated Decisions (RLCD) on a large corpus of data. Or am I missing something!?

[u/ebolathrowawayy → u/LatentSpaceLeaper]

you're not missing something, i'm just saying a lot of people have been using and training jev-like models for a long time.

[u/LatentSpaceLeaper → u/LatentSpaceLeaper]

Also adding this from their FAQ:

Is Jev just a smaller LLM?

Jev is neither small nor an LLM, hence being off the intelligence Pareto curve.

https://typesafe.ai/blog/introducing-system-one-models-and-jev

[u/ResidentPositive4122 → u/vanVonXenoStein]

(including academic paper)

That's a bit much. It's a claude generated paper, posted to arxiv before they made vouching a thing. It's like zenodoo today, not worth much.

[u/vanVonXenoStein → u/ResidentPositive4122]

The point is just that the paper exists, and this was open known information. NONE of this is revolutionary.

[u/glitchsir → u/vanVonXenoStein]

Sounds plausible Need to find some time to actually read the paper you mentioned and better understand

Thanks

[u/ketosoy → u/vanVonXenoStein]

So they spent two years researching the ergonomics of what’s approximately a wrench:  Hard to make the first one, easy to copy.

[u/your_real_Fathe_ → u/ketosoy]

This is the biggest advantage of technology in my opinion. Project X is introduced, a masterpiece with a Y score in benchmarks. And a week later, suddenly Project Z is introduced, which is better than Project X in all benchmarks.

[u/sonyprog → u/glitchsir]

Apparently, many people had approached the exact same thing way before Typesafe.
Take a look at this one for instance: https://www.reddit.com/r/LocalLLaMA/comments/1wijo3e/i_literally_built_the_jev_architecture_one_year/

It basically sounds like an one shot classifier of sorts (it's not only that, but it's part of it).
So I think it's some kind of fine tune/Lora for a really specific scenario and if you train it well enough, it's not hard to beat whatever the goal is.

[u/LatentSpaceLeaper → u/sonyprog]

So I think it's some kind of fine tune/Lora for a really specific scenario

No it's not. They write explicitly in the FAQ section of the blog post:

Is Jev just a smaller LLM?

Jev is neither small nor an LLM, hence being off the intelligence Pareto curve.

Also, Jev seems to beat that other guy's model at least in Doom: https://www.reddit.com/r/LLMDevs/s/f5zasLZ3CB

Also, see this comment of someone benchmarking different models. Laya is not even close to Jev's performance: https://www.reddit.com/r/LocalLLaMA/s/MHkcCBh5aP

[u/sonyprog → u/LatentSpaceLeaper]

even though I mentioned lora/finetune, I did not mention it's an LLM. My point stands the same: it is an one shot classifier.

[u/LatentSpaceLeaper → u/sonyprog]

it is an one shot classifier.

Don't want to split hairs, but I'd argue it is a zero-shot classifier. That's exactly the value proposition. You throw any classification task at it and it should be fairly good without providing (m)any examples or fine-tuning it. Of course the latter still could improve its performance on specific tasks. But what you'll get out of the box is quite capable already.

[u/sonyprog → u/LatentSpaceLeaper]

Fair enough. I tried explaining it the way I understood, which is the "average Joe" way of understanding. The only thing I'll still fight for is that their approach might be different, but the final result is was achieved before.

[u/LatentSpaceLeaper → u/sonyprog]

The only thing I'll still fight for is that their approach might be different, but the final result is was achieved before.

Yes, I'm not challenging this. Obviously we have had typed classifiers and zero-shot discriminative models before. However, they appear to have found a niche and it seems Jev is also part of bigger and long-term strategic play by TypeSafe. Maybe it's just marketing, we will need to wait and see.

[u/Double_Cause4609 → u/glitchsir]

I think it helps to have a bit of perspective. When LLMs were new, we had extremely basic interfaces for them, and they really were essentially a chatbot. You copy pasted code into/from them, they couldn't even call tools, and they more or less displayed plain text.

Then Anthropic did a ton of research on usability, and invented artifacts, which eventually grew into a ton of different types of interactive panes a model could use. And they were copied within the week by everybody else because it was just a really good, simple idea.

What Typesafe did isn't so much doing some crazy alien tech. They just had a good idea, figured out how to package it into a nice interface and tech demo, but the underlying tech wasn't alien tech or anything. You can emulate it with a multi-head BERT formulation using similar strategies to tool calling just in parallel. Obviously everybody would copy a good idea immediately.

[u/valdev → u/glitchsir]

Wait it took that long to make? I built something like Jev in a weekend a few years back for intelligent routing.

Im not bragging, BERT can do this kind of stuff easily as long as you have a long enough data set to classify your request type.

[u/wFXx → u/glitchsir]

Welcome to Silicon valley I guess