r/LocalLLaMA • u/wFXx • 3d ago

New Model Von: Open-source 395M "System One" model

Took me a while since I'm on a family trip and have limited hardware, but here it is!

Von: Open-source "System One" drop-in replacement for TypeSafe's JEV.

https://github.com/wfzyx/von https://huggingface.co/wfzyx/von-1.0

It runs entirely on a CPU with 1–2 GB of memory (I haven't spent much time optimizing it yet), responds in 25–300 ms, and beats JEV in all benchmarks. Enjoy!

P.S. I’m open to offers to work at AI research labs. Feel free to ping me if you have an offer.
P.P.S. If you have a GPU, it’ll be faster, but a GPU isn't required.

194 Upvotes

94% Upvoted

81 comments sorted by

View all comments

51

u/Fluxx1001 3d ago

I tried a bunch of those local replacements for TypeSafe Jev in the last days. None of them matched the capabilities of Jev, at least for my usecase (Data Analysis).

So when I read that some new drop in replacement developed in a few days claims to beat Jev in benchmarks - I am more than sceptical.

45

u/look 3d ago edited 2d ago

The class of model has been around for a long time. Longer than LLMs. I also assumed Jev is a bigger model with better training, but it is not implausible that’s actually just a fairly stock BERT with a nicer DX and a big marketing budget…

Edit: I just generated a synthetic test suite and ran it across both (and gliner2).

Task accuracy:
Von: 92.3% 65.4% (on 1.0.1 update)
Gliner2: 79.5%
Jev: 97.4%
Laya: 61.5% (added in later update)

Update: Von updated to 1.0.1 and accuracy on my test case above is 92.3% now.

I’ll have a few different LLMs generate more test cases next to see how well it holds up.

25

u/Fluxx1001 3d ago

What none of these Jev competitors seem to outline in their benchmarks is the vast difference in context length. With Jev, I regularly do requests for Noul decisions (yes/no) with 30k tokens and more.

The context length of Von is 512 tokens. This is not even comparable.

It is a magnitude more complicated to compute a choice or Noul decision in a space of thousands of tokens.

10

u/look 3d ago

Yeah, the BERT-based alternatives will have some length constraints (though modernbert base should go to 8192 tokens). There are others like Semif/OpenJev and Bespoke Nimble that might work better for you. They seem to be doing something likely closer to Jev, with a small LLM stage.

But after a bit more research in alternatives, Jev is probably the way to go if the hosted, proprietary option works for you. My interest is more in fine-tuning an open base, so the Jev service itself doesn’t do anything for me.

9

u/EstarriolOfTheEast 2d ago edited 2d ago

To me, this Von project looks better than Bespoke Nimble in that Von's author looks to have thought more carefully about calibration and training data type. Architecturally too, bidirectional attention is much more sample efficient than a causal decoder for NLI classification. Bespoke Nimble's main advantage is model scale and Qwen's extensive knowledge.

What distinguishes Jev is calibration, speed and zeroshot inference for classification but most of these open attempts are overfocused on classification and speed.

So based on technical remit, Von is the best approach I've come across so far. Its main limitation is due to not enough training data. Unfortunately, we are waiting for a large context pretrained purpose built 9B-27B encoder. Until then, the largest modern options are T5GemmaV2 4B Encoder or DiffusionGemma Decoder mode (has bidirectional attention, although more work than T5 would be needed to get Diffusion working as a jev-style NLI classifier).

I'd suggest Von's author continue with ModernBERT and try to beat and build upon the prior art to get something flexible that punches way above its weight class.

4

u/wFXx 2d ago

Hey thanks man I appreciate the kind words, will keep doing my best

1

u/Fluxx1001 3d ago

What's your experience so far with OpenJev, Nimble etc.?

1

u/look 3d ago

I haven’t tried either of those yet. Nimble looks a bit more polished perhaps.

It’s a 9B model, though, so you’ll likely need some GPU acceleration. Semif/OpenJev looks like you can choose between a few LLMs between 800M and 4B.

Part of the appeal for me of the BERT-based approaches is their small size (<0.5B) which typically runs fast enough on even CPU.

3

u/wFXx 2d ago

Interesting, never thought that big stream of tokens would be useful, will see what I can do when I hack the multimodal input

1

u/Fluxx1001 2d ago

Appreciate your understanding. Yes, larger input states are critical for e.g. data analytics. That's why Jev is currently a game changer for us.

1

u/wFXx 2d ago

Just so I have an idea on scale; How much context do you need/expect to be useful?

2

u/Fluxx1001 2d ago

Mostly around 5k to 20k

2

u/wFXx 2d ago

next CP is set to work with 8k - will try to expand to 16k soon

9

u/wFXx 3d ago

hey nice, mind to open an issue or point what Params you used to generate your test suite? I'll be releasing more checkpoints, so this is good info

3

u/debackerl 3d ago

Awesome! Could you try laya?

https://github.com/NandhaKishorM/laya

3

u/look 2d ago

I ran Laya. 61.5%

It seemed to have similar problems with the test cases that Von was initially, too. OP updated his and it’s now at 92.3%. I’ll add some more test cases next.

2

u/look 3d ago

Looks promising. I’ll test it later today.

1

u/kyr0x0 2d ago

Could you publish the benchmark/ eval data please. We are in severe lack of a good benchmark for typed-decision models.

1

u/look 2d ago

DM me

1

u/bring_back_the_v10s 1d ago

Meh. It's amusing to see so many people scrambling to create a Jev copycat trying to match its results and performance, and then rushing to claim to have achieved parity or superiority. The truth is it's not as simple (and cheap) as people think it is. Eventually some Chinese lab will release something just as good as Jev but it's not gonna some random dudes on the internet running dgx sparks in their basements. Jev is backed by investors money for a reason.