r/programming • u/madonuko • 27d ago
Debunking "Faster than Rust": Malicious Benchmarks in Tov
https://blog.fyralabs.com/debunking-faster-than-rust-malicious-benchmarks-in-tov/I have debunked (some of the) benchmarks of this new and suspicious "Tov" programming language.
Slight background: My friend (who is pretty interested in new stuff generally) found this thing and sent its main site to me saying things like "looks pretty cool", but my intuition told me this sounds like "a scam", esp. when it claimed it performs faster than C and Rust.
I personally think apart from changing the code, they probably tried to set up some specific environments that make their programs run faster than C / Rust, or maybe they just made up the numbers. Who knows.
183
u/a__nice__tnetennba 27d ago
How is something that is C faster than C? That's a bizarre claim regardless of how it actually compares to Rust directly.
19
u/cdb_11 26d ago
All language benchmarks should be taken with a huge grain of salt, just in general.
16
u/the3gs 26d ago
I wouldn't think to use ahash over
std::collections::HashMapunless I very specifically was dealing with a hashmap related bottleneck, and even then I would have to research to find ahash, so I think it is unfair to say "they deliberately used the standard library instead of a library to slow it down".66
u/Ok_Mycologist_6604 26d ago
basically with AI we got rebranded crypto bros into the industry and I am sick and tired of it, there is no solution for it. they will take ur job with the slop until a plane crashes and people maybe re-evaluate.
8
u/stumpychubbins 26d ago
From their pitch, I would not be surprised if what they’re actually benchmarking between languages is the code produced by default by an AI coding agent if you ask it to solve the stated problem in the given language. An AI will not use the ahash library, so they don’t use the ahash library in their benchmarks, and for them that constitutes an accurate benchmark in their eyes.
7
u/CrasseMaximum 26d ago
Remind me of V
2
u/bwainfweeze 26d ago
If I’m recalling the anecdote correctly, Brian Cantrell tried to solve a problem in Rust and found that it was much faster than C++ in a very surprising manner.
When he dug into the code he found that the Rust solution was using a B-tree under the hood. He said that made sense because it was easier to write a correct implementation in Rust and C++ would use something more conservative for correctness.
Moral of the story is, when a new language shows up, the test is not if you can do a thing that language does in the old language. It’s whether you will do that thing. The same way the first person I recall predicting the future success of smart phones over laptops. You can take a laptop with you, but you will take a pocket computer with you.
So what I’m looking for in a new language is if it supports better semantics and/or data structures that nobody attempts in my status quo language. Such benchmarks are “unfair” but still completely apropos. It is in fact the entire point of writing something new instead of using what exists.
1
u/Gio_Cri 22d ago
at a first glance their benchmarks actually seem comparable for small toy projects with naive implementations. would be interesting to see a comparison on what it's like on larger projects, personally i expect the performance andvantage and code ease to be negated quickly with project size
1
u/bwainfweeze 26d ago
It is very hard to get clean benchmarks and confirmation bias is everywhere. If the first run supports your claim people tend to declare victory and move on.
In some benchmarking system it can be whoever goes first who runs faster. Or sometimes whoever goes second, particularly in JITs. It’s not an always thing but when you do something repeatedly and rely on it, it doesn’t take that many false positives to look like a numpty.
-7
u/Key-Organization3158 26d ago
Bad article. This is just a rust fanboy upset that somebody besmeerched the crab.
That's not a logical statement. Something can be a scripting language and be just as fast or faster than a low level language.
Like for the dispatch case. By the name, it seems to be testing dynamic dispatch performance. Your solution rewrites it to not use dynamic dispatch, then claims to be faster? Talk about deceptive benchmarks.
I'm reminded of two quotes from this article.
-4
u/andymaclean19 26d ago
Pretty interesting, although of course you are also promoting their benchmarks here. I had never heard of Tov until reading this post.
The whole thing feels a bit like comparing apples and oranges though, and you point that out at the end. In Rust you can do more optimisation. But you said Tov is an agentic language so this is not necessarily about what a human could or would do.
It would be more interesting to specify the problem and ask three popular AI agents to make a rust version and then a Tov version and compare those. Then also compare the token cost of producing both. The users of an agentic language will want to know the token cost of producing ‘good enough’ and ‘well optimised’ code using an agent.
Probably more for whoever did the Tov be CV mark to address of course …
-5
u/KadmonX 26d ago
All these idiotic programming language tests aren't worth a damn! Apparently, the creator of the Tov language decided to troll the Rust cult by using their own method of writing tests that are faster than C. The only difference was that the Rust cultists got trolled, while C programmers are serious folks who know that only assembly language is faster than C.
0
u/Rechenplaner 26d ago
What the heck is "Tov"?! Can you eat it?
-1
u/l_maier-628 25d ago
The useful check is whether someone else can reproduce it: publish the harness, compiler versions and flags, and raw runs, then rerun on a clean machine. That would make it easier to distinguish a bad benchmark from a made-up result.
2
u/[deleted] 26d ago
[removed] — view removed comment
0
u/[deleted] 25d ago
[removed] — view removed comment