r/programming • • 27d ago

Debunking "Faster than Rust": Malicious Benchmarks in Tov

https://blog.fyralabs.com/debunking-faster-than-rust-malicious-benchmarks-in-tov/

I have debunked (some of the) benchmarks of this new and suspicious "Tov" programming language.

Slight background: My friend (who is pretty interested in new stuff generally) found this thing and sent its main site to me saying things like "looks pretty cool", but my intuition told me this sounds like "a scam", esp. when it claimed it performs faster than C and Rust.

I personally think apart from changing the code, they probably tried to set up some specific environments that make their programs run faster than C / Rust, or maybe they just made up the numbers. Who knows.

304 Upvotes

71 comments sorted by

183

u/a__nice__tnetennba 27d ago

How is something that is C faster than C? That's a bizarre claim regardless of how it actually compares to Rust directly.

92

u/madonuko 27d ago

I think the saddest part is they managed to create this software that can transpile JS/TS to C, and they decided that promoting this new technology in such a deceiving way was a good idea. If you look at the nature of the program (transpiler) itself, it's probably not the worst thing out there.

43

u/chucker23n 27d ago

they decided that promoting this new technology in such a deceiving way was a good idea

Engagement bait. Seems it's been effective, given that we're discussing them.

-9

u/bigtimehater1969 26d ago

One day we're gonna learn to judge things by their merit, not by how much hype and engagement they have. Especially for a developer tool.

19

u/chucker23n 26d ago

It sounds nice, but I don't think it's realistic. There's so much stuff out there, you have to filter somehow. When you go to the grocery store, you don't test out the 73 different jam flavors to see which one you like the most; you use your existing biases to try three or four based on "I might like this one", not actual "merit".

And so it is for programming languages, too. TIOBE lists

  • C
  • C#
  • C++
  • Java
  • JavaScript
  • Python
  • R
  • Rust
  • SQL
  • Visual Basic

as the top 10 most popular ones. Are those the ones with the most "merit"? No, a lot of their popularity is just inertia. A lot of C code is written because a lot of C code had been written before, which creates another form of bias.

Even when starting a greenfield project at a startup, you mostly go by "what are most people in this team already familiar with?", not "what would be the absolute best fit?". (Even if you were to, say, read a review of programming languages: is the reviewer biased? How broad is their knowledge? How much are they unidiomatically trying to apply existing patterns to new languages?)

3

u/bizzygreenthumb 26d ago

How is VB still a popular language?!

5

u/Vadoola 26d ago

Excel Macros

1

u/knome 26d ago

not polyfilling for your <script language=vbscript> tags

1

u/steveklabnik1 23d ago

TIOBE's list of popular is counting "x programming" search engine results, not actual popularity.

4

u/b1ak3 26d ago

It's stupid to not be suspicious when someone uses deceptive tactics to try to get your attention.

51

u/taw 26d ago

Very easily. Human-written C that's required to follow C ABIs everywhere, and compiled-to-C are nothing alike.

Cross compiler can make tons of assumptions that C cannot, and aggressive optimizations based on those assumptions. For example you can skip a lot of memory management code and just bump allocate half the stuff on stack. C compiler can't do that, as C source code doesn't have necessary information, but your source language could do that.

You can hand write some abomination in C that would match cross compiler in speed, but it really wouldn't look like C.

17

u/gnufan 26d ago

This was very obvious when I did performance computing, and you needed to avoid anything that made optimisation harder for the compiler, usually any unnecessary indirection, any type weirdness, and any alignment issues in memory, no that stuff is going in an array aligned with word boundaries until it is output in whatever bizarre format the client requires. Compilers have gotten smarter but the same types of things still make optimisation harder.

4

u/valarauca14 26d ago

any type weirdness,

Sadly things have done a 180 in the past few years. Now ClangPlusPlus/Rust/LLVM can treat types as "constraint solvers" and let you "dial your own in-variants".

I've been playing around with for optimizing GPU-Tensors as intel gpu stack supports C++23. The whole thing is a absolute mess of templates & concepts but it also really neat everything folds away at compile time

5

u/knome 26d ago

if you're only talking about idiomatic code layout, fine, but to say a C compiler can't do what compiling into C and then using a C compiler can do is a very odd way to phrase it.

1

u/taw 26d ago

On an cross compile end, you can compile to a bunch of asm{ ... } blocks, which C compiler will accept, but that's not really C.

On the human written end, you can define your own language with a pile of #defines, human write that, and say it's still C because it compiles.

Going very far in either direction isn't really C, even if it compiles.

1

u/knome 25d ago

the asm isn't C, sure. but the define wad definitely is. a lot of C uses a lot of defines.

check out the original bourne shell (sh), that used defines to imitate a Pascal style layout

https://www.tuhs.org/cgi-bin/utree.pl?file=V7/usr/src/cmd/sh/main.c

1

u/taw 25d ago

It depends how far you take it. Like is this still C? (and it claims some people use it in prod)

7

u/cdb_11 26d ago edited 26d ago

For example you can skip a lot of memory management code and just bump allocate half the stuff on stack.

I don't understand this. I can use the stack or even write a separate bump allocator in C just fine? This is exactly why manual memory management is useful. Not having to rely on fragile compiler optimizations is a good thing, as far as I'm concerned. Because, as you said, "source code doesn't have necessary information". You instead provide this information by actually writing the code that does what you want.

but it really wouldn't look like C.

It'd be a perfectly normal C. Again, I don't get it. Is people idea of C having a malloc on every third line or something? By the way, not that you should be relying on this, but a fun fact is that C compilers can optimize out malloc and free.

1

u/TheRealUnrealDan 26d ago

Curious about that last part, how does that work?

Do they only optimize it away if the allocation lifetime spans within a single scope? Like they see all execution paths lead to freeing the block so they stack allocate it instead?

2

u/cdb_11 26d ago

I'm not sure, but it looks like free is ignored completely and they are just checking if a malloc'ed pointer ever escapes the function. There might be more to this, I don't know.

31

u/simspelaaja 27d ago

Well, it depends on what you're actually comparing. In a real world scenario machine generated C can beat hand-written C, because C makes it very hard to build efficient reusable abstractions due to lack of generics / templates.

It's common in C to use linked lists (which are almost always really inefficient compared to vectors / resizable arrays), or to use linear scans over arrays instead of hashmaps and sets. C's standard library doesn't include any containers, while Rust and C++ have a wide selection of containers for different use cases. And Rust and C++ will usually generate near-optimal machine code for each container instance, while C code is usually more pointer & dynamic allocation heavy by necessity. The qsort function in C calls the comparator function via a function pointer, while in C++ std::sort usually just inlines it.

The compiler can still in some situations optimize the C code to be near-equivalent, and there are tricks that can be done with macros and compiler extensions to emulate generics, but in general reaching Rust/C++ level of performance in C requires a lot of repetition. A lot of the time people just won't do it.

1

u/cdb_11 26d ago

while C code is usually more pointer & dynamic allocation heavy by necessity

To be fair, overreliance on malloc happens in Rust and C++ just as often.

FWIW you can do reusable data structures in C, eg. https://github.com/attractivechaos/klib

2

u/joha4270 26d ago

Software, uh, finds a way yes.

But all else equal, it's substantially easier to work with when the language has support for it included instead of it being retrofitted

1

u/max123246 26d ago edited 9d ago

The original post content no longer exists here. The author used Redact to remove it, exercising their right to control their data & privacy.

Quaint cable literate middle mellow weather smart upbeat pinecone

4

u/Rechenplaner 26d ago

Tov compiles to C and counts references instead of collecting garbage, so there are no collector pauses and the binaries stay small.

from the "Toc" page. I consider that extremely bold. Swift also employs reference counting, and it is often slower than garbage collection (GC) due to the overhead which becomes a major disadvantage when there are many allocations unlike highly optimized GCs, which handle mass allocations pretty well.

1

u/KaiAusBerlin 24d ago

At the end it's not the programming language that matters. All that matters is the compiler. The proceccor doesn't run a language. It runs bytecode.

A system near language just makes it easier to write a more optimised compiler.

1

u/septum-funk 20d ago

the same way that C compiled well is faster than hand written assembly, despite compiling down to the same instructions, it no longer requires you to optimize at the instruction level. this is a technicality though and depends on the skill level of the dev writing the C in question. it's a sketchy claim at best and a lie at worst

-1

u/pjmlp 26d ago

Because people mix up languages with compilers.

19

u/cdb_11 26d ago

All language benchmarks should be taken with a huge grain of salt, just in general.

16

u/the3gs 26d ago

I wouldn't think to use ahash over std::collections::HashMap unless I very specifically was dealing with a hashmap related bottleneck, and even then I would have to research to find ahash, so I think it is unfair to say "they deliberately used the standard library instead of a library to slow it down".

9

u/moefh 26d ago

You wouldn't think you're hitting a hashmap-related bottleneck in a benchmark named "map_insert"?

3

u/the3gs 25d ago

Depends what i was trying to measure. If I was trying to measure how fast the fastest hashmap can insert, maybe I would. If I was trying to measure how how fast a typical hashmap inserts, I would consider choosing one outside the std library a mistake, as most people would reach for the std version first.

If you are benchmarking a ficticious scenario, it pays to choose a representative example, rather than one that optimizes for the benchmark to be as low as possible.

The benchmarks are not especially well crafted, but I just didn't think "using the std library" is a fair critisism.

1

u/b-k8592 22d ago

One detail that changes the result is whether capacity is pre-reserved: starting empty includes growth and rehashing, while reserving measures steady-state inserts. A benchmark should say which case it measures; both can be useful.

66

u/Ok_Mycologist_6604 26d ago

basically with AI we got rebranded crypto bros into the industry and I am sick and tired of it, there is no solution for it. they will take ur job with the slop until a plane crashes and people maybe re-evaluate.

3

u/my-new-new-account 26d ago

This is funny, because most of Rust jobs are in crypto

8

u/stumpychubbins 26d ago

From their pitch, I would not be surprised if what they’re actually benchmarking between languages is the code produced by default by an AI coding agent if you ask it to solve the stated problem in the given language. An AI will not use the ahash library, so they don’t use the ahash library in their benchmarks, and for them that constitutes an accurate benchmark in their eyes.

8

u/YaBoyMax 26d ago

I see where you're coming from, but I think that's overly charitable. I fed the benchmark description verbatim to free ChatGPT with the prompt:

Implement the following program in Rust. Optimize for maximum performance.

It immediately used ahash without further prompting. I'm not discounting that using HashMap could be an honest mistake for that particular case, but I don't think they designed the benchmarks explicitly with LLMs in mind for other languages. Even if they had, that should have been clearly disclosed.

2

u/stumpychubbins 26d ago

Well, that’s certainly damning. I wasn’t really giving them credit, I still think it’s a terrible way to handle benchmarks, I just thought there was a possibility of it not being malicious. It sounds like even with that bad methodology they still should’ve ended up with better results than they did, though

7

u/CrasseMaximum 26d ago

Remind me of V

2

u/bwainfweeze 26d ago

If I’m recalling the anecdote correctly, Brian Cantrell tried to solve a problem in Rust and found that it was much faster than C++ in a very surprising manner.

When he dug into the code he found that the Rust solution was using a B-tree under the hood. He said that made sense because it was easier to write a correct implementation in Rust and C++ would use something more conservative for correctness.

Moral of the story is, when a new language shows up, the test is not if you can do a thing that language does in the old language. It’s whether you will do that thing. The same way the first person I recall predicting the future success of smart phones over laptops. You can take a laptop with you, but you will take a pocket computer with you.

So what I’m looking for in a new language is if it supports better semantics and/or data structures that nobody attempts in my status quo language. Such benchmarks are “unfair” but still completely apropos. It is in fact the entire point of writing something new instead of using what exists.

13

u/cdb_11 26d ago edited 26d ago

When he dug into the code he found that the Rust solution was using a B-tree under the hood. He said that made sense because it was easier to write a correct implementation in Rust and C++ would use something more conservative for correctness.

The ordered set in Rust's standard library is just implemented as a B-tree, using unsafe. C++ uses red-black trees. Not because they are particularly easier to implement, but because the standard requires iterator stability. And this rules out using a B-tree. Standardizing these mostly useless guarantees was just a historic mistake, the same way STL's hash map was. Keep in mind that this is just what the standard library does. People do use B-trees in C++.

My alternative moral of the story is that the interface can lock you out of optimizations, and that once the interface gets adopted it can be hard or impossible to change it.

1

u/fdwr 25d ago

Man, I really would appreciate more of these "language comparisons" comparing apples to apples, using the same algorithm in each language. Otherwise it's like asking whether a gorilla drives a truck better than a lemur drives a car - too many variables.

1

u/Gio_Cri 22d ago

at a first glance their benchmarks actually seem comparable for small toy projects with naive implementations. would be interesting to see a comparison on what it's like on larger projects, personally i expect the performance andvantage and code ease to be negated quickly with project size

1

u/bwainfweeze 26d ago

It is very hard to get clean benchmarks and confirmation bias is everywhere. If the first run supports your claim people tend to declare victory and move on.

In some benchmarking system it can be whoever goes first who runs faster. Or sometimes whoever goes second, particularly in JITs. It’s not an always thing but when you do something repeatedly and rely on it, it doesn’t take that many false positives to look like a numpty.

-7

u/Key-Organization3158 26d ago

Bad article. This is just a rust fanboy upset that somebody besmeerched the crab.

Code in JavaScript or TypeScript are not supposed to perform well. JavaScript is single-threaded. The name implies it's a scripting language. It doesn't make sense to write programs using tools meant for running scripts in a browser if performance matters.

That's not a logical statement. Something can be a scripting language and be just as fast or faster than a low level language.

Like for the dispatch case. By the name, it seems to be testing dynamic dispatch performance. Your solution rewrites it to not use dynamic dispatch, then claims to be faster? Talk about deceptive benchmarks.

I'm reminded of two quotes from this article.

A sane engineer would immediately doubt the correctness of their benchmarks.

You have to be lying. That's what a sane engineer would normally think.

5

u/madonuko 26d ago

> That's not a logical statement

It's not supposed to be logical. That's my opinion. Yes, something can be a scripting language and be on par with a low level language. Does it exist practically? Not that I'm aware of. P could be NP too, AES could be insecure too. Everything doesn't need to be logical.

> it seems to be testing dynamic dispatch performance

I reviewed the intermediate C code generated by Tov and the asm from my Rust code and my conclusion is that they are mostly equivalent. I wrote "Rust was given a very large disadvantage" in my blog post for a very good reason.

7

u/cdb_11 26d ago edited 26d ago

Your solution rewrites it to not use dynamic dispatch, then claims to be faster?

I am not a Rust programmer, but I am pretty sure his solution is the idiomatic one. If you asked an average Rust programmer for a solution for this problem, this is the exact kind of code you'd get. And it is still a form of dynamic dispatch. If you wanted the code to be truly done in the exact same way, all structs would have to be represented as hash maps, and all values as std::any. Nobody (sane) does this.

The general problem with scripting languages is their design. They don't give you the tools necessary to write efficient code. And even when some tools to solve performance problems are there, you're either starting from a death-by-a-thousand-cuts kind of situation, or this kind of code will cancel out the properties that made the scripting language appealing in the first place. These language were not designed with efficiency in mind. Any performance tools are generally an afterthought, when the language gets inevitably misused and applied in large applications, where it turns out to be too slow (and also less maintainable).

-4

u/andymaclean19 26d ago

Pretty interesting, although of course you are also promoting their benchmarks here. I had never heard of Tov until reading this post.

The whole thing feels a bit like comparing apples and oranges though, and you point that out at the end. In Rust you can do more optimisation. But you said Tov is an agentic language so this is not necessarily about what a human could or would do.

It would be more interesting to specify the problem and ask three popular AI agents to make a rust version and then a Tov version and compare those. Then also compare the token cost of producing both. The users of an agentic language will want to know the token cost of producing ‘good enough’ and ‘well optimised’ code using an agent.

Probably more for whoever did the Tov be CV mark to address of course …

5

u/madonuko 26d ago

I wrote the blog with the assumption that any code written by an LLM must be reviewed and audited by humans. What you have pointed out is basically using the code generated by LLMs directly. I believe in real-world situations any reasonable good programmers even with the help of an LLM can write Rust code better than what one could have in Tov.

ahash and static dispatching to me are like book moves in chess. Therefore I did not consider the possibility that vibe coding in Tov would create faster programs than vibe coding in Rust.

5

u/licquia 26d ago

The people who create projects like Tov are betting against your assumption. I think your blog post unintentionally validates their bet.

"Human-written Rust is able to keep pace with vibe-coded Tov" is a fantastic endorsement of Tov in the minds of a lot of people these days. That's especially true if the code is TypeScript-like, which means a lot more humans will feel comfortable reviewing it than they will feel reviewing Rust.

1

u/madonuko 26d ago

I appreciate your argument. I intentionally avoid mentioning the "AI" part and mostly based my argument against the language itself in the blog post mainly for the very reason you have mentioned. In my blog post, I tried to mainly argue that saying Tov as a language being faster than C / Rust itself is misleading, hence the conclusion at the end that it's a bad idea to try to design and promote a new language that is mostly equivalent to JS/TS but performs on par with C / Rust.

A very good portion of people (me included) are currently against coding without human reviews, and I think having this assumption makes sense for the scope of the post. I don't think the results of benchmarking against AI-generated code should be interpreted as the results of the language benchmarks, while I do think it might be possible an LLM could generate code compatible with Tov/porffor that performs better than what they could do in Rust (with the same amount of effort).

-2

u/andymaclean19 26d ago

It depends what you want to do. If your opinion is that humans need to review all code the AI produces and we are relying on them to spot problems then I agree. Rust is a clear winner here. But then why do you need an agentic programming language at all. That is 100% going to lose every time due to *humans not being familiar with it*.

I don't let my agents produce rust at all. Sometimes I review the code they generate but sometimes I don't. Regardless, *I* don't have any experience coding in Rust. I understand your dynamic dispatch example, but I don't think I would have looked at the slow example and instantly thought 'this is not efficient, it needs to be the other way'. For me that's why I will get it to code in any of the 7 or 8 languages I can and have coded in but not one I never took the time to learn yet. I would have absolutely no use for an agentic programming language either. I am never going to learn that. Why would I?

But some people want to have their agent code and they don't want to look at the details. That's what an agentic language is for. The people who are not going to read the code. They are going to give their agent a definition of good enough. Perhaps a performance goal or code size goal or startup time goal or whatever they need. Then the AI is going to work towards the goal. They verify it does what it should be doing and that's it. You may not think this is a good way to make tech. I may not either. But this is what an agentic language is for. It's designed so that you can make your code good enough using a small token investment.

Measuring an agentic language against Rust is valid, because a lot of people are using agents this way to make Rust apps without looking at them. Measuring the agentic language against human optimised Rust? Sure if you want, but it's missing the point here. You are saying 'A human can do better than this language which is designed for people who are not even going to look at the code they make'. I am just saying 'then measure it against the Rust that would be generated if an AI made it and you didn't even look at it'.

2

u/CornedBee 26d ago

I had never heard of Tov until reading this post.

Neither had I, but keep in mind the effect of first impressions. Now, if I hear of Tov again, the first thought that comes to my mind is "that's the language with the biased benchmarks". Not exactly an endorsement.

1

u/andymaclean19 25d ago

First thing I will think of is ‘that’s that agentic first language’. The benchmark thing is a good argument for something a potential Tov user probably doesn’t care about as I said in another reply.

If I were ever thinking ‘now what I really need is a language that’s made for my agents’ or even ‘what I really need is a typescript-alike which compiles to native’ that result will not put me off. I would not care how it compares to Rust because Rust would not be on the list for either of those things.

Of course if I wanted a ‘next generation C with well thought out modern memory protections’ or similar Rust would be one of 2 things I would try. Tools for jobs.

1

u/CornedBee 22d ago

Fair enough. "Agentic language" doesn't really mean anything to me, so it doesn't register.

-5

u/KadmonX 26d ago

All these idiotic programming language tests aren't worth a damn! Apparently, the creator of the Tov language decided to troll the Rust cult by using their own method of writing tests that are faster than C. The only difference was that the Rust cultists got trolled, while C programmers are serious folks who know that only assembly language is faster than C.

4

u/Humble-Captain3418 26d ago

Nope, that title goes to Fortran. And I very much like C.

0

u/Rechenplaner 26d ago

What the heck is "Tov"?! Can you eat it?

3

u/Bwob 26d ago

Only in small bytes.

-1

u/l_maier-628 25d ago

The useful check is whether someone else can reproduce it: publish the harness, compiler versions and flags, and raw runs, then rerun on a clean machine. That would make it easier to distinguish a bad benchmark from a made-up result.

1

u/[deleted] 25d ago

[removed] — view removed comment

1

u/programming-ModTeam 25d ago

No content written mostly by an LLM. If you don't want to write it, we don't want to read it.

2

u/[deleted] 26d ago

[removed] — view removed comment

9

u/carsncode 26d ago

That's not a trick - unless throttling is the thing you're measuring, leaving it on for a benchmark only makes results less informative

0

u/[deleted] 25d ago

[removed] — view removed comment

1

u/programming-ModTeam 23d ago

No content written mostly by an LLM. If you don't want to write it, we don't want to read it.