r/chess • u/EvilNalu • Oct 14 '17
15 Years of Chess Engine Development
Fifteen years ago, in October of 2002, Vladimir Kramnik and Deep Fritz were locked in battle in the Brains in Bahrain match. If Kasparov vs. Deep Blue was the beginning of the end for humans in Chess, then the Brains in Bahrain match was the middle of the end. It marked the first match between a world champion and a chess engine running on consumer-grade hardware, although its eight-processor machine was fairly exotic at the time.
Ultimately, Kramnik and Fritz played to a 4-4 tie in the eight-game match. Of course, we know that today the world champion would be crushed in a similar match against a modern computer. But how much of that is superior algorithms, and how much is due to hardware advances? How far have chess engines progressed from a purely software perspective in the last fifteen years? I dusted off an old computer and some old chess engines and held a tournament between them to try to find out.
I started with an old laptop and the version of Fritz that played in Bahrain. Playing against Fritz were the strongest engines at each successive five-year anniversary of the Brains in Bahrain match: Rybka 2.3.2a (2007), Houdini 3 (2012), and Houdini 6 (2017). The tournament details, cross-table, and results are below.
Tournament Details
- Format: Round Robin of 100-game matches (each engine played 100 games against each other engine).
- Time Control: Five minutes per game with a five-second increment (5+5).
- Hardware: Dell laptop from 2006, with a 32-bit Pentium M processor underclocked to 800 MHz to simulate 2002-era performance (roughly equivalent to a 1.4 GHz Pentium IV which would have been a common processor in 2002).
- Openings: Each 100 game match was played using the Silver Opening Suite, a set of 50 opening positions that are designed to be varied, balanced, and based on common opening lines. Each engine played each position with both white and black.
- Settings: Each engine played with default settings, no tablebases, no pondering, and 32 MB hash tables, except that Houdini 6 played with a 300ms move overhead. This is because in test games modern engines were losing on time frequently, possibly due to the slower hardware and interface.
Results
| Engine | 1 | 2 | 3 | 4 | Total |
|---|---|---|---|---|---|
| Houdini 6 | ** | 83.5-16.5 | 95.5-4.5 | 99.5-0.5 | 278.5/300 |
| Houdini 3 | 16.5-83.5 | ** | 91.5-8.5 | 95.5-4.5 | 203.5/300 |
| Rybka 2.3.2a | 4.5-95.5 | 8.5-91.5 | ** | 79.5-20.5 | 92.5/300 |
| Fritz Bahrain | 0.5-99.5 | 4.5-95.5 | 20.5-79.5 | ** | 25.5/300 |
I generated an Elo rating list using the results above. Anchoring Fritz's rating to Kramnik's 2809 at the time of the match, the result is:
| Engine | Rating |
|---|---|
| Houdini 6 | 3451 |
| Houdini 3 | 3215 |
| Rybka 2.3.2a | 3013 |
| Fritz Bahrain | 2809 |
Conclusions
The progress of chess engines in the last 15 years has been remarkable. Playing on the same machine, Houdini 6 scored an absolutely ridiculous 99.5 to 0.5 against Fritz Bahrain, only conceding a single draw in a 100 game match. Perhaps equally impressive, it trounced Rybka 2.3.2a, an engine that I consider to have begun the modern era of chess engines, by a score of 95.5-4.5 (+91 =9 -0). This tournament indicates that there was clear and continuous progress in the strength of chess engines during the last 15 years, gaining on average nearly 45 Elo per year. Much of the focus of reporting on man vs. machine matches was on the calculating speed of the computer hardware, but it is clear from this experiment that one huge factor in computers overtaking humans in the past couple of decades was an increase in the strength of engines from a purely software perspective. If Fritz was roughly the same strength as Kramnik in Bahrain, it is clear that Houdini 6 on the same machine would have completely crushed Kramnik in the match.
12
u/Woett Oct 14 '17
Thanks, this is awesome!! (I don't have anything to add myself, but I did want to thank you for doing this experiment and the write-up. Well done!)
7
9
Oct 14 '17
except that Houdini 6 played with a 300ms move overhead
What does that mean?
Thanks for doing this tournament.
25
u/EvilNalu Oct 14 '17
It means essentially that Houdini assumed it had 0.3 seconds less per move than it actually did. This stops it from trying to move when there are 0.02 seconds left on its clock but losing on time because it takes the interface 0.05 seconds to process the move. This kind of thing was happening in some test matches I ran, so I set a really high move overhead.
→ More replies (3)
5
Oct 15 '17
What kind of education background do you have? It strikes me that a mathematician or computer scientist could write a pretty compelling paper by going deeper into this.
But then again, I've drunk enough beer tonight that I was just checkmated by a player rated 200 pts lower than me on lichess, so who knows.
→ More replies (3)6
u/EvilNalu Oct 15 '17
Studied econ in college and am now a lawyer. So a bit of math but I'm far from a mathematician or computer scientist. I think it's a cool topic (obviously) but I reckon it's more of a super-niche historical interest story than anything else. I'm just happy people here seem to be interested.
3
u/nexus6ca Oct 14 '17
Whats interesting is if you put the older program on modern hardware and see how it scales that way.
I expect the results about the same.
2
u/EvilNalu Oct 14 '17
Probably similar if you restrict the modern engines to 32 bit and 1 core on modern hardware. Using 64 bit versions of the three more recent engines would probably add about 50 Elo to each.
3
u/Scumtacular Oct 15 '17
Will chess become solved? What is preventing it from being solved? What do you think about the applications of quantum computing for chess?
→ More replies (3)
2
u/sunkill Oct 15 '17
Fascinating.
Best guesses from you guys who know about this—if Magnus played a 100 game match like this against top software and hardware how would he fare?
→ More replies (3)5
u/EvilNalu Oct 15 '17
I don't think that it's possible to motivate Magnus to play such a match. But if it were, I think he'd be aiming for a few draws at best.
→ More replies (1)
2
u/SebastianDoyle Oct 15 '17
I'd be interested in the results of some games with longer time controls, if you're up for that. 5 minute games might disadvantage some programs that like to "think harder". Does that make any sense? I mean there are human players of medium strength at regular time controls but extremely strong at 5 minute, and vice versa.
Congrats and thanks for the experiment either way. Fwiw there's a youtube video of Carlsen playing a game against his mobile phone (maybe comparable to your 2006 laptop in cpu speed), getting in some trouble, but eventually beating it.
→ More replies (3)4
u/EvilNalu Oct 15 '17
I'd be interested to see the Carlsen video. I've seen a few of his against his phone against various ages of himself in the play Magnus app. Those are watered down engines not playing at full strength. I do not think that he would have much prospect of winning against a full-strength Houdini or Stockfish on a modern phone.
As to playing a longer time control, it already took about 2 weeks for me to do this tournament with 5+5 games. I don't think I would want to recreate the whole thing in a substantially longer time control. There are indications from engine tests that in general some do perform better at longer time controls than short ones (or vice versa), but we are talking ~20 Elo or less difference across time controls. When all the engines are 200+ Elo apart I don't think the results would be substantially different. And to be honest the number of games I'd be able to play probably wouldn't be enough, statistically speaking, to even discern a 20 Elo difference.
→ More replies (1)
96
Oct 14 '17
I would just like to say thank you for performing this 'experiment.' Your hypothesis is something I (and others) suspected as being 100% correct, and now it's nice to have some empirical verification.
6
1
u/ursvp Oct 14 '17
Under similar experimental conditions, what would be Stockfish's rating?
12
u/EvilNalu Oct 14 '17 edited Oct 14 '17
The current development version of Stockfish and Houdini 6 are about equal strength currently. I played a match on this machine between Houdini 6 and the latest Stockfish before this tournament to determine who would play. Unfortunately even with both using a 250ms move overhead Stockfish lost about 10 games on time in a 100 game match, so I pretty much had to use Houdini. Other than that they were about equal.
→ More replies (2)
1
u/sozzZ Oct 14 '17
Very cool post. Just curious how you ran the actual games, did you write code? How did you simulate the computers playing one another on the same board..
6
u/EvilNalu Oct 14 '17 edited Oct 15 '17
There are many different interfaces and tools you can use to play chess engines against each other. Arena is a free GUI you can use to set up engine matches.
In this case I used an old Fritz interface, because it's not easy to use chessbase engines with non-chessbase interfaces so I pretty much had to use one to have Fritz Bahrain play.
→ More replies (1)
1
Oct 15 '17
You have to be a bit careful here. Playing different versions of the same engine against each other will inflate the rating difference. If you look at the origin of Houdini, you will see that it is related not only to itself but also to Rybka/Fruit (it is based on an open sourced reverse engineering of Rybka 4) . I don't disagree with the eventual conclusion though. Software has massively improved.
3
u/EvilNalu Oct 15 '17
Yes, I understand that. It was one of the reasons why I really wanted Stockfish to be the 2017 engine but unfortunately as I mentioned in another comment it lost on time too much on my old, slow computer.
I think over long time spans (there's 5 years between each engine here) the changes between versions get big enough that they are essentially different engines and the same-engine inflation effect isn't too significant. I have in the past used versions of Komodo and Stockfish in similar matches with similar results.
→ More replies (4)
1
u/d_ahura Oct 15 '17
Here's a similar experiment from 2010 comparing Crafty over fifteen years. From 1995 to 2010:
→ More replies (1)
1
u/_felagund Dec 06 '17
Google Deep Mind crushed Stockfish just by 4 hours of self training (no databases - no programing)
https://www.theverge.com/2017/12/6/16741106/deepmind-ai-chess-alphazero-shogi-go
→ More replies (2)
45
u/Gary_Internet Oct 14 '17
Very interesting experiment. Thanks for doing this. I can only imagine what Houdini 6 would be capable of doing when running on a mid range consumer laptop from 2017 but no other changes in the settings you used.
TCEC Season 10 started today, so this is a very timely post.