The Week in Chess

Thursday, May 2, 2013

Stockfish 3 Killed While Sleeping On Threads

Stockfish 3 was released on April 30, 2013 and I immediately got to work to prepare it for my computer chess engines rating list publication. I got the surprise of my life when the early results showed similar problem I have with Stockfish 2.3.1 a few weeks ago when ran with multiple instances of simultaneous tournaments.  Stockfish 2.3.1 was convincingly crushed by Fire and Firenzina which falsely led me to believe that it is not the most powerful open source chess engine.

And here is the final result of the initial test:

Stockfish 3 Tests
RankEngineScoreStStStS-B
1Stockfish 130429 x64 m10i pgo87.5/100· ·· ·· ··37-3-1043-2-5 2726.50 
2Stockfish 3 x64 ja 33.5/1003-37-10· ·· ·· ··13-12-25 1439.50 
3Stockfish 3 x64 m10i pgo 29.0/1002-43-512-13-25· ·· ·· ·· 1214.50 


150 games played / Tournament finished

Tournament start: 2013.05.01, 15:59:15
Latest update: 2013.05.02, 17:39:58
Level: Blitz 0:10/1
Hardware: AMD Phenom(tm) II X4 945 Processor 3000 MHz with 4 GB Memory
Operating system: Microsoft Windows XP 64 Bit Professional Service Pack 2 (Build 3790)
Table created with: Arena 3.0

Notes:

1. Stockfish 130429 x64 m101 pgo was my last compilation based on release dated April 29, 2013, a day before Stockfish 3 was officially released.

2. Stockfish 3 x64 ja was the official release compiled by +Jim Ablett.

3. Stockfish 3 x64 m10i pgo was my other compilation created to compare with the official release.

This was a 50 round robin tournament and the result showed that  Stockfish 130429 won by a lopsided score.  Both Stockfish 3 versions were at the bottom where it only managed to score 5 wins combined against the pre-release version.

I was curious why it behaved that way and wanted to know what changed on the day of the release.  So, I fired the diff command on PclinuxOS and got the following output:

diff /v/hda/src29/misc.cpp /v/hda/src30/misc.cpp
36c36
< static const string Version = "";
---
> static const string Version = "3";
diff /v/hda/src29/uci.cpp /v/hda/src30/uci.cpp
30d29
< #include "tt.h"
106c105
<       else if (token == "ucinewgame") TT.clear();
---
>       else if (token == "ucinewgame") { /* Avoid returning "Unknown command" */ }
diff /v/hda/src29/ucioption.cpp /v/hda/src30/ucioption.cpp
78c78
<   o["Use Sleeping Threads"]        = Option(true);
---
>   o["Use Sleeping Threads"]        = Option(false);



The src29 was the one released on April 29 and src30 was version 3 released on April 30, 2013. Not sure what this thing was all about because this was my first serious use of the diff command, I also compared the official Stockfish 2.3.1 release against the Stockfish 2.3.1 base which I happened to see and downloaded yesterday from github.com/mcostalba.  Almost the same output was produced, so I had the hunch that somewhere in this output lies the culprit.  The nearest thing that came to my mind was "threads" which I mentioned in the earlier posts about the Stockfish 2.3.1 being a moron. Without hesitation, I opened up the ucioption.cpp file and changed the Option(false) to Option(true) in the "Use Sleeping Threads" option and then re-compiled the Stockfish 3 source.

Here is the result with the changed option:

Stockfish 3 - Threads Test
RankEngineScoreSt
1Stockfish 3 x64 m10it pgo29.0/3028-0-2
2Stockfish 3 x64 m10i pgo 1.0/30· ·· ··


30 games played / Tournament finished

Tournament start: 2013.05.02, 21:10:27
Latest update: 2013.05.02, 22:17:39
Level: Blitz 0:10/1
Hardware: AMD Phenom(tm) II X4 945 Processor 3000 MHz with 4 GB Memory
Operating system: Microsoft Windows XP 64 Bit Professional Service Pack 2 (Build 3790)
Table created with: Arena 3.0
Notes:

1. Stockfish 3 x64 m10it pgo was the one with changed sleeping threads option to true.

2. Stockfish 3 x64 m10i pgo without the 't' was compiled with the unchanged source.

From the match result where the original compilation did not score a single win out of 30 games, it is very clear that the single change of the "use sleeping threads" option from false to true was the solution to the mystery of Stockfish being killed in tournaments where there are many threads running simultaneously.

Searching for reasons in the Internet on why there was a ritual of changing at the last moment of official release regarding sleeping threads, I found this link from Stockfishchess blog dated January 1, 2011:

Added new option “Use Sleeping Threads” that modifies how threads are managed during multi CPU searches. Setting to ON should greatly help on Hyper Thread enabled machines and could help also for other SMP cases. It is off by default because is not tested a lot (also because we don’t have big hardware) and results depend on the specific machine. So to be on the safe side we release 2.0 with the option disabled (here specific tests by users would be very welcomed and perhaps there could be some nice surprise especially on powerful hardware). 

Perhaps, it is now time to review the above statement and also for Stockfish team to release a version of Stockfish 3 that will address the problem I described on this post.

One thorn has been extracted and now I could use my computer resources in full without tending Stockfish for cases that would give it abnormally very low score in my tournaments.

For those who are curious, I provided the download links below for the games and the fix to the sleeping threads issue.

Download the "Stockfish 3 x64 m10i pgo.exe" that fix the problem here.

Download the Stockfish 3 test matches here.

2 comments:

  1. You can toggle sleeping threads from the GUI, it is an UCI parameter.

    ReplyDelete
  2. Marco, thank you very much for your response. I become aware of this option after realizing that I changed a parameter that is visible from a GUI that is UCI compliant.

    I am less interested in the exposed GUI parameters because changing any one of them could alter the performance of a chess engine which will distort my rating list, unless it could fix a problem that prevents a chess engine from executing properly.

    Your Stockfish is progressing steadily. Maybe one day, it could be number 1. Keep up the good work.

    ReplyDelete

Chessdom News