2020/12/05 Huge experimental test (3x 7000 games) of the Eman 6.60 learning-feature.
I was curious, if (and how many) Eman 6.60 will gain by using it's learning-function. Eman 6.60 writes an experience-file, when he is playing. So I did 3 7000 games testruns, starting with no experience and then let Eman learn and learn. Each of the 3 testruns were 100% identical to the others, expcept, that Eman was allowed to learn and to keep the Eman.exp-file for the next testrun. All conditions like a normal Stockfish-testrun (see main-site): 3'+1'', singlecore, Hash 256MB, no ponder, 500 HERT openings.
As you can see below, the results are a complete disappointment. The second testrun (using experience-file of the first testrun) gave +4 Elo, the third testrun (using experience-file of the first and the second testrun) gave no progress at all, even though the Eman.exp-file size was 72 Megabyte, after the third testrun was finished...
Download the 21000 test-games here
Program Elo + - Games Score Av.Op. Draws
1 CFish 12 3xCerebellum : 3726 9 9 7000 86.1 % 3389 27.3 % 2 Stockfish 201115 avx2 : 3724 8 8 7000 78.3 % 3473 41.9 % 3 Stockfish 201126 avx2 : 3722 8 8 7000 78.1 % 3473 42.6 % 4 Eman 6.60 avx2 3rd_run : 3716 8 8 7000 77.6 % 3473 43.4 % 5 Eman 6.60 avx2 2nd_run : 3716 8 8 7000 77.6 % 3473 43.0 % 6 Eman 6.60 avx2 : 3712 8 8 7000 77.3 % 3473 43.1 % 7 CFish 12 avx2 : 3703 8 8 7000 84.6 % 3389 29.1 % 8 Stockfish 12 200902 : 3684 4 4 25000 74.1 % 3470 45.4 %
*** rest of the ratinglist deleted ***
Individual statistics:
Eman 6.60 avx2 3rd_run: 3716 7000 (+3914,=3035,- 51), 77.6 %
RubiChess 1.9dev nnue : 1000 (+803,=195,- 2), 90.0 % KomodoDragon 1.0 avx2 : 1000 (+227,=758,- 15), 60.6 % Slow Chess 2.4 popc : 1000 (+726,=274,- 0), 86.3 % Ethereal 12.75 avx2 : 1000 (+706,=291,- 3), 85.2 % Nemorino 6.00 avx2 : 1000 (+634,=363,- 3), 81.5 % Stockfish 12 200902 : 1000 (+123,=850,- 27), 54.8 % Houdini 6 pext : 1000 (+695,=304,- 1), 84.7 % ********************************************************** Eman 6.60 avx2 2nd_run: 3716 7000 (+3927,=3007,- 66), 77.6 %
RubiChess 1.9dev nnue : 1000 (+802,=196,- 2), 90.0 % KomodoDragon 1.0 avx2 : 1000 (+202,=782,- 16), 59.3 % Slow Chess 2.4 popc : 1000 (+743,=254,- 3), 87.0 % Ethereal 12.75 avx2 : 1000 (+712,=285,- 3), 85.5 % Nemorino 6.00 avx2 : 1000 (+645,=353,- 2), 82.2 % Stockfish 12 200902 : 1000 (+110,=854,- 36), 53.7 % Houdini 6 pext : 1000 (+713,=283,- 4), 85.5 % ********************************************************** Eman 6.60 avx2 : 3712 7000 (+3900,=3018,- 82), 77.3 %
RubiChess 1.9dev nnue : 1000 (+803,=197,- 0), 90.2 % KomodoDragon 1.0 avx2 : 1000 (+205,=770,- 25), 59.0 % Slow Chess 2.4 popc : 1000 (+740,=254,- 6), 86.7 % Ethereal 12.75 avx2 : 1000 (+718,=277,- 5), 85.7 % Nemorino 6.00 avx2 : 1000 (+620,=378,- 2), 80.9 % Stockfish 12 200902 : 1000 (+113,=849,- 38), 53.8 % Houdini 6 pext : 1000 (+701,=293,- 6), 84.8 % ********************************************************** |