Find good MM tenuki threshold

[lemonsqueeze]

Right now for MM patterns used in tree search we do one pass looking for local patterns, and if nothing big matches another one ignoring distance this time to find good tenuki moves. The LOW_PATTERN_RATING threshold was chosen low enough that patterns' prediction rate (t-predict) isn't affected but there's probably plenty of room for increasing it. Looking at Pachi nodcnn games with human players it often tends to play too much locally, continuing even though it's hopeless (opponent breaking through with tons of cutting points ...)

If someone has computing resources to spare, play-testing with different LOW_PATTERN_RATING values might yield better style / winrate. Could use CLOP games to find the best value too.

#define LOW_PATTERN_RATING 6.0

floating_t
pattern_rate_moves(struct pattern_config *pc, struct board *b, enum stone color,
                   struct pattern *pats, floating_t *probs, struct ownermap *ownermap)
{
        /* Try local moves first. */
        floating_t max = pattern_max_rating(pc, b, color, pats, probs, ownermap, true);

        /* Nothing big matches ? Try again ignoring distance so we get good tenuki moves. */
        if (max < LOW_PATTERN_RATING)
                max = pattern_max_rating(pc, b, color, pats, probs, ownermap, false);

        return rescale_probs(b, probs, max);
}

[sthalik]

If someone has computing resources to spare, play-testing with different LOW_PATTERN_RATING values might yield better style / winrate. Could use CLOP games to find the best value too.

How much resources over how long a period of time?

[lemonsqueeze]

How much resources over how long a period of time?

Say 200 19x19 games against hira for different threshold values, should give a good idea. Can shortcut if the games are obviously worse. This is pachi nodcnn with 5k playouts so very fast, gpu not needed. Can run them in parallel, one on each core single threaded if there's enough ram for all the hiras. I'd begun testing but it takes forever on raspberry pi =) Probably a day or two on a fast machine.

[pnprog]

Hi!
I volunteer for that. my computer is old, but it is not used during the day, so it can be put to better use.

The restriction I have is that I will be using the CPU version of Hira (Hiratuka10_37B), not the CUDA one, under wine.

So if my understanding is good, I need to compile Pachi for different values of LOW_PATTERN_RATING, then have each of those versions play 200 matches against Hira and report the win rates?

If so:

• Could you provide the list of values to test? I will test progressively, and report results for each values days after days.
• What command line parameters should I use for Pachi and Hira?
• Any GTP commands to use for both?
• Komi?

[lemonsqueeze]

Hi,
Thanks for volunteering,

No problem, cpu version is fine. It's possible to build hira from source also to avoid wine.

So if my understanding is good, I need to compile Pachi for different values of LOW_PATTERN_RATING, then have each of those versions play 200 matches against Hira and report the win rates?

Yes, basically that's it.

You can use the t-play framework in Pachi to manage the games if you like, but there are other options like CLOP. Not sure what they use for leela-zero these days but probably some nice tools there as well.

For t-play, README should be a good start: Basically setup a build directory with different pachi versions, edit the rc file:

pachi_opts="--nodcnn -t =5000 max_tree_size=40"

player pachi            "runpachi ~/build/pachi           $pachi_opts" 0
player pachi_tenuki8    "runpachi ~/build/pachi_tenuki10  $pachi_opts" 0

pairing 19  h4 2  hira_1k pachi
#pairing 19  h4 2  hira_1k pachi_tenuki10

So 4 stones handicap games between hira 1kyu level and pachi nodcnn (5 stones maybe better actually)

For LOW_PATTERN_RATING values try big changes first.
I have an example where it needs 14 to tenuki well, so try maybe 10, 15, 20.

Good luck !

[pnprog]

It's possible to build hira from source also to avoid wine.

Wait, what? isn't Hira closed source software? or Aren't we talking about the same bot? I thought is was this Hira: https://senseis.xmp.net/?HiraBot
But then, where can I find the source code?

so try maybe 10, 15, 20
Ok, so let's try those value first, and see the result.

[lemonsqueeze]

Yes, it's not exactly opensource but it used to be shareware and the full version came with source code. Somehow got it to compile on unix so wine's not needed anymore. I can give you a copy for the purpose of play-testing but please don't distribute it without the author's permission.

so try maybe 10, 15, 20

Actually, try infinity first (just the second pattern_max_rating() call) that could be interesting

[pnprog]

I can give you a copy for the purpose of play-testing but please don't distribute it without the author's permission.

It's fine, I will do it with wine.

Actually, try infinity first

I guess that means that?

#define LOW_PATTERN_RATING 6.0

floating_t
pattern_rate_moves(struct pattern_config *pc,
                   struct board *b, enum stone color,
                   struct pattern *pats, floating_t *probs,
		   struct ownermap *ownermap)
{
#ifdef PATTERN_FEATURE_STATS
	pattern_stats_new_position();
#endif

	/* Try local moves first. */
	floating_t max = pattern_max_rating(pc, b, color, pats, probs, ownermap, true);

	/* Nothing big matches ? Try again ignoring distance so we get good tenuki moves. */
	//if (max < LOW_PATTERN_RATING)
	max = pattern_max_rating(pc, b, color, pats, probs, ownermap, false);
	
	return rescale_probs(b, probs, max);
}

[lemonsqueeze]

Yes, in this case you can also comment out the first call for speed.

[pnprog]

Hi!
So here are the first results. Those were handicap games, with Pachi as black taking 4 handicap stones.

• Threshold 10: Pachi: 76, Hira 126; average game length: 199/183
• Threshold 15: Pachi: 67, Hira: 133; average game length: 192/176
• Threshold 20: Pachi, 63, Hira: 137; average game length: 198/181
• Threshold infinity: Pachi: 53, Hira: 147; average game length: 183/173

I will upload the games somewhere also, I you want to check.

[lemonsqueeze]

Ok, seems local patterns do help.
Also did an experiment letting minipachi play with threshold infinity on kgs starting Feb 26,
rank graph starts going down at about that time, although not catastrophic.
So looks like 10 is pretty good.
If you feel like running more games you could try threshold 6 (current default) to see which is better,
otherwise i'm happy with that =)

[pnprog]

I propose to continue with 4, 6 and 8 to see if there is a maximum around there.
Once we have narrowed on that maximum value, we should also have that version of Pachi play against the current version of Pachi (so without handicap) to see if there is a real improvement.