[RL-baseline] Model v5, experiment #1

[ziritrion]

Action set #0 was chosen for this experiment:
[0.0, 0.0, 0.0], # no action
[0.0, 0.8, 0.0], # throttle
[0.0, 0.3, 0.0], # throttle
[0.0, 0.0, 0.6], # break
[0.0, 0.0, 0.2], # break
[-0.9, 0.0, 0.0], # left
[-0.5, 0.0, 0.0], # left
[-0.2, 0.0, 0.0], # left
[0.9, 0.0, 0.0], # right
[0.5, 0.0, 0.0], # right
[0.2, 0.0, 0.0], # right

Loss, Entropy and Running Reward were very low between the 10k and 15k episode marks but the model managed to overcome it and ended up with a maximum running reward of 528, which is also the final running reward at the end of the 20k episodes. It's likely that the network could keep improving if we run it for a few more episodes.

Tensorboard captures below:

Sample video below. As with all samples before, the model hasn't managed to learn to break before a sharp turn, but in this example the road appears again by sheer luck and managed to get back in.
https://user-images.githubusercontent.com/1465235/113404852-2128a500-93a9-11eb-8881-05101c3cd0e0.mp4

[ziritrion]

Updated the experiment with 10k additional episodes for a total of 30k. We hoped that due to the upward trend in the Running Reward right before 20k episodes we could see an even better value, but this has not been the case.

The Running Reward improved slightly right at the begging but then steadily dropped and never recovered. The final Running Reward value is 250.

Sample video below.
https://user-images.githubusercontent.com/1465235/113481938-a640b680-949c-11eb-916a-5d2aaecf5857.mp4