Skip to content

Implement reproducible seeding #1288

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Merged
merged 124 commits into from
Aug 11, 2020
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
124 commits
Select commit Hold shift + click to select a range
e31b00d
First pass on reproducible matches and parallel tournaments with rand…
marcharper Feb 27, 2020
a518d09
Fix many tests and errors for seed setting and propagation
marcharper Feb 28, 2020
96cef67
Rewrite many tests to use new seeding method. Rewrite Evolveable play…
marcharper Mar 2, 2020
59f156d
Update tournament test to only compare ranked_names
marcharper Mar 2, 2020
06255f3
Undo or remove various commented lines
marcharper Mar 2, 2020
bad199e
Remove deadline for filtering test
marcharper Mar 2, 2020
423b59f
Cleanup post rebase, update some imports
marcharper Apr 21, 2020
af3c387
Update many tests with new random seeds and values, up to axelrod_fir…
marcharper Apr 21, 2020
b2dc6cd
Fix many more tests
marcharper Apr 21, 2020
ca05063
Map TestPlayer.versus_test onto TestMatch.
marcharper Jul 2, 2020
5195f31
Add data classes and YAML serialization of test matches
marcharper Jul 3, 2020
b910f94
Add dacite to requirements.txt
marcharper Jul 3, 2020
5823a63
Fix broken import
marcharper Jul 3, 2020
9ffc176
Fix broken import
marcharper Jul 3, 2020
3ee324f
Fix test watching for headsup tests
marcharper Jul 3, 2020
22491d7
Fix post merge import conflicts
marcharper Jul 3, 2020
d428499
Rerun isort
marcharper Jul 3, 2020
35a5e3a
Pin a new version of hypothesis
marcharper Jul 3, 2020
7669803
Post testing branch merge
marcharper Jul 3, 2020
3caa9c7
Add set_seed function to Match
marcharper Jul 3, 2020
e19ac93
Rework test_adaptive
marcharper Jul 3, 2020
059106c
Rework test_adaptor
marcharper Jul 3, 2020
9eac5a1
Rework test_alternator
marcharper Jul 3, 2020
162814c
Split tests in test_ann
marcharper Jul 3, 2020
585148b
Split tests in test_apavlov
marcharper Jul 3, 2020
0d24c3e
Split tests in test_appeaser
marcharper Jul 3, 2020
84db8e9
Rework test_averagecopier
marcharper Jul 4, 2020
28f7e82
Rename some tests in test_apavlov
marcharper Jul 4, 2020
77a7587
Change opponent in test_alternator
marcharper Jul 4, 2020
8615f1d
New seeds for test_gambler
marcharper Jul 4, 2020
d784afe
Fix test_hmm
marcharper Jul 4, 2020
a39a78b
Fixseeds for test_memoryone
marcharper Jul 4, 2020
be374ac
Fix seeds for test_meta
marcharper Jul 4, 2020
2cf50b6
Add exclusions for cloning test for Mind* strategies
marcharper Jul 4, 2020
13437ce
Update seeds for test_prober
marcharper Jul 4, 2020
9f2a967
Update seeds for test_qlearner
marcharper Jul 4, 2020
7983d93
Update seeds for test_selfsteem
marcharper Jul 4, 2020
cec3538
Update seeds for test_stalker
marcharper Jul 4, 2020
2f6b43b
Update seeds for test_titfortat
marcharper Jul 4, 2020
f124448
Update seeds for test_worse_and_worse
marcharper Jul 4, 2020
4d5fee0
Update seeds for test_zero_determinant
marcharper Jul 4, 2020
5d1cff0
Rework test_calculator
marcharper Jul 4, 2020
d9d95a0
Update seeds for several tests in axelrod_second
marcharper Jul 4, 2020
f0e0713
Rework tests in axlrod_second
marcharper Jul 4, 2020
9c6da82
Update seeds in test_axelrod_first
marcharper Jul 4, 2020
e308d21
Fix alternator test
marcharper Jul 4, 2020
eaf2d79
Fix Various Typos
marcharper Jul 4, 2020
7811434
Dataclasses and yaml parsing of versus_test
marcharper Jul 4, 2020
8a6930f
Add deadline=None to hypothesis tests
marcharper Jul 4, 2020
add852f
Fix several doc tests and seeds
marcharper Jul 4, 2020
1363ea5
Fix remaining doctests
marcharper Jul 4, 2020
ca78119
Make EvolvablePlayers properly seed reproducible
marcharper Jul 5, 2020
fef4a40
Rework test_strategy_transformers and update seeds
marcharper Jul 6, 2020
02ecf68
Add search_seeds method to TestMatch
marcharper Jul 6, 2020
14ba128
Make JossAnn and Mixed transformers avoid use of _random when they ar…
marcharper Jul 6, 2020
69ac7f4
Rework Random strategy to avoid using random numbers if actually dete…
marcharper Jul 6, 2020
28419c6
Remove redundant code from Random
marcharper Jul 6, 2020
e6f18f3
Fix several issues with MetaMixer
marcharper Jul 6, 2020
f7e0b88
Fix various issues, including stochastic classification, for MetaWinn…
marcharper Jul 6, 2020
a796a2b
Avoid use of _random in deterministic probers
marcharper Jul 8, 2020
559f35b
Avoid use of _random in deterministic HMMs
marcharper Jul 8, 2020
637c59b
Add comments to Meta
marcharper Jul 8, 2020
9e47ce3
In Match, only set seeds on players if they report that they are stoc…
marcharper Jul 8, 2020
950f1a5
Rewrite parts of strategy transformers to fix issues with classificat…
marcharper Jul 9, 2020
d8b1c75
Fix flip_history bug with LimitedHistory
marcharper Jul 9, 2020
6987307
Rework Random to avoid calls to _random
marcharper Jul 9, 2020
5897fdb
Add more test for Random strategy
marcharper Jul 9, 2020
83b6db8
Cleanup evolvable_player.py and propagate forward seeds
marcharper Jul 9, 2020
fa2ca0c
Update seeds for TFT tests and add new tests
marcharper Jul 29, 2020
7b503ac
Fix fingerprint tests
marcharper Jul 29, 2020
a8e6469
Update and optimize HMM
marcharper Jul 29, 2020
b9a0d30
Update HMM tests
marcharper Jul 29, 2020
fde9e4f
Update a Meta player test
marcharper Jul 29, 2020
dc1d867
Update seeds for strategy transformer tests
marcharper Jul 29, 2020
c4a7dac
Update Moran process to use propagate random seeds
marcharper Jul 29, 2020
a2d2826
Remove defunct tests in test_tournament
marcharper Jul 29, 2020
5ec9b70
Update fingerpring docs test
marcharper Jul 29, 2020
c196545
Update docstrings in property.py
marcharper Jul 29, 2020
50a0735
Have match watcher skip transformed players, which don't currently se…
marcharper Jul 29, 2020
b156230
Cleanup eigen.py
marcharper Jul 29, 2020
b396f0d
Update Axelrod First Tournament documentation tests with new seeds
marcharper Jul 29, 2020
79c3097
Rework memory one players
marcharper Jul 30, 2020
e42041e
Rework memory two players
marcharper Jul 30, 2020
1810617
Add EvolvablePlayer seed propagationtests
marcharper Jul 30, 2020
1c2a8a8
Update fingerprints and strategy_transformers to work properly with M…
marcharper Jul 31, 2020
d0859ab
Update Meta strategies to properly handle Joss-Ann edge cases
marcharper Jul 31, 2020
47b1871
Update various docstrings and other cleanup
marcharper Jul 31, 2020
c3a0786
Update classification docs tests
marcharper Jul 31, 2020
61f2c0e
Fix merge conflicts post merge of upstream master
marcharper Jul 31, 2020
9b2efb3
Update Joss Ann reclassifier + a test
marcharper Jul 31, 2020
5128e4d
Remove yaml testing code and data classes dependent on py3, used to f…
marcharper Jul 31, 2020
e9bb9ce
Fix requirements.txt typo
marcharper Jul 31, 2020
44bc97d
Remove imports associated to yaml and dataclasses
marcharper Jul 31, 2020
19b36f7
Remove pinned hypothesis version from github workflow
marcharper Jul 31, 2020
750f73c
Specify random seed type in attempt to make Windows happy
marcharper Jul 31, 2020
ea2599d
Add deadline=None to two tests to prevent errors on CI
marcharper Jul 31, 2020
c725138
Attempt 2 to make windows happy with int64 random seed type
marcharper Jul 31, 2020
e43d76a
Attempt 3 to make windows happy with int64 random seed type
marcharper Jul 31, 2020
693de4f
Remove unused requirements and increase minimum versions
marcharper Jul 31, 2020
7ef70e1
Add deadline=None to another test
marcharper Jul 31, 2020
40d9cb7
Attempt 4 to make windows happy with uint64 random seed type
marcharper Jul 31, 2020
9ea9f98
Add toolz back to requirements.txt for dask
marcharper Jul 31, 2020
f8821fc
Be more explicit about dask dependencies
marcharper Jul 31, 2020
398052a
Add tests to improve coverage
marcharper Aug 1, 2020
2fad16a
Fix TricklyLevelPunisher tests
marcharper Aug 1, 2020
dcd9d92
Fix seed warning test
marcharper Aug 1, 2020
08987ae
Fix a few more tests and cover some remaining lines
marcharper Aug 1, 2020
c547b56
Remove leftover testing code.
marcharper Aug 1, 2020
853b652
Fix doctest output
marcharper Aug 1, 2020
2e0e663
Fix type annotations in hmm.py
marcharper Aug 1, 2020
761f887
Run isort
marcharper Aug 1, 2020
1fd9dd6
Rework classifiers for transformers, rebuild expected classifiers to …
marcharper Aug 1, 2020
7c233a7
Mark line as no cover
marcharper Aug 1, 2020
96c3352
Fix no-op isort command in github checks
marcharper Aug 1, 2020
3eac7c2
Pin specific version of isort
marcharper Aug 1, 2020
c37fd74
Cleanup some comments
marcharper Aug 1, 2020
23b1d50
Find seed for original doctest
marcharper Aug 1, 2020
1e4ee69
Remove seeding from filter integration test
marcharper Aug 4, 2020
7163497
Fix docstrings, remove comments, and update a few tests based on roun…
marcharper Aug 4, 2020
57d4034
Updates following Nik's initial review
marcharper Aug 5, 2020
b0e00fe
Rerun isort
marcharper Aug 5, 2020
11a6f1c
A few minor updates following some additional review
marcharper Aug 8, 2020
23a7cf4
Update two tests following review comments.
marcharper Aug 9, 2020
82a22ca
Various updates following Nik's review
marcharper Aug 9, 2020
File filter

Filter by extension

Filter by extension


Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
6 changes: 2 additions & 4 deletions .github/workflows/config.yml
Original file line number Diff line number Diff line change
Expand Up @@ -9,7 +9,6 @@ on:

jobs:
build:

runs-on: ${{ matrix.os }}
strategy:
max-parallel: 4
Expand Down Expand Up @@ -37,7 +36,6 @@ jobs:
- name: Run tests
run: |
python -m pip install coverage
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

We have hypothesis in requirements.txt (and specifically excluded from setup.py: https://github.com/Axelrod-Python/Axelrod/blob/master/setup.py#L4). Should we do the same with coverage, mypy, isort? (Just do be consistent.)

(Happy for this to be done in a later PR - just a thought.)

Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Sounds good. It would be good to also have a "presubmit" script to run isort, mypy, etc. locally in line with config.yml.

Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Good idea.

Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Opened #1360

python -m pip install hypothesis==3.2
coverage run --source=axelrod -m unittest discover
- name: Report coverage
run: |
Expand All @@ -51,8 +49,8 @@ jobs:
python run_mypy.py
- name: Check imports are sorted
run: |
python -m pip install isort
python -m isort --check-only
python -m pip install "isort==4.3.21"
python -m isort --check-only --recursive axelrod/.
- name: Check that all strategies are indexed
run: |
python run_strategy_indexer.py
Expand Down
3 changes: 1 addition & 2 deletions README.rst
Original file line number Diff line number Diff line change
Expand Up @@ -89,9 +89,8 @@ Quick Start
The following runs a basic tournament::

>>> import axelrod as axl
>>> axl.seed(0) # Set a seed
>>> players = [s() for s in axl.demo_strategies] # Create players
>>> tournament = axl.Tournament(players) # Create a tournament
>>> tournament = axl.Tournament(players, seed=1) # Create a tournament
>>> results = tournament.play() # Play the tournament
>>> results.ranked_names
['Defector', 'Grudger', 'Tit For Tat', 'Cooperator', 'Random: 0.5']
Expand Down
12 changes: 10 additions & 2 deletions axelrod/__init__.py
Original file line number Diff line number Diff line change
Expand Up @@ -3,10 +3,18 @@

# The order of imports matters!
from axelrod.version import __version__
from axelrod.action import Action
from axelrod.random_ import Pdf, RandomGenerator, BulkRandomGenerator

# Initialize module level Random
# This is initially seeded by the clock / OS entropy pool
# It is not used if user specifies seeds everywhere and should only be
# used internally by the library and in certain tests that need to set
# its seed.
_module_random = RandomGenerator()
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

What's the rational of calling this _module_random ? Would _random_generator or _runtime_rng be a bit more descriptive? (Working through test_filtering and was not immediately sure what _module_random was.)

Copy link
Member Author

@marcharper marcharper Aug 4, 2020

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I didn't think much about the name, we can change it. The idea was that it's a "module level" RNG and so reproducibility wouldn't be guaranteed. But I think we could also probably eliminate it altogether, if we're willing to toss a few more warnings / exceptions at the user. The biggest issue in that case is instantiating evolvable players with insufficient parameters (for random initialization).

Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Turns out we can remove the need for a seed from the current filter test and it still works.


from axelrod.load_data_ import load_pso_tables, load_weights
from axelrod import graph
from axelrod.action import Action
from axelrod.random_ import random_choice, random_flip, seed, Pdf
from axelrod.plot import Plot
from axelrod.game import DefaultGame, Game
from axelrod.history import History, LimitedHistory
Expand Down
8 changes: 5 additions & 3 deletions axelrod/_strategy_utils.py
Original file line number Diff line number Diff line change
Expand Up @@ -137,7 +137,7 @@ def _calculate_scores(p1, p2, game):
return s1, s2


def look_ahead(player_1, player_2, game, rounds=10):
def look_ahead(player1, player2, game, rounds=10):
"""Returns a constant action that maximizes score by looking ahead.

Parameters
Expand All @@ -160,8 +160,10 @@ def look_ahead(player_1, player_2, game, rounds=10):
possible_strategies = {C: Cooperator(), D: Defector()}
for action, player in possible_strategies.items():
# Instead of a deepcopy, create a new opponent and replay the history to it.
opponent_ = player_2.clone()
for h in player_1.history:
opponent_ = player2.clone()
if opponent_.classifier["stochastic"]:
opponent_.set_seed(player2._seed)
for h in player1.history:
_limited_simulate_play(player, opponent_, h)

# Now play forward with the constant strategy.
Expand Down
4 changes: 2 additions & 2 deletions axelrod/action.py
Original file line number Diff line number Diff line change
Expand Up @@ -8,7 +8,7 @@

from enum import Enum
from functools import total_ordering
from typing import Iterable
from typing import Iterable, Tuple


class UnknownActionError(ValueError):
Expand Down Expand Up @@ -70,7 +70,7 @@ def from_char(cls, character):
raise UnknownActionError('Character must be "C" or "D".')


def str_to_actions(actions: str) -> tuple:
def str_to_actions(actions: str) -> Tuple[Action, ...]:
"""Converts a string to a tuple of actions.

Parameters
Expand Down
2 changes: 1 addition & 1 deletion axelrod/classifier.py
Original file line number Diff line number Diff line change
Expand Up @@ -13,9 +13,9 @@
Union,
)

import yaml
from axelrod.makes_use_of import makes_use_of
from axelrod.player import Player
import yaml

ALL_CLASSIFIERS_PATH = "data/all_classifiers.yml"

Expand Down
45 changes: 34 additions & 11 deletions axelrod/data/all_classifiers.yml
Original file line number Diff line number Diff line change
Expand Up @@ -961,39 +961,47 @@ Meta Hunter Aggressive:
Meta Majority:
inspects_source: false
long_run_time: true
makes_use_of: !!set {}
makes_use_of: !!set
game: null
length: null
manipulates_source: false
manipulates_state: false
memory_depth: .inf
stochastic: true
Meta Majority Finite Memory:
inspects_source: false
long_run_time: true
makes_use_of: !!set {}
makes_use_of: !!set
game: null
manipulates_source: false
manipulates_state: false
memory_depth: .inf
stochastic: true
Meta Majority Long Memory:
inspects_source: false
long_run_time: true
makes_use_of: !!set {}
makes_use_of: !!set
game: null
length: null
manipulates_source: false
manipulates_state: false
memory_depth: .inf
stochastic: true
Meta Majority Memory One:
inspects_source: false
long_run_time: true
makes_use_of: !!set {}
makes_use_of: !!set
game: null
manipulates_source: false
manipulates_state: false
memory_depth: .inf
stochastic: true
Meta Minority:
inspects_source: false
long_run_time: true
makes_use_of: !!set {}
makes_use_of: !!set
game: null
length: null
manipulates_source: false
manipulates_state: false
memory_depth: .inf
Expand All @@ -1011,6 +1019,7 @@ Meta Winner:
long_run_time: true
makes_use_of: !!set
game: null
length: null
manipulates_source: false
manipulates_state: false
memory_depth: .inf
Expand All @@ -1020,6 +1029,7 @@ Meta Winner Deterministic:
long_run_time: true
makes_use_of: !!set
game: null
length: null
manipulates_source: false
manipulates_state: false
memory_depth: .inf
Expand All @@ -1029,6 +1039,7 @@ Meta Winner Ensemble:
long_run_time: true
makes_use_of: !!set
game: null
length: null
manipulates_source: false
manipulates_state: false
memory_depth: .inf
Expand All @@ -1047,6 +1058,7 @@ Meta Winner Long Memory:
long_run_time: true
makes_use_of: !!set
game: null
length: null
manipulates_source: false
manipulates_state: false
memory_depth: .inf
Expand All @@ -1065,6 +1077,7 @@ Meta Winner Stochastic:
long_run_time: true
makes_use_of: !!set
game: null
length: null
manipulates_source: false
manipulates_state: false
memory_depth: .inf
Expand Down Expand Up @@ -1130,40 +1143,47 @@ N Tit(s) For M Tat(s):
NMWE Deterministic:
inspects_source: false
long_run_time: true
makes_use_of: &id001 !!set
makes_use_of: !!set
game: null
length: null
manipulates_source: false
manipulates_state: false
memory_depth: .inf
stochastic: true
NMWE Finite Memory:
inspects_source: false
long_run_time: true
makes_use_of: *id001
makes_use_of: !!set
game: null
manipulates_source: false
manipulates_state: false
memory_depth: .inf
stochastic: true
NMWE Long Memory:
inspects_source: false
long_run_time: true
makes_use_of: *id001
makes_use_of: !!set
game: null
length: null
manipulates_source: false
manipulates_state: false
memory_depth: .inf
stochastic: true
NMWE Memory One:
inspects_source: false
long_run_time: true
makes_use_of: *id001
makes_use_of: !!set
game: null
manipulates_source: false
manipulates_state: false
memory_depth: .inf
stochastic: true
NMWE Stochastic:
inspects_source: false
long_run_time: true
makes_use_of: *id001
makes_use_of: !!set
game: null
length: null
manipulates_source: false
manipulates_state: false
memory_depth: .inf
Expand Down Expand Up @@ -1197,14 +1217,17 @@ Nice Meta Winner:
long_run_time: true
makes_use_of: !!set
game: null
length: null
manipulates_source: false
manipulates_state: false
memory_depth: .inf
stochastic: true
Nice Meta Winner Ensemble:
inspects_source: false
long_run_time: true
makes_use_of: *id001
makes_use_of: !!set
game: null
length: null
manipulates_source: false
manipulates_state: false
memory_depth: .inf
Expand Down
28 changes: 14 additions & 14 deletions axelrod/eigen.py
Original file line number Diff line number Diff line change
Expand Up @@ -7,24 +7,24 @@

from typing import Tuple

import numpy
import numpy as np


def _normalise(nvec: numpy.ndarray) -> numpy.ndarray:
def _normalise(nvec: np.ndarray) -> np.ndarray:
"""Normalises the given numpy array."""
with numpy.errstate(invalid="ignore"):
result = nvec / numpy.sqrt((nvec @ nvec))
with np.errstate(invalid="ignore"):
result = nvec / np.sqrt((nvec @ nvec))
return result


def _squared_error(vector_1: numpy.ndarray, vector_2: numpy.ndarray) -> float:
def _squared_error(vector_1: np.ndarray, vector_2: np.ndarray) -> float:
"""Computes the squared error between two numpy arrays."""
diff = vector_1 - vector_2
s = diff @ diff
return numpy.sqrt(s)
return np.sqrt(s)


def _power_iteration(mat: numpy.array, initial: numpy.ndarray) -> numpy.ndarray:
def _power_iteration(mat: np.array, initial: np.ndarray) -> np.ndarray:
"""
Generator of successive approximations.

Expand All @@ -33,7 +33,7 @@ def _power_iteration(mat: numpy.array, initial: numpy.ndarray) -> numpy.ndarray:
mat: numpy.array
The matrix to use for multiplication iteration
initial: numpy.array, None
The initial state. Will be set to numpy.array([1, 1, ...]) if None
The initial state. Will be set to np.array([1, 1, ...]) if None

Yields
------
Expand All @@ -42,13 +42,13 @@ def _power_iteration(mat: numpy.array, initial: numpy.ndarray) -> numpy.ndarray:

vec = initial
while True:
vec = _normalise(numpy.dot(mat, vec))
vec = _normalise(np.dot(mat, vec))
yield vec


def principal_eigenvector(
mat: numpy.array, maximum_iterations=1000, max_error=1e-3
) -> Tuple[numpy.ndarray, float]:
mat: np.array, maximum_iterations=1000, max_error=1e-3
) -> Tuple[np.ndarray, float]:
"""
Computes the (normalised) principal eigenvector of the given matrix.

Expand All @@ -66,12 +66,12 @@ def principal_eigenvector(
ndarray
Eigenvector estimate for the input matrix
float
Eigenvalue corresonding to the returned eigenvector
Eigenvalue corresponding to the returned eigenvector
"""

mat_ = numpy.array(mat)
mat_ = np.array(mat)
size = mat_.shape[0]
initial = numpy.ones(size)
initial = np.ones(size)

# Power iteration
if not maximum_iterations:
Expand Down
Loading