Test cases should be parameterizable based on command-line flags (and other things)

[georgevdd]

Enumeration of the cases for a parameterized test method in an absl.testing.parameterized.TestCase is eager: it happens at the point where the class is defined.

Under normal circumstances, this is before googletest.main() has been called, meaning that the sequence of cases cannot depend on the values of flags defined by absl.flags (or, indeed, on anything that is not meaningfully defined at the point that the TestCase class is created).

Thus, the following structure will not work:

from absl import flags
from absl.testing import absltest
from absl.testing import parameterized

MY_FLAG = flags.DEFINE_bool(...)

def _gen_cases():
  return (x for x in ...) if MY_FLAG.value else (y for y in ...)  # But MY_FLAG won't have been set yet

class MyTest(parameterized.TestCase):

  @parameterized.parameters(_gen_cases()):
  def test_my_condition(self, case):
    ...

if __name__ == '__main__':
  absltest.main()

Possible Fix

Either of two changes could fix this: making command-line flag parsing more eager, or making test case enumeration lazier. The latter seems simpler.

Each of parameterized.parameters(), parameterized.named_parameters(), parameterized.product() and so on could be widened to accept, in addition to the types that it currently accepts, a callable that returns one of those types. If it finds its argument is callable, it could defer enumeration of the cases until some point after command-line flags have been parsed.

Then the above snippet could be amended:

  @parameterized.parameters(_gen_cases):  # Note the generator is not called at this point
  def test_my_condition(self, case):

[yilei]

(We also have this FR internally.)

The idea of an implementation would be to make absl.testing.absltest.TestLoader aware of parameterized, and evaluate the parameterized test cases there. It needs to continue support test filtering, test order randomization.