Appropriately Handle Invariances

[Scienfitz]

An Issue to specify and collect progress around handling invariances. Implemented parts will be ticked off.

What is Invariance
Invariance happens when the nature of the optimization problem means certain points in the searchspace are identical despite having different parameter values. In practice, this means that the two experiments corresponding to the two searchspace points are identical and cannot be distinguished.

In BayBE, we currently focus on the two most commonly encountered invariances:

• Permutation invariance: Exchanging the values of two parameters does not change the resulting experiment.
• Dependency invariance: If one parameter has a certain value, the value(s) of some other parameters become irrelevant. Example: A switch can be "on" or "off". If the switch is "on", another piece of equipment with 3 parameters becomes relevant. However, these 3 parameters can have any value without affecting the resulting experiment if the switch is "off"

How Invariance Enters BayBE
Invariance is simply activated by using the respective constraint DiscreteDependenciesConstraint or DiscretePermutationInvarianceConstraint. This should activate the appropriate machinery.

Needed Machinery in the Presence of Invariances

• Remove invariant search space points to reduce the degenerate search space points.

  • Implement filtering
  • As consequence of the filtering, one point per degenerate set of points remains, effectively displaying a random value of the affected parameters. This might be confusing and an output via placeholder(s) will be clearer

• Augment incoming measurements such that they comprise all possible invariant combination
  Advantage: Can be used for any surrogate model and requires no changes to the latter. Also brings invariance to continuous parameters.
  Disadvantage: This is jut learned invariance. Increases the amount of data points, possibly problematic for badly scaling models
  Conclusion: Should probably be activated by default and maybe deactivated for GP's

• Make the GP kernel parts corresponding to the respective parameters the needed invariant character
  Advantage: Best possible treatment of invariance via model architecture. Also brings invariance to continuous parameters.
  Disadvantage: Only easily possible for GP. The kernels might be very expensive if there are many affected paramters
  Conclusion: Should be implemented for GP