Remove default check for is_set (#61)

* Remove automatically checking for is_set. Now only explicit is_set rules are checked. Also, is_valid does not generate a warning for a rule violation of a subfield. This is somewhat redundant.
* Set OSI submodule to according branch
* Add option to rules2yml to add is_set rule to all fields for a full interface check. The documentation has been adjusted accordingly
* Add comment to restructure subfield iteration
* Change trace file to a one with more test cases. Remove txt trace file
* Update osi submodule to current master branch
* Rename is_valid to check_children, because outputting a validity warning for every field with non-valid sub-fields does not make any sense and clutters up the warning output.
* Adjusted usage documentation according to the changes
* Fix call to rule_checker
* Fix doc references to example trace file
* Fix example trace file name
* Remove parallel example from usage doc
---------

Signed-off-by: ClemensLinnhoff <[email protected]>

Author: ClemensLinnhoff

Diff for: .github/workflows/ci.yml

@@ -78,5 +78,5 @@ jobs:
       run: |
         source .venv/bin/activate
         pip install .
-        osivalidator --data data/20210818T150542Z_sv_312_50_one_moving_object.osi -r rules
-        osivalidator --data data/20210818T150542Z_sv_312_50_one_moving_object.osi -r rules --parallel
+        osivalidator --data data/20240221T141700Z_sv_300_2112_10_one_moving_object.osi -r rules
+        osivalidator --data data/20240221T141700Z_sv_300_2112_10_one_moving_object.osi -r rules --parallel

Diff for: README.md

@@ -80,5 +80,5 @@ $ source .venv/Scripts/activate
 ## Example command
 
 ```bash
-$ osivalidator --data data/20210818T150542Z_sv_312_50_one_moving_object.osi --rules rules/
+$ osivalidator --data data/20240221T141700Z_sv_300_2112_10_one_moving_object.osi --rules rules/
 ```

Diff for: data/20210818T150542Z_sv_312_50_one_moving_object.osi

@@ -2,19 +2,37 @@
 
 == Folder structure
 
-Currently the rules are contained in `+*.yml+` files in the folder
-https://github.com/OpenSimulationInterface/osi-validation/tree/master/rules[rules].
-The organization of the files in this folder follows the architecture of
-OSI for consistency. In the future the rules will be ported directly
-into the `+*.proto+` files of
+Official rules for individual OSI fields are defined in the `+*.proto+` files of
 https://github.com/OpenSimulationInterface/open-simulation-interface[OSI].
+The first step to using and editing rules is to extract them from the proto files with `+rules2yml.py+`.
+It has the following options:
+[source,bash]
+----
+usage: python3 rules2yml.py [-h] [--dir DIR] [--full-osi]
+
+Export the rules of *.proto files into the *.yml format so it can be used by
+the validator.
+
+options:
+  -h, --help         show this help message and exit
+  --dir DIR, -d DIR  Name of the directory where the yml rules will be stored.
+  --full-osi, -f     Add is_set rule to all fields that do not contain it already.
+----
+
+The following example command will generate the yml rule files in a folder `rules` and add the `is_set` rule to every OSI field that does not have this rule anyways.
+This will result in a rule base to check for a full interface.
+
+[source,bash]
+----
+python3 rules2yml.py --dir rules --full-osi
+----
+
+The resulting yml rule files can be freely edited and are explained in the following in more detail.
 
 == File structure
 
-Below you can see an example of the
-https://github.com/OpenSimulationInterface/osi-validation/blob/master/rules/osi_detectedlane.yml[osi_detectedlane.yml]
-rule file for
-https://github.com/OpenSimulationInterface/open-simulation-interface/blob/master/osi_detectedlane.proto[osi_detectedlane.proto]:
+Below you can see an example of the osi_detectedlane.yml  rule file for
+https://github.com/OpenSimulationInterface/open-simulation-interface/blob/master/osi_detectedlane.proto[osi_detectedlane.proto] without the `--full-osi` option.
 
 [source,YAML]
 ----
@@ -46,10 +64,11 @@ apply to that specific field. For example the probability of the message
 
 The rules can either be with or without any parameters.
 
-....
+[source,YAML]
+----
 is_greater_than_or_equal_to: 0.0
 is_equal: 1
-....
+----
 
 In the case a rule has a parameter, it is written as a
 [.title-ref]#mapping# ( a [.title-ref]#scalar# followed by a colon ":").
@@ -62,15 +81,16 @@ https://github.com/OpenSimulationInterface/osi-validation/blob/master/osivalidat
 See also examples for available rules which can be defined in `+*.yml+`
 files below:
 
-[source,python]
+[source,YAML]
 ----
+is_set:
 is_greater_than: 1
 is_greater_than_or_equal_to: 1
 is_less_than_or_equal_to: 10
 is_less_than: 2
 is_equal: 1
 is_different: 2
-is_globally_unique
+is_globally_unique:
 refers_to: Lane
 is_iso_country_code:
 first_element: {is_equal: 0.13, is_greater_than: 0.13}
@@ -80,12 +100,13 @@ check_if: [{is_equal: 2, is_greater_than: 3, target: this.y}, {do_check: {is_equ
 
 == Severity
 
-When an attribute does not comply with a rule, a Warning is throw. An
-Error will be throw if an exclamation mark is written at the end of the
+When an attribute does not comply with a rule, a warning is thrown. An
+error will be thrown if an exclamation mark is written at the end of the
 verb of a rule.
 
 === Example
 
-....
+[source,YAML]
+----
 is_greater_than!: 0
-....
+----

Diff for: data/20240221T141700Z_sv_300_2112_10_one_moving_object.osi

@@ -184,7 +184,7 @@ def process_message(message, timestep, data_type):
     LOGGER.info(None, f"Analyze message of timestamp {timestamp}", False)
 
     # Check common rules
-    getattr(rule_checker, "is_valid")(
+    getattr(rule_checker, "check_children")(
         linked_proto_field.LinkedProtoField(message, name=data_type),
         VALIDATION_RULES.get_rules().get_type(data_type),
     )

Diff for: doc/usage.adoc

@@ -24,15 +24,7 @@ def add_default_rules_to_subfields(message, type_rules):
         )
 
         if descriptor.message_type:
-            field_rules.add_rule(osi_rules.Rule(verb="is_valid"))
-
-        is_set_severity = (
-            osi_rules.Severity.WARN
-            if field_rules.has_rule("is_optional")
-            else osi_rules.Severity.ERROR
-        )
-
-        field_rules.add_rule(osi_rules.Rule(verb="is_set", severity=is_set_severity))
+            field_rules.add_rule(osi_rules.Rule(verb="check_children"))
 
 
 # DECORATORS
@@ -100,7 +92,7 @@ def wrapper(self, field, rule, **kwargs):
 
 
 @rule_implementation
-def is_valid(self, field, rule):
+def check_children(self, field, rule):
     """Check if a field message is valid, that is all the inner rules of the
     message in the field are complying.
 
@@ -122,20 +114,19 @@ def is_valid(self, field, rule):
     else:
         subfield_rules = rule.root.get_type(field.message_type)
 
-    result = True
     # Add default rules for each subfield that can be validated (default)
     add_default_rules_to_subfields(field, subfield_rules)
 
     # loop over the fields in the rules
     for subfield_rules in subfield_rules.fields.values():
         for subfield_rule in subfield_rules.rules.values():
-            result = self.check_rule(field, subfield_rule) and result
+            self.check_rule(field, subfield_rule)
 
         # Resolve ID and references
     if not field.parent:
         self.id_manager.resolve_unicity(self.timestamp)
         self.id_manager.resolve_references(self.timestamp)
-    return result
+    return True
 
 
 @rule_implementation

Diff for: doc/writing-rules.adoc

@@ -23,11 +23,18 @@ def command_line_arguments():
         required=False,
         type=str,
     )
+    parser.add_argument(
+        "--full-osi",
+        "-f",
+        help="Add is_set rule to all fields that do not contain it already.",
+        action="store_true",
+        required=False,
+    )
 
     return parser.parse_args()
 
 
-def gen_yml_rules(dir_name="rules"):
+def gen_yml_rules(dir_name="rules", full_osi=False):
     with open(r"open-simulation-interface/rules.yml") as file:
         yaml = YAML(typ="safe")
         rules_dict = yaml.load(file)
@@ -161,6 +168,15 @@ def gen_yml_rules(dir_name="rules"):
                                         yml_file.write(
                                             (2 * numMessage) * " " + f"{field}:\n"
                                         )
+                                        # If option --full-osi is enabled:
+                                        # Check if is_set is already a rule for the current field, if not, add it.
+                                        if full_osi and not any(
+                                            "is_set" in rule for rule in rules
+                                        ):
+                                            yml_file.write(
+                                                (2 * numMessage + 2) * " "
+                                                + f"- is_set:\n"
+                                            )
 
                                         if shiftCounter:
                                             for rule in rules:
@@ -217,11 +233,21 @@ def gen_yml_rules(dir_name="rules"):
                                                         (2 * numMessage + 8) * " "
                                                         + f"- {rule_list[2]}: {rule_list[3]}\n"
                                                     )
-                                                elif "is_globally_unique" in rule_list:
+                                                # Standalone rules
+                                                elif any(
+                                                    list_item
+                                                    in [
+                                                        "is_globally_unique",
+                                                        "is_set",
+                                                        "is_iso_country_code",
+                                                    ]
+                                                    for list_item in rule_list
+                                                ):
                                                     yml_file.write(
                                                         (2 * numMessage + 2) * " "
                                                         + f"-{rule}:\n"
                                                     )
+                                                # Values or parameters of rules
                                                 else:
                                                     yml_file.write(
                                                         (2 * numMessage + 2) * " "
@@ -235,7 +261,7 @@ def gen_yml_rules(dir_name="rules"):
 def main():
     # Handling of command line arguments
     args = command_line_arguments()
-    gen_yml_rules(args.dir)
+    gen_yml_rules(args.dir, args.full_osi)
 
 
 if __name__ == "__main__":