feat: add more scenarios to o3de manipulation benchmark (#452)

Co-authored-by: Bartłomiej Boczek <[email protected]>

Author: jmatejcz

demos.repos

@@ -6,4 +6,4 @@ repositories:
   src/examples/rai-manipulation-demo:
     type: git
     url: https://github.com/RobotecAI/rai-manipulation-demo.git
-    version: development
+    version: kd/wait_for_clock

src/rai_bench/README.md

@@ -1,36 +1,56 @@
 ## RAI Benchmark
 
-## Description
+### Description
 
 The RAI Bench is a package including benchmarks and providing frame for creating new benchmarks
 
-## Frame Components
-
-Frame components can be found in `src/rai_bench/rai_bench/benchmark_model.py`
+### Frame Components
 
 - `Task`
 - `Scenario`
 - `Benchmark`
 
-For more information about these classes go to -> `src/rai_bench/rai_bench/benchmark_model.py`
+For more information about these classes go to -> [benchmark_model](./rai_bench/benchmark_model.py)
+
+### O3DE Test Benchmark
+
+The O3DE Test Benchmark [o3de_test_benchmark_module](./rai_bench/o3de_test_bench/) provides tasks and scene configurations for robotic arm manipulation task. The tasks use a common `ManipulationTask` logic and can be parameterized, which allows for many task variants. The current tasks include:
+
+- **MoveObjectToLeftTask**
+- **GroupObjectsTask**
+- **BuildCubeTowerTask**
+- **PlaceObjectAtCoordTask**
+- **RotateObjectTask** (currently not applicable due to limitations in the ManipulatorMoveTo tool)
+
+The result of a task is a value between 0 and 1, calculated like initially_misplaced_now_correct / initially_misplaced. This score is calculated at the end of each scenario.
 
-### O3DE TEST BENCHMARK
+Current O3DE simulation binaries:
 
-O3DE Test Benchmark (`src/rai_bench/rai_bench/o3de_test_bench/`), contains 2 Tasks(`tasks/`) - GrabCarrotTask and PlaceCubesTask (these tasks implement calculating scores) and 4 scene_configs(`configs/`) for O3DE robotic arm simulation.
+### Running
 
-Both tasks calculate score, taking into consideration 4 values:
+1. Download O3DE simulation binary and unzip it.
 
-- initially_misplaced_now_correct
-- initially_misplaced_still_incorrect
-- initially_correct_still_correct
-- initially_correct_now_incorrect
+   - [ros2-humble](https://robotec-ml-rai-public.s3.eu-north-1.amazonaws.com/RAIManipulationDemo_jammyhumble.zip)
+   - [ros2-jazzy](https://robotec-ml-rai-public.s3.eu-north-1.amazonaws.com/RAIManipulationDemo_noblejazzy.zip)
 
-The result is a value between 0 and 1, calculated like (initially_misplaced_now_correct + initially_correct_still_correct) / number_of_initial_objects.
-This score is calculated at the beggining and at the end of each scenario.
+2. Follow step 2 from [Manipulation demo Setup section](../../docs/demos/manipulation.md#setup)
+
+3. Adjust the path to the binary in: [o3de_config.yaml](./rai_bench/o3de_test_bench/configs/o3de_config.yaml)
+4. Run benchmark with:
+
+   ```bash
+   cd rai
+   source setup_shell.sh
+   python src/rai_bench/rai_bench/examples/o3de_test_benchmark.py
+   ```
+
+> [!NOTE]
+> For now benchmark runs all available scenarios (~160). See [Examples](#example-usege)
+> section for details.
 
 ### Example usage
 
-Example of how to load scenes, define scenarios and run benchmark can be found in `src/rai_bench/rai_bench/examples/o3de_test_benchmark.py`
+Example of how to load scenes, define scenarios and run benchmark can be found in [o3de_test_benchmark_example](./rai_bench/examples/o3de_test_benchmark.py)
 
 Scenarios can be loaded manually like:
 
@@ -52,3 +72,33 @@ scenarios = Benchmark.create_scenarios(
 ```
 
 which will result in list of scenarios with combination of every possible task and scene(task decides if scene config is suitable for it).
+
+or can be imported from exisitng packets [scenarios_packets](./rai_bench/o3de_test_bench/scenarios.py):
+
+```python
+t_scenarios = trivial_scenarios(
+        configs_dir=configs_dir, connector_path=connector_path, logger=bench_logger
+    )
+e_scenarios = easy_scenarios(
+    configs_dir=configs_dir, connector_path=connector_path, logger=bench_logger
+)
+m_scenarios = medium_scenarios(
+    configs_dir=configs_dir, connector_path=connector_path, logger=bench_logger
+)
+h_scenarios = hard_scenarios(
+    configs_dir=configs_dir, connector_path=connector_path, logger=bench_logger
+)
+vh_scenarios = very_hard_scenarios(
+    configs_dir=configs_dir, connector_path=connector_path, logger=bench_logger
+)
+```
+
+which are grouped by their subjective difficulty. For now there are 10 trivial, 42 easy, 23 medium, 38 hard and 47 very hard scenarios.
+Check docstrings and code in [scenarios_packets](./rai_bench/o3de_test_bench/scenarios.py) if you want to know how scenarios are assigned to difficulty level.
+
+### Development
+
+When creating new task or changing existing ones, make sure to add unit tests for score calculation in [rai_bench_tests](../../tests/rai_bench/).
+This applies also when you are adding or changing the helper methods in `Task` or `ManipulationTask`.
+
+The number of scenarios can be easily extened without writing new tasks, by increasing number of variants of the same task and adding more simulation configs but it won't improve variety of scenarios as much as creating new tasks.