|
4 | 4 |
|
5 | 5 | The RAI Bench is a package including benchmarks and providing frame for creating new benchmarks
|
6 | 6 |
|
7 |
| -### Components |
| 7 | +## Frame Components |
8 | 8 |
|
9 |
| -- `Scenario` - class Defined by a Scene and Task |
10 |
| -- `Benchamrk` - class responsible for running scenarios. |
11 |
| -- `Task` - abstract class for creating specific task. It introduces helper funtions that make it easier to calculate metrics/scores |
| 9 | +Frame components can be found in `src/rai_bench/rai_bench/benchmark_model.py` |
12 | 10 |
|
13 |
| -### Example implementation |
| 11 | +- `Task` - abstract class for creating specific task. It introduces helper funtions that make it easier to calculate metrics/scores. Your custom tasks must implement a prompt got agent to do, a way to calculate a result and a validation if given scene config suits the task. |
| 12 | +- |
| 13 | +- `Scenario` - class defined by a Scene and Task. Can be created manually like: |
14 | 14 |
|
15 |
| -- `o3de_test_bench` - it's example of creating benchmark. |
| 15 | + ```python |
| 16 | + |
| 17 | + ``` |
| 18 | + |
| 19 | +- `Benchmark` - class responsible for running and logging scenarios. |
| 20 | + |
| 21 | +### O3DE TEST BENCHMARK |
| 22 | + |
| 23 | +O3DE Test Benchmark (src/rai_bench/rai_bench/o3de_test_bench/), contains 2 Tasks(tasks/) - GrabCarrotTask and PlaceCubesTask (these tasks implement calculating scores) and 4 scene_configs(configs/) for O3DE robotic arm simulation. |
| 24 | + |
| 25 | +Both tasks calculate score, taking into consideration 4 values: |
| 26 | + |
| 27 | +- initially_misplaced_now_correct - when the object which was in the incorrect place at the start, is in a correct place at the end |
| 28 | +- initially_misplaced_still_incorrect - when the object which was in the incorrect place at the start, is in a incorrect place at the end |
| 29 | +- initially_correct_still_correct - when the object which was in the correct place at the start, is in a correct place at the end |
| 30 | +- initially_correct_now_incorrect - when the object which was in the correct place at the start, is in a incorrect place at the end |
| 31 | + |
| 32 | +The result is a value between 0 and 1, calculated like (initially_misplaced_now_correct + initially_correct_still_correct) / number_of_initial_objects. |
| 33 | +This score is calculated at the beggining and at the end of each scenario. |
| 34 | + |
| 35 | +### Example usage |
| 36 | + |
| 37 | +Example of how to load scenes, define scenarios and run benchmark can be found in `src/rai_bench/rai_bench/benchmark_main.py` |
| 38 | + |
| 39 | +Scenarios can be loaded manually like: |
| 40 | + |
| 41 | +```python |
| 42 | +one_carrot_simulation_config = O3DExROS2SimulationConfig.load_config( |
| 43 | + base_config_path=Path("path_to_scene.yaml"), |
| 44 | + connector_config_path=Path("path_to_o3de_config.yaml"), |
| 45 | + ) |
| 46 | + |
| 47 | +Scenario(task=GrabCarrotTask(logger=some_logger), simulation_config=one_carrot_simulation_config) |
| 48 | +``` |
| 49 | + |
| 50 | +or automatically like: |
| 51 | + |
| 52 | +```python |
| 53 | +scenarios = Benchmark.create_scenarios( |
| 54 | + tasks=tasks, simulation_configs=simulations_configs |
| 55 | + ) |
| 56 | +``` |
| 57 | + |
| 58 | +which will result in list of scenarios with combination of every possible task and scene(task decides if scene config is suitable for it). |
| 59 | + |
| 60 | +Both approaches can be found in `main.py` |
0 commit comments