Skip to content

Commit 82ed909

Browse files
committed
docs: extended readme
1 parent 285817b commit 82ed909

File tree

1 file changed

+51
-6
lines changed

1 file changed

+51
-6
lines changed

src/rai_bench/README.md

+51-6
Original file line numberDiff line numberDiff line change
@@ -4,12 +4,57 @@
44

55
The RAI Bench is a package including benchmarks and providing frame for creating new benchmarks
66

7-
### Components
7+
## Frame Components
88

9-
- `Scenario` - class Defined by a Scene and Task
10-
- `Benchamrk` - class responsible for running scenarios.
11-
- `Task` - abstract class for creating specific task. It introduces helper funtions that make it easier to calculate metrics/scores
9+
Frame components can be found in `src/rai_bench/rai_bench/benchmark_model.py`
1210

13-
### Example implementation
11+
- `Task` - abstract class for creating specific task. It introduces helper funtions that make it easier to calculate metrics/scores. Your custom tasks must implement a prompt got agent to do, a way to calculate a result and a validation if given scene config suits the task.
12+
-
13+
- `Scenario` - class defined by a Scene and Task. Can be created manually like:
1414

15-
- `o3de_test_bench` - it's example of creating benchmark.
15+
```python
16+
17+
```
18+
19+
- `Benchmark` - class responsible for running and logging scenarios.
20+
21+
### O3DE TEST BENCHMARK
22+
23+
O3DE Test Benchmark (src/rai_bench/rai_bench/o3de_test_bench/), contains 2 Tasks(tasks/) - GrabCarrotTask and PlaceCubesTask (these tasks implement calculating scores) and 4 scene_configs(configs/) for O3DE robotic arm simulation.
24+
25+
Both tasks calculate score, taking into consideration 4 values:
26+
27+
- initially_misplaced_now_correct - when the object which was in the incorrect place at the start, is in a correct place at the end
28+
- initially_misplaced_still_incorrect - when the object which was in the incorrect place at the start, is in a incorrect place at the end
29+
- initially_correct_still_correct - when the object which was in the correct place at the start, is in a correct place at the end
30+
- initially_correct_now_incorrect - when the object which was in the correct place at the start, is in a incorrect place at the end
31+
32+
The result is a value between 0 and 1, calculated like (initially_misplaced_now_correct + initially_correct_still_correct) / number_of_initial_objects.
33+
This score is calculated at the beggining and at the end of each scenario.
34+
35+
### Example usage
36+
37+
Example of how to load scenes, define scenarios and run benchmark can be found in `src/rai_bench/rai_bench/benchmark_main.py`
38+
39+
Scenarios can be loaded manually like:
40+
41+
```python
42+
one_carrot_simulation_config = O3DExROS2SimulationConfig.load_config(
43+
base_config_path=Path("path_to_scene.yaml"),
44+
connector_config_path=Path("path_to_o3de_config.yaml"),
45+
)
46+
47+
Scenario(task=GrabCarrotTask(logger=some_logger), simulation_config=one_carrot_simulation_config)
48+
```
49+
50+
or automatically like:
51+
52+
```python
53+
scenarios = Benchmark.create_scenarios(
54+
tasks=tasks, simulation_configs=simulations_configs
55+
)
56+
```
57+
58+
which will result in list of scenarios with combination of every possible task and scene(task decides if scene config is suitable for it).
59+
60+
Both approaches can be found in `main.py`

0 commit comments

Comments
 (0)