Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Initial refactoring to use sane OSITrace reader #70

Merged
merged 2 commits into from
May 15, 2024
Merged

Initial refactoring to use sane OSITrace reader #70

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
2 commits
Select commit Hold shift + click to select a range
ce4ec62
Initial refactoring to use sane OSITrace reader
pmai Apr 13, 2024
75ebad2
Refactor left-over manual indexing after code review
pmai May 10, 2024
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
28 changes: 2 additions & 26 deletions .github/workflows/ci.yml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -48,42 +48,20 @@ jobs:
- name: Checkout
uses: actions/checkout@v4
with:
submodules: recursive
lfs: true

- name: Set up Python ${{ matrix.python-version }}
uses: actions/setup-python@v5
with:
python-version: ${{ matrix.python-version }}

- name: Cache Dependencies
id: cache-depends
uses: actions/cache@v3
with:
path: protobuf-3.20.1
key: ${{ runner.os }}-v2-depends

- name: Download ProtoBuf
if: steps.cache-depends.outputs.cache-hit != 'true'
run: curl -OL https://github.com/protocolbuffers/protobuf/releases/download/v3.20.1/protobuf-all-3.20.1.tar.gz && tar xzvf protobuf-all-3.20.1.tar.gz

- name: Build ProtoBuf
if: steps.cache-depends.outputs.cache-hit != 'true'
working-directory: protobuf-3.20.1
run: ./configure DIST_LANG=cpp --prefix=/usr && make

- name: Install ProtoBuf
working-directory: protobuf-3.20.1
run: sudo make install && sudo ldconfig

- name: Install Open Simulation Interface
shell: bash
- name: Set up Virtual Environment
run: |
git submodule update --init
python -m venv .venv
source .venv/bin/activate
python -m pip install --upgrade pip
pip install -r requirements_develop.txt
cd open-simulation-interface && pip install . && cd ..

- name: Generate parsed rules
run: |
Expand All @@ -100,7 +78,5 @@ jobs:
run: |
source .venv/bin/activate
pip install .
osivalidator --data data/20210818T150542Z_sv_312_50_one_moving_object.txt -r rules
osivalidator --data data/20210818T150542Z_sv_312_50_one_moving_object.txt -r rules --parallel
osivalidator --data data/20210818T150542Z_sv_312_50_one_moving_object.osi -r rules
osivalidator --data data/20210818T150542Z_sv_312_50_one_moving_object.osi -r rules --parallel
14 changes: 7 additions & 7 deletions README.md
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -11,7 +11,7 @@ The full documentation on the validator and customization of the rules is availa
## Usage

```bash
usage: osivalidator [-h] [--data DATA] [--rules RULES] [--type {SensorView,GroundTruth,SensorData}] [--output OUTPUT] [--timesteps TIMESTEPS] [--debug] [--verbose] [--parallel] [--format {separated,None}]
usage: osivalidator [-h] --data DATA [--rules RULES] [--type {SensorView,GroundTruth,SensorData}] [--output OUTPUT] [--timesteps TIMESTEPS] [--debug] [--verbose] [--parallel] [--format {None}]
[--blast BLAST] [--buffer BUFFER]

Validate data defined at the input
Expand All @@ -29,13 +29,13 @@ optional arguments:
Number of timesteps to analyze. If -1, all.
--debug Set the debug mode to ON.
--verbose, -v Set the verbose mode to ON.
--parallel, -p Set parallel mode to ON.
--format {separated,None}, -f {separated,None}
Set the format type of the trace.
--parallel, -p (Ignored) Set parallel mode to ON.
--format {None}, -f {None}
(Ignored) Set the format type of the trace.
--blast BLAST, -bl BLAST
Set the in-memory storage count of OSI messages during validation.
Set the maximum in-memory storage count of OSI messages during validation.
--buffer BUFFER, -bu BUFFER
Set the buffer size to retrieve OSI messages from trace file. Set it to 0 if you do not want to use buffering at all.
(Ignored) Set the buffer size to retrieve OSI messages from trace file. Set it to 0 if you do not want to use buffering at all.
```

## Installation
Expand Down Expand Up @@ -80,5 +80,5 @@ $ source .venv/Scripts/activate
## Example command

```bash
$ osivalidator --data data/20210818T150542Z_sv_312_50_one_moving_object.txt --rules rules/
$ osivalidator --data data/20210818T150542Z_sv_312_50_one_moving_object.osi --rules rules/
```
Binary file removed data/20210818T150542Z_sv_312_50_one_moving_object.txt
View file
Open in desktop
Binary file not shown.
49 changes: 15 additions & 34 deletions doc/usage.adoc
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -7,65 +7,46 @@ terminal which has the following usage:

[source,bash]
----
usage: osivalidator [-h] [--data DATA] [--rules RULES] [--type {SensorView,GroundTruth,SensorData}] [--output OUTPUT] [--timesteps TIMESTEPS] [--debug] [--verbose] [--parallel] [--format {separated,None}]
usage: osivalidator [-h] --data DATA [--rules RULES] [--type {SensorView,GroundTruth,SensorData}] [--output OUTPUT] [--timesteps TIMESTEPS] [--debug] [--verbose] [--parallel] [--format {None}]
[--blast BLAST] [--buffer BUFFER]

Validate data defined at the input

mandatory arguments:
--data DATA Path to the file with OSI-serialized data.

optional arguments:
-h, --help show this help message and exit
--data DATA Path to the file with OSI-serialized data.
--rules RULES, -r RULES
Directory with text files containig rules.
Directory with text files containig rules.
--type {SensorView,GroundTruth,SensorData}, -t {SensorView,GroundTruth,SensorData}
Name of the type used to serialize data.
Name of the type used to serialize data.
--output OUTPUT, -o OUTPUT
Output folder of the log files.
Output folder of the log files.
--timesteps TIMESTEPS
Number of timesteps to analyze. If -1, all.
Number of timesteps to analyze. If -1, all.
--debug Set the debug mode to ON.
--verbose, -v Set the verbose mode to ON.
--parallel, -p Set parallel mode to ON.
--format {separated,None}, -f {separated,None}
Set the format type of the trace.
--parallel, -p (Ignored) Set parallel mode to ON.
--format {None}, -f {None}
(Ignored) Set the format type of the trace.
--blast BLAST, -bl BLAST
Set the in-memory storage count of OSI messages during validation.
Set the maximum in-memory storage count of OSI messages during validation.
--buffer BUFFER, -bu BUFFER
Set the buffer size to retrieve OSI messages from trace file. Set it to 0 if you do not want to use buffering at all.
(Ignored) Set the buffer size to retrieve OSI messages from trace file. Set it to 0 if you do not want to use buffering at all.
----

To run the validation first you need an OSI trace file which consists of
multiple OSI messages. In the directory `+data+` of the repository we
already provide an example trace file which is called
`+20210818T150542Z_sv_312_50_one_moving_object.txt+`. Use the
https://github.com/OpenSimulationInterface/open-simulation-interface/blob/master/format/txt2osi.py[txt2osi.py]
of the OSI repo in the format directory to convert from `+*.txt+` to
`+*.osi+` files. See usage below:

[source,bash]
----
usage: txt2osi converter [-h] [--file FILE]
[--type {SensorView,GroundTruth,SensorData}]
[--output OUTPUT] [--compress]

Convert txt trace file to osi trace files.

optional arguments:
-h, --help show this help message and exit
--file FILE, -f FILE Path to the file with serialized data.
--type {SensorView,GroundTruth,SensorData}, -t {SensorView,GroundTruth,SensorData}
Name of the type used to serialize data.
--output OUTPUT, -o OUTPUT
Output name of the file.
----
`+20210818T150542Z_sv_312_50_one_moving_object.osi+`.

To validate the trace files you simply call `+osivalidator+` and provide
the path to the trace:

[source,bash]
----
osivalidator --data data/20210818T150542Z_sv_312_50_one_moving_object.osi
osivalidator --data data/20210818T150542Z_sv_312_50_one_moving_object.txt
----

You can also validate the traces in parallel to increase the speed of
Expand All @@ -84,7 +65,7 @@ need to generate them and then specify them:
[source,bash]
----
python rules2yml.py # Generates the rule directory
osivalidator --data data/20210818T150542Z_sv_312_50_one_moving_object.txt --rules rules/ -p
osivalidator --data data/20210818T150542Z_sv_312_50_one_moving_object.osi --rules rules -p
----

After successfully running the validation the following output is
Expand Down
Loading