Jb3982/#929 test plan running script

[jb3982]

Description

• Resolves PATH: Script to run test plans + display results #929
• Added a run_test_plans command for running local pathfinding test plans sequentially.
• Supports selecting plans by name or number, randomly filling remaining slots, and reproducible selection using a seed.
• Displays live progress, including elapsed time, local and global waypoint counts, remaining distance, and boat speed.
• Redirects noisy ros2 launch terminal output to a per-test log.
• Detects child ROS process crashes and stops the test immediately to avoid wasting time and resources.
• Handles successful completion, early exits, timeouts, and user interruption.
• Preserves logs for failures and interruptions while deleting them for COMPLETED and TIMEOUT results.
• Writes results for each test and a batch summary containing all tests from the current execution as JSON.
• Records a ROS bag for each selected test plan.

Verification

• Ran ros2 run local_pathfinding run_test_plans -t 1 2 3 and verified plan selection, random filling, and live progress output.
• Exercise crash-message detection using representative ROS process has died output.
• Force a child ROS node to crash and confirm the test stops with FAILED.
• Confirm a completed test deletes its launch log.
• Confirm a shortened timeout stops the test and deletes its launch log.

Guidance on using run_test_plans.py

Build and source the workspace before running the command.

Run the script without arguments to select five test plans randomly:

ros2 run local_pathfinding run_test_plans

By default, five test plans are run. Users can select some or all of those plans, and any remaining slots are filled randomly.

List all available test plans

ros2 run local_pathfinding run_test_plans --list

Run selected test plans

Plans can be selected by number, full filename, or filename without the .yaml extension:

ros2 run local_pathfinding run_test_plans --tests basic, 2, crossing.yaml  --num_tests 3

The short argument names can also be used and the list of tests space-separated rather than comma-separated:

ros2 run local_pathfinding run_test_plans -t basic 2 crossing.yaml -n 3

If --num_tests is omitted, the selected plans are supplemented with random plans until five tests are selected.

Available arguments

Argument	Description	Default
-t, --tests	Plans selected by number, filename, or filename without .yaml	None
-n, --num_tests	Total number of plans to run, including randomly filled slots	5
--seed	Seed used for reproducible random selection	Random
--list	List available plans and exit	Disabled
--mode	Launch mode passed to global_launch	development
--log_level	ROS log level: debug, info, warn, error, or fatal	info
--timeout_hours	Maximum duration of each test	5.0
--goal_threshold_m	Distance in meters from a waypoint that counts as reached	300.0
--save_path	Workspace-relative output directory	notebooks/local_pathfinding/session_recordings/test_plans_results

Show command help

ros2 run local_pathfinding run_test_plans --help

Resources

• ROS 2 Launch documentation

[raghumanimehta]

The table should be documented in a confluence page.
Use this one: https://ubcsailbot.atlassian.net/wiki/spaces/prjt22/pages/2996568069/Test+Plans

[raghumanimehta]

"Confirm a shortened timeout stops the test and deletes its launch log."
Why do you want to this?

[raghumanimehta]

Some parts are unnecessarily complex for example, why have so much stuff being logged?
What is the need for so many states? why delete the logs when "TIMEOUT" occurs? This just assumes that all the test plans will terminate in 5 hours.
You should also aim at reducing the size of PR.

[jb3982]

"Confirm a shortened timeout stops the test and deletes its launch log." Why do you want to this?

I just wanted to test the system for different cases but not on a 5hour long session rather a smaller timeout value. To quickly test all the cases

[jb3982]

Some parts are unnecessarily complex for example, why have so much stuff being logged? What is the need for so many states? why delete the logs when "TIMEOUT" occurs? This just assumes that all the test plans will terminate in 5 hours. You should also aim at reducing the size of PR.

To clarify the logging idea: my alternate idea was to use a rolling log window and retain it only when a test failed, but that would have increased the PR’s complexity. The current implementation redirects ROS launch output to a file to keep the terminal uncluttered and saves the full file. But since we were running tests sequentially and pathfinding is in final trials, I thought we would rarely encounter FAILURE cases so implementing rolling window did make much sense to me.(Sorry I forgot to mention this in meeting today).

Now as to why we delete the logs in case of TIMEOUT, my idea was:
TIMEOUT would mean that either the boat did not reach the final destination given in the plan or one of the ros node crashed but the ros launch process kept running and that would have tricked my old implementation to assume that test plan is running correctly. Eventhough in that case too, the boat would have not reached the final deatination and caused us to log the terminal outputs but to save that time and resoruces I put a check.
And this check marked the 2nd case mentioned above as "FAILED" and save the log

[raghumanimehta]

Some parts are unnecessarily complex for example, why have so much stuff being logged? What is the need for so many states? why delete the logs when "TIMEOUT" occurs? This just assumes that all the test plans will terminate in 5 hours. You should also aim at reducing the size of PR.

To clarify the logging idea: my alternate idea was to use a rolling log window and retain it only when a test failed, but that would have increased the PR’s complexity. The current implementation redirects ROS launch output to a file to keep the terminal uncluttered and saves the full file. But since we were running tests sequentially and pathfinding is in final trials, I thought we would rarely encounter FAILURE cases so implementing rolling window did make much sense to me.(Sorry I forgot to mention this in meeting today).

Now as to why we delete the logs in case of TIMEOUT, my idea was: TIMEOUT would mean that either the boat did not reach the final destination given in the plan or one of the ros node crashed but the ros launch process kept running and that would have tricked my old implementation to assume that test plan is running correctly. Eventhough in that case too, the boat would have not reached the final deatination and caused us to log the terminal outputs but to save that time and resoruces I put a check. And this check marked the 2nd case mentioned above as "FAILED" and save the log

Timeout part is unnecessary. The assumption is poor. Remove that please. Don't delete any logs whatsoever.

[raghumanimehta]

why do you have three selection criteria?
Why not just run the plans in the same order as user input? Or in th case of no input, in the order that code sees it?

[jb3982]

That is not a selection criteria but a lookup logic {dictionaries} which does not affect execution order. They allow user's each test entry to be resolved by index (1), filename (basic.yaml), or stem (basic).
Selected tests are still run in the order provided by the user. And in the case of no input, resolve_test_plans() randomly selects the requested number of plans, which is separate from this lookup logic.

[raghumanimehta]

I had to manually add the documentation which you were supposed to do and i asked you to remove the deletion for all termination cases, which I had to edit manually as well.

Please make sure that you review all the comments properly from next time onwards.