WOSAC Why feeding GT can't have likelihood=1.0 for kinematic_metrics?

[pengzhenghao]

I try to feed GT (repeated 32 times) into the WOSAC eval pipe and find:

scenario_metrics/metametric: 0.77725
scenario_metrics/average_displacement_error: 0.00000
scenario_metrics/min_average_displacement_error: 0.00000
scenario_metrics/linear_speed_likelihood: 0.39830
scenario_metrics/linear_acceleration_likelihood: 0.47793
scenario_metrics/angular_speed_likelihood: 0.45490
scenario_metrics/angular_acceleration_likelihood: 0.64707
scenario_metrics/distance_to_nearest_object_likelihood: 0.21332
scenario_metrics/collision_indication_likelihood: 0.99997
scenario_metrics/time_to_collision_likelihood: 0.99965
scenario_metrics/distance_to_road_edge_likelihood: 0.57061
scenario_metrics/offroad_indication_likelihood: 0.99997
aggregate_metrics/realism_meta_metric: 0.77725
aggregate_metrics/kinematic_metrics: 0.49455
aggregate_metrics/interactive_metrics: 0.82509
aggregate_metrics/map_based_metrics: 0.87730
aggregate_metrics/min_ade: 0.00000

Is this a normal behavior?

[pengzhenghao]

I wonder if there any issue inside the histogram_estimate of
waymo_open_dataset/wdl_limited/sim_agents_metrics/estimators.py:

If I asking for the log likelihood of all zeros vs all zeros:

If I asking for the log likelihood of the same array:

(in case you need it, I just randomly grab an agent's heading data:

You can find the output of histogram_estimate is quite wierd.

[nicomon24]

Hi @pengzhenghao,

This is by design, and it's an effect of having independent_timesteps=True in the metric config.
Why and how is also explained in section 4.2.1 of the paper (https://arxiv.org/pdf/2305.12032)
The effect of this is that the ground truth will always have a score lower than 1.0 (if you look at Figure 4, this was 0.722 in the 2023 metrics).

We are looking into making new changes to this, so look out for how the metrics are configured.

Just as a double-check that it's not the histogram_estimate, could you try to set independent_timesteps=False and re-run that eval? If I'm not mistaken, that should put everything to roughly 1.0