Log broken invariant as soon as it is found

[aviggiano]

Component

Forge

Describe the feature you would like

I am trying to benchmark Foundry invariant tests against other fuzzers, and it would be really important to have real time information of the exact time an invariant breaks.

Additional context

N/A

[grandizzy]

@aviggiano sure, are you looking for how long it took or the exact time when was found (without shrunk phase)? Please provide all the details you'd like to get, thank you!

[aviggiano]

I was thinking about adding an extra "invariant failed" to the metrics JSON that is already outputted

Then by simply parsing the logs we'd be able to tell.

Another option is to keep an "invariants failed" array that would keep growing as time passes, but this would make the "metrics" JSON structure a bit less flatten, so different than the current on

[0xalpharush]

We should not really consider this log stable as when we are asserting all invariants in all threads, it will change (#12587). Rn it has measurements per invariant test instead of globally

[Kushmanmb]

lol what do you want?

[aviggiano]

fyi I'm using this vibe coded quick fix so that I don't depend on the referenced PR for my analysis master...aviggiano:foundry:master

[0xalpharush]

I think it'd be useful to add to the metrics the aggregate failures and unique_failures. For unique crashes, there can be a separate event that tells which invariant is broken.

{"timestamp":1770241125,"event":"pulse","metrics":{"unique_failures": 0, "failures": 0, cumulative_edges_seen":273,"cumulative_features_seen":0,"corpus_count":78,"favored_items":0}}
{"timestamp":1770241126,"event":"failure","invariant":"invariant_*", "target": "InvariantTest"}
{"timestamp":1770241130,"event":"pulse","metrics":{"unique_failures": 1 "failures": 1, {"cumulative_edges_seen":273,"cumulative_features_seen":0,"corpus_count":78,"favored_items":0}}