Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Improve QCheck2 list_size shrinker #324

Open
jmid opened this issue Feb 19, 2025 · 0 comments
Open

Improve QCheck2 list_size shrinker #324

jmid opened this issue Feb 19, 2025 · 0 comments

Comments

@jmid
Copy link
Collaborator

jmid commented Feb 19, 2025

From #319:

This PR takes a first step towards improving the QCheck2 list shrinker.
The current one uses the input size tree and offers only prefixes, e.g., shrinking [1;2;3;4;5;6;7;8] to [], [1;2;3;4], [1;2;3;4;5;6], [1;2;3;4;5;6;7] (or something like that). This results in needlessly large counterexamples, if the property is essentially "lists do not contain the int 7". 😬
[...]
The PR does not improve the QCheck2.Gen.{list_size, array_size, bytes_size, string_size} generators.
These take a size generator parameter, and I've not figured out how to cleanly respect that (read: preserve invariants), e.g., if a user provides a size generator of "even numbers", "multiples of 7", or "prime numbers". As such,
the shrinking behaviour and performance of those remain untouched.

and from #321:

The exception is the shrinker for QCheck2.Gen.list_size (and derivatives) which I've not attempted to improve.
For reference below I include a shrink_bench.log output for OCaml 4.14.2 on my laptop dominated by

  • long_shrink - 2.070s
  • lists shorter than 432 - 1.034s
  • lists shorter than 4332 - 7.541s
  • fold_left test, fun first - 1.908s

Of these the 3 first use QCheck2.Gen.list_size - and the last uses a function first (a known pain point also for QCheck(1)),

For a start, on a size 8 list generating say [1;2;3;4;5;6;7;8]
for a size shrink tree output of, e.g., 4, in addition to the prefix [1;2;3;4] one could try the suffix [5;6;7;8], the middle [3;4;5;6] and/or dropping elements, e.g., yielding [1;3;5;7]. For a smaller size shrink tree output of, e.g., 2, the challenge will be not to emit too many shrinking candidates as that may ruin performance as previously discussed for QCheck(1) in #268 #242 #235. The lesson from the QCheck(1) improvements is to yield in the order of O(log n) shrinking candidates, as that both (a) composes well and (b) scales better to large lists.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

No branches or pull requests

1 participant