Fewer tests running for Jax than other frameworks

[simonetgordon]

At Ivy, we have noticed that a relatively small number of examples are tested for Jax compared to the other frameworks e.g. currently, for Tensorflow there are 3000+ test examples running:

309 failed, 3032 passed, 275 skipped, 183 deselected, 5 warnings in 5041.71s (1:24:01)

Whereas for Jax, most are being deselected :

201 passed, 11 skipped, 3587 deselected, 67 warnings in 2801.07s (0:46:41)

I can't find anything in the hypothesis docs about test deselection and was redirected here by a member of their repo. We're a bit puzzled as we handle the testing of each of the frameworks in the same way. Do you have any insight into why we're experiencing this?

[honno]

Discussion in #115 might be relevant.

Yeah so running your master branch of ivy on in the internal Hypothesis test suite produces some errors, most of which seem to indeed be because of the timings you demonstrated, and these issues ultimately propagates in filtered examples. All those tests definitely need to be passing for Ivy's JAX backend to be reliably tested in array-api-tests too.

FYI to run ivy's JAX backend on the internal Hypothesis test suite, I replaced these lines with:

import ivy
ivy.set_framework("jax")
params = [pytest.param(ivy, make_strategies_namespace(ivy), id="ivyjax")]

Now this could very well be a hard limitation of JAX's use of JIT, where we might struggle without indeed introducing a flag like --deadline=... (0 could mean disable). I would want a JAX maintainer to start implementing a namespace before supporting this use case, seeing as Ivy is third-party and could indeed be handling things like device management inefficiently, which would exacerbate slow first-time array creation into something not practical to test with in the first place.

In any case, will explore this but low-prio.

[asmeurer]

I'm also unclear what "deselected" means. Is that a hypothesis thing, or is that something that we're doing?

[simonetgordon]

Update

This can be closed. It was due to an issue in how we were running the tests on CI.