Skip to content

Commit 6a92447

Browse files
committed
Add some additional planned features to the tests-coverage page
1 parent 9d2bf25 commit 6a92447

File tree

1 file changed

+28
-0
lines changed

1 file changed

+28
-0
lines changed

tests-coverage.md

+28
Original file line numberDiff line numberDiff line change
@@ -65,3 +65,31 @@ The following things are tested
6565
| var | Yes | Yes | Yes | Yes | | |
6666

6767
[^1]: `sum` and `prod` have special type promotion rules.
68+
69+
## Additional Planned Features
70+
71+
In addition to getting full coverage of the spec, there are some additional
72+
features and improvements for the test suite that are planned. Work on these features
73+
will be guided primarily by concrete needs from library implementers, so if
74+
you are someone using this test suite to test your library, please [let us
75+
know](https://github.com/data-apis/array-api-tests/issues) the limitations you
76+
come across.
77+
78+
- Making the test suite more usable for partially conforming libraries. Many
79+
tests rely on various functions in the array library to function. This means
80+
that if certain functions aren't implemented, for example, `asarray()` or
81+
`equals()`, then many tests will not function at all. We want to improve
82+
this situation, so that tests that don't strictly require these functions can
83+
still be run.
84+
85+
- Better reporting. The pytest output can be difficult to parse, especially
86+
when there are many failures. Additionally some error messages can be
87+
difficult to understand without prior knowledge of the test internals.
88+
Better reporting can also make it easier to compare different
89+
implementations by their conformance.
90+
91+
- Better tests for numerical outputs. Right now numerical outputs are either
92+
not tested at all, or only tested against very rough epsilons. This is
93+
partly due to the fact that the spec does not mandate any level of precision
94+
for most functions. However, it may be useful to, for instance, give a
95+
report of how off a given function is from the "expected" exact output.

0 commit comments

Comments
 (0)