Skip to content

Break out optional tests into requirement-level directories #708

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Open
wants to merge 3 commits into
base: main
Choose a base branch
from
Open

Break out optional tests into requirement-level directories #708

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
3 commits
Select commit Hold shift + click to select a range
934616a
split format tests between MAY and SHOULD requirements
gregsdennis Nov 22, 2023
008726c
sort remaining tests
gregsdennis Nov 22, 2023
c1f1712
remove questions file
gregsdennis Nov 22, 2023
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
15 changes: 15 additions & 0 deletions tests/draft-next/may/format-configured-assertion/README.md
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -0,0 +1,15 @@
This folder contains tests to cover the case when the `format-annotation` vocabulary is present, meaning `format` should not be validated, but the implementation is configured to validate it anyway.

_Implementations must be configured to validate `format` when running these tests in order for the tests to be effective._ If the implementation does not support such a configuration, these tests may be skipped.

<!-- we'll need to update this link when draft/next is published -->
[Validation Section 7.2.1](https://json-schema.org/draft/2020-12/json-schema-validation#section-7.2.1) states

> When the implementation is configured for assertion behavior, it:
>
> - SHOULD provide an implementation-specific best effort validation for each format attribute defined below;
> - MAY choose to implement validation of any or all format attributes as a no-op by always producing a validation result of true;

The tests in herein cover the second point. As such, these test cases are invalid data for their respective formats, so validation is expected to fail.

Returns a passing validation for these scenarios is permitted.
71 changes: 71 additions & 0 deletions tests/draft-next/may/format-configured-assertion/date-time.json
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -0,0 +1,71 @@
[
{
"description": "validation of invalid date-time strings",
"schema": {
"$schema": "https://json-schema.org/draft/next/schema",
"format": "date-time"
},
"tests": [
{
"description": "an invalid date-time past leap second, UTC",
"data": "1998-12-31T23:59:61Z",
"valid": false
},
{
"description": "an invalid date-time with leap second on a wrong minute, UTC",
"data": "1998-12-31T23:58:60Z",
"valid": false
},
{
"description": "an invalid date-time with leap second on a wrong hour, UTC",
"data": "1998-12-31T22:59:60Z",
"valid": false
},
{
"description": "an invalid day in date-time string",
"data": "1990-02-31T15:59:59.123-08:00",
"valid": false
},
{
"description": "an invalid offset in date-time string",
"data": "1990-12-31T15:59:59-24:00",
"valid": false
},
{
"description": "an invalid closing Z after time-zone offset",
"data": "1963-06-19T08:30:06.28123+01:00Z",
"valid": false
},
{
"description": "an invalid date-time string",
"data": "06/19/1963 08:30:06 PST",
"valid": false
},
{
"description": "only RFC3339 not all of ISO 8601 are valid",
"data": "2013-350T01:01:01",
"valid": false
},
{
"description": "invalid non-padded month dates",
"data": "1963-6-19T08:30:06.283185Z",
"valid": false
},
{
"description": "invalid non-padded day dates",
"data": "1963-06-1T08:30:06.283185Z",
"valid": false
},
{
"description": "invalid non-ASCII '৪' (a Bengali 4) in date portion",
"data": "1963-06-1৪T00:00:00Z",
"valid": false
},
{
"description": "invalid non-ASCII '৪' (a Bengali 4) in time portion",
"data": "1963-06-11T0৪:00:00Z",
"valid": false
}
]
}
]
105 changes: 0 additions & 105 deletions tests/draft-next/optional/format/date.json → ...may/format-configured-assertion/date.json
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -6,166 +6,66 @@
"format": "date"
},
"tests": [
{
"description": "all string formats ignore integers",
"data": 12,
"valid": true
},
{
"description": "all string formats ignore floats",
"data": 13.7,
"valid": true
},
{
"description": "all string formats ignore objects",
"data": {},
"valid": true
},
{
"description": "all string formats ignore arrays",
"data": [],
"valid": true
},
{
"description": "all string formats ignore booleans",
"data": false,
"valid": true
},
{
"description": "all string formats ignore nulls",
"data": null,
"valid": true
},
{
"description": "a valid date string",
"data": "1963-06-19",
"valid": true
},
{
"description": "a valid date string with 31 days in January",
"data": "2020-01-31",
"valid": true
},
{
"description": "a invalid date string with 32 days in January",
"data": "2020-01-32",
"valid": false
},
{
"description": "a valid date string with 28 days in February (normal)",
"data": "2021-02-28",
"valid": true
},
{
"description": "a invalid date string with 29 days in February (normal)",
"data": "2021-02-29",
"valid": false
},
{
"description": "a valid date string with 29 days in February (leap)",
"data": "2020-02-29",
"valid": true
},
{
"description": "a invalid date string with 30 days in February (leap)",
"data": "2020-02-30",
"valid": false
},
{
"description": "a valid date string with 31 days in March",
"data": "2020-03-31",
"valid": true
},
{
"description": "a invalid date string with 32 days in March",
"data": "2020-03-32",
"valid": false
},
{
"description": "a valid date string with 30 days in April",
"data": "2020-04-30",
"valid": true
},
{
"description": "a invalid date string with 31 days in April",
"data": "2020-04-31",
"valid": false
},
{
"description": "a valid date string with 31 days in May",
"data": "2020-05-31",
"valid": true
},
{
"description": "a invalid date string with 32 days in May",
"data": "2020-05-32",
"valid": false
},
{
"description": "a valid date string with 30 days in June",
"data": "2020-06-30",
"valid": true
},
{
"description": "a invalid date string with 31 days in June",
"data": "2020-06-31",
"valid": false
},
{
"description": "a valid date string with 31 days in July",
"data": "2020-07-31",
"valid": true
},
{
"description": "a invalid date string with 32 days in July",
"data": "2020-07-32",
"valid": false
},
{
"description": "a valid date string with 31 days in August",
"data": "2020-08-31",
"valid": true
},
{
"description": "a invalid date string with 32 days in August",
"data": "2020-08-32",
"valid": false
},
{
"description": "a valid date string with 30 days in September",
"data": "2020-09-30",
"valid": true
},
{
"description": "a invalid date string with 31 days in September",
"data": "2020-09-31",
"valid": false
},
{
"description": "a valid date string with 31 days in October",
"data": "2020-10-31",
"valid": true
},
{
"description": "a invalid date string with 32 days in October",
"data": "2020-10-32",
"valid": false
},
{
"description": "a valid date string with 30 days in November",
"data": "2020-11-30",
"valid": true
},
{
"description": "a invalid date string with 31 days in November",
"data": "2020-11-31",
"valid": false
},
{
"description": "a valid date string with 31 days in December",
"data": "2020-12-31",
"valid": true
},
{
"description": "a invalid date string with 32 days in December",
"data": "2020-12-32",
Expand Down Expand Up @@ -211,11 +111,6 @@
"data": "2021-02-29",
"valid": false
},
{
"description": "2020 is a leap year",
"data": "2020-02-29",
"valid": true
},
{
"description": "invalid non-ASCII '৪' (a Bengali 4)",
"data": "1963-06-1৪",
Expand Down
61 changes: 61 additions & 0 deletions tests/draft-next/may/format-configured-assertion/duration.json
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -0,0 +1,61 @@
[
{
"description": "validation of duration strings",
"schema": {
"$schema": "https://json-schema.org/draft/next/schema",
"format": "duration"
},
"tests": [
{
"description": "an invalid duration string",
"data": "PT1D",
"valid": false
},
{
"description": "no elements present",
"data": "P",
"valid": false
},
{
"description": "no time elements present",
"data": "P1YT",
"valid": false
},
{
"description": "no date or time elements present",
"data": "PT",
"valid": false
},
{
"description": "elements out of order",
"data": "P2D1Y",
"valid": false
},
{
"description": "missing time separator",
"data": "P1D2H",
"valid": false
},
{
"description": "time element in the date position",
"data": "P2S",
"valid": false
},
{
"description": "weeks cannot be combined with other units",
"data": "P1Y2W",
"valid": false
},
{
"description": "invalid non-ASCII '২' (a Bengali 2)",
"data": "P২Y",
"valid": false
},
{
"description": "element without unit",
"data": "P1",
"valid": false
}
]
}
]
41 changes: 41 additions & 0 deletions tests/draft-next/may/format-configured-assertion/email.json
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -0,0 +1,41 @@
[
{
"description": "validation of e-mail addresses",
"schema": {
"$schema": "https://json-schema.org/draft/next/schema",
"format": "email"
},
"tests": [
{
"description": "an invalid e-mail address",
"data": "2962",
"valid": false
},
{
"description": "dot before local part is not valid",
"data": "[email protected]",
"valid": false
},
{
"description": "dot after local part is not valid",
"data": "[email protected]",
"valid": false
},
{
"description": "two subsequent dots inside local part are not valid",
"data": "[email protected]",
"valid": false
},
{
"description": "an invalid domain",
"data": "joe.bloggs@invalid=domain.com",
"valid": false
},
{
"description": "an invalid IPv4-address-literal",
"data": "joe.bloggs@[127.0.0.300]",
"valid": false
}
]
}
]
Loading