Skip to content

Commit bb9e720

Browse files
mmccoolmmccool
committed
initial checkin, arch assertions only, no results
1 parent 6c4c7e1 commit bb9e720

Some content is hidden

Large Commits have some content hidden by default. Use the searchbox below for content that may be hidden.

42 files changed

+13859
-35
lines changed

‎testing/README.md

+219
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,219 @@
1+
# Implementation Report
2+
3+
This directory contains the
4+
[WoT Thing Description Implementation report](https://w3c.github.io/wot-thing-description/testing/report.html)
5+
and the sources for it.
6+
The report itself is in [report.html](report.html).
7+
Do not edit this file,
8+
it is autogenerated by executing `npm run assertions` from the main directory,
9+
and any edits would be overwritten the next time you generate it.
10+
11+
## Adding Your Results
12+
13+
If you want to update your input for a Member organization, please go to [WoT IG testing](https://github.com/w3c/wot/tree/master/testing/tests/2019-05) and see the [README](https://github.com/w3c/wot/blob/master/testing/tests/2019-05/README.md) there.
14+
15+
## Updating Implementation Report
16+
17+
### w3c/wot Repository: Updating Results from Inputs
18+
19+
Test inputs are collected via the WoT IG and are currently in `wot/testing/tests/2019-05`.
20+
21+
---
22+
23+
#### First Time Only
24+
To update a test run to incorporate new results or test cases, first check out the `wot` repo,
25+
```
26+
git clone https://github.com/w3c/wot.git
27+
```
28+
and install the dependencies for the AssertionTester tool:
29+
```
30+
cd wot/testing/tools/thingweb-playground/AssertionTester
31+
npm install
32+
```
33+
34+
---
35+
36+
Go to the `wot/testing/tests/2019-05` directory.
37+
It contains a subdirectory for `descriptions` and one for `inputs`, which in turn contains TDs (TDs of the same implementation must be grouped in a further subdirectory) and manual assertion results as CSV files.
38+
Each is organized by Member organization.
39+
40+
To run all tests, while in the `wot/testing/tests/2019-05` directory,
41+
run the following:
42+
```
43+
./update.sh | tee update.log
44+
```
45+
This puts all the results in the `outputs` directory and also logs
46+
diagnostic output to `update.log`. To check for errors do
47+
```
48+
grep ERROR update.log > errors.log
49+
grep INVALID update.log >> errors.log
50+
```
51+
If there are no fatal errors (for example, a TD that does not validate or
52+
is not JSON), this file should be empty.
53+
54+
The output data files are now in `outputs/*/*.csv`.
55+
Check that there are no zero-length files in this list using `ls -l`.
56+
If there are it means something has gone wrong, most likely an input
57+
CSV file with a syntax error. An easy way to check for CSV syntax errors
58+
is to open the file in github. If it does not render properly as a table,
59+
there is an error, and GitHub will even point to the offending line.
60+
61+
### w3c/wot-thing-description: Importing Results
62+
63+
Go back to the `wot-thing-description/testing` directory and update the
64+
inputs for the report generator in the `inputs/implementations` subdirectory:
65+
```
66+
cd ./wot-thing-description/testing/inputs/implementations
67+
rm *.html
68+
cp ../../../../wot/testing/tests/2019-05/descriptions/*/*.html .
69+
```
70+
And in the `inputs/results` subdirectory:
71+
```
72+
cd ../results
73+
rm *.csv
74+
cp ../../../../wot/testing/tests/2019-05/outputs/*/*.csv .
75+
```
76+
77+
### w3c/wot-thing-description: Generating Report
78+
79+
Now return to the main `wot-thing-description` directory and regenerate the report:
80+
```
81+
cd ../../../
82+
npm run assertions
83+
```
84+
85+
If you need to update the at-risk highlighting in the TD specification,
86+
you may also need to
87+
run `npm run render` afterwards.
88+
If new assertions have been added to the
89+
TD specification then you will need to run `npm run render` BEFORE generating
90+
the report.
91+
92+
## Suppressing Assertions
93+
The file
94+
[suppressed.csv](suppressed.csv)
95+
can be used to list assertions for which
96+
test results should be ignored. Such results will also not be used for
97+
"roll-up" results (when a child assertion, one with an underscore in its
98+
name, is used to break down assertions with multiple options into simpler,
99+
separately testable assertions).
100+
101+
## Manual Assertions
102+
As already mentioned,
103+
[manual.csv](manual.csv)
104+
can be used to identify assertions that need
105+
manual testing or declarations.
106+
107+
## At-Risk Highlighting
108+
The file
109+
[atrisk.css](atrisk.css) provides highlighting for at-risk elements in
110+
the report but should NOT be edited directly as it is autogenerated.
111+
Instead edit
112+
[inputs/atrisk.csv](inputs/atrisk.csv).
113+
114+
## Report Template
115+
The HTML template for the Implementation Report is in
116+
[inputs/template.html](inputs/template.html).
117+
If you wish to edit the main explanatory text of the report or
118+
update the metadata (date, authors, etc.), do so here.
119+
120+
## Implementation Descriptions
121+
Implementations are described in HTML files in
122+
[inputs/implementations](inputs/implementations), with one
123+
file per contributing organization.
124+
The IDs declared in these descriptions should be unique and
125+
descriptive as they will be used elsewhere, for example,
126+
in the interoperability data files referring to those implementations.
127+
To add a new description, use the template in
128+
[inputs/implementations/template.html.t](inputs/implementations/template.html.t).
129+
130+
Implementations should also be entered into the table
131+
in [inputs/impl.csv](inputs/impl.csv) with identifiers, titles, and roles
132+
consistent with those assigned in the above HTML files.
133+
An implementation can either be a consumer,
134+
a producer (eg exposing a TD, a Thing), or both (eg a Servient).
135+
136+
## Test Specifications
137+
A procedure for testing each normative assertion should be given in
138+
[inputs/testspec.html](inputs/testspec.html).
139+
These may be included as an appendix in the report,
140+
but are currently suppressed.
141+
142+
Test specifications can be given both for
143+
assertions given in the specification
144+
(see [inputs/results/template.csv.t](inputs/results/template.csv.t)
145+
for an automatically
146+
generated list of identifiers) and for any extra assertions in
147+
[inputs/extra-asserts.html](inputs/extra-asserts.html).
148+
149+
## Extra Assertions
150+
Assertions used for testing but not (yet) included in the specification
151+
may be listed in
152+
[inputs/extra-asserts.html](inputs/extra-asserts.html).
153+
The intention is that these assertions should
154+
eventually, and before final release, be inserted into the final specification.
155+
156+
## At-Risk Assertions
157+
Assertions related to features that are at risk of being deleted from the final
158+
CR should be identified in the [inputs/atrisk.csv](inputs/atrisk.csv) file.
159+
The assertion text for these will be given a special color in the report table.
160+
161+
## Categories
162+
Assertions can be assigned to a category in
163+
[inputs/categories.csv](inputs/categories.csv).
164+
Eventually the table _may_ be sorted to group categories
165+
into sections; for now it is just an extra column.
166+
167+
## Dependencies
168+
Dependencies between assertions can be recorded in
169+
[inputs/depends.csv](inputs/depends.csv).
170+
The "Parents" column relates detailed assertions to more general assertions.
171+
the "Contexts" column indicates assertions that only need to be considered in
172+
a particular context (either syntactic, if pointing at another vocabulary item,
173+
or logical, if pointing at another optional assertion).
174+
Both "Parents" and "Contexts" may have multiple items separated by spaces.
175+
Entries should be IDs of other assertions.
176+
Use `"null"` if there is no dependency.
177+
178+
## Result Data
179+
Each implementation should record
180+
which features they have implemented and tested under the
181+
[inputs/results](inputs/results) directory.
182+
All data will be read and merged into the report.
183+
Mark each implemented feature with a status of either
184+
`"pass"` (if it satisfies the specification)
185+
or
186+
`"fail"` (if it does not).
187+
If you have not implemented a feature, list its status as `"not-impl"`.
188+
189+
Features not listed will not be included in the sums; this is
190+
distinct from `"not-impl"` as absence of a feature is meant to be used
191+
to allow different features to be reported in different files.
192+
193+
If you did not implement a feature on purpose please indicate this explicitly.
194+
195+
Any other status will be ignored (e.g., `"null"` as used in the template).
196+
If you have tested a feature in multiple
197+
implementations check in one file per implementation, using as a filename
198+
the id given in the template for the implementations' description.
199+
Use a convention
200+
like `ORG-IMPL-MODULE.csv` for the filename.
201+
The filename should also be used as an id in the
202+
description of each implementation.
203+
204+
The [inputs/results/template.csv](inputs/results/template.csv)
205+
file lists all features but with a `"null"` status.
206+
*Do not edit this file; it is autogenerated.*
207+
It is provided so
208+
you can use it as a reference and as a basis for your own data files.
209+
210+
Files should be in CSV format, including headers as defined in
211+
[inputs/results/template.csv](results/tempate.csv),
212+
and will be parsed by the `csvtojson` Node.js library.
213+
214+
## To Dos
215+
* Test specs need to be completed.
216+
* Interop and results data needs to be collected and collated.
217+
* Sort categories together in the output and put the category in a section header instead of a column.
218+
* Sanity check that parent assertions are only marked as "pass" if all sub-assertions are also marked as "pass".
219+

‎testing/assertions.csv

+35-35
Original file line numberDiff line numberDiff line change
@@ -1,35 +1,35 @@
1-
"ID","Status","Comment","Assertion"
2-
"arch-td-metadata","null","not testable with Assertion Tester","In W3C WoT, the description metadata for a Thing instance MUST be available as a WoT Thing Description (TD) [[?WOT-THING-DESCRIPTION]]."
3-
"arch-other-thing-representations","null","not testable with Assertion Tester","There MAY be other representations of a Thing such as an HTML-based user interface, simply an image of the physical entity, or even non-Web representations in closed systems."
4-
"arch-td-mandatory","null","not testable with Assertion Tester","To be considered a Thing, however, at least one TD representation MUST be available."
5-
"arch-td-linking","null","not testable with Assertion Tester","WoT Thing Descriptions MAY link to other Things and other resources on the Web to form a Web of Things."
6-
"arch-id-correlation","null","not testable with Assertion Tester","An identifier in the WoT Thing Description MUST allow for the correlation of multiple TDs representing the same original Thing or ultimately unique physical entity."
7-
"arch-affordances","null","not testable with Assertion Tester","In addition to navigation affordances (i.e., Web links), Things MAY offer three other types of Interaction Affordances defined by this specification: Properties, Actions, and Events."
8-
"arch-property-dataschema","null","not testable with Assertion Tester","If the data is not fully specified by the Protocol Binding used (e.g., through a media type), Properties MAY contain one data schema for the exposed state."
9-
"arch-action-functions","null","not testable with Assertion Tester","An Action MAY manipulate state that is not directly exposed (cf. Properties), manipulate multiple Properties at a time, or manipulate Properties based on internal logic (e.g., toggle)."
10-
"arch-action-process","null","not testable with Assertion Tester","Invoking an Action MAY also trigger a process on the Thing that manipulates state (including physical state through actuators) over time."
11-
"arch-action-dataschema","null","not testable with Assertion Tester","If the data is not fully specified by the Protocol Binding used (e.g., through a media type), Actions MAY contain data schemas for optional input parameters and output results."
12-
"arch-event-trigger","null","not testable with Assertion Tester","Events MAY be triggered through conditions that are not exposed as Properties."
13-
"arch-event-dataschema","null","not testable with Assertion Tester","If the data is not fully specified by the Protocol Binding used (e.g., through a media type), Events MAY contain data schemas for the event data and possible subscription control messages (e.g., to subscribe with a Webhook callback URI)."
14-
"arch-rel-types","null","not testable with Assertion Tester","Extension relation types MUST be compared as strings using a case-insensitive comparison. (If they are serialized in a different format they are to be converted to URIs)."
15-
"arch-rel-type-lowercase","null","not testable with Assertion Tester","Nevertheless, all-lowercase URIs SHOULD be used for extension relation types. [[!RFC8288]]"
16-
"arch-form-iris","null","not testable with Assertion Tester","Form contexts and submission targets MUST both be Internationalized Resource Identifiers (IRIs) [[!RFC3987]]."
17-
"arch-form-iris2","null","not testable with Assertion Tester","Form context and submission target MAY point to the same resource or different resources, where the submission target resource implements the operation for the context."
18-
"arch-op-wellknown","null","not testable with Assertion Tester","Well-known operation types MUST follow the ABNF LOALPHA *( LOALPHA / DIGIT / ""."" / ""-"" )."
19-
"arch-op-extension-uri","null","not testable with Assertion Tester","Extension operation types MUST be URIs [[!RFC3986]] that uniquely identify the type."
20-
"arch-op-extension-comparison","null","not testable with Assertion Tester","Extension operation types MUST be compared as strings using a case-insensitive comparison."
21-
"arch-op-extension-lowercase","null","not testable with Assertion Tester","Nevertheless, all-lowercase URIs SHOULD be used for extension operation types."
22-
"arch-op-request-method","null","not testable with Assertion Tester","The request method MUST identify one method of the standard set of the protocol identified by the submission target URI scheme."
23-
"arch-op-expected-request","null","not testable with Assertion Tester","Form fields are optional and MAY further specify the expected request message for the given operation."
24-
"arch-op-form-fields-protocol","null","not testable with Assertion Tester","Form fields MAY depend on the protocol used for the submission target as specified in the URI scheme."
25-
"arch-hypermedia","null","not testable with Assertion Tester","Interaction Affordances MUST include one or more Protocol Bindings."
26-
"arch-hypermedia-protocol-binding","null","not testable with Assertion Tester","Protocol Bindings MUST be serialized as hypermedia controls to be self-descriptive on how to activate the Interaction Affordance."
27-
"arch-hypermedia-caching","null","not testable with Assertion Tester","The hypermedia controls MAY be cached outside the Thing and used for offline processing if caching metadata is available to determine the freshness."
28-
"arch-media-type","null","not testable with Assertion Tester","All data (a.k.a. content) exchanged when activating Interaction Affordances MUST be identified by a media type [[!RFC2046]] in the Protocol Binding."
29-
"arch-media-type-extra","null","not testable with Assertion Tester","Protocol Bindings MAY have additional information that specifies representation formats in more detail than the media type alone."
30-
"arch-schema","null","not testable with Assertion Tester","Thus, the corresponding Interaction Affordances SHOULD declare a data schema to provide more detailed syntactic metadata for the data exchanged."
31-
"arch-intermediary-td-extra-protocols","null","not testable with Assertion Tester","If necessary, a TD generated by an Intermediary MAY contain interfaces for other communication protocols."
32-
"arch-discovery-no-metadata-in-introduction","null","not testable with Assertion Tester","Any introduction mechanism used for WoT Discovery MUST NOT provide metadata but instead should simply result in one or more opaque URLs at which metadata may be accessed, if and only if the user can authenticate and has the appropriate authorization (access rights)."
33-
"arch-discovery-no-metadata-in-introduction-urls","null","not testable with Assertion Tester","Introduction URLs MUST NOT themselves include any metadata, e.g. a human-readable device type or name."
34-
"arch-discovery-metadata-after-authorization","null","not testable with Assertion Tester","Any WoT Discovery ""exploration"" implementation MUST only provide metadata and TDs after appropriate best-practice authentication and authorization."
35-
"arch-discovery-should-use-standard","null","not testable with Assertion Tester","Whenever possible, the Discovery mechanisms defined in [[!WOT-DISCOVERY]] SHOULD be used to distribute TDs."
1+
"ID","Status","Comment"
2+
"arch-action-dataschema","null",
3+
"arch-action-functions","null",
4+
"arch-action-process","null",
5+
"arch-affordances","null",
6+
"arch-discovery-metadata-after-authorization","null",
7+
"arch-discovery-no-metadata-in-introduction","null",
8+
"arch-discovery-no-metadata-in-introduction-urls","null",
9+
"arch-discovery-should-use-standard","null",
10+
"arch-event-dataschema","null",
11+
"arch-event-trigger","null",
12+
"arch-form-iris","null",
13+
"arch-form-iris2","null",
14+
"arch-hypermedia","null",
15+
"arch-hypermedia-caching","null",
16+
"arch-hypermedia-protocol-binding","null",
17+
"arch-id-correlation","null",
18+
"arch-intermediary-td-extra-protocols","null",
19+
"arch-media-type","null",
20+
"arch-media-type-extra","null",
21+
"arch-op-expected-request","null",
22+
"arch-op-extension-comparison","null",
23+
"arch-op-extension-lowercase","null",
24+
"arch-op-extension-uri","null",
25+
"arch-op-form-fields-protocol","null",
26+
"arch-op-request-method","null",
27+
"arch-op-wellknown","null",
28+
"arch-other-thing-representations","null",
29+
"arch-property-dataschema","null",
30+
"arch-rel-type-lowercase","null",
31+
"arch-rel-types","null",
32+
"arch-schema","null",
33+
"arch-td-linking","null",
34+
"arch-td-mandatory","null",
35+
"arch-td-metadata","null",

‎testing/atrisk.css

Whitespace-only changes.

‎testing/inputs/atrisk.csv

+1
Original file line numberDiff line numberDiff line change
@@ -0,0 +1 @@
1+
"ID","Fix"

0 commit comments

Comments
 (0)