Update from upstream schema.org master 2020-07-07 (v.8.x)

.gitignore

@@ -4,6 +4,12 @@
 /scripts/spec/schema.rb
 /scripts/spec/schema_context.rb
 /scripts/spec/examples_spec.rb
+/staticbuild/preversion.txt
+/staticbuild/handlers.yaml
+/staticbuild/handlers-local.yaml
+/staticbuild/docs
+/staticbuild/releases
+/staticbuild/terms
 /.byebug_history
 /scripts/rspec-failures.txt
 /rspec-failures.txt

.gitmodules

@@ -0,0 +1,4 @@
+[submodule "sdopythonapp"]
+	path = sdopythonapp
+	url = https://github.com/schemaorg/sdopythonapp.git
+	branch = master

.travis.yml

@@ -7,15 +7,6 @@ cache: bundler
 
 jobs:
   include:
-    - stage: xmllint
-      name: "XML Lint RDFa"
-      language: python
-      python: "2.7"
-      before_script:
-        - sudo apt-get install -y libxml2-utils
-      script:
-        - xmllint --noout data/*.rdfa data/ext/*/*.rdfa
-
     - stage: python
       name: "Python tests"
       language: python

README.md

@@ -4,6 +4,8 @@ Welcome to Schema.org
 
 This is the Schema.org project repository. It contains all the schemas, examples and software used to publish schema.org. For the site itself, please see [Schema.org](https://schema.org/) instead.
 
+Note: Much of the supporting software is imported from a sub module: 'sdopythonapp'
+
 Issues and proposals are managed here by participants of the [W3C Schema.org Community Group](https://www.w3.org/community/schemaorg/). If you are interested to participate please
 join the group at the [W3C](https://www.w3.org/community/schemaorg/), introduce yourself and find or file issues here that engage your interest. If you are new to Git and GitHub, there's a useful [introduction to Github](https://www.w3.org/2006/tools/wiki/Github) in the W3C Wiki.
 
@@ -12,15 +14,14 @@ There is also a [Travis-CI integration](https://travis-ci.org/schemaorg/schemaor
 [Issue #1](https://github.com/schemaorg/schemaorg/issues/1) in Github is an entry point for release planning. It
 should provide an overview of upcoming work, in terms of broad themes, specific issues and release milestones.
 
-With every release, we change the default Github branch to match the code name for the release.
 [Issue #1](https://github.com/schemaorg/schemaorg/issues/1) will link to per-release entry points, or else navigate issues via label or milestone within Github.
 Every change to the site comes via discussions here. Substantive changes are recorded in our [release notes](https://schema.org/docs/releases.html).
 A preview of the [draft new release notes](http://webschemas.org/docs/releases.html) can be found as part of the test site for our next release.
-Every month or so, after final review by the Schema.org Steering Group, we make a formal release.
+Every month or so, after final review by the Schema.org Steering Group and wider community, we make a formal release.
 
 Regarding CC and opensource licenses for documents and software, see our [FAQ entry](https://schema.org/docs/faq.html#18).
 
-Proposing schemas
+Improving schemas
 =================
 
 We are always interested in practical suggestions for improvements to schema.org, and our collection of schemas has been [growing gradually](http://schema.org/docs/releases.html) since our launch in 2011.
@@ -32,7 +33,7 @@ allow for [independent extension](http://schema.org/docs/extension.html) (for ex
 
 We are also highly unlikely to take on large scale reorganizations of Schema.org's terminology, if they are motivated solely  by considerations of elegance, "proper modeling", ontological purity or conceptual unification. Although the project founders and team are familiar with - and respectful of - the traditions behind such concerns, the scale, scope and nature of Schema.org has required us to trade elegance and global consistency for a somewhat scruffier notion of incremental evolution and a pragmatic tolerance for a style that would be out of place in a formal ontology. Proposals for unifying, cross-domain logic-based knowledge structures may be better received e.g. in the [Ontolog](https://groups.google.com/forum/#!forum/ontolog-forum) community.
 
-We sometimes introduce types without dedicated property associations, simply for markup usability reasons. In a formal ontology, this is often considered poor modeling. However, logically equivalent structures can result in many more errors from publishers/webmasters unfamiliar with the underlying formal concepts behind JSON-LD or RDF/S. Schema.org is not a closed system, and other initiatives e.g. Wikidata or [GS1](http://gs1.org/voc/) have defined many other terms that can be mixed in alongside those we define at schema.org. We also make efforts to align our designs to relevant external standards and initiatives, even when it reduces the global elegance of Schema.org considered alone. For example in a bibliographic or cultural heritage context we may be influenced by initiatives like MARC, BibFrame, and FRBR, while for e-commerce we collaborated with [Good Relations](http://blog.schema.org/2012/11/good-relations-and-schemaorg.html) and GS1. Schema.org's [news](https://schema.org/docs/news.html)-related terms were heavily influenced by incorporating [IPTC's rNews](https://iptc.org/standards/rnews/) design, alongside collaborations with [fact checkers](https://courses.poynter.org/courses/course-v1:newsu+nwsu_ClaimReview2017+2017_1/about), the [Trust Project](https://thetrustproject.org/), and others. Our TV and Music related vocabularies are heavily influenced by working with the [BBC and the European Broadcasting Union](https://www.bbc.co.uk/ontologies/po), alongside [the Music ontology](http://musicontology.com/docs/getting-started.html) and [MusicBrainz](https://musicbrainz.org/doc/LinkedBrainz/RDF); our schemas reflect these prior designs. We prefer to collaborate in this way, improving Schema.org incrementally and working to polish, integrate and blend such designs rather than produce our own pure model in isolation. The result may lack global elegance but brings our work into alignment with related efforts worldwide. 
+We sometimes introduce types without dedicated property associations, simply for markup usability reasons. In a formal ontology, this is often considered poor modeling. However, logically equivalent structures can result in many more errors from publishers/webmasters unfamiliar with the underlying formal concepts behind JSON-LD or RDF/S. Schema.org is not a closed system, and other initiatives e.g. Wikidata or [GS1](http://gs1.org/voc/) have defined many other terms that can be mixed in alongside those we define at schema.org. We also make efforts to align our designs to relevant external standards and initiatives, even when it reduces the global elegance of Schema.org considered alone. For example in a bibliographic or cultural heritage context we may be influenced by initiatives like MARC, BibFrame, and FRBR, while for e-commerce we collaborated with [Good Relations](http://blog.schema.org/2012/11/good-relations-and-schemaorg.html) and GS1. Schema.org's [news](https://schema.org/docs/news.html)-related terms were heavily influenced by incorporating [IPTC's rNews](https://iptc.org/standards/rnews/) design, alongside collaborations with [fact checkers](https://courses.poynter.org/courses/course-v1:newsu+nwsu_ClaimReview2017+2017_1/about), the [Trust Project](https://thetrustproject.org/), and others. Our TV and Music related vocabularies are heavily influenced by working with the [BBC and the European Broadcasting Union](https://www.bbc.co.uk/ontologies/po), alongside [the Music ontology](http://musicontology.com/docs/getting-started.html) and [MusicBrainz](https://musicbrainz.org/doc/LinkedBrainz/RDF); our schemas reflect these prior designs. We prefer to collaborate in this way, improving Schema.org incrementally and working to polish, integrate and blend such designs rather than produce our own pure model in isolation. The result may lack global elegance but brings our work into alignment with related efforts worldwide.
 
 We always welcome issues that track usability and readability issues, but encourage a focus on concrete situations (e.g. how to describe repeating events) rather than global philosophical concerns (e.g. whether a Reservation or Action is "really" an Event). We prioritize local coherence (having sensible ways to describe many common situations) over global elegance (having a global theory in which everything has a sensible place). This doesn't mean we never have cleanups, but they are balanced against (and often outweighed by) other considerations.
 
@@ -44,7 +45,7 @@ When we add terms, often into the "Pending" area, we strongly encourage feedback
  * While many Schema.org improvements have been proposed via Github's "[Pull request](https://help.github.com/articles/about-pull-requests/)" mechanism (see also our list of [PRs](https://github.com/schemaorg/schemaorg/pulls)), please do not undertake any substantial development work without agreeing it with the project team here first.
  * All Pull Requests should reference specific issues that they're fixes or solutions for. This lets the schema.org community discuss problems and topics without it being tied too closely to a specific (and easily outdated) proposed fix.
  * Please note that some changes are much easier to make than others: the wording/phrasing in definitions is relatively easy to amend, whereas the exact spelling of a type or property ('Person', 'startDate' etc.) is much more disruptive to change.
- * There are many other projects developing schemas and ontologies for the Web, e.g. [Wikidata](http://wikidata.org/) or the vocabulary projects in the [Linked Data](http://lov.okfn.org/) community. Many of these projects go into more expressive detail than is possible for a project like Schema.org. To keep Schema.org manageable, we have a strong bias towards designs that are grounded in large scale usage on the Web, in particular [usage](https://github.com/schemaorg/schemaorg/issues/652) by data-consuming applications since these in turn motivate data publishers. Other schema initiatives have different priorities and make different tradeoffs. 
+ * There are many other projects developing schemas and ontologies for the Web, e.g. [Wikidata](http://wikidata.org/) or the vocabulary projects in the [Linked Data](http://lov.okfn.org/) community. Many of these projects go into more expressive detail than is possible for a project like Schema.org. To keep Schema.org manageable, we have a strong bias towards designs that are grounded in large scale usage on the Web, in particular [usage](https://github.com/schemaorg/schemaorg/issues/652) by data-consuming applications since these in turn motivate data publishers. Other schema initiatives have different priorities and make different tradeoffs.
 
 See more on ["How we work"](https://schema.org/docs/howwework.html)
 
@@ -54,7 +55,7 @@ Software
 
 For most collaborators, all you need to know about the software is how to run it. Essentially you will need to have the Python version of Google App Engine SDK running on the platform of your choice. You can then make test builds of schema.org running on your own machine accessible as http://localhost:8080/ or else post them on appspot.com for collaboration. See the [Appengine documentation](https://cloud.google.com/appengine/docs) for details.
 
-More information about the software is also available in [SOFTWARE_README.md](SOFTWARE_README.md)
+More information about the software is also available in [SOFTWARE_README.md](SOFTWARE_README.md) and [STATICBUILD_README.md](./staticbuild/STATICBUILD_README.md)
 
 See also notes in the wiki: https://github.com/schemaorg/schemaorg/wiki/Contributing
 

RELEASING.TXT

@@ -8,32 +8,33 @@ Code, schema, examples and supporting documentation should be in a stable state.
 Pay particular attention to release-specific structures and docs/releases.html
 (e.g. no stray '2.x', '2.*' etc. in hidden markup or tables)
 
-In sdoapp.py we declare the current version. This should match the number
+We declare the current version. This should match the number
 chosen in /docs/releases.html (check markup assigns an HTML ID too).
 
 The following steps assume a general healthy freeze (tested / QA as below).
 
-in sdoapp.py:
-
-    SCHEMA_VERSION=2.2 # or 2.x, as above.
-
-in sdoapp.py:
-
-  releaselog = { "2.0": "2015-05-13", "2.1": "2015-08-06" }
+in versions.json:
 
+    "schemaversion": "3.9",
+    "releaseLog": {
+.
+.
+        "3.8": "2019-07-01",
+        "3.9": "2019-08-01"
+    }
 becomes
-
-  releaselog = { "2.0": "2015-05-13", "2.1": "2015-08-06", "2.2": "2015-11-05" }
-
+    "schemaversion": "4.0",
+    "releaseLog": {
+.
+.
+        "3.8": "2019-07-01",
+        "3.9": "2019-08-01",
+        "4.0": "2019-10-01"
+    }
 
 ... this should have a release date for the current release and all
 previous releases that are archived under data/releases/{version}/*
 
-TODO: Add a unit test to ensure this.
-
-TODO: Re-order this document to avoid forward references.
-
-Under data/releases/{version}/ we need a versioned-snapshot structure:
 
 e.g.
     data/releases/2.1
@@ -54,18 +55,6 @@ Example usage:
     scripts/buildreleasefiles.sh 3.9
 
 
-* Inspect snapshot, add to git and push to repo.
- * ls -l data/releases/2.2/
- * git add data/releases/2.2/*
- * git push
- * TODO: more QA and automation would be highly appropriate here.
-
-
-
-TODO: list optional tasks post-launch (blog, email, refresh dydra sparql db etc.).
-
-
-
 
 1) General preconditions / process and QA for release.
 
@@ -116,71 +105,39 @@ e.g. see http://webschemas.org/docs/qa.html
 There is often some final tweaking, bugs, etc. When things settle down, a
 final round of sanity-check tests is useful:
 
-* Run again './scripts/buildsitemap.py' make sure sitemap is up to date
-* Run again: PYTHONPATH=/usr/local/google_appengine ./scripts/run_tests.py
-* Run again 'git status', confirm no uncommited files or changes.
-* Run again: './scripts/updateAppEngine.sh' for development/upstream site sync.
-
-3.) Previously: "Merge release branch into master"; now we develop in "master."
-
-4.) Publish from a clean checkout of 'master' branch to schema.org.
-
-Using gcloud app deploy
-~~~~~~~~~~~~~~~~~~~~~~~
-
-From 3.4.1 onwards use app deploy script (does a bit of housekeeping):
-	scripts/appdeploy.sh --project schemaorgae  --no-promote --version=3-5-1 schemaorg.yaml
-
-
-Recommended Method For Making Release Live
-~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
+* Run again './scripts/buildreleasefiles.py'
 
-1) Use app deploy script as above providing a version identifier not in use on schemaorgae appengine.
 
-2) Access that version directly via the URL of following format:
-      https://3-5-1-dot-schemaorgae.appspot.com
-
-      Note: The application will take a few minutes before responding to its first request.
-            If you get a timeout message, try again in 30 seconds or so.
+Using gcloud deployment script
+~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
 
-3) Try a few samples requests to the direct URL 
+Use python deploment script:
 
-4) In directory from which the deploy took place run the following script, with the approriate site (-s) value:
-	scripts/exercisesite.py -s https://3-5-1-dot-schemaorgae.appspot.com
-
-	The script requests a page for each Type and property, causing them to be cached in the application's storage
-	area.  It averages a few seconds per page, the whole process takes 60-90 minutes.
-
-5) When exercisesite is complete, use the Traffic Spiting feature of the App Engine Console to progressively increase 
-   the percentage of traffic being allocated to the new version (5%, 10%, 25%, 50%, 75%, 100%) with at least a few minutes 
-   between each.
-
-During all the above steps, monitor the log for the new version, filtering for errors being the min requirement.  Note: receiving occasional errors from requests such as _start, _stop, can be expected.
+        ./scripts/deployschema.org.sh - to deploy to main Schema.org site
+        ./scripts/deploywebschemas.org.sh - to deploy to main webschemas.org site
 
+        * Choose Local option to deploy local configuration files
+        * Choose Remote option to pull files from a URL (eg. the github repository)
 
-##### original deploy: gcloud app deploy --no-promote --project schemaorgae --version=3.4 schemaorg.yaml
+        * Provide an appropriate version string - different to currently running version
+        * Optionally pass -e to bypass site exercise option
+        * Optionally pass -m to bypass merge of new version
 
-Note: the --no-promote flage deploys the version but does not transfer traffic to it.
-This can then be done when required via the appengine console, versions screen:
-   https://console.cloud.google.com/appengine/versions?project=schemaorgae&serviceId=default&versionssize=50
 
-===========OLD============
-Either by editing app.yaml to make sure 'application: schemaorgae' is active,
-or by passing the name of the live appengine ID on the commandline.
-===========OLD============
+Recommended Method For Making Release Live
+~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
 
-Currently @rvguha (as Steering Group chair) and @danbri (as Community Group chair and
-webmaster / dogsbody) have write access to the corresponding appengine account.
+For Webschema.org
+    Use ./scripts/deploywebschemas.org.sh -e
+        No real benefit in exercising low traffic site
 
-===========OLD============
- * e.g. update schemaorg.yaml then:  appcfg.py update schemaorg.yaml -A schemaorgae
- * Count to thirty. Log into appengine and you should see a new version.
-* Send 5% of traffic to this version to catch any errors, then move to 100%.
-===========OLD============
+For Schema.org.
 
-Note that appengine versions are visible using URLs with this pattern:
+    Either run ./scripts/deployschema.org.sh
+        This will put the new site live when site exercise is complete (takes some time)
+    or run ./scripts/deployschema.org.sh -m
+        At some time after site exercise is complete login to console and manually migrate traffic
 
-http://3-3.schemaorgae.appspot.com/docs/releases.html
 
 5.) Tag the code at the current relese state
 * Use a local copy of the relesed version of the 'master' branch:
@@ -198,24 +155,10 @@ Note: If master has moved on from the the released version before tagging
 * Blog post.
 * There's a SPARQL database at DYDRA that can load NQuad dumps from
 https://github.com/schemaorg/schemaorg/blob/sdo-phobos/data/releases/2.2/all-layers.nq
-* Make a new per-release branch, reserve an appspot.com account for it.
-* Configure GitHub with the new default branch.
-  See https://github.com/schemaorg/schemaorg/settings/branches
 * Update issue tracker moving issues to new releases.
  * Close issues that were addressed in this release.
  * Make careful considered adjustment to expected release for the rest. Or
    just tag them all for the next release.
-* Update git release tags (TODO: we have no history of handling this yet).
-
-For example, if we are in 'master' and 'git status' shows it in a clean state,
-
-* git checkout -b sdo-deimos
-* This tells us we can do:
- *     git push --set-upstream origin sdo-deimos
-
-> To https://github.com/schemaorg/schemaorg.git
->  * [new branch]      sdo-deimos -> sdo-deimos
-> Branch sdo-deimos set up to track remote branch sdo-deimos from origin.
 
 TODO: