Merge pull request datalad#440 from datalad/knitting-threads

Knitting threads: bringing changes from all over into upstream

Author: jsheunis

datalad_catalog/catalog/README.md

@@ -20,16 +20,16 @@ The `artwork` and `assets` directories contain images and web assets (such as Ja
 
 ## Serving the content
 
-Since this site is self-contained and static, no further build processes, server-side implementations, or access to content delivery networks (CDNs) are necessary in order to serve the content. All that is needed is a simple HTTP server.
+Since this site is self-contained and static, no further build processes or access to content delivery networks (CDNs) are necessary in order to serve the content. All that is needed is a simple HTTP server with one specific addition - a custom redirect. This is required because the application makes use of [Vue Router's history mode](https://v3.router.vuejs.org/guide/essentials/history-mode.html#html5-history-mode), which requires a server-side redirect configuration to deal with the fact that a VueJS application is actually a single-page app.
 
-This can be achieved locally, for example using Python:
+For serving the content locally, this is already taken care of in `datalad-catalog`, and you can simply run the following:
 
 ```bash
 cd path/to/catalog/directory
-python3 -m http.server
+datalad catalog-serve -c .
 ```
 
-The content can also be hosted and served online. A straightforward and free way to achieve this is via GitHub and [GitHub Pages](https://pages.github.com/). After publishing this content as a GitHub repository, you can activate GitHub Pages in the repository's settings. See detailed instructions [here](https://docs.github.com/en/pages/getting-started-with-github-pages/creating-a-github-pages-site).
+The content can also be hosted and served online. A straightforward way to achieve this, and one which has a free tier, is via [Netlify](https://www.netlify.com/) (the common alternative, [GitHub Pages](https://pages.github.com/), does not currently support server-side redirects). After publishing this content, e.g. as a GitHub repository, you can link that repository to a site on Netlify. [See here](https://docs.netlify.com/routing/redirects/) how to set up redirects with Netlify. Of course, the site can also be served from your preferred server setup, as long as page redirects can be supported.
 
 ## Maintaining content
 

datalad_catalog/catalog/assets/app_component_dataset.js

@@ -34,7 +34,6 @@ const datasetView = () =>
             files_ready: false,
             tags_ready: false,
             description_ready: false,
-            description_display: "",
             citation_busy: false,
             citation_text: "",
             invalid_doi: false,
@@ -43,14 +42,13 @@ const datasetView = () =>
         watch: {
           subdatasets_ready: function (newVal, oldVal) {
             if (newVal) {
-              console.log("subdatasets fetched!");
+              console.debug("Watched property: subdatasets_ready = true")
+              console.debug("Subdatasets have been fetched:");
               this.subdatasets = this.selectedDataset.subdatasets;
-              console.log("from watcher");
-              console.log(this.subdatasets);
+              console.debug(this.subdatasets);
               tags = this.tag_options;
               if (this.subdatasets) {
                 this.subdatasets.forEach((subds, index) => {
-                  console.log(index + "; " + subds);
                   if (subds.available == "true" && subds.keywords) {
                     tags = tags.concat(
                       subds.keywords.filter((item) => tags.indexOf(item) < 0)
@@ -67,8 +65,10 @@ const datasetView = () =>
           dataset_ready: function (newVal, oldVal) {
             // TODO: some of these methods/steps should be moved to the generatpr tool. e.g. shortname
             if (newVal) {
+              console.debug("Watched property: dataset_ready = true")
+              console.debug("Active dataset:");
               dataset = this.selectedDataset;
-              console.log(this.selectedDataset);
+              console.debug(this.selectedDataset);
               disp_dataset = {};
               // Set name to unknown if not available
               if (!dataset.hasOwnProperty("name") || !dataset["name"]) {
@@ -156,6 +156,13 @@ const datasetView = () =>
                 }
               } else {
                 disp_dataset.url = dataset.url;
+                if (disp_dataset.url && dataset.url.toLowerCase().indexOf("gin.g-node") >= 0) {
+                  disp_dataset.is_gin = true;
+                  disp_dataset.url = disp_dataset.url.replace('ssh://', '');
+                  disp_dataset.url = disp_dataset.url.replace('[email protected]:', 'https://gin.g-node.org');
+                  disp_dataset.url = disp_dataset.url.replace('[email protected]', 'https://gin.g-node.org');
+                  disp_dataset.url = disp_dataset.url.replace('.git', '');
+                }
               }
               // Description
               if (
@@ -200,6 +207,19 @@ const datasetView = () =>
               else {
                 disp_dataset.show_export = false
               }
+              // Determine show/hide confirg for "Request access" button
+              if (dataset.config?.hasOwnProperty("dataset_options") && dataset.config.dataset_options.hasOwnProperty("include_access_request")) {
+                disp_dataset.show_access_request = dataset.config.dataset_options.include_access_request
+              }
+              else {
+                // default should be to display the access request button, if access request contact/url are included
+                disp_dataset.show_access_request = true
+              }
+              // Show / hide binder button: if disp_dataset.url exists OR if dataset has a notebook specified in metadata
+              disp_dataset.show_binder_button = false
+              if ( disp_dataset.url || dataset.hasOwnProperty("notebooks") && current_dataset.notebooks.length > 0 ) {
+                disp_dataset.show_binder_button = true
+              }
               // Write main derived variable and set to ready
               this.displayData = disp_dataset;
               this.display_ready = true;
@@ -338,6 +358,8 @@ const datasetView = () =>
             }, 1000);
           },
           async selectDataset(event, obj, objId, objVersion, objPath) {
+            console.debug("Inside selectDataset")
+            console.debug(event)
             event.preventDefault()
             var newBrowserTab = event.ctrlKey || event.metaKey || (event.button == 1)
             if (obj != null) {
@@ -375,29 +397,6 @@ const datasetView = () =>
             this.search_tags = []
             this.clearSearchTagText()
           },
-          selectDescription(desc) {
-            if (desc.content.startsWith("path:")) {
-              this.description_ready = false;
-              filepath = desc.content.split(":")[1];
-              extension = "." + filepath.split(".")[1];
-              desc_file = getFilePath(
-                this.selectedDataset.dataset_id,
-                this.selectedDataset.dataset_version,
-                desc.path,
-                extension
-              );
-              fetch(desc_file)
-                .then((response) => response.blob())
-                .then((blob) => blob.text())
-                .then((markdown) => {
-                  this.description_display = marked.parse(markdown);
-                  this.description_ready = true;
-                });
-            } else {
-              this.description_display = desc.content;
-              this.description_ready = true;
-            }
-          },
           gotoHome() {
             // if there is NO home page set:
             // - if there is a tab name in the URL, navigate to current
@@ -464,12 +463,19 @@ const datasetView = () =>
           gotoURL(url) {
             window.open(url);
           },
-          openWithBinder(dataset_url) {
+          openWithBinder(dataset_url, current_dataset) {
             const environment_url =
               "https://mybinder.org/v2/gh/datalad/datalad-binder/main";
             const content_url = "https://github.com/jsheunis/datalad-notebooks";
             const content_repo_name = "datalad-notebooks";
             const notebook_name = "download_data_with_datalad_python.ipynb";
+            if (current_dataset.hasOwnProperty("notebooks") && current_dataset.notebooks.length > 0) {
+              // until including the functionality to select from multiple notebooks in a dropdown, just select the first one
+              notebook = current_dataset.notebooks[0]
+              content_url = notebook.git_repo_url.replace(".git", "")
+              content_repo_name = content_url.substring(content_url.lastIndexOf('/') + 1)
+              notebook_name = notebook.notebook_path
+            }
             binder_url =
               environment_url +
               "?urlpath=git-pull%3Frepo%3D" +
@@ -558,8 +564,9 @@ const datasetView = () =>
                 fetch(doi, { headers })
                   .then((response) => response.text())
                   .then((data) => {
-                    this.selectedDataset.citation_text = data;
-                    console.log(data);
+                    // strip html tags from response text
+                    let doc = new DOMParser().parseFromString(data, 'text/html');
+                    this.selectedDataset.citation_text = doc.body.textContent || "";
                     this.citation_busy = false;
                   });
               } else {
@@ -582,7 +589,7 @@ const datasetView = () =>
             // If a tab parameter is supplied via the router, navigate to that tab if
             // part of available tabs, otherwise default tab
             else {
-              selectTab = tabs.indexOf(tab_param.toLowerCase())
+              selectTab = available_tabs.indexOf(tab_param.toLowerCase())
               if (selectTab >= 0) {
                 this.tabIndex = selectTab;
               } else {
@@ -598,6 +605,7 @@ const datasetView = () =>
           },
         },
         async beforeRouteUpdate(to, from, next) {
+          console.debug("Executing navigation guard: beforeRouteUpdate")
           this.subdatasets_ready = false;
           this.dataset_ready = false;
 
@@ -624,7 +632,6 @@ const datasetView = () =>
           this.$root.selectedDataset.keywords = this.$root.selectedDataset.keywords
             ? this.$root.selectedDataset.keywords
             : [];
-          this.dataset_ready = true;
 
           if (
             this.$root.selectedDataset.hasOwnProperty("subdatasets") &&
@@ -706,7 +713,7 @@ const datasetView = () =>
             this.$root.selectedDataset.has_files = false;
           }
           // Now list all tabs and set the correct one
-          // order in DOM: subdatasets, content, publications, funding, provenance,
+          // order in DOM: content, subdatasets, publications, funding, provenance,
           sDs = this.$root.selectedDataset
           available_tabs = ['content', 'subdatasets']
           standard_tabs = ['publications', 'funding', 'provenance']
@@ -741,9 +748,11 @@ const datasetView = () =>
             available_tabs_lower,
             this.$root.selectedDataset.config?.dataset_options?.default_tab
           )
+          this.dataset_ready = true;
           next();
         },
         async created() {
+          console.debug("Executing lifecycle hook: created")
           // fetch superfile in order to set id and version on $root
           homefile = metadata_dir + "/super.json";
           homeresponse = await fetch(homefile);
@@ -772,7 +781,6 @@ const datasetView = () =>
           }
           text = await response.text();
           app.selectedDataset = JSON.parse(text);
-          this.dataset_ready = true;
           if (
             this.$root.selectedDataset.hasOwnProperty("subdatasets") &&
             this.$root.selectedDataset.subdatasets instanceof Array &&
@@ -872,8 +880,10 @@ const datasetView = () =>
             available_tabs_lower,
             this.$root.selectedDataset.config?.dataset_options?.default_tab
           )
+          this.dataset_ready = true;
         },
         mounted() {
+          console.debug("Executing lifecycle hook: mounted")
           this.tag_options_filtered = this.tag_options;
           this.tag_options_available = this.tag_options;
         }

datalad_catalog/catalog/metadata/1882e2e6-fbbf-4ade-a65f-3a1615235f51/e5d2f8368fc5f6717d8ef131041c6d943298d0c7/cc9/4e43626749e8e368f1526a44a8fcf.json

@@ -47,12 +47,7 @@
         ]
     },
     "name": "Studyforrest Structural MRI scans",
-    "description": [
-        {
-            "source": "datacite_gin",
-            "content": "\"This dataset contains T1 and T2 weighted MRI scans, susceptibility weighted\n images, and diffusion imaging scans. These data are suitable for analyses\n of brain structure.\n\n For more information about the project visit: http://studyforrest.org\""
-        }
-    ],
+    "description": "\"This dataset contains T1 and T2 weighted MRI scans, susceptibility weighted\n images, and diffusion imaging scans. These data are suitable for analyses\n of brain structure.\n\n For more information about the project visit: http://studyforrest.org\"",
     "license": {
         "name": "Open Data Commons Public Domain Dedication and License (PDDL)",
         "url": "https://opendatacommons.org/licenses/pddl/"

datalad_catalog/catalog/metadata/3a8648b3-7df8-413f-8efb-4d39040ac174/10e23aafa8271f742e9022a3522d9a88d7fe30cf/e3a/40711c6157b40133f72c8598b00a2.json

@@ -165,12 +165,7 @@
             "value": 11
         }
     ],
-    "description": [
-        {
-            "source": "datacite_gin",
-            "content": "\"Extension of the dataset published in Hanke et al. (2014; doi:10.1038/sdata.2014.3) with\n additional acquisitions for 6 of the original 20 participants (and one additional participant).\n Participants performed a central fixation task while being stimulated with oriented gratings.\n The experiment was repeated in four different sessions with 7T BOLD fMRI acquisition, each\n with a different scan resolution (0.8, 1.4, 2.0, and 3.0 mm).\n \n This research was supported by a grant from the German Research Foundation (DFG) awarded to\n S. Pollmann and O. Speck (DFG PO 548/15-1). This research was, in part, also supported by the\n German Federal Ministry of Education and Research (BMBF) as part of a US-German collaboration\n in computational neuroscience (CRCNS), co-funded by the BMBF and the US National Science\n Foundation (BMBF 01GQ1112; NSF 1129855). Work on the data-sharing technology employed for this\n research was supported by US-German CRCNS project, co-funded by the BMBF and the US National\n Science Foundation (BMBF 01GQ1411; NSF 1429999).\""
-        }
-    ],
+    "description": "\"Extension of the dataset published in Hanke et al. (2014; doi:10.1038/sdata.2014.3) with\n additional acquisitions for 6 of the original 20 participants (and one additional participant).\n Participants performed a central fixation task while being stimulated with oriented gratings.\n The experiment was repeated in four different sessions with 7T BOLD fMRI acquisition, each\n with a different scan resolution (0.8, 1.4, 2.0, and 3.0 mm).\n \n This research was supported by a grant from the German Research Foundation (DFG) awarded to\n S. Pollmann and O. Speck (DFG PO 548/15-1). This research was, in part, also supported by the\n German Federal Ministry of Education and Research (BMBF) as part of a US-German collaboration\n in computational neuroscience (CRCNS), co-funded by the BMBF and the US National Science\n Foundation (BMBF 01GQ1112; NSF 1129855). Work on the data-sharing technology employed for this\n research was supported by US-German CRCNS project, co-funded by the BMBF and the US National\n Science Foundation (BMBF 01GQ1411; NSF 1429999).\"",
     "publications": [
         {
             "type": "",

datalad_catalog/catalog/metadata/5b1081d6-84d7-11e8-b00a-a0369fb55db0/b623351b43eb2715331ac59ad4cf41682e84ff7d/ed9/46e96a98769a46c7cb92b89d071a7.json

@@ -172,12 +172,7 @@
             "value": 10
         }
     ],
-    "description": [
-        {
-            "source": "datacite_gin",
-            "content": "\"Extension of a matching fMRI dataset (Sengupta, et al., 2017; OpenFMRI ds000113c) on participants performing a\n central fixation task while being stimulated with oriented visual gratings. This dataset extends the previous one\n with acquisitions for 3 matching spatial resolutions (1.4, 2.0, and 3.0 mm) at 3T (complementing the previous 7T\n acquisitions at 0.8, 1.4, 2.0, and 3.0 mm). Five of the total of seven participants are identical in both datasets.\n All participants are part of the studyforrest.org project.\n\n- This research was enabled by a grant from the German Research Foundation (DFG) awarded to S. Pollmann and\n  O. Speck (DFG PO 548/15-1).\n\n- This research was, in part, also supported by the German Federal Ministry of Education and Research (BMBF) as\n  part of a US-German collaboration in computational neuroscience (CRCNS), co-funded by the BMBF and the US\n  National Science Foundation (BMBF 01GQ1112; NSF 1129855).\n\n- Data management technology (DataLad) employed for this research was supported by US-German CRCNS project,\n  co-funded by the BMBF and the US National Science Foundation (BMBF 01GQ1411; NSF 1429999).\""
-        }
-    ],
+    "description": "\"Extension of a matching fMRI dataset (Sengupta, et al., 2017; OpenFMRI ds000113c) on participants performing a\n central fixation task while being stimulated with oriented visual gratings. This dataset extends the previous one\n with acquisitions for 3 matching spatial resolutions (1.4, 2.0, and 3.0 mm) at 3T (complementing the previous 7T\n acquisitions at 0.8, 1.4, 2.0, and 3.0 mm). Five of the total of seven participants are identical in both datasets.\n All participants are part of the studyforrest.org project.\n\n- This research was enabled by a grant from the German Research Foundation (DFG) awarded to S. Pollmann and\n  O. Speck (DFG PO 548/15-1).\n\n- This research was, in part, also supported by the German Federal Ministry of Education and Research (BMBF) as\n  part of a US-German collaboration in computational neuroscience (CRCNS), co-funded by the BMBF and the US\n  National Science Foundation (BMBF 01GQ1112; NSF 1129855).\n\n- Data management technology (DataLad) employed for this research was supported by US-German CRCNS project,\n  co-funded by the BMBF and the US National Science Foundation (BMBF 01GQ1411; NSF 1429999).\"",
     "publications": [
         {
             "type": "",