Design analysis and technology options for jupyFAIR status and findings

[jurra]

Summary:

Most repositories have open apis to upload data and metadata, but they lack on managing large datasets. Figshare is the only one that has a better mechanism to handle large datasets.

The first impression is that we can support many data repositories from the beginning. This is because there are metadata standards in place and generic tools to interact with different repositories.

What the tool (jypyterFAIR) should provide? open to discussion:

• A python package
• CLI for advance users build on top of the package
• Jupyter lab extension that interacts with the CLI

Things to try:

• SWORD, can a common api be used? Some considerations
• What can we learn from SWORD that can be used for this project
• Where is the benefit from a GUI? -> in validating the schema
• Upload data and metadata on 4TU Data archive/figshare
• Can the tool include a way to search/pull datasets? Give priority to pull datasets, the user can provide the id for the dataset
• Draft the python package structure

[jurra]

About the goals we are trying to achieve

From the current status of our analysis it is my impression that:

1. We are looking towards a client application(package, cli, gui for jupyter, or hybrid) that allows to perform two main things.
2. Pushing data and metadata,
3. and pulling data and metadata.
4. Currently we have in the scope to try to support not just one repository type but as much as we can.

This has a lot of implications and considerations specially if it goes beyond the scope of just one repository like figshare (4TU).

This brings us to the next questions

Are there protocols, standards and maybe packages that can be used for many important repositories?

• Partial answer for pulling data: We know now that OAI-PMH can do job for harvesting and pulling data.
• Partial answer for pushing data: SWORD (This protocol also has a server, data verse has implemented a SWORD server, for example).

I highly recommend to read this, which has helped me grasp many things:
https://wiki.surfnet.nl/display/standards/PEER+Use+of+SWORD+deposit+protocol

[manuGil]

Regarding (1), and considering how Jupyter Hub is built (my understanding is they follow a microservices architecture). We will be building a service: parts of the application will act as a server and parts as a client, depending on the tasks.

[girgink]

As we discussed, I think 3-level approach (package, CLI, extension) will be quite useful not only for the initial development, but also for further development and outreach. The package can be utilized by other software, experienced users (e.g. who already use git CLI) can use CLI from their own working environment, and for Jupyter users we can facilitate the workflow through the extension.

Here are my thoughts regarding different components. They are not in any specific order, but I will try to keep related ones close to each other:

• Configuration files: There are different information we need to store (e.g. user accounts, data manifests, status of uploads). One option could be to use a file-based database (e.g. sqlite), but this limits direct editing by the users, which is quite handy once they get experienced. I think it is better if we choose YAML or JSON, and use it as the default format. Although we use one, in the future it might be possible to provide support for the other(s) for user-related files (e.g. data manifests), so that they can create/edit them more easily. My suggestion is YAML.

• User account management: The default username/password or API token can be stored in a configuration file under the home directory. We can use a .dot folder to store the file. Although our support for different data repositories might be limited at the beginning, it can be good to support multiple accounts initially so that we don't change the configuration schema. I think we should also support local user account information stored under the root folder of the working directory (i.e. where data to be published is located), which might be different from the default one. Also support for environmental variables (e.g. FAIR_USERNAME, FAIR_PASSWORD, etc.) can be nice. My suggestion is to support all, i.e. environmental variables (first), local .dot folder under data home (second), default .dot folder under user home (last).

• A typical workflow seems to be as follows:

  1. Development of research code and data
  2. Selection of the files and folders that are part of the dataset (i.e. to be uploaded to the data repository)
  3. Preparation of metadata
  4. Creation of a draft record in the data repository
  5. Upload of data to the repository
  6. Editing of the record for publishing
  7. Sending the record for review
  8. Review process
  9. Final publishing

  Steps 1 is outside of scope. I think Step 7 is also outside of scope.
  Steps 4-9 should be repeated if data should be published in different repositories.
  Steps 7 and 8 might be optional (or not exist) for some data repositories.

• Step 2: Files and folders of the dataset can be indicated as a list in the manifest file. Support for folders is really important, because otherwise the dataset structure as designed by the researcher cannot be replicated in the repository. Allowing wildcards (e.g. *.csv) to select multiple files/folders will be very useful. Support for files to be excluded should also be considered.

• Step 3: Metadata can be indicated by the researcher in the manifest file. Instead of repository-specific metadata fields we can target standards such as Dublin Core and Datacite. Information on the depositor (i.e. researcher) can be taken from the user configuration. Also for other collaborators we can support researcher IDs (e.g. ORCID), which can be utilized to retrieve related information (e.g. name, surname, organization) automatically. Metadata regarding the dataset (e.g. file format, geographical extend, available data columns, etc.) can be extracted automatically. I think we can foresee a step-wise process here:

  1. Creation of a basic metadata by the researcher
  2. Enrichment of the metadata by the system (e.g. full researcher information, collaborators, information on dataset)
  3. Final touches by the researcher

  I think providing a validator can be useful to check if metadata is valid / complete. It can be used by the system, but also by the user whenever necessary.

• Step 4: Based on the metadata crated in Step 3 and by using user credentials the system should be able to create a draft record in the repository and store necessary information to interact with the record in the future. Some of this information can be stored in the original manifest, but for some may be it can be better to use a separate file as not all information might be human-readable and should be protected from editing. If possible it will be good to have this step fully automated and non-interactive. But if the repository requires some interaction may be we can use CLI (or Jupyter stdin/stdout) as well for interaction. If the repository requires a specific standard, then one option could be to translate related fields automatically by using a mapping table (e.g. https://schema.datacite.org/meta/kernel-4.4/doc/DataCite_DublinCore_Mapping.pdf).

• Step 5: Once the record is created in Step 4, files/folders of the dataset can be uploaded. I suggest we decouple this from Step 3, so that it can be done separately, unless the repository requires data upload during the creation of the record. One issue of using Web UIs of the repositories is that the user needs to monitor the uploading of the files and should keep the web browser open for that. But in our case it can be possible to do it in the background and the system can keep track of the uploads, sending files one by one, re-sending them if some error occurs, etc. It might be also possible to pause uploading and continue another time. The library should have a mechanism to communicate status of the uploads, which can be used by the CLI and extension to inform the user. For the extension one nice option could be a status indicator in the status bar with percentage of upload completed and providing more detailed information through a dialog, e.g. if status indicator is clicked. I think being able to pause and continue to upload will be a key feature, especially for large datasets. For this we should be able to track the status of each file (e.g. which parts are uploaded, which parts are still missing) and also make sure that the files are in sync (i.e. not changed since the start of the upload). If they are not in sync we should stop the process and give a warning (or ask if the file should be re-uploaded). If folders are not supported by the repository we can provide an automated archive creation functionality to create a single archive file (e.g. ZIP).

• Step 6: For final editing of the metadata we can simply open the webpage of the record on the repository portal, so that the research can check the record, add additional information, etc. I think the critical point here will be to update the local manifest with the latest data available on the repository. So a two-way communication will be necessary. Considering conflicting information possibilities, I think we need a conflict solution process, e.g. like versioning systems do.

• Steps 7-9: I think it is better to leave them to the researcher and data stewards, by making sure that the local manifest is up-to-date at the end (like Step 6).

Because keeping local manifest requires two-way communication of metadata, if we extend this also with two-way file/folder handling (e.g. downloading files/folders from the repository), then the system can also be used to clone datasets available on the repositories. I think this can be a very useful feature. Probably the download logic (i.e. pause/continue, keeping track of the progress) will be similar to uploading, so it might be a low hanging fruit.

[manuGil]

"It might be also possible to pause uploading and continue another time."
This is a very nice and convenient feature indeed! The feasibility of implementing this will depend on the API of the target data archive (e.g., connection timeout), and the granularity of the requests available on the target API. The FigShare API allows to send data in parts (as I remember), but uploading a part probably has to be done from scratch if it fails.

An approach that might work, will be to tack uploads per file (the app keeps track of which files were uploaded so far) and mark upload files as failed if the upload wasn't completed (for any reason). However, this won't be so useful if the datasets are very large (e.g. > 1GB?) We also have to consider the payload limits for the API endpoints themselves.

This is not to say such a feature is not possible. Only to highlight some of the potential challenges. Similar considerations apply to the sync functionality. To keep things light wait, we don't want the app/extension to create too much traffic between itself and the data archive's API.

[manuGil]

I think we need a conflict solution process, e.g. like versioning systems do.

A naive method to avoid conflicts will be to give priority to one of the steps, and assume the user knows best. Add a confirmations button and warning message, and make the app overwrite existing info on the manifest.

[jurra]

One general remark about the ideas we have:

Lets frame the ideas as hypothesis we want to validate. Then we can design experiments of different kind to validate that they are feasible or relevant.

There are two key concepts I like when developing ideas:

validation: We check that we are doing the right thing, this is that we check our users really a certain tool.
verification: We check that it is feasible to develop such idea.

Both things go hand in hand and it is an iterative process.
My proposal is that we collect as much of our ideas as possible, and formulate them as testable hypohtesis. We can then visualize easily and prioritize easily which of these hypothesis we want to target first. For instance the package, or the cli are general hypothesis that can be formulate in ways that we can endup with testable statements... I will show an example later ;)

[manuGil]

As the next step, should I try to draft a first version of the app's architecture? or is it better to wait after the user's meeting?

[jurra]

The architecture design, should be a result of early partial tests of different components and packages, and addressing partial assumptions we have made about what can be beneficial for users. In fact I think it is more of a final result of the project, than an early one ;)

[manuGil]

I think I don't understand your point. Software architecture is part of the design step. Surely, it might not be perfect at the first iteration. But it is not a pure consequence of the implementation.

[manuGil]

SWORD, can a common API be used?

I looked into some tech aspects of the SWORD protocol. Here's a summary of my impressions.
Original SWORD follows a 'fire-and-forget protocol philosophy (e.i., send a bunch of information to the data repository, and then surrender control, and hope for the best.) A caveat for our app: this might not enable tack uploads on the archive side. This was improved in newer versions of the protocol.

Versions

• There are three versions of the technical specification of SWORD. The latest version for SWORD is 3.0
• Because version 3.0 is very different from version 2.0, not many data archives might have been implemented yet. For example, Dataverse.nl implements version 2 and not fully.
• A key difference between versions 1.0/2.0 and version 3 is the way they model/handle data and metadata. Versions 1.0 and 2.0 define containers, which 'contain' metadata and data. Version 3.0 defines Objects, which contain metadata resources, file sets, packaged content, and extra files.

Version 1.0, 2.0

Version 3.0

• Other key differences in version 3.0:
  • Support for By-Reference file deposit
  • Support for Segmented file deposit
  • More advanced native packaging and metadata formats
  • Supports segmented file upload, and report file/object status from the repository (archive).

Implementations and Tools

There are implementations in different languages including (Wikipedia):
PHP SWORD client library[18][19]
Ruby SWORD client library [20]
Java SWORD client and server library[21]
Python client library[22]

Python for version 2.0

• Implementation for server and client sides
• Offers Full CRUD support). While the SWORD2 profile only names Dublin Core Terms (dcterms) as a potential namespace with which to include other metadata, it does not limit this (other metadata fields can be added)
• Missing some documentation.

Python for version 3.0

• The implementation for version 3.0 deviates radically from version 2.0 (more strict to REST+JSON), USE JSON-LD to for linked-data.
• Common library (sever and client sides)
• Client implementation
• Examples on how to use the client

[jurra]

Hello @girgink I found this resource on how to formulate user stories. They provide templates from simple ot more complex.

Now that you have distilled the tasks, pains and gains into a single list, maybe we can formulate them in such a format. We could use this list as statements to be presented by users. They can say if the statement is true or false, or provide them with a scale of importance from 1 to 5.