Update development with hot fix and changelog (#44)

* Add CL items for 1.3 release

* try to find existing ms sample in case of replicate data

* Add CL information for patch 1.3.1

Affects the register-mzml-dropbox. Checks the
given project for existing samples with the provided
sample code of type "Q_MS_RUN". A new sample will be only
created in openBIS, if this sample does not yet exist.

* Provide format documentation for single-end / paired-end data registration (#40)

This CL introduces some documentation for the data structure in preparation to transfer data to QBIC
and register them into qPortal.

* Update Changelog

Co-authored-by: Sven Fillinger <[email protected]>
Co-authored-by: Sven F <[email protected]>

Author: wow-such-code

CHANGELOG.md

@@ -1,5 +1,19 @@
 # Changelog
 
+## 1.4.0
+
+* Register unclassified pooling data of Nanopore experiments directly at the experiment level (no copies are added to sample-based datasets)
+
+## 1.3.1
+
+* Avoid sample registration for existing mass spectrometry data
+
+## 1.3
+
+* Provide metadata schema in JSON for the IMGAG dropbox
+* Register checksums for Oxford Nanopore datasets
+* Register unclassified read data for Oxford Nanopore datasets
+
 ## 1.2
 
 * Provide ETL routine for Oxford Nanopore NGS data

README.md

@@ -2,3 +2,55 @@
 
 This repository holds a collection of Jython ETL (extract-transform-load) scripts that are used at QBiC that define the behaviour of openBIS dropboxes.
 The ETL processes combine some quality control measures for incoming data and data transformation to facilitate the registration in openBIS.
+
+## Data format guidelines
+
+These guidelines describe the necessary file structure for different
+data types to be met in order to ingest and register them correctly in
+openBIS.
+
+Formats:
+
+- [NGS single-end / paired-end data](#ngs-single-end--paired-end-data)
+
+### NGS single-end / paired-end data
+
+**Responsible dropbox:**
+[QBiC-register-fastq-dropbox](drop-boxes/register-fastq-dropbox)
+
+**Resulting data model in openBIS**  
+Q_TEST_SAMPLE -> Q_NGS_RAW_DATA (with sample code) -> DataSet (directory
+with files contained)
+
+**Description**  
+For paired-end sequencing reads in FASTQ format, the file structure
+needs to look like this
+
+```
+<QBIC sample code>.fastq // Directory
+    |-- <QBIC sample code>_R1.fastq
+    |-- <QBIC sample code>_R1.fastq.sha256sum
+    |-- <QBIC sample code>_R2.fastq
+    |-- <QBIC sample code>_R2.fastq.sha256sum
+```
+
+or in the case of gzipped FASTQ files:
+
+```
+<QBIC sample code>.fastq.gz // Directory
+    |-- <QBIC sample code>_R1.fastq.gz
+    |-- <QBIC sample code>_R1.fastq.gz.sha256sum
+    |-- <QBIC sample code>_R2.fastq.gz
+    |-- <QBIC sample code>_R2.fastq.gz.sha256sum
+```
+
+In the case of single-end sequencing data, the file structure needs to
+look like this:
+
+```
+<QBIC sample code>.fastq.gz // Directory
+    |-- <QBIC sample code>.fastq.gz
+    |-- <QBIC sample code>.fastq.gz.sha256sum
+```
+
+

drop-boxes/register-mzML-dropbox/register-mzml-dropbox.py

@@ -81,6 +81,10 @@ def process(transaction):
                 experimentIDs.append(exp.getExperimentIdentifier())
                 if exp.getExperimentType() == expType:
                         msExperiment = exp
+        msSampleID = '/' + space + '/' + 'MS' + parentCode
+        msSample = transaction.getSampleForUpdate(msSampleID)
+        if msSample:
+                msExperiment = msSample.getExperiment()
         # no existing experiment for samples of this sample preparation found
         if not msExperiment:
                 expID = experimentIDs[0]
@@ -91,13 +95,14 @@ def process(transaction):
                         expID = '/' + space + '/' + project + '/' + project + 'E' + str(expNum)
                 msExperiment = transaction.createNewExperiment(expID, expType)
 
-        newMSSample = transaction.createNewSample('/' + space + '/' + 'MS'+ parentCode, "Q_MS_RUN")
-        newMSSample.setParentSampleIdentifiers([sa.getSampleIdentifier()])
-        newMSSample.setExperiment(msExperiment) 
+        if not msSample:
+                msSample = transaction.createNewSample('/' + space + '/' + 'MS'+ parentCode, "Q_MS_RUN")
+                msSample.setParentSampleIdentifiers([sa.getSampleIdentifier()])
+                msSample.setExperiment(msExperiment) 
         # create new dataset 
         dataSet = transaction.createNewDataSet("Q_MS_MZML_DATA")
         dataSet.setMeasuredData(False)
-        dataSet.setSample(newMSSample)
+        dataSet.setSample(msSample)
 
         transaction.moveFile(incomingPath, dataSet)