Release/1.7.0 (#78)

* Adds completed version of Omero imaging data registration script:
* General metadata is stored in openBIS (imaging experiments and samples)
* Data and additional metadata key-value-pairs are stored in OMERO
* Add readme for Omero etl script
* Added some comments to code
* Minor code cleanup

Co-authored-by: Sven F <[email protected]>
Co-authored-by: luiskuhn <[email protected]>
Co-authored-by: luiskuhn <[email protected]>
Co-authored-by: Tobias Koch <[email protected]>

Author: 4 people

CHANGELOG.md

@@ -1,5 +1,10 @@
 # Changelog
 
+## 1.7.0 2021-03-19
+
+* Provides fully tested functionality to register generic imaging data, with OMERO server support (v5.4.10). [Link to PR](https://github.com/qbicsoftware/etl-scripts/pull/78)
+* Uses an omero-importer-cli (with Bio-formats) for image file registration into an OMERO server instance
+* Uses an initial version of the openBIS-OMERO metadata model
 
 ## 1.6.0 2021-01-22
 

README.md

@@ -44,6 +44,7 @@ Formats:
 - [NGS single-end / paired-end data with metadata (deprecated)](#ngs-single-end--paired-end-data-with-metadata)
 - [Attachment Data](#attachment-data)
 - [Mass Spectrometry mzML conversion and registration](#mass-spectrometry-mzml-conversion-and-registration)
+- [Imaging data with an OMERO server instance](#imaging-data-with-an-omero-server-instance)
 
 ### NGS single-end / paired-end data
 
@@ -279,3 +280,56 @@ Q_MS_LCMS_METHODS - openBIS code from the vocabulary of LCMS methods
 technical_replicate - free text to denote replicates
 
 workflow_type - DDA or DIA
+
+
+### Imaging data with an OMERO server instance
+
+**Responsible dropbox:**
+[QBiC-register-omero-metadata](drop-boxes/register-omero-metadata)
+
+**Resulting data model in openBIS**  
+For each tissue sample multiple images (the data files) can be created, so multiple Q_BMI_GENERIC_IMAGING_RUN samples are created and attached to that tissue sample
+...Q_BIOLOGICAL_SAMPLE -> one Q_BMI_GENERIC_IMAGING_RUN per data file
+
+**Expected data structure**
+In every use case, the data structure needs to contain a top folder around the respective data in order to accommodate metadata files.
+
+The sample code found in the top folder is of type `Q_BIOLOGICAL_SAMPLE` (tissue imaging).
+
+**Valid file types**:
+Valid files in the folder are any imaging files that can be handled by the OMERO server
+
+**Incoming structure overview:**
+```
+QABCD002A8
+|-- QABCD002A8
+|   |-- Est-B1a.lif
+|   |-- Image_1.czi
+|   |-- Image_2.czi
+|   |-- Image7246.tif
+|   |-- metadata_3.tsv
+|   |-- rubisco_avg.mrc
+|   `-- tomogram_x.mrc
+|-- QABCD002A8.sha256sum
+`-- source_dropbox.txt
+```
+
+The metadata file, ending in `.tsv` has tab-separated columns:
+```
+IMAGE_FILE_NAME  IMAGING_MODALITY  IMAGED_TISSUE  INSTRUMENT_MANUFACTURER  INSTRUMENT_USER  IMAGING_DATE
+tomogram_x.mrc   NCIT_C18113       cell           FEI                      Dr. Horrible     01.03.2021
+rubisco_avg.mrc  NCIT_C18113       cell           FEI                      Max Mustermann   01.04.2021
+Image7246.tif    NCIT_C18216       leaf           Zeiss                    Max Mustermann   23.02.2021
+Est-B1a.lif      NCIT_C17753       root           Zeiss                    Max Mustermann   01.02.2021
+Image_1.czi      NCIT_C17753       leaf           Zeiss                    Max Mustermann   11.02.2021
+Image_2.czi      NCIT_C17753       leaf           Zeiss                    Max Mustermann   01.02.2021
+```
+
+column name | description
+--------------|----------------
+`IMAGE_FILE_NAME`| one of the file names found in the incoming folder per line
+`IMAGING_MODALITY`| Ontology Identifier for the imaging modality, currently from the [NCI Thesaurus](https://ncit.nci.nih.gov/ncitbrowser/pages/home.jsf?version=21.02d). **Examples:** NCIT_C18113 (Cryo-Electron Microscopy), NCIT_C18216 (Transmission Electron Microscopy), NCIT_C17753 (Confocal Microscopy)
+`IMAGED_TISSUE` | the imaged tissue
+`INSTRUMENT_MANUFACTURER` | the imaging instrument manufacturer
+`INSTRUMENT_USER` | the person who measured the data file using the imaging instrument
+`IMAGING_DATE` | the date of the measurement in dd.mm.yyyy format (days and months with leading zeroes)

drop-boxes/register-attachments-dropbox/register-attachment-dropbox.py

@@ -84,10 +84,10 @@ def process(transaction):
 	sa = transaction.getSampleForUpdate(sampleID)
 	space = sa.getSpace()
 	if not attachmentReady:
-		expID = '/' + space + '/' + project + '/'+ project+'_INFO'
-		exp = transaction.getExperimentForUpdate(expID)
-		if not exp:
-			exp = transaction.createNewExperiment(expID, "Q_PROJECT_DETAILS")
+		infoSampleID = "/"+space+"/"+code
+		sa = transaction.getSampleForUpdate(infoSampleID)
+	if not sa:
+		exp = transaction.createNewExperiment('/' + space + '/' + project + '/'+ project+'_INFO', "Q_PROJECT_DETAILS")
 		sa = transaction.createNewSample('/' + space + '/'+ code, "Q_ATTACHMENT_SAMPLE")
 		sa.setExperiment(exp)
 	info = None

drop-boxes/register-omero-metadata/backendinterface.py

@@ -14,7 +14,6 @@
     
 """
 
-
 def omero_connect(usr, pwd, host, port):
     """
     Connects to the OMERO Server with the provided username and password.
@@ -182,9 +181,7 @@ def register_image_file_with_dataset_id(file_path, dataset_id, usr, pwd, host, p
     ds_id = dataset_id
 
     if ds_id != -1:
-
         cmd = "omero-importer -s " + host + " -p " + str(port) + " -u " + usr + " -w " + pwd + " -d " + str(int(ds_id)) + " " + file_path
-
         proc = subprocess.Popen(cmd,
                             stdout=subprocess.PIPE,
                             stderr=subprocess.PIPE,
@@ -193,17 +190,20 @@ def register_image_file_with_dataset_id(file_path, dataset_id, usr, pwd, host, p
 
         std_out, std_err = proc.communicate()
 
-        if int(proc.returncode) == 0:
-
-            fist_line = std_out.splitlines()[0]
-            image_ids = fist_line[6:].split(',')
+        # the terminal output of the omero-importer tool provides a lot of information on the registration process 
+        # we are looking for a line with this format: "Image:id_1,1d_2,id_3,...,id_n"
+        # where id_1,...,id_n are a list of ints, which denote the unique OMERO image IDs for the image file
+        # (one file can have many images)
 
+        if int(proc.returncode) == 0:
+            for line in std_out.splitlines():
+                if line[:6] == "Image:":
+                    image_ids = line[6:].split(',')
+                    break
         else:
-            image_ids = -1
-
+            image_ids = []
     else:
-        image_ids = -1
-
+        image_ids = []
     return image_ids
 
 
@@ -315,11 +315,18 @@ def get_image_array(conn, image_id):
 
     return hypercube
 
-################################
-
 def add_annotations_to_image(conn, image_id, key_value_data):
     """
-    TODO
+    This function is used to add key-value pair annotations to an image
+    Example:
+        key_value_data = [["Drug Name", "Monastrol"], ["Concentration", "5 mg/ml"]]
+        add_annotations_to_image(conn, image_id, key_value_data)
+    Args:
+        conn: Established Connection to the OMERO Server via a BlitzGateway
+        image_id (int): An OMERO image ID
+        key_value_data (list of lists): list of key-value pairs
+    Returns:
+        int: not relevant atm
     """
 
     import omero
@@ -339,15 +346,19 @@ def add_annotations_to_image(conn, image_id, key_value_data):
 
 
 #########################
-##app
 
 from optparse import OptionParser
+import ConfigParser
+
+config = ConfigParser.RawConfigParser()
+config.read("imaging_config.properties")
+
 
 ###OMERO server info
-USERNAME = "usr"
-PASSWORD = "pwd"
-HOST = "host"
-PORT = 4064
+USERNAME = config.get('OmeroServerSection', 'omero.username')
+PASSWORD = config.get('OmeroServerSection', 'omero.password')
+HOST = config.get('OmeroServerSection', 'omero.host')
+PORT = int(config.get('OmeroServerSection', 'omero.port'))
 
 
 def get_args():
@@ -358,6 +369,10 @@ def get_args():
     parser.add_option('-p', '--project', dest='project_id', default="None", help='project id for dataset id retrieval')
     parser.add_option('-s', '--sample', dest='sample_id', default="None", help='sample id for dataset id retrieval')
 
+    parser.add_option('-i', '--image', dest='image_id', default="None", help='image id for key-value pair annotation')
+    parser.add_option('-a', '--annotation', dest='ann_str', default="None", help='annotation string')
+
+
     (options, args) = parser.parse_args()
     return options
 
@@ -373,9 +388,26 @@ def get_args():
             id_str = id_str + id_i + " "
 
         print id_str
-    else:
+
+    elif args.project_id != "None":
 
         conn = omero_connect(USERNAME, PASSWORD, HOST, str(PORT))
         ds_id = get_omero_dataset_id(conn, str(args.project_id), str(args.sample_id))
 
         print ds_id
+
+    elif args.image_id != "None":
+
+        conn = omero_connect(USERNAME, PASSWORD, HOST, str(PORT))
+
+        #string format: key1::value1//key2::value2//key3::value3//...
+        key_value_data = []
+        pair_list = args.ann_str.split("//")
+        for pair in pair_list:
+            key_value = pair.split("::")
+            key_value_data.append(key_value)
+
+
+        add_annotations_to_image(conn, str(args.image_id), key_value_data)
+
+        print "0"

drop-boxes/register-omero-metadata/image_registration_process.py

@@ -7,30 +7,47 @@
 from subprocess import Popen, PIPE
 
 barcode_pattern = re.compile('Q[a-zA-Z0-9]{4}[0-9]{3}[A-Z][a-zA-Z0-9]')
+conda_home_path = "/home/qeana10/miniconda2/"
+omero_lib_path = "/home/qeana10/openbis/servers/core-plugins/QBIC/1/dss/drop-boxes/register-omero-metadata/OMERO.py-5.4.10-ice36-b105"
+etl_home_path = "/home/qeana10/openbis/servers/core-plugins/QBIC/1/dss/drop-boxes/register-omero-metadata/"
+
 
 class ImageRegistrationProcess:
 
-    def __init__(self, transaction, env_name="omero_env_0", project_code="", sample_code=""):
+    def __init__(self, transaction, env_name="omero_env_0", project_code="", sample_code="", conda_path=None, omero_path=None, etl_path=None):
 
         self._transaction = transaction
         self._incoming_file_name = transaction.getIncoming().getName()
+        self._search_service = transaction.getSearchService()
 
         self._project_code = project_code
         self._sample_code = sample_code
 
+        ### set exec. env
+        self._conda_path = conda_home_path
+        if not conda_path is None:
+            self._conda_path = conda_path
+
+        self._omero_path = omero_lib_path
+        if not omero_path is None:
+            self._omero_path = omero_path
+
+        self._etl_path= etl_home_path
+        if not etl_path is None:
+            self._etl_path = etl_path
+
         self._init_cmd_list = []
-        self._init_cmd_list.append('eval "$(/home/qeana10/miniconda2/bin/conda shell.bash hook)"')
+        self._init_cmd_list.append('eval "$(' + self._conda_path + 'bin/conda shell.bash hook)"')
         self._init_cmd_list.append('conda activate ' + env_name)
 
-        self._init_cmd_list.append('export OMERO_PREFIX=/home/qeana10/openbis/servers/core-plugins/QBIC/1/dss/drop-boxes/register-omero-metadata/OMERO.py-5.4.10-ice36-b105')
+        self._init_cmd_list.append('export OMERO_PREFIX=' + self._omero_path)
         self._init_cmd_list.append('export PYTHONPATH=$PYTHONPATH:$OMERO_PREFIX/lib/python')
 
-        #now use the omero-importer app packaged in the conda env
-        #self._init_cmd_list.append('export PATH=$PATH:/home/qeana10/openbis/servers/core-plugins/QBIC/1/dss/drop-boxes/register-omero-metadata/OMERO.server-5.4.10-ice36-b105/bin')
-        self._init_cmd_list.append('export PATH=$PATH:/home/qeana10/miniconda2/envs/' + env_name + '/bin')
+        # now use the omero-importer app packaged in the conda env
+        self._init_cmd_list.append('export PATH=$PATH:' + self._conda_path + 'envs/' + env_name + '/bin')
 
-        #move to the dir where backendinterface.py lives
-        self._init_cmd_list.append('cd /home/qeana10/openbis/servers/core-plugins/QBIC/1/dss/drop-boxes/register-omero-metadata/')
+        # move to the dir where backendinterface.py lives for exec.
+        self._init_cmd_list.append('cd ' + self._etl_path)
 
     def fetchOpenBisSampleCode(self):
         found = barcode_pattern.findall(self._incoming_file_name)
@@ -43,7 +60,17 @@ def fetchOpenBisSampleCode(self):
             raise SampleCodeError(self._sample_code, "The sample code seems to be invalid, the checksum could not be confirmed.")
 
         return self._project_code, self._sample_code
-    
+
+    def searchOpenBisSample(self, sample_code):
+        # find specific sample
+        sc = SearchCriteria()
+        sc.addMatchClause(SearchCriteria.MatchClause.createAttributeMatch(SearchCriteria.MatchClauseAttribute.CODE, sample_code))
+        foundSamples = self._search_service.searchForSamples(sc)
+        if len(foundSamples) == 0:
+            raise SampleNotFoundError(sample_code, "Sample could not be found in openBIS.")
+        sample = foundSamples[0]
+        return sample
+
     def _isValidSampleCode(self, sample_code):
         try:
             id = sample_code[0:9]
@@ -73,6 +100,7 @@ def requestOmeroDatasetId(self, project_code=None, sample_code=None):
         return ds_id
 
     def registerImageFileInOmero(self, file_path, dataset_id):
+        
         cmd_list = list(self._init_cmd_list)
         cmd_list.append( "python backendinterface.py -f " + file_path + " -d " + str(dataset_id) )
 
@@ -84,23 +112,26 @@ def registerImageFileInOmero(self, file_path, dataset_id):
         out, err = process.communicate( commands )
 
         id_list = str(out).split()
+        for img_id in id_list:
+            if not img_id.isdigit():
+                return []
 
         return id_list
 
 
     def triggerOMETiffConversion(self):
         pass
 
-    #ToDo Check if Metadata file is provided as was suggested in test.tsv provided by LK
-    def extractMetadataFromTSV(self, tsvFilePath):
+    #ToDo Check if Metadata file is provided as defined
+    def extractMetadataFromTSV(self, tsv_file_path):
         tsvFileMap = {}
         try:
-            with open(tsvFilePath) as tsvfile:
+            with open(tsv_file_path) as tsvfile:
                 reader = csv.DictReader(tsvfile, delimiter='\t', strict=True)
                 for row in reader:
                     tsvFileMap.update(row)
         except IOError:
-            print "Error: No file found at provided filepath " + tsvFilePath
+            print "Error: No file found at provided filepath " + tsv_file_path
         except csv.Error as e:
             print 'Could not gather the Metadata from TSVfile %s, in line %d: %s' % (tsvfile, reader.line_num, e)
 
@@ -109,6 +140,30 @@ def extractMetadataFromTSV(self, tsvFilePath):
     def registerExperimentDataInOpenBIS(self):
         pass
 
+    def registerOmeroKeyValuePairs(self, image_id, property_map):
+        """Registers the property map as key-value pairs in the OMERO server.
+        """
+        
+        cmd_list = list(self._init_cmd_list)
+
+        # string format: key1::value1//key2::value2//key3::value3//...
+        key_value_str = ""
+        for key in property_map.keys(): 
+            key_value_str = key_value_str + str(key) + "::" + str(property_map[key]) + "//"
+        key_value_str = key_value_str[:len(key_value_str)-2] #remove last two chars
+
+        cmd_list.append( "python backendinterface.py -i " + str(image_id) + " -a " + key_value_str )
+
+        commands = ""
+        for cmd in cmd_list:
+            commands = commands + cmd + "\n"
+
+        process = Popen( "/bin/bash", shell=False, universal_newlines=True, stdin=PIPE, stdout=PIPE, stderr=PIPE )
+        out, err = process.communicate( commands )
+
+
+        return 0
+
 
 class SampleCodeError(Exception):
 
@@ -120,4 +175,12 @@ def __init__(self, sample_code, message):
     def test(self):
         pass
 
+class SampleNotFoundError(Exception):
+    
+    def __init__(self, sample_code, message):
+        self.sample_code = sample_code
+        self.message = message
+        super().__init__(self.message)
 
+    def test(self):
+        pass