bpm handling to master

BPMReader.py

@@ -68,14 +68,18 @@ def __init__(self, manifest_reader, loci_to_filter, skip_indels, logger):
 
     def get_bpm_records(self):
         for record in self._manifest_reader.get_bpm_records():
-            if record.chromosome == "0" or record.pos == 0:
-                continue
-            if self._loci_to_filter and record.name in self._loci_to_filter:
-                continue
-            if record.is_indel() and self._skip_indels:
-                self._logger.warning("Skipping indel " + record.name)
-                continue
-            yield record
+            try:
+                if record.chromosome == "0" or record.pos == 0:
+                    continue
+                if self._loci_to_filter and record.name in self._loci_to_filter:
+                    continue
+                if record.is_indel() and self._skip_indels:
+                    self._logger.warning("Skipping indel " + record.name)
+                    continue
+                yield record
+            except Exception as error:
+                self._logger.warn("Failed to process entry for record %s: %s", record.name, str(error))
+
 
 class CSVManifestReader(object):
     """

CODEOWNERS

@@ -0,0 +1 @@
+* @selfdecode/team2

Dockerfile

@@ -0,0 +1,23 @@
+# Stage 1: Compile & build wheels for Python deps
+FROM alpine:3.11
+RUN apk update && apk add --no-cache make git py2-pip python2-dev gcc g++ zlib-dev libcurl curl-dev cython libc-dev ncurses-dev xz-dev bzip2-dev linux-headers
+RUN ln -s /usr/include/locale.h /usr/include/xlocale.h
+RUN pip install wheel
+RUN pip install PyVCF==0.6.8 
+RUN pip install pysam==0.11.0 
+RUN pip install numpy==1.11.2 
+RUN pip wheel PyVCF pysam numpy
+RUN mkdir /src && mv *.whl /src
+
+# Step 2: Install Python deps from wheels, add code and tests
+FROM alpine:3.11
+ADD scripts/* ./*.py /usr/local/bin/
+ADD tests /tests
+COPY --from=0 /src/*.whl /src/
+RUN apk update && apk add --no-cache py2-pip python2 xz-dev libcurl
+RUN pip install /src/*.whl && rm /src/*.whl
+RUN mkdir /data && chmod 777 /data
+
+# Run
+WORKDIR /data
+ENTRYPOINT ["gtc_to_vcf.py"]

README.md

@@ -1,6 +1,37 @@
 # GTC to VCF converter
 
+## Semi-archived state
+As of May 2023, it is now recommended to use the fully-supported Illumina solution:
+[DRAGEN Array](https://support.illumina.com/array/array_software/dragen-array-secondary-analysis.html) for GTC to VCF conversion. That tool is the most up-to-date with bug fixes and improvements.
+This repository will remain open, but may be fully archived in the near future.
+
+## Requirements
+
+**GTCtoVCF is currently only known to work with python2.7.**
+The following packages are also required.
+Note that the pysam package is currently not supported on the Windows platform.
+
+Requirement | Version
+--- | ---
+pysam | 0.9.0
+numpy | 1.11.2
+pyvcf | 0.6.8
+
+An easy way to obtain a version of python with these dependencies available is to use "Miniconda". Miniconda is a minimal python installation along with a package manager ("conda") that can be used to install additional packages. First, obtain the 64-bit python 2.7 installer from https://conda.io/miniconda.html and install
+```bash
+bash Miniconda2-latest-Linux-x86_64.sh
+```
+This will run the installer and ask you where you would like to install Miniconda. Then, install numpy, pyvcf, and pysam
+```bash
+conda install -c miniconda numpy=1.11.2
+conda install -c bioconda pyvcf=0.6.8
+conda install -c bioconda pysam=0.9.0
+```
+where conda is a the package manager binary located in the installation location specified in the first step.
+
+
 ## Usage
+
 ```none
 usage: gtc_to_vcf.py [-h] [--gtc-paths GTC_PATHS [GTC_PATHS ...]]
                      --manifest-file MANIFEST_FILE --genome-fasta-file
@@ -51,8 +82,8 @@ Input GTC files are specified with the --gtc-paths option. One or more paths may
 
 The --output-vcf-path option may either be a file or directory. If the argument is a directory, the program will automatically determine an appropriate name for the VCF output file created within that directory. The behavior is summarized in the following table
 
-| # GTC files | --vcf-output-path | Behavior | 
-| - | - | - | 
+| # GTC files | --vcf-output-path | Behavior |
+| - | - | - |
 | 0 | directory  | VCF filename determined from manifest filename  |
 | 1 | directory  | VCF filename determined from input GTC file  |
 | 2+ | directory  | VCF filenames determined from input GTC files  |
@@ -67,15 +98,15 @@ The supplied manifest file may either be in CSV or BPM format; however, a CSV fo
 * For an insertion relative to the reference, the position of the base immediately 5' to the insertion (on the plus strand) is given.
 * For a deletion relative to the reference, the position of the most 5' deleted based (on the plus strand) is given.
 
-Any standard product manifest provided by Illumina will already follow these conventions. 
+Any standard product manifest provided by Illumina will already follow these conventions.
 ### Reference genome
 The contig identifiers in the provided genome FASTA file must match exactly the chromosome identifiers specified in the provided manifest. For a standard human product manifest, this means that the contig headers should read ">1" rather than ">chr1". For compatibility with BaseSpace Variant Interpreter (https://www.illumina.com/informatics/research/biological-data-interpretation/variant-interpreter.html), the specified path of the reference must contain either contain the string GrCh37 or GrCh38, for build 37 and 38, respectively. Suitable whole genome FASTA files can be built with the download_reference.sh script located within the scripts directory. This bash script is dependent on samtools (http://www.htslib.org/download/). If running on OSX, you may need to install coreutils (e.g. `brew install coreutils`).
 
 ### Squashing duplicates
-In the manifest, there can be cases where the same variant is probed by multiple different assays. These assays may be the same design or alternate designs for the same locus. In the default mode of operation, these duplicates will be "squashed" into a single record in the VCF. The method used to incorporate information across multiple assays is under the latter "Output description" heading. When the "--unsquash-duplicates" option is provided, this "squashing" behavior is disabled, and each duplicate assay will be reported in a separate entry in the VCF file. This option is helpful when you are interested in investigating or validating the performance of individual assays, rather than trying to generate genotypes for specific variants. Note that if a locus has more than two alleles and is also queried with duplicated designs, the duplicates will not be unsquashed. 
+In the manifest, there can be cases where the same variant is probed by multiple different assays. These assays may be the same design or alternate designs for the same locus. In the default mode of operation, these duplicates will be "squashed" into a single record in the VCF. The method used to incorporate information across multiple assays is under the latter "Output description" heading. When the "--unsquash-duplicates" option is provided, this "squashing" behavior is disabled, and each duplicate assay will be reported in a separate entry in the VCF file. This option is helpful when you are interested in investigating or validating the performance of individual assays, rather than trying to generate genotypes for specific variants. Note that if a locus has more than two alleles and is also queried with duplicated designs, the duplicates will not be unsquashed.
 
 ### Genome cache
-By default, the entire reference genome will be read into memory. Generally, this will be more efficient than reading data from the indexed reference on disk at the expense of greater memory utilization. For situations in which the genome caching is not desirable (low memory availability or a small input manifest), it is possible to disable this default behavior with the "--disable-genome-cache" option. 
+By default, the entire reference genome will be read into memory. Generally, this will be more efficient than reading data from the indexed reference on disk at the expense of greater memory utilization. For situations in which the genome caching is not desirable (low memory availability or a small input manifest), it is possible to disable this default behavior with the "--disable-genome-cache" option.
 
 ### Auxiliary loci
 Certain classes of variant types (such as multi-nucleotide variants) are not currently supported in the upstream analysis software that produces GTC files. However, it is possible to query this type of variant by creating a SNP design that differentiates the specific multi-nucleotide alleles of interest. For example, if the true source sequence is
@@ -101,28 +132,35 @@ The VCF file output follows VCF4.1 format (https://samtools.github.io/hts-specs/
 * Genotypes are adjusted to reflect the sample ploidy. Calls are haploid for loci on Y, MT and non-PAR chromosome X for males.
 * Multiple SNPs in the input manifest which are mapped to the same chromosomal coordinate (e.g. tri-allelic loci or duplicated sites) are collapsed into one VCF entry and a combined genotype generated. To produce the combined genotype, the set of all possible genotypes is enumerated based on the queried alleles. Genotypes which are not possible based on called alleles and assay design limitations (e.g. infiniumII designs cannot distinguish between A/T and C/G calls) are filtered. If only one consistent genotype remains after the filtering process, then the site is assigned this genotype  Otherwise, the genotype is ambiguous (more than 1) or inconsistent (less than 1) and a no-call is returned.  Please see 'test_class.py' in the 'tests' folder for unit tests and examples demonstrating  the genotype merging process.
 
-## Requirements
-The following python packages are required. Note that the pysam package is currently not supported on the Windows platform.
+## Docker
 
-Requirement | Version
---- | ---
-pysam | 0.9.0
-numpy | 1.11.2
-pyvcf | 0.6.8
+### Build
 
-An easy way to obtain a version of python with these dependencies available is to use "Miniconda". Miniconda is a minimal python installation along with a package manager ("conda") that can be used to install additional packages. First, obtain the 64-bit python 2.7 installer from https://conda.io/miniconda.html and install
-```bash
-bash Miniconda2-latest-Linux-x86_64.sh 
+Build the Docker image:
+
+```sh
+docker build -t gtc_to_vcf .
 ```
-This will run the installer and ask you where you would like to install Miniconda. Then, install numpy, pyvcf, and pysam
-```bash
-conda install -c miniconda numpy=1.11.2
-conda install -c bioconda pyvcf=0.6.8
-conda install -c bioconda pysam=0.9.0
+
+### Usage
+
+Set the full path to the location of the GTC files (`gtc_dir`), manifest csv or bpm (`manifest`), and reference fasta (`ref`). The reference fasta must be indexed with `samtools faidx`, with an accompanying `.fasta.fai` file. 
+
+```sh
+gtc_dir=/path/to/gtcs
+manifest=/data/projects/cag/iscan/gsa-manifest-clusterfile/GSA-24v3-0_A1.csv
+ref=/data/projects/cag/reference-data/reference-genomes/human_g1k_v37.fasta
+
+docker run --rm \
+-u $(id -u):$(id -g) \
+-v $gtc_dir:/data \
+-v $manifest:/tmp/$(basename ${manifest}) \
+-v $ref:/tmp/ref.fasta \
+-v ${ref}.fai:/tmp/ref.fasta.fai \
+gtc_to_vcf --gtc-paths . --manifest-file /tmp/$(basename ${manifest}) --genome-fasta-file /tmp/ref.fasta --output-vcf-path .
+
 ```
-where conda is a the package manager binary located in the installation location specified in the first step. 
 
-##
 ## License
 
 Copyright 2018 Illumina