Skip to content

Commit 9922f23

Browse files
nservantnservant
authored
Merge pull request #391 from nservant/devel
Devel
2 parents 9209ddf + b6ac648 commit 9922f23

File tree

142 files changed

+1253
-102943
lines changed

Some content is hidden

Large Commits have some content hidden by default. Use the searchbox below for content that may be hidden.

142 files changed

+1253
-102943
lines changed

‎.travis.yml

+35
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,35 @@
1+
sudo: required
2+
language: python
3+
python: '3.7'
4+
cache: pip
5+
matrix:
6+
fast_finish: true
7+
8+
before_install:
9+
# install conda
10+
- wget http://repo.continuum.io/miniconda/Miniconda-latest-Linux-x86_64.sh -O miniconda.sh
11+
- chmod +x miniconda.sh
12+
- ./miniconda.sh -b
13+
- export PATH=/home/travis/miniconda2/bin:$PATH
14+
- conda update --yes conda
15+
16+
17+
install:
18+
# use the conda env to install all dependencies
19+
- mkdir -p /tmp/conda
20+
- conda env create -p /tmp/conda/hicpro -f environment.yml
21+
- export PATH=/tmp/conda/hicpro/bin:$PATH
22+
# Install HiC-Pro
23+
- mkdir /tmp/apps
24+
- make configure prefix=/tmp/apps
25+
- make install
26+
27+
script:
28+
# Get test data
29+
- mkdir -p $HOME/test-op && cd $HOME/test-op
30+
- wget https://zerkalo.curie.fr/partage/HiC-Pro/hicpro_testop_travis.tar.gz -P $HOME/test-op/ && tar -zxvf hicpro_testop_travis.tar.gz
31+
- /bin/rm -f hicpro_testop_travis.tar.gz
32+
# Run HiC-Pro
33+
- INSTALLED_HICPRO_VERSION=$(find /tmp/apps -name HiC-Pro | xargs dirname)
34+
- time $INSTALLED_HICPRO_VERSION/HiC-Pro -i test_data -o $HOME/test-op/results -c config_testop_yeast.txt
35+

‎CHANGELOG

+27
Original file line numberDiff line numberDiff line change
@@ -1,3 +1,30 @@
1+
***********************************
2+
CHANGES IN VERSION 3.0.0
3+
4+
NEW FEATURES
5+
6+
o vcf file can be zipped in vcf.gz
7+
8+
o Add a Dockerfile for automatic Docker build on DockerHub
9+
10+
o Set up Travis continuous testing
11+
12+
o iced has been removed from HiC-Pro and is now an independant module
13+
14+
o Add a environment.yml file for installation with conda
15+
16+
o HiC-Pro is now based on python3. /!\ Python2 is no longer maintained /!\
17+
18+
SIGNIFICANT USER-VISIBLE CHANGES
19+
20+
o SORT_RAM is not divided by N_CPU for samtools (#369)
21+
22+
o Fix bug in make_viewpoints.py if multiple viewpoints are specified in the bed file
23+
24+
o Add '--float' option for hicpro2higlass to manage normalized Hi-C data
25+
26+
o Update of the Singularity image
27+
128
***********************************
229
CHANGES IN VERSION 2.11.4
330

‎Dockerfile

+39
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,39 @@
1+
FROM ubuntu:18.04
2+
ENV LANG=C.UTF-8 LC_ALL=C.UTF-8
3+
4+
LABEL authors="Nicolas Servant" \
5+
description="Docker image containing all requirements for the HiC-Pro pipeline"
6+
7+
## Install system tools
8+
RUN apt-get update \
9+
&& apt-get install -y build-essential \
10+
wget \
11+
unzip \
12+
bzip2 \
13+
gcc \
14+
g++ && apt-get clean
15+
16+
17+
## Install miniconda.
18+
RUN wget https://repo.continuum.io/miniconda/Miniconda3-py37_4.8.2-Linux-x86_64.sh -O ~/anaconda.sh
19+
RUN bash ~/anaconda.sh -b -p /usr/local/anaconda
20+
RUN rm ~/anaconda.sh
21+
ENV PATH /usr/local/anaconda/bin:$PATH
22+
23+
24+
## Install all dependencies using conda
25+
COPY environment.yml /
26+
RUN conda env create -f /environment.yml && conda clean -a
27+
ENV PATH /usr/local/anaconda/envs/HiC-Pro_v3.0.0/bin:$PATH
28+
29+
## Install HiCPro
30+
RUN cd /tmp && \
31+
echo "devel.zip" | wget --base=http://github.com/nservant/HiC-Pro/archive/ -i - -O hicpro_latest.zip && \
32+
unzip hicpro_latest.zip && \
33+
cd HiC-Pro-devel && \
34+
make configure prefix=/ && \
35+
make install && \
36+
cd .. && \
37+
rm -fr HiC-Pro*
38+
39+
RUN /HiC-Pro_3.0.0/bin/HiC-Pro -h

‎INSTALL

+53-8
Original file line numberDiff line numberDiff line change
@@ -14,22 +14,67 @@ The installation process will generate a config-system.txt file which defines al
1414
##
1515

1616
Dependencies are checked during installation.
17-
A couple of tools such as bowtie2 and samtools (>1.0) can be automatically installed if not detected.
17+
A couple of tools such as `bowtie2` and `samtools` (>=1.9) can be automatically installed if not detected.
1818
Others have to be installed before starting the installation :
1919

2020
- R (http://www.r-project.org/) with the following packages
2121
--- ggplot2 (>2.2.1)
2222
--- RColorBrewer
2323
--- grid
24-
- Python libraries :
25-
--- bx-python (>=0.5.0) - https://pypi.python.org/pypi/bx-python
26-
--- numpy (>=1.8.2) - http://www.scipy.org/scipylib/download.html
27-
--- scipy (>=0.15.1) - http://www.scipy.org/scipylib/download.html
28-
--- pysam (>=0.8.3) - https://github.com/pysam-developers/pysam
29-
--- argparse if python <= 2.6
24+
- Python (>3.7) libraries :
25+
--- bx-python (>=0.8.8) - https://pypi.python.org/pypi/bx-python
26+
--- numpy (>=1.18.1) - http://www.scipy.org/scipylib/download.html
27+
--- scipy (>=1.4.1) - http://www.scipy.org/scipylib/download.html
28+
--- pysam (>=0.15.4) - https://github.com/pysam-developers/pysam
29+
--- argparse
3030

3131
Note that the iced module is also required (https://github.com/hiclib/iced).
32-
Iced is part of the HiC-Pro source code, but can also be installed independantly (>=2.2)
32+
Iced is no longer part of the HiC-Pro source code, and should be independantly installed
33+
34+
35+
##
36+
## Containers
37+
##
38+
39+
If you do not want to install all dependencies manually, HiC-Pro version 3.X now provides `conda`, `Docker` and `Singularity` recipes.
40+
41+
### Using HiC-Pro through `conda`
42+
43+
In order to ease the installation of HiC-Pro dependancies, we provide a `yml` file for conda with all required tools.
44+
In order to build your conda environment, first install [miniconda](https://docs.conda.io/en/latest/miniconda.html) and use :
45+
46+
```
47+
conda env create -f MY_INSTALL_PATH/HiC-Pro/environment.yml -p WHERE_TO_INSTALL_MY_ENV
48+
conda activate WHERE_TO_INSTALL_MY_ENV
49+
```
50+
51+
### Using the HiC-Pro `Docker` image
52+
53+
A docker image is automatically build and available on [Docker Hub](https://hub.docker.com/repository/docker/nservant/hicpro)
54+
To pull a Docker image, simply use :
55+
56+
```
57+
docker pull nservant/hicpro:latest
58+
```
59+
60+
Note that the `tag` may depend on the HiC-Pro version.
61+
62+
You can also build your own image from the root folder using
63+
64+
```
65+
docker build -t hicpro:3.0.0 .
66+
```
67+
68+
### Using HiC-Pro through `Singularity`
69+
70+
HiC-Pro provides a Singularity container to ease its installation process.
71+
A ready-to-use container is available [here](https://zerkalo.curie.fr/partage/HiC-Pro/singularity_images/hicpro_latest_ubuntu.img).
72+
73+
In order to build you own Singularity image, use :
74+
75+
```
76+
sudo singularity build hicpro_latest_ubuntu.img MY_INSTALL_PATH/HiC-Pro/envs/Singularity
77+
```
3378

3479
##
3580
## Install HiC-Pro

‎Makefile

+14-9
Original file line numberDiff line numberDiff line change
@@ -14,7 +14,8 @@ CONFIGURE_OUT=$(wildcard ./config-system.txt)
1414
CONFIG_SYS=$(wildcard ./config-install.txt)
1515
RUNNER=$(shell whoami)
1616

17-
install : config_check mapbuilder readstrimming iced cp
17+
#install : config_check mapbuilder readstrimming iced cp
18+
install : config_check mapbuilder readstrimming cp
1819

1920
######################################
2021
## Config file
@@ -33,7 +34,11 @@ endif
3334
######################################
3435
configure:
3536
ifneq ("$(CONFIG_SYS)","")
37+
ifneq ("$(prefix)","")
38+
make -f ./scripts/install/Makefile CONFIG_SYS=$(CONFIG_SYS) prefix=$(prefix)
39+
else
3640
make -f ./scripts/install/Makefile CONFIG_SYS=$(CONFIG_SYS)
41+
endif
3742
else
3843
$(error config-install.txt file not found !)
3944
endif
@@ -51,14 +56,14 @@ readstrimming: $(INST_SOURCES)/cutsite_trimming.cpp
5156
(g++ -Wall -O2 -std=c++0x -o cutsite_trimming ${INST_SOURCES}/cutsite_trimming.cpp; mv cutsite_trimming ${INST_SCRIPTS})
5257

5358
## Build Python lib
54-
iced: $(INST_SOURCES)/ice_mod
55-
ifeq ("$(RUNNER)","root")
56-
@echo "Installing the iced package as root"
57-
(cd $(INST_SOURCES)/ice_mod/; ${PYTHON_PATH}/python setup.py install;)
58-
else
59-
@echo "Installing the iced package in --user repository [runner=$(RUNNER)]"
60-
(cd $(INST_SOURCES)/ice_mod/; ${PYTHON_PATH}/python setup.py install --user;)
61-
endif
59+
#iced: $(INST_SOURCES)/ice_mod
60+
#ifeq ("$(RUNNER)","root")
61+
# @echo "Installing the iced package as root"
62+
# (cd $(INST_SOURCES)/ice_mod/; ${PYTHON_PATH}/python setup.py install;)
63+
#else
64+
# @echo "Installing the iced package in --user repository [runner=$(RUNNER)]"
65+
# (cd $(INST_SOURCES)/ice_mod/; ${PYTHON_PATH}/python setup.py install --user;)
66+
#endif
6267

6368
test: config_check
6469
@echo ${PYTHON_PATH}

‎README.md

+46-21
Original file line numberDiff line numberDiff line change
@@ -3,8 +3,13 @@
33

44
### An optimized and flexible pipeline for Hi-C data processing
55

6-
![MultiQC](https://img.shields.io/badge/MultiQC-1.6-blue.svg)
7-
![Singularity](https://img.shields.io/badge/Singularity-build-brightgreen.svg)
6+
[![Build Status](https://travis-ci.com/nservant/HiC-Pro.svg?branch=devel_py3)](https://travis-ci.com/nservant/HiC-Pro)
7+
8+
![Conda](https://img.shields.io/badge/Conda-build-brightgreen.svg)
9+
![Singularity](https://img.shields.io/badge/Singularity-build-brightgreen.svg)
10+
[![Docker](https://img.shields.io/badge/Docker-manual-yellow.svg)](https://hub.docker.com/repository/docker/nservant/hicpro)
11+
12+
![MultiQC](https://img.shields.io/badge/MultiQC-1.8-blue.svg)
813
[![Forum](https://img.shields.io/badge/Groups-%20join%20chat%20%E2%86%92-4fb99a.svg?style=flat-square)](https://groups.google.com/forum/#!forum/hic-pro)
914
[![DOI](https://img.shields.io/badge/DOI-10.1186%2Fs13059--015--0831--x-lightgrey.svg?style=flat-square)](https://doi.org/10.1186/s13059-015-0831-x)
1015

@@ -25,7 +30,36 @@ If you use HiC-Pro, please cite :
2530

2631
*Servant N., Varoquaux N., Lajoie BR., Viara E., Chen CJ., Vert JP., Dekker J., Heard E., Barillot E.* HiC-Pro: An optimized and flexible pipeline for Hi-C processing. Genome Biology 2015, 16:259 [doi:10.1186/s13059-015-0831-x](https://doi.org/10.1186/s13059-015-0831-x)
2732

28-
## Using HiC-Pro through Singularity
33+
## Containers
34+
35+
### Using HiC-Pro through `conda`
36+
37+
In order to ease the installation of HiC-Pro dependancies, we provide a `yml` file for conda with all required tools.
38+
In order to build your conda environment, first install [miniconda](https://docs.conda.io/en/latest/miniconda.html) and use :
39+
40+
```
41+
conda env create -f MY_INSTALL_PATH/HiC-Pro/environment.yml -p WHERE_TO_INSTALL_MY_ENV
42+
conda activate WHERE_TO_INSTALL_MY_ENV
43+
```
44+
45+
### Using the HiC-Pro `Docker` image
46+
47+
A docker image is automatically build and available on [Docker Hub](https://hub.docker.com/repository/docker/nservant/hicpro)
48+
To pull a Docker image, simply use :
49+
50+
```
51+
docker pull nservant/hicpro:latest
52+
```
53+
54+
Note that the `tag` may depend on the HiC-Pro version.
55+
56+
You can also build your own image from the root folder using
57+
58+
```
59+
docker build -t hicpro:3.0.0 .
60+
```
61+
62+
### Using HiC-Pro through `Singularity`
2963

3064
HiC-Pro provides a Singularity container to ease its installation process.
3165
A ready-to-use container is available [here](https://zerkalo.curie.fr/partage/HiC-Pro/singularity_images/hicpro_latest_ubuntu.img).
@@ -41,7 +75,7 @@ In order to build you own Singularity image;
4175
2- Build the singularity HiC-Pro image using the 'Singularity' file available in the HiC-Pro root directory.
4276

4377
```
44-
sudo singularity build hicpro_latest_ubuntu.img MY_INSTALL_PATH/HiC-Pro/Singularity
78+
sudo singularity build hicpro_latest_ubuntu.img MY_INSTALL_PATH/HiC-Pro/envs/Singularity
4579
```
4680

4781
3- Run HiC-pro
@@ -64,16 +98,16 @@ HiC-Pro -h
6498
The HiC-Pro pipeline requires the following dependencies :
6599

66100
- The [bowtie2](http://bowtie-bio.sourceforge.net/bowtie2/index.shtml) mapper
67-
- Python (>2.7) with *pysam (>=0.8.3)*, *bx-python(>=0.5.0)*, *numpy(>=1.8.2)*, and *scipy(>=0.15.1)* libraries.
68-
**Note that the current version does not support python 3**
101+
- Python (>3.7) with *pysam (>=0.15.4)*, *bx-python(>=0.8.8)*, *numpy(>=1.18.1)*, and *scipy(>=1.4.1)* libraries.
102+
**Note that the current version no longer supports python 2**
69103
- R with the *RColorBrewer* and *ggplot2 (>2.2.1)* packages
70104
- g++ compiler
71-
- samtools (>1.1)
105+
- samtools (>1.9)
72106
- Unix sort (**which support -V option**) is required ! For Mac OS user, please install the GNU core utilities !
73107

74108
Note that Bowtie >2.2.2 is strongly recommanded for allele specific analysis.
75109

76-
To install HiC-Pro (>=2.7.8), be sure to have the appropriate rights and run :
110+
To install HiC-Pro, be sure to have the appropriate rights and run :
77111

78112
```
79113
tar -zxvf HiC-Pro-master.tar.gz
@@ -83,15 +117,6 @@ make configure
83117
make install
84118
```
85119

86-
For older version (<2.7.8), the following process can be used
87-
88-
```
89-
tar -zxvf HiC-Pro-master.tar.gz
90-
cd HiC-Pro-master
91-
## Edit config-install.txt file if necessary
92-
make CONFIG_SYS=config-install.txt install
93-
```
94-
95120
Note that if some of these dependencies are not installed (i.e. not detected in the $PATH), HiC-Pro will try to install them.
96121
You can also edit the *config-install.txt* file and manually defined the paths to dependencies.
97122

@@ -100,9 +125,9 @@ You can also edit the *config-install.txt* file and manually defined the paths t
100125
|---------------|-------------------------------------------------------------------------------|
101126
| PREFIX | Path to installation folder |
102127
| BOWTIE2_PATH | Full path the bowtie2 installation directory |
103-
| SAMTOOLS_PATH | Full path to the samtools installation directory (>1.1 ) |
128+
| SAMTOOLS_PATH | Full path to the samtools installation directory |
104129
| R_PATH | Full path to the R installation directory |
105-
| PYTHON_PATH | Full path to the python installation directory (>2.7 - python3 not supported) |
130+
| PYTHON_PATH | Full path to the python installation directory |
106131
| CLUSTER_SYS | Scheduler to use for cluster submission. Must be TORQUE, SGE, SLURM or LSF |
107132

108133

@@ -155,7 +180,7 @@ First have a look at the help message !
155180
usage : HiC-Pro -i INPUT -o OUTPUT -c CONFIG [-s ANALYSIS_STEP] [-p] [-h] [-v]
156181
Use option -h|--help for more information
157182
158-
HiC-Pro 2.11.3
183+
HiC-Pro 3.0.0
159184
---------------
160185
OPTIONS
161186
@@ -238,7 +263,7 @@ Small fastq files (2M reads) extracted from the Dixon et al. 2012 paper are avai
238263
## Run HiC-Pro
239264
time HICPRO_INSTALL_DIR/bin/HiC-Pro -c config_test_latest.txt -i test_data -o hicpro_latest_test
240265
241-
Run HiC-Pro 2.11.3
266+
Run HiC-Pro 3.0.0
242267
--------------------------------------------
243268
Thu Mar 19, 12:18:10 (UTC+0100)
244269
Bowtie2 alignment step1 ...

‎Singularity

+11-10
Original file line numberDiff line numberDiff line change
@@ -40,15 +40,16 @@ From: ubuntu:latest
4040

4141
# external tools
4242
echo "Installing dependancies ... "
43-
conda install -y bowtie2
44-
conda install -y samtools
43+
conda install -y bowtie2=2.3.5
44+
conda install -y samtools=1.9
4545

46-
# Python (>2.7) with *pysam (>=0.8.3)*, *bx(>=0.5.0)*, *numpy(>=1.8.2)*, and *scipy(>=0.15.1)* libraries
47-
conda install -y -c conda-forge python=2.7.15
48-
conda install -y -c anaconda scipy=1.2.1
49-
conda install -y -c anaconda numpy=1.16.3
50-
conda install -y -c bcbio bx-python=0.8.2
51-
conda install -y -c bioconda pysam=0.15.2
46+
# Python (>3.7.0) with *pysam (>=0.8.3)*, *bx(>=0.5.0)*, *numpy(>=1.8.2)*, and *scipy(>=0.15.1)* libraries
47+
conda install -y -c conda-forge python=3.7.6
48+
conda install -y -c conda-forge scipy
49+
conda install -y -c conda-forge numpy
50+
conda install -y -c bioconda bx-python
51+
conda install -y -c bioconda pysam
52+
conda install -y -c bioconda iced
5253

5354
# Install R
5455
conda update readline
@@ -59,13 +60,13 @@ From: ubuntu:latest
5960
conda install -c r r-gridbase=0.4_7
6061

6162
# Install MultiQC
62-
conda install -c bioconda multiqc=1.7
63+
conda install -c bioconda multiqc=1.8
6364

6465
# Install HiC-pro
6566
echo "Installing latest HiC-Pro release ..."
6667
#VERSION=$(curl -s https://github.com/nservant/HiC-Pro/releases/latest | egrep -o '2.[0-9]*.[0-9]*')
6768
#echo "v"$VERSION".zip" | wget --base=http://github.com/nservant/HiC-Pro/archive/ -i - -O hicpro_latest.zip && unzip hicpro_latest.zip
68-
VERSION="devel"
69+
VERSION="devel_py3"
6970
echo $VERSION".zip" | wget --base=http://github.com/nservant/HiC-Pro/archive/ -i - -O hicpro_latest.zip && unzip hicpro_latest.zip
7071

7172
cd $(echo HiC-Pro-$VERSION)

0 commit comments

Comments
 (0)