Merge pull request #150 from marbl/make-partition-default

Make partition default

Author: bkille

Diff for: README.md

@@ -1,14 +1,52 @@
 Parsnp is a command-line-tool for efficient microbial core genome alignment and SNP detection. Parsnp was designed to work in tandem with Gingr, a flexible platform for visualizing genome alignments and phylogenetic trees; both Parsnp and Gingr form part of the Harvest suite :
 
 
+
 # Installation
 ## From conda
 Parsnp is available on the [Bioconda](https://bioconda.github.io/user/install.html#set-up-channels) channel. This is the recommended method of installation. Once you have [added the Bioconda channel](https://bioconda.github.io/user/install.html#set-up-channels) to your conda environment, `parsnp` can be installed via
 ```
 conda install parsnp
 ```
 
-## From source
+Instructions for building Parsnp from source are available towards the end of this README.
+
+# Running Parsnp
+Parsnp can be run multiple ways, but the most common is with a set of genomes and a reference. 
+```
+parsnp -g <reference_genbank> -d <genomes> 
+```
+```
+parsnp -r <reference_fasta> -d <genomes> 
+```
+For example, 
+```
+parsnp -r examples/mers_virus/ref/England1.fna -d examples/mers_virus/genomes/*.fna -o examples-out
+```
+
+## Partition mode
+Parsnp 2 will group query genomes up into random partitions of at least `--min-partition-size` genomes each (50 by default). Parsnp is then run independently on each group, and the resulting alignment of each group is merged into a single alignment of all input genomes. This limits the input size for an individual "core" Parsnp step, leading to significantly less memory and CPU usage. We've also shown, on simulated and empirical data, that this partitioning step often leads to increased core-genome size and better phylogenetic signal.
+
+The `--no-partition` flag allows users to run all query genomes at once.
+
+## Output files
+* `parsnp.xmfa` is the core-genome alignment.
+* `parsnp.ggr` is the compressed representation of the alignment generated by the harvest-toolkit. This file can be used to visualize alignments with Gingr.
+* `parsnp.snps.mblocks` is the core-SNP signature of each sequence in fasta format. This is the file which is used to generate `parsnp.tree`
+* `parsnp.tree` is the resulting phylogeny.
+* If run in partition mode, Parsnp will produce a `partition` folder in the output directory, which contains the output of each of the partitioned runs. 
+
+
+### XMFA format
+The output XMFA file contains a header section mapping contig names to indices. Following the header section, the LCBs/clusters are reported in the XMFA format, where the ID for each record in an LCB is formatted as:
+
+```
+[fileidx]:[concat_start]-[concat_end] [strand] cluster[x] s[contig_idx]:p[contig_pos]
+```
+
+The `concat_start` and `concat_end` values are internal to parsnp. The sequence for this record can be found in the file at index `fileidx` (these are declared at the top of the xmfa) on the `contig_idx`th contig starting at position `contig_pos`. 
+
+## Building from source 
 
 To build Parsnp from source, users must have automake 1.15, autoconf, and libtool installed. Parsnp also requires RaxML (or FastTree), Harvest-tools, and numpy. Some additional features require  pySPOA, Mash, FastANI, and Phipack. All of these packages are available via Conda (many on the Bioconda channel).
 
@@ -44,44 +82,6 @@ Note that the `parsnp` executable in `bin/` is not the same as the one in the ro
 ## OSX Users (Catalina)
 Recent OSX have a Gatekeeper, that's designed to ensure that only softwre from known developers runs on  tour Mac. Please refer to this link to enable the binaries shipped with Parsnp to run: https://support.apple.com/en-us/HT202491
 
-# Running Parsnp
-Parsnp can be run multiple ways, but the most common is with a set of genomes and a reference. 
-```
-parsnp -g <reference_genbank> -d <genomes> 
-```
-```
-parsnp -r <reference_fasta> -d <genomes> 
-```
-For example, 
-```
-parsnp -r examples/mers_virus/ref/England1.fna -d examples/mers_virus/genomes/*.fna -o examples-out
-```
-
-## Partition mode
-Parsnp 2 includes a new mode which can be activated with `--partition`. This mode randomly splits the input genomes up into groups of *p* genomes each, where *p* defaults to 50 and can be changed with `--partition-size=p`. Parsnp is then run independently on each group, and the resulting alignment of each group is merged into a single alignment of all input genomes. This mode is intended for large datasets, as it reduces the computational requirements. 
-
-```
-parsnp -r examples/mers_virus/ref/England1.fna -d examples/mers_virus/genomes/*.fna --partition --partition-size 10 -o examples-out-partitioned
-```
-
-More examples can be found in the [readthedocs tutorial](https://harvest.readthedocs.io/en/latest/content/parsnp/tutorial.html)
-
-## Output files
-* `parsnp.xmfa` is the core-genome alignment.
-* `parsnp.ggr` is the compressed representation of the alignment generated by the harvest-toolkit. This file can be used to visualize alignments with Gingr.
-* `parsnp.snps.mblocks` is the core-SNP signature of each sequence in fasta format. This is the file which is used to generate `parsnp.tree`
-* `parsnp.tree` is the resulting phylogeny.
-* If run in partition mode, Parsnp will produce a `partition` folder in the output directory, which contains the output of each of the partitioned runs. 
-
-### XMFA format
-The output XMFA file contains a header section mapping contig names to indices. Following the header section, the LCBs/clusters are reported in the XMFA format, where the ID for each record in an LCB is formatted as:
-
-```
-[fileidx]:[concat_start]-[concat_end] [strand] cluster[x] s[contig_idx]:p[contig_pos]
-```
-
-The `concat_start` and `concat_end` values are internal to parsnp. The sequence for this record can be found in the file at index `fileidx` (these are declared at the top of the xmfa) on the `contig_idx`th contig starting at position `contig_pos`. 
-
 ## Misc
 
 CITATION provides details on how to cite Parsnp.

Diff for: parsnp

@@ -430,14 +430,14 @@ def parse_args():
 
     partition_args = parser.add_argument_group("Sequence Partitioning")
     partition_args.add_argument(
-        "--partition",
+        "--no-partition",
         action='store_true',
-        help="Evenly split input sequences across separate runs of parsnp, then merge results")
+        help="Run all query genomes in single parsnp alignment, no partitioning.")
     partition_args.add_argument(
-        "--partition-size",
+        "--min-partition-size",
         type=int,
         default=50,
-        help="Number of sequences in a partitioned run of parsnp")
+        help="Minimum size of a partition. Input genomes will be split evenly across partitions at least this large.")
 
     extend_args = parser.add_argument_group("LCB Extension")
     extend_args.add_argument(
@@ -1459,7 +1459,9 @@ SETTINGS:
     run_recomb_filter = 0
 
     #3)run parsnp (cores, grid?)
-    if not args.partition:
+    if args.no_partition or len(finalfiles) < 2*args.min_partition_size:
+        if len(finalfiles) < 2*args.min_partition_size:
+            logger.info(f"Too few genomes to run partitions of size >{args.min_partition_size}. Running all genomes at once.")
         # Editing the ini file to be used for parnsp-aligner (which is different from parsnip as a mumi finder)
         if not inifile_exists:
             write_inifile_2(inifiled, outputDir, unaligned, args, auto_ref, query, finalfiles, ref, args.threads)
@@ -1620,16 +1622,15 @@ SETTINGS:
 
     else:
         import partition 
-
-        if len(finalfiles) % args.partition_size == 1:
-            logger.warning("Incrementing partition size by 1 to avoid having a remainder partition of size 1")
-            args.partition_size += 1
+        full_partitions = len(finalfiles) // args.min_partition_size
+        effective_partition_size = len(finalfiles) // full_partitions
+        logger.info(f"Setting the partition size to {effective_partition_size}")
         partition_output_dir = f"{outputDir}/partition"
         partition_list_dir = f"{partition_output_dir}/input-lists"
         os.makedirs(partition_list_dir, exist_ok=True)
-        for partition_idx in range(math.ceil(len(finalfiles) / args.partition_size)):
+        for partition_idx in range(math.ceil(len(finalfiles) / effective_partition_size)):
             with open(f"{partition_list_dir}/{partition.CHUNK_PREFIX}-{partition_idx:010}.txt", 'w') as part_out:
-                for qf in finalfiles[partition_idx*args.partition_size : (partition_idx+1)*args.partition_size]:
+                for qf in finalfiles[partition_idx*effective_partition_size : (partition_idx+1)*effective_partition_size]:
                     part_out.write(f"{qf}\n")
 
         chunk_label_parser = re.compile(f'{partition.CHUNK_PREFIX}-(.*).txt')