Merge pull request #114 from shenwei356/reformat2

v0.19.0

Author: shenwei356

README.md

@@ -53,7 +53,7 @@ Related projects:
 - **Versatile commands** 
     - [Usage and examples](http://bioinf.shenwei.me/taxonkit/usage/)
     - Featured command: [tracking monthly changelog of all TaxIds](https://github.com/shenwei356/taxid-changelog)
-    - Featured command: [reformating lineage into format of seven-level ("superkingdom/kingdom, phylum, class, order, family, genus, species"](https://bioinf.shenwei.me/taxonkit/usage/#reformat)
+    - Featured command: [reformating lineage into format of seven-level ("superkingdom/kingdom, phylum, class, order, family, genus, species"](https://bioinf.shenwei.me/taxonkit/usage/#reformat), and [even all possible ranks](https://bioinf.shenwei.me/taxonkit/usage/#reformat2)
     - Featured command: [filtering taxiDs by a rank range](http://bioinf.shenwei.me/taxonkit/usage/#filter), e.g., at or below genus rank.
     - Featured command: [**Create NCBI-style taxdump files for custom taxonomy, e.g., GTDB and ICTV**](https://bioinf.shenwei.me/taxonkit/usage/#create-taxdump)
 
@@ -64,7 +64,7 @@ Subcommand                                                                    |F
 [`list`](https://bioinf.shenwei.me/taxonkit/usage/#list)                      |List taxonomic subtrees (TaxIds) bellow given TaxIds
 [`lineage`](https://bioinf.shenwei.me/taxonkit/usage/#lineage)                |Query taxonomic lineage of given TaxIds
 [`reformat`](https://bioinf.shenwei.me/taxonkit/usage/#reformat)              |Reformat lineage in canonical ranks
-[`reformat2`](https://bioinf.shenwei.me/taxonkit/usage/#reformat2)            |Reformat lineage in chosen ranks, allowing more ranks than 'reformat'
+[`reformat2`](https://bioinf.shenwei.me/taxonkit/usage/#reformat2)<sup>*</sup>|Reformat lineage in chosen ranks, allowing more ranks than 'reformat'
 [`name2taxid`](https://bioinf.shenwei.me/taxonkit/usage/#name2taxid)          |Convert taxon names to TaxIds
 [`filter`](https://bioinf.shenwei.me/taxonkit/usage/#filter)                  |Filter TaxIds by taxonomic rank range
 [`lca`](https://bioinf.shenwei.me/taxonkit/usage/#lca)                        |Compute lowest common ancestor (LCA) for TaxIds
@@ -79,23 +79,23 @@ Note: <sup>*</sup>New commands since the publication.
 
 ## Benchmark
 
-1. Getting complete lineage for given TaxIds
+1. Getting complete lineage for given TaxIds (this plot is very old).
 
    <img src="bench/bench.get_lineage.reformat.tsv.png" alt="" width="600" align="center" />
 
    Versions: ETE=3.1.2, taxopy=0.5.0 ([faster since 0.6.0](https://github.com/shenwei356/taxonkit/issues/47)), TaxonKit=0.7.2.
 
 ## Dataset
 
-1. Download and uncompress `taxdump.tar.gz`: ftp://ftp.ncbi.nih.gov/pub/taxonomy/taxdump.tar.gz 
+1. Download and uncompress `taxdump.tar.gz`: https://ftp.ncbi.nih.gov/pub/taxonomy/taxdump.tar.gz 
 2. Copy `names.dmp`, `nodes.dmp`, `delnodes.dmp` and `merged.dmp` to data directory: `$HOME/.taxonkit`,
 e.g., `/home/shenwei/.taxonkit` ,
 3. Optionally copy to some other directories, and later you can refer to using flag `--data-dir`,
 or environment variable `TAXONKIT_DB`.
 
 All-in-one command:
 
-    wget -c ftp://ftp.ncbi.nih.gov/pub/taxonomy/taxdump.tar.gz 
+    wget -c https://ftp.ncbi.nih.gov/pub/taxonomy/taxdump.tar.gz 
     tar -zxvf taxdump.tar.gz
 
     mkdir -p $HOME/.taxonkit
@@ -141,9 +141,9 @@ And then:
 
 1. [Install go](https://go.dev/doc/install)
 
-        wget https://go.dev/dl/go1.17.13.linux-amd64.tar.gz
+        wget https://go.dev/dl/go1.24.1.linux-amd64.tar.gz
 
-        tar -zxf go1.17.13.linux-amd64.tar.gz -C $HOME/
+        tar -zxf go1.24.1.linux-amd64.tar.gz -C $HOME/
 
         # or 
         #   echo "export PATH=$PATH:$HOME/go/bin" >> ~/.bashrc

doc/docs/download.md

@@ -6,12 +6,11 @@
 
 ## Current Version
 
-- [TaxonKit v0.18.0](https://github.com/shenwei356/taxonkit/releases/tag/v0.18.0)
-[![Github Releases (by Release)](https://img.shields.io/github/downloads/shenwei356/taxonkit/v0.18.0/total.svg)](https://github.com/shenwei356/taxonkit/releases/tag/v0.18.0)
+- [TaxonKit v0.19.0](https://github.com/shenwei356/taxonkit/releases/tag/v0.19.0)
+[![Github Releases (by Release)](https://img.shields.io/github/downloads/shenwei356/taxonkit/v0.19.0/total.svg)](https://github.com/shenwei356/taxonkit/releases/tag/v0.19.0)
+    - new command `taxonkit reformat2`: Reformat lineage in chosen ranks, allowing more ranks than 'reformat'
     - `taxonkit reformat`:
-        - Add a placeholder for rank "realm", "{r}", which is common in Virus taxonomy like [ictv](https://github.com/shenwei356/ictv-taxdump). [#102](https://github.com/shenwei356/taxonkit/issues/102)
-    - `taxonkit name2taxid`:
-        - Show warning for names with multiple taxids. [#103](https://github.com/shenwei356/taxonkit/issues/103)
+        - Fix `-T/--trim` which did not work for `-r/--miss-rank-repl`. [#106](https://github.com/shenwei356/taxonkit/issues/106)
 
 ### Please cite
 
@@ -28,11 +27,11 @@
 
 OS     |Arch      |File, 中国镜像                                                                                                                                                                              |Download Count
 :------|:---------|:-------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------|:-----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------
-Linux  |**64-bit**|[**taxonkit_linux_amd64.tar.gz**](https://github.com/shenwei356/taxonkit/releases/download/v0.18.0/taxonkit_linux_amd64.tar.gz),<br/> [中国镜像](http://app.shenwei.me/data/taxonkit/taxonkit_linux_amd64.tar.gz)                  |[![Github Releases (by Asset)](https://img.shields.io/github/downloads/shenwei356/taxonkit/latest/taxonkit_linux_amd64.tar.gz.svg?maxAge=3600)](https://github.com/shenwei356/taxonkit/releases/download/v0.18.0/taxonkit_linux_amd64.tar.gz)
-Linux  |**arm64** |[**taxonkit_linux_arm64.tar.gz**](https://github.com/shenwei356/taxonkit/releases/download/v0.18.0/taxonkit_linux_arm64.tar.gz),<br/> [中国镜像](http://app.shenwei.me/data/taxonkit/taxonkit_linux_arm64.tar.gz)                  |[![Github Releases (by Asset)](https://img.shields.io/github/downloads/shenwei356/taxonkit/latest/taxonkit_linux_arm64.tar.gz.svg?maxAge=3600)](https://github.com/shenwei356/taxonkit/releases/download/v0.18.0/taxonkit_linux_arm64.tar.gz)
-macOS  |**64-bit**|[**taxonkit_darwin_amd64.tar.gz**](https://github.com/shenwei356/taxonkit/releases/download/v0.18.0/taxonkit_darwin_amd64.tar.gz),<br/> [中国镜像](http://app.shenwei.me/data/taxonkit/taxonkit_darwin_amd64.tar.gz)               |[![Github Releases (by Asset)](https://img.shields.io/github/downloads/shenwei356/taxonkit/latest/taxonkit_darwin_amd64.tar.gz.svg?maxAge=3600)](https://github.com/shenwei356/taxonkit/releases/download/v0.18.0/taxonkit_darwin_amd64.tar.gz)
-macOS  |**arm64** |[**taxonkit_darwin_arm64.tar.gz**](https://github.com/shenwei356/taxonkit/releases/download/v0.18.0/taxonkit_darwin_arm64.tar.gz),<br/> [中国镜像](http://app.shenwei.me/data/taxonkit/taxonkit_darwin_arm64.tar.gz)               |[![Github Releases (by Asset)](https://img.shields.io/github/downloads/shenwei356/taxonkit/latest/taxonkit_darwin_arm64.tar.gz.svg?maxAge=3600)](https://github.com/shenwei356/taxonkit/releases/download/v0.18.0/taxonkit_darwin_arm64.tar.gz)
-Windows|**64-bit**|[**taxonkit_windows_amd64.exe.tar.gz**](https://github.com/shenwei356/taxonkit/releases/download/v0.18.0/taxonkit_windows_amd64.exe.tar.gz),<br/> [中国镜像](http://app.shenwei.me/data/taxonkit/taxonkit_windows_amd64.exe.tar.gz)|[![Github Releases (by Asset)](https://img.shields.io/github/downloads/shenwei356/taxonkit/latest/taxonkit_windows_amd64.exe.tar.gz.svg?maxAge=3600)](https://github.com/shenwei356/taxonkit/releases/download/v0.18.0/taxonkit_windows_amd64.exe.tar.gz)
+Linux  |**64-bit**|[**taxonkit_linux_amd64.tar.gz**](https://github.com/shenwei356/taxonkit/releases/download/v0.19.0/taxonkit_linux_amd64.tar.gz),<br/> [中国镜像](http://app.shenwei.me/data/taxonkit/taxonkit_linux_amd64.tar.gz)                  |[![Github Releases (by Asset)](https://img.shields.io/github/downloads/shenwei356/taxonkit/latest/taxonkit_linux_amd64.tar.gz.svg?maxAge=3600)](https://github.com/shenwei356/taxonkit/releases/download/v0.19.0/taxonkit_linux_amd64.tar.gz)
+Linux  |**arm64** |[**taxonkit_linux_arm64.tar.gz**](https://github.com/shenwei356/taxonkit/releases/download/v0.19.0/taxonkit_linux_arm64.tar.gz),<br/> [中国镜像](http://app.shenwei.me/data/taxonkit/taxonkit_linux_arm64.tar.gz)                  |[![Github Releases (by Asset)](https://img.shields.io/github/downloads/shenwei356/taxonkit/latest/taxonkit_linux_arm64.tar.gz.svg?maxAge=3600)](https://github.com/shenwei356/taxonkit/releases/download/v0.19.0/taxonkit_linux_arm64.tar.gz)
+macOS  |**64-bit**|[**taxonkit_darwin_amd64.tar.gz**](https://github.com/shenwei356/taxonkit/releases/download/v0.19.0/taxonkit_darwin_amd64.tar.gz),<br/> [中国镜像](http://app.shenwei.me/data/taxonkit/taxonkit_darwin_amd64.tar.gz)               |[![Github Releases (by Asset)](https://img.shields.io/github/downloads/shenwei356/taxonkit/latest/taxonkit_darwin_amd64.tar.gz.svg?maxAge=3600)](https://github.com/shenwei356/taxonkit/releases/download/v0.19.0/taxonkit_darwin_amd64.tar.gz)
+macOS  |**arm64** |[**taxonkit_darwin_arm64.tar.gz**](https://github.com/shenwei356/taxonkit/releases/download/v0.19.0/taxonkit_darwin_arm64.tar.gz),<br/> [中国镜像](http://app.shenwei.me/data/taxonkit/taxonkit_darwin_arm64.tar.gz)               |[![Github Releases (by Asset)](https://img.shields.io/github/downloads/shenwei356/taxonkit/latest/taxonkit_darwin_arm64.tar.gz.svg?maxAge=3600)](https://github.com/shenwei356/taxonkit/releases/download/v0.19.0/taxonkit_darwin_arm64.tar.gz)
+Windows|**64-bit**|[**taxonkit_windows_amd64.exe.tar.gz**](https://github.com/shenwei356/taxonkit/releases/download/v0.19.0/taxonkit_windows_amd64.exe.tar.gz),<br/> [中国镜像](http://app.shenwei.me/data/taxonkit/taxonkit_windows_amd64.exe.tar.gz)|[![Github Releases (by Asset)](https://img.shields.io/github/downloads/shenwei356/taxonkit/latest/taxonkit_windows_amd64.exe.tar.gz.svg?maxAge=3600)](https://github.com/shenwei356/taxonkit/releases/download/v0.19.0/taxonkit_windows_amd64.exe.tar.gz)
 
 ## Installation
 
@@ -72,9 +71,9 @@ And then:
 
 1. [Install go](https://go.dev/doc/install)
 
-        wget https://go.dev/dl/go1.17.13.linux-amd64.tar.gz
+        wget https://go.dev/dl/go1.24.1.linux-amd64.tar.gz
 
-        tar -zxf go1.17.13.linux-amd64.tar.gz -C $HOME/
+        tar -zxf go1.24.1.linux-amd64.tar.gz -C $HOME/
 
         # or 
         #   echo "export PATH=$PATH:$HOME/go/bin" >> ~/.bashrc
@@ -153,7 +152,12 @@ All-in-one command:
 
 ## Release history
 
-
+- [TaxonKit v0.18.0](https://github.com/shenwei356/taxonkit/releases/tag/v0.18.0)
+[![Github Releases (by Release)](https://img.shields.io/github/downloads/shenwei356/taxonkit/v0.18.0/total.svg)](https://github.com/shenwei356/taxonkit/releases/tag/v0.18.0)
+    - `taxonkit reformat`:
+        - Add a placeholder for rank "realm", "{r}", which is common in Virus taxonomy like [ictv](https://github.com/shenwei356/ictv-taxdump). [#102](https://github.com/shenwei356/taxonkit/issues/102)
+    - `taxonkit name2taxid`:
+        - Show warning for names with multiple taxids. [#103](https://github.com/shenwei356/taxonkit/issues/103)
 - [TaxonKit v0.17.0](https://github.com/shenwei356/taxonkit/releases/tag/v0.17.0)
 [![Github Releases (by Release)](https://img.shields.io/github/downloads/shenwei356/taxonkit/v0.17.0/total.svg)](https://github.com/shenwei356/taxonkit/releases/tag/v0.17.0)
     - `taxonkit filter`:

doc/docs/tutorial.md

@@ -11,6 +11,7 @@
 - [Making nr blastdb for specific taxids](#making-nr-blastdb-for-specific-taxids)
 - [Summaries of taxonomy data](#summaries-of-taxonomy-data)
 - [Merging GTDB and NCBI taxonomy](#merging-gtdb-and-ncbi-taxonomy)
+- [Filtering or subsetting taxdmp files to make a custom taxdmp with given TaxIDs](#filtering-or-subsetting-taxdmp-files-to-make-a-custom-taxdmp-with-given-taxids)
 
 <!-- END doctoc generated TOC please keep comment here to allow auto update -->
 
@@ -915,6 +916,89 @@ Some tests:
     1187493883   genus          Escherichia
     1945799576   species        Escherichia coli
 
+## Filtering or subsetting taxdmp files to make a custom taxdmp with given TaxIDs
+
+> You want to create a smaller version of the official NCBI taxonomy taxdmp filtered or subset to just the lineages of certain species, for purposes such as creating small test data for testing of tools using taxdmp files.
+>
+> https://github.com/shenwei356/taxonkit/issues/112
+
+Step 1:  preparing taxids in the subset tree
+
+    # here, only keep nodes at the rank of species
+    taxonkit list --ids 707,9606 -I "" \
+        | taxonkit filter -E species \
+        | taxonkit lineage -t \
+        | cut -f 3 \
+        | sed -s 's/;/\n/g' \
+        > taxids.txt
+
+    # the root node
+    echo 1 >> taxids.txt
+
+Step 2: extracting data of needed nodes
+
+    mkdir subset
+
+    grep -w -f <(awk '{print "^"$1}' taxids.txt) ~/.taxonkit/nodes.dmp > subset/nodes.dmp
+    grep -w -f <(awk '{print "^"$1}' taxids.txt) ~/.taxonkit/names.dmp > subset/names.dmp
+
+    touch subset/delnodes.dmp subset/merged.dmp
+
+
+Checking it. Since there are only two leaves here, we just dump the whole tree
+
+    $ wc -l subset/*.dmp
+       0 subset/delnodes.dmp
+       0 subset/merged.dmp
+     144 subset/names.dmp
+      39 subset/nodes.dmp
+     183 total
+
+    $ taxonkit list --ids 1 --data-dir subset/ -nr
+    1 [no rank] root
+      131567 [no rank] cellular organisms
+        2 [superkingdom] Bacteria
+          1224 [phylum] Pseudomonadota
+            1236 [class] Gammaproteobacteria
+              135623 [order] Vibrionales
+                641 [family] Vibrionaceae
+                  662 [genus] Vibrio
+                    28174 [species] Vibrio ordalii
+        2759 [superkingdom] Eukaryota
+          33154 [clade] Opisthokonta
+            33208 [kingdom] Metazoa
+              6072 [clade] Eumetazoa
+                33213 [clade] Bilateria
+                  33511 [clade] Deuterostomia
+                    7711 [phylum] Chordata
+                      89593 [subphylum] Craniata
+                        7742 [clade] Vertebrata
+                          7776 [clade] Gnathostomata
+                            117570 [clade] Teleostomi
+                              117571 [clade] Euteleostomi
+                                8287 [superclass] Sarcopterygii
+                                  1338369 [clade] Dipnotetrapodomorpha
+                                    32523 [clade] Tetrapoda
+                                      32524 [clade] Amniota
+                                        40674 [class] Mammalia
+                                          32525 [clade] Theria
+                                            9347 [clade] Eutheria
+                                              1437010 [clade] Boreoeutheria
+                                                314146 [superorder] Euarchontoglires
+                                                  9443 [order] Primates
+                                                    376913 [suborder] Haplorrhini
+                                                      314293 [infraorder] Simiiformes
+                                                        9526 [parvorder] Catarrhini
+                                                          314295 [superfamily] Hominoidea
+                                                            9604 [family] Hominidae
+                                                              207598 [subfamily] Homininae
+                                                                9605 [genus] Homo
+                                                                  9606 [species] Homo sapiens
+   
+
+    $ echo 28174 | taxonkit lineage -nr --data-dir subset/
+    28174   cellular organisms;Bacteria;Pseudomonadota;Gammaproteobacteria;Vibrionales;Vibrionaceae;Vibrio;Vibrio ordalii       Vibrio ordalii  species
+
 
 <div id="disqus_thread"></div>
 <script>

doc/docs/usage.md

@@ -1029,7 +1029,7 @@ Input:
 
   - List of TaxIds, one record per line.
   - Or tab-delimited format.
-    Plese specify the TaxId field with flag -I/--taxid-field (default 1)
+    Please specify the TaxId field with flag -I/--taxid-field (default 1)
   - Supporting (gzipped) file or STDIN.
 
 Output:
@@ -1040,7 +1040,7 @@ Output:
 
 Output format:
 
-  1. it can contains some escape charactors like "\t".
+  1. it can contain some escape characters like "\t".
   2. For subspecies nodes, the rank might be "subpecies", "strain", or "no rank".
      You can use "|" to set multiple ranks, and the first valid one will be outputted.
      For example,
@@ -1060,7 +1060,7 @@ Differences from 'taxonkit reformat':
   - do not automatically add prefixes, but you can set in the format
 
 Usage:
-  taxonkit reformat2 [flags]
+  taxonkit reformat2 [flags] 
 
 Flags:
   -f, --format string            output format, placeholders of rank are needed (default
@@ -1073,7 +1073,7 @@ Flags:
   -t, --show-lineage-taxids      show corresponding taxids of reformated lineage
   -I, --taxid-field int          field index of taxid. input data should be tab-separated. it overrides
                                  -i/--lineage-field (default 1)
-  -T, --trim                     do not replace missing ranks lower than the rank of current node
+  -T, --trim                     do not replace missing ranks lower than the rank of the current node
 
 ```
 

taxonkit/cmd/reformat2.go

@@ -43,18 +43,18 @@ Input:
 
   - List of TaxIds, one record per line.
   - Or tab-delimited format.
-    Plese specify the TaxId field with flag -I/--taxid-field (default 1)
+    Please specify the TaxId field with flag -I/--taxid-field (default 1)
   - Supporting (gzipped) file or STDIN.
 
 Output:
 
   1. Input line data.
   2. Reformated lineage.
   3. (Optional) TaxIds taxons in the lineage (-t/--show-lineage-taxids)
-   
+
 Output format:
 
-  1. it can contains some escape charactors like "\t".
+  1. it can contain some escape characters like "\t".
   2. For subspecies nodes, the rank might be "subpecies", "strain", or "no rank".
      You can use "|" to set multiple ranks, and the first valid one will be outputted.
      For example,
@@ -310,7 +310,7 @@ func init() {
 	reformat2Cmd.Flags().StringP("format", "f", "{superkingdom};{phylum};{class};{order};{family};{genus};{species}", "output format, placeholders of rank are needed")
 	reformat2Cmd.Flags().StringP("miss-rank-repl", "r", "", `replacement string for missing rank`)
 	reformat2Cmd.Flags().StringP("miss-taxid-repl", "R", "", `replacement string for missing taxid`)
-	reformat2Cmd.Flags().BoolP("trim", "T", false, "do not replace missing ranks lower than the rank of current node")
+	reformat2Cmd.Flags().BoolP("trim", "T", false, "do not replace missing ranks lower than the rank of the current node")
 
 	reformat2Cmd.Flags().IntP("taxid-field", "I", 1, "field index of taxid. input data should be tab-separated. it overrides -i/--lineage-field")
 	reformat2Cmd.Flags().BoolP("show-lineage-taxids", "t", false, `show corresponding taxids of reformated lineage`)