DD(s,mu) function for mocks/theory (#130) (#132)

* DD(s,mu) function for mocks/theory (#130) * Updated README.rst [ci skip] Weights are on `pip`. Changed repo files to be links. * add a DDsmu mocks function * remove extra slash * add tests of DDsmu_mocks * add name to authors * update to add a mu_max function parameter * bug fixes; verified output against kdcount for different mu_max for AVX, SSE42, and fallback * adding theory DDsmu; verified for different mu_max and all ISA against kdcount * update docs * include DDsmu in theory/tests * fix type error * forgot to remove other variable definition * Updating the docs for (theory) DDsmu * Reviewed the theory functions (still need comprehensive tests and update RTD docs) * Adding the new file, tests_common.h, to allow integration tests (exhaustive tests for new pair-counters). * My (broken) mocks code * Fixing bugs uncovered by doctests (which are still not failing the build) * Trying to solve the doctests failures and the warnings raised during compiling the docs for DDsmu * I have a suspicion that doctests are not failing the build because they are in the 'after_success' part. Moved the doctests into the tests section. Might solve #143 * Attempting to fix #144 * Fixed the Makefile for DDsmu tests * Added the tests for DDsmu_mocks into the Makefile * Whitespace changes only for better readability [ci skip] * Corrected the variable type for nmu_bins and some small changes for better code readability * The output file for DDsmu_mocks.DD really corresponds to DDsmu_mocks.RR (see #132) * Fixed the DDsmu_mocks tests * Changed the name of the DDsmu_mocks test from DD->RR. Put the name of each test on a new line * Attempting to fix travis failure (from doctest failure) * Another attempt at fixing the doctest failure on travis * Next attempt at fixing doctest failure * Small change to the auto-generated docs [ci skip] * Doctests are failing because numpy does not honour set_printoptions for structured arrays (numpy issue #5606). This numpy issue seems to have been solved in 1.12. Bumping the default travis numpy version to 1.12 * Still trying to fix doctest failures. Now removed testing for python3.3 and added python3.6 * Missed the 'then' in the if condition. Added a xcode9 image for osx tests * Added a python3.6 for osx and changed the python version to python2.7 for xcode6 and xcode7 * Corrected the miniconda installer filenames for python2 * Added the numpy version=1.7 for testing the minimum requirements on osx * Added C mode declaration for syntax highlighting [ci skip] * Made sure that mu_max is specified before nmu_bins. Changed the ordering in the python extension as well * Added example C codes for the DDsmu and DDsmu_mocks pair-counters * The case of a mis-placed dot (or how to break the build) * Enforce that mu_max is scalar and greater than 0 * pimax is not required for DDsmu_mocks. Correctly added the parx/pary/parz components into the pair-weight struct for DDsmu_mocks and DDrppi_mocks. Renamed variables to make context clearer (will need to be done for DDrppi_mocks as well) * Renamed sqr_sep to sqr_s and removed checks for pimax * Changed the kernel parameters to smax/smin from sqr_smax/sqr_smin * The AVX tests pass now for DDsmu_mocks * Fixed the INTEGRATION_TEST section for DDtheta_mocks * Updated docstrings in python bindings for DDsmu and DDsmu_mocks * Added docs for DDsmu and DDsmu_mocks. Fixed the docstring formatting (removed notes within function docstrings) * Added the missing variable for doctests * Renamed w(theta) to DD(theta) and changed some text formatting * I forgot to fix the DDsmu_mocks file for the doctest failure * DDsmu PR is now ready to be merged. Bumping version to 2.1 * README updated to show that github pages are no longer being published [ci skip] * Filled in some more missing docs/docstrings * Remove further references to github pages site [ci skip]
manodeep · Jan 16, 2018 · c931429 · c931429
1 parent 0d66e22
commit c931429
Show file tree

Hide file tree

Showing 73 changed files with 8,527 additions and 1,385 deletions.
diff --git a/.travis.yml b/.travis.yml
@@ -10,7 +10,8 @@ notifications:
 
 env:
   global:
-    - NUMPY_VERSION=1.9
+    - NUMPY_VERSION=1.12
+    - DOCTEST=TRUE
 
 addons: 
   apt:
@@ -32,12 +33,6 @@ matrix:
     #     - brew update
     #     - brew tap homebrew/versions && brew reinstall gcc49 --without-multilib
     #     - wget http://repo.continuum.io/miniconda/Miniconda-latest-MacOSX-x86_64.sh -O miniconda.sh
-    #     - bash miniconda.sh -b -p $HOME/miniconda
-    #     - export PATH="$HOME/miniconda/bin:$PATH"
-    #     - conda create -q --yes -n test python=$PYTHON_VERSION numpy=$NUMPY_VERSION sphinx
-    #     - source activate test
-    #     - conda install -q --yes -c asmeurer gsl
-
 
     # - os: osx
     #   compiler: clang
@@ -47,74 +42,42 @@ matrix:
     #     - brew outdated xctool || brew upgrade xctool
     #     - brew tap homebrew/versions && brew install clang-omp
     #     - wget http://repo.continuum.io/miniconda/Miniconda-latest-MacOSX-x86_64.sh -O miniconda.sh
-    #     - bash miniconda.sh -b -p $HOME/miniconda
-    #     - export PATH="$HOME/miniconda/bin:$PATH"
-    #     - conda create -q --yes -n test python=$PYTHON_VERSION numpy=$NUMPY_VERSION sphinx
-    #     - source activate test
-    #     - conda install -q --yes -c asmeurer gsl        
 
     - os: osx
-      osx_image: xcode8
+      osx_image: xcode9
       compiler: clang
-      env: COMPILER=clang FAMILY=clang V='Apple LLVM 8.0.0' PYTHON_VERSION=3.5
+      env: COMPILER=clang FAMILY=clang V='Apple LLVM 7.0.0' PYTHON_VERSION=3.6 DOCTEST=FALSE
       before_install:
         - wget http://repo.continuum.io/miniconda/Miniconda3-latest-MacOSX-x86_64.sh -O miniconda.sh
-        - bash miniconda.sh -b -p $HOME/miniconda
-        - export PATH="$HOME/miniconda/bin:$PATH"
-        - conda create -q --yes -n test python=$PYTHON_VERSION numpy=$NUMPY_VERSION sphinx
-        - source activate test
-        - conda install -q --yes -c asmeurer gsl
+
 
     - os: osx
-      osx_image: xcode7.3
+      osx_image: xcode8
       compiler: clang
-      env: COMPILER=clang FAMILY=clang V='Apple LLVM 7.0.0' PYTHON_VERSION=3.5
+      env: COMPILER=clang FAMILY=clang V='Apple LLVM 7.0.0' PYTHON_VERSION=3.5 DOCTEST=FALSE
       before_install:
         - wget http://repo.continuum.io/miniconda/Miniconda3-latest-MacOSX-x86_64.sh -O miniconda.sh
-        - bash miniconda.sh -b -p $HOME/miniconda
-        - export PATH="$HOME/miniconda/bin:$PATH"
-        - conda create -q --yes -n test python=$PYTHON_VERSION numpy=$NUMPY_VERSION sphinx
-        - source activate test
-        - conda install -q --yes -c asmeurer gsl
-
 
     - os: osx
-      osx_image: xcode9
+      osx_image: xcode7.3
       compiler: clang
-      env: COMPILER=clang FAMILY=clang V='Apple LLVM 9.0.0' PYTHON_VERSION=3.5
+      env: COMPILER=clang FAMILY=clang V='Apple LLVM 7.0.0' PYTHON_VERSION=2.7 DOCTEST=FALSE
       before_install:
-        - wget http://repo.continuum.io/miniconda/Miniconda3-latest-MacOSX-x86_64.sh -O miniconda.sh
-        - bash miniconda.sh -b -p $HOME/miniconda
-        - export PATH="$HOME/miniconda/bin:$PATH"
-        - conda create -q --yes -n test python=$PYTHON_VERSION numpy=$NUMPY_VERSION sphinx
-        - source activate test
-        - conda install -q --yes -c asmeurer gsl
-
+        - wget http://repo.continuum.io/miniconda/Miniconda2-latest-MacOSX-x86_64.sh -O miniconda.sh
 
     - os: osx
       osx_image: xcode6.4
       compiler: clang
-      env: COMPILER=clang FAMILY=clang V='Apple LLVM 7.0.0' PYTHON_VERSION=3.5
+      env: COMPILER=clang FAMILY=clang V='Apple LLVM 7.0.0' PYTHON_VERSION=2.6 NUMPY_VERSION=1.7 DOCTEST=FALSE
       before_install:
-        - wget http://repo.continuum.io/miniconda/Miniconda3-latest-MacOSX-x86_64.sh -O miniconda.sh
-        - bash miniconda.sh -b -p $HOME/miniconda
-        - export PATH="$HOME/miniconda/bin:$PATH"
-        - conda create -q --yes -n test python=$PYTHON_VERSION numpy=$NUMPY_VERSION sphinx
-        - source activate test
-        - conda install -q --yes -c asmeurer gsl
-
+        - wget http://repo.continuum.io/miniconda/Miniconda2-latest-MacOSX-x86_64.sh -O miniconda.sh
 
     # - os: osx
     #   compiler: gcc
     #   env: COMPILER=gcc-4.8 V='4.8' PYTHON_VERSION=3.5 FAMILY=gcc
     #   before_install:
     #     - brew update && brew tap homebrew/versions && brew install gcc48 --without-multilib
     #     - wget http://repo.continuum.io/miniconda/Miniconda3-latest-MacOSX-x86_64.sh -O miniconda.sh
-    #     - bash miniconda.sh -b -p $HOME/miniconda
-    #     - export PATH="$HOME/miniconda/bin:$PATH"
-    #     - conda create -q --yes -n test python=$PYTHON_VERSION numpy=$NUMPY_VERSION sphinx
-    #     - source activate test
-    #     - conda install -q --yes -c asmeurer gsl
 
     # - os: linux
     #   dist: trusty
@@ -127,10 +90,6 @@ matrix:
     #   env: COMPILER=clang-3.6 V=3.6 PYTHON_VERSION=2.7 
     #   before_install:
     #     - wget http://repo.continuum.io/miniconda/Miniconda-latest-Linux-x86_64.sh -O miniconda.sh
-    #     - bash miniconda.sh -b -p $HOME/miniconda
-    #     - export PATH="$HOME/miniconda/bin:$PATH"
-    #     - conda create -q --yes -n test python=$PYTHON_VERSION numpy=$NUMPY_VERSION sphinx
-    #     - source activate test
 
     # - os: linux
     #   dist: trusty
@@ -143,22 +102,14 @@ matrix:
     #   env: COMPILER=clang-3.6 V=3.6 PYTHON_VERSION=3.5
     #   before_install:
     #     - wget http://repo.continuum.io/miniconda/Miniconda3-latest-Linux-x86_64.sh -O miniconda.sh
-    #     - bash miniconda.sh -b -p $HOME/miniconda
-    #     - export PATH="$HOME/miniconda/bin:$PATH"
-    #     - conda create -q --yes -n test python=$PYTHON_VERSION numpy=$NUMPY_VERSION sphinx
-    #     - source activate test
 
     - os: linux
       dist: trusty
       sudo: required
       compiler: gcc
-      env: COMPILER=gcc PYTHON_VERSION=2.6 NUMPY_VERSION=1.7
+      env: COMPILER=gcc PYTHON_VERSION=2.6 NUMPY_VERSION=1.7 DOCTEST=FALSE
       before_install:
         - wget http://repo.continuum.io/miniconda/Miniconda-latest-Linux-x86_64.sh -O miniconda.sh
-        - bash miniconda.sh -b -p $HOME/miniconda
-        - export PATH="$HOME/miniconda/bin:$PATH"
-        - conda create -q --yes -n test python=$PYTHON_VERSION numpy=$NUMPY_VERSION sphinx
-        - source activate test
 
     - os: linux
       dist: trusty
@@ -167,22 +118,14 @@ matrix:
       env: COMPILER=gcc PYTHON_VERSION=2.7
       before_install:
         - wget http://repo.continuum.io/miniconda/Miniconda-latest-Linux-x86_64.sh -O miniconda.sh
-        - bash miniconda.sh -b -p $HOME/miniconda
-        - export PATH="$HOME/miniconda/bin:$PATH"
-        - conda create -q --yes -n test python=$PYTHON_VERSION numpy=$NUMPY_VERSION sphinx
-        - source activate test
 
     - os: linux
       dist: trusty
       sudo: required
       compiler: gcc
-      env: COMPILER=gcc PYTHON_VERSION=3.3 
+      env: COMPILER=gcc PYTHON_VERSION=3.4 
       before_install:
         - wget http://repo.continuum.io/miniconda/Miniconda3-latest-Linux-x86_64.sh -O miniconda.sh
-        - bash miniconda.sh -b -p $HOME/miniconda
-        - export PATH="$HOME/miniconda/bin:$PATH"
-        - conda create -q --yes -n test python=$PYTHON_VERSION numpy=$NUMPY_VERSION sphinx
-        - source activate test
 
     - os: linux
       dist: trusty
@@ -191,22 +134,28 @@ matrix:
       env: COMPILER=gcc PYTHON_VERSION=3.5
       before_install:
         - wget http://repo.continuum.io/miniconda/Miniconda3-latest-Linux-x86_64.sh -O miniconda.sh
-        - bash miniconda.sh -b -p $HOME/miniconda
-        - export PATH="$HOME/miniconda/bin:$PATH"
-        - conda create -q --yes -n test python=$PYTHON_VERSION numpy=$NUMPY_VERSION sphinx
-        - source activate test
-
+
+    - os: linux
+      dist: trusty
+      sudo: required
+      compiler: gcc
+      env: COMPILER=gcc PYTHON_VERSION=3.6
+      before_install:
+        - wget http://repo.continuum.io/miniconda/Miniconda3-latest-Linux-x86_64.sh -O miniconda.sh
 
 install:
+  - bash miniconda.sh -b -p $HOME/miniconda
+  - export PATH="$HOME/miniconda/bin:$PATH"
+  - conda create -q --yes -n test python=$PYTHON_VERSION numpy=$NUMPY_VERSION sphinx
+  - source activate test
+  - conda install -q --yes -c asmeurer gsl
   - make -r CC=$COMPILER
   - make install CC=$COMPILER
   - python setup.py install
 
 script:
   - make tests CC=$COMPILER
-
-after_success:
-  - cd docs && make html && cd ../ 
-  - if [[ "$TRAVIS_OS_NAME" != "osx" ]]; then cd docs && make doctest && cd ../; fi
+  - make -C docs html
+  - if [[ "${DOCTEST}" == "TRUE" ]]; then make -C docs doctest ; fi
 
 
diff --git a/Corrfunc/__init__.py b/Corrfunc/__init__.py
@@ -10,7 +10,7 @@
                         unicode_literals)
 import os
 
-__version__ = "2.0.1"
+__version__ = "2.1.0"
 __author__ = "Manodeep Sinha <[email protected]>"
 
 

diff --git a/Corrfunc/call_correlation_functions.py b/Corrfunc/call_correlation_functions.py
@@ -22,7 +22,8 @@
     countpairs_rp_pi as DDrppi_extn,\
     countpairs_wp as wp_extn,\
     countpairs_xi as xi_extn,\
-    countspheres_vpf as vpf_extn
+    countspheres_vpf as vpf_extn,\
+    countpairs_s_mu as DDsmu_extn
 
 
 def main():
@@ -74,6 +75,26 @@ def main():
               .format(items[0], items[1], items[2], items[3], items[4], items[5]))
     print("------------------------------------------------------------------------")
 
+    mu_max = 0.5
+    nmu_bins = 10
+
+    print("\nRunning 2-D correlation function DD(s,mu)")
+    results_DDsmu, _ = DDsmu_extn(autocorr, nthreads, binfile,
+                                    mu_max, nmu_bins,
+                                    x, y, z,
+                                    weights1=np.ones_like(x), weight_type='pair_product',
+                                    verbose=True, periodic=periodic,
+                                    boxsize=boxsize, output_savg=True)
+    print("\n#            ****** DD(s,mu): first {0} bins  *******      "
+          .format(numbins_to_print))
+    print("#      smin        smax       savg     mu_max     npairs    weightavg")
+    print("########################################################################")
+    for ibin in range(numbins_to_print):
+        items = results_DDsmu[ibin]
+        print("{0:12.4f} {1:12.4f} {2:10.4f} {3:10.1f} {4:10d} {5:10.4f}"
+              .format(items[0], items[1], items[2], items[3], items[4], items[5]))
+    print("------------------------------------------------------------------------")
+
     print("\nRunning 2-D projected correlation function wp(rp)")
     results_wp, _, _ = wp_extn(boxsize, pimax, nthreads,
                             binfile, x, y, z,

diff --git a/Corrfunc/call_correlation_functions_mocks.py b/Corrfunc/call_correlation_functions_mocks.py
@@ -19,6 +19,7 @@ def main():
     from Corrfunc.io import read_catalog
     from Corrfunc._countpairs_mocks import\
         countpairs_rp_pi_mocks as rp_pi_mocks_extn,\
+        countpairs_s_mu_mocks as s_mu_mocks_extn,\
         countpairs_theta_mocks as theta_mocks_extn,\
         countspheres_vpf_mocks as vpf_mocks_extn
 
@@ -52,7 +53,7 @@ def main():
                                          output_rpavg=True, verbose=True)
     print("\n#            ****** DD(rp,pi): first {0} bins  *******      "
           .format(numbins_to_print))
-    print("#      rmin        rmax       rpavg     pi_upper     npairs    weightavg")
+    print("#      rmin        rmax       rpavg     pi_upper     npairs   weightavg")
     print("########################################################################")
     for ibin in range(numbins_to_print):
         items = results_DDrppi[ibin]
@@ -61,20 +62,40 @@ def main():
 
     print("------------------------------------------------------------------------")
 
+    nmu_bins = 10
+    mu_max = 1.0
+
+    print("\nRunning 2-D correlation function xi(s,mu)")
+    results_DDsmu, _ = s_mu_mocks_extn(autocorr, cosmology, nthreads,
+                                       mu_max, nmu_bins, binfile,
+                                       ra, dec, cz, weights1=np.ones_like(ra),
+                                       output_savg=True, verbose=True,
+                                       weight_type='pair_product')
+    print("\n#            ****** DD(s,mu): first {0} bins  *******      "
+          .format(numbins_to_print))
+    print("#      smin        smax       savg     mu_upper       npairs    weight_avg")
+    print("###########################################################################")
+    for ibin in range(numbins_to_print):
+        items = results_DDsmu[ibin]
+        print("{0:12.4f} {1:12.4f} {2:10.4f} {3:10.1f} {4:10d} {5:12.4f}"
+              .format(items[0], items[1], items[2], items[3], items[4], items[5]))
+
+    print("--------------------------------------------------------------------------")
+
     binfile = pjoin(dirname(abspath(__file__)),
                     "../mocks/tests/", "angular_bins")
-    print("\nRunning angular correlation function w(theta)")
+    print("\nRunning angular correlation function DD(theta)")
     results_wtheta, _ = theta_mocks_extn(autocorr, nthreads, binfile,
                                          ra, dec, RA2=ra, DEC2=dec,
                                          weights1=np.ones_like(ra),
                                          weights2=np.ones_like(ra),
                                          weight_type='pair_product',
                                          output_thetaavg=True, fast_acos=True,
                                          verbose=1)
-    print("\n#         ******  wtheta: first {0} bins  *******        "
+    print("\n#         ******  DD(theta): first {0} bins  *******        "
           .format(numbins_to_print))
-    print("#      thetamin        thetamax       thetaavg      npairs    weightavg")
-    print("#######################################################################")
+    print("#      thetamin        thetamax       thetaavg        npairs      weightavg")
+    print("############################################################################")
     for ibin in range(numbins_to_print):
         items = results_wtheta[ibin]
         print("{0:14.4f} {1:14.4f} {2:14.4f} {3:14d} {4:14.4f}"

diff --git a/Corrfunc/io.py b/Corrfunc/io.py
@@ -10,6 +10,10 @@
 from os.path import dirname, abspath, splitext, exists as file_exists,\
     join as pjoin
 import numpy as np
+try:
+    import pandas as pd
+except ImportError:
+    pd = None
 
 
 __all__ = ('read_fastfood_catalog', 'read_ascii_catalog', 'read_catalog')
@@ -220,11 +224,6 @@ def read_ascii_catalog(filename, return_dtype=None):
 
     # check if pandas is available - much faster to read in the data
     # using pandas
-    try:
-        import pandas as pd
-    except ImportError:
-        pd = None
-
     if pd is not None:
         df = pd.read_csv(filename, header=None,
                          engine="c",