Merge pull request #319 from bsc-wdc/release-0.6

Release 0.6

CHANGELOG.md

@@ -4,6 +4,36 @@ All notable changes to this project will be documented in this file.
 The format is based on [Keep a Changelog](https://keepachangelog.com/en/1.0.0/),
 and this project adheres to [Semantic Versioning](https://semver.org/spec/v2.0.0.html).
 
+## [0.6.0] - 2020-10-09
+### Added
+- User guide and glossary
+- Method to read from npy files
+- Support for one-dimensional data in ds-array
+- Parametrized ds-array tests
+- identity, full and zeros methods that generate ds-arrays filled with a value
+- ds-array operators: subtraction, division, conjugate, transpose, item setting, etc.
+- matmul, kronecker product and rechunk methods for of ds-arrays
+- Automatic deletion of ds-arrays when the GC is called
+- Multivariate linear regression
+- SVD (Singular Value Decomposition)
+- PCA using SVD
+- ADMM Lasso algorithm
+- Daura clustering algorithm
+
+### Changed
+- Improved performance testing scripts and added new tests
+- Allow executing applications with params using dislib exec
+- Extended and improved the tutorial notebook
+- Moved data loading routines to a different file as array.py was getting too big
+- apply_along_axis for sparse data now returns sparse ds-arrays
+- Updated dislib-base docker image
+- Replaced COLLECTION_INOUT parameters with COLLECTION_OUT when possible for improving performance
+- Updated requirement PyCOMPSs >= 2.7
+
+### Fixed
+- Some bugs in the ds-array
+- Internal inconsistencies in transformed_array of PCA
+
 ## [0.5.0] - 2019-11-25
 ### Added
 - Grid search and randomized search with cross-validation
@@ -116,7 +146,8 @@ process.
 - Moved the quickstart guide to a separate file and included it in the documentation
 - Fixed several bugs
 
-[Unreleased]: https://github.com/bsc-wdc/dislib/compare/v0.5.0...HEAD
+[Unreleased]: https://github.com/bsc-wdc/dislib/compare/v0.6.0...HEAD
+[0.5.0]: https://github.com/bsc-wdc/dislib/compare/v0.5.0...v0.6.0
 [0.5.0]: https://github.com/bsc-wdc/dislib/compare/v0.4.0...v0.5.0
 [0.4.0]: https://github.com/bsc-wdc/dislib/compare/v0.3.0...v0.4.0
 [0.3.0]: https://github.com/bsc-wdc/dislib/compare/v0.2.0...v0.3.0

NOTICE

@@ -1,4 +1,4 @@
-Copyright 2019 Barcelona Supercomputing Center (BSC)
+Copyright 2019-2020 Barcelona Supercomputing Center (BSC)
 
 Licensed under the Apache License, Version 2.0 (the "License");
 you may not use this file except in compliance with the License.

QUICKSTART.md

@@ -22,7 +22,7 @@ numpydoc >= 0.8.0 is requried to build the documentation.
 #### Installation steps 
 
 1. Check which PyCOMPSs version to install. 
-    * Latest dislib release requires **PyCOMPSs 2.5** or greater (check [here](https://github.com/bsc-wdc/dislib/releases) for information about other releases).
+    * Latest dislib release requires **PyCOMPSs 2.7** or greater (check [here](https://github.com/bsc-wdc/dislib/releases) for information about other releases).
 
 2. Install PyCOMPSs following these [instructions](https://compss-doc.readthedocs.io/en/2.7/Sections/01_Installation.html).
 

VERSION

@@ -1 +1 @@
-0.5.0
+0.6.0

dislib/__init__.py

@@ -22,7 +22,7 @@
 
         __version__ = pkg_resources.require("dislib")[0].version
     except Exception as e:
-        print("Could not get installed dislib version. "
+        print("This dislib installation does not have a version number. "
               "Probably it was not installed with setup.py.\n%s" % e)
         __version__ = 'unknown'
 

docs/source/index.rst

@@ -54,20 +54,24 @@ Labels on the horizontal axis represent algorithm-dataset, where:
 - ALS = AlternatingLeastSquares
 - CSVM = CascadeSVM
 - GMM = GaussianMixture
+- Load = :meth:`load_svmlight_file <dislib.load_svmlight_file>`
 - RF = RandomForestClassifier
+- Shuf = :meth:`shuffle <dislib.utils.base.shuffle>`
+- TR = :meth:`Array.transpose <dislib.data.array.Array.transpose>`
 
 and:
 
 - Netflix = The Netflix Prize `dataset <https://www.kaggle
   .com/netflix-inc/netflix-prize-data>`_.
-- ijcnn1 = The `ijcnn1 <https://www.csie.ntu.edu
-  .tw/~cjlin/libsvmtools/datasets/binary.html#ijcnn1>`_ dataset.
 - KDD99 = The `KDDCUP 1999 <http://kdd.ics.uci.edu/databases
   /kddcup99/kddcup99.html>`_ dataset.
+- ijcnn1 = The `ijcnn1 <https://www.csie.ntu.edu
+  .tw/~cjlin/libsvmtools/datasets/binary.html#ijcnn1>`_ dataset.
 - gaia = The Tycho-Gaia Astrometric Solution dataset [1]_.
-- 1M and 3M = 1 and 3 million random samples.
+- 100M and 300M = 100 and 300 million random samples, with 100 features each.
 - mnist = The `mnist <https://www.csie.ntu.edu
   .tw/~cjlin/libsvmtools/datasets/multiclass.html#mnist>`_ dataset.
+- 20K = Square matrix of 20 thousand rows and 20 thousand columns, with random values.
 
 
 Source code

notebooks/estimators.ipynb

@@ -10,7 +10,7 @@
     "\n",
     "## Requirements\n",
     "\n",
-    "Apart from dislib, this notebook requires [PyCOMPSs 2.5](https://www.bsc.es/research-and-development/software-and-apps/software-list/comp-superscalar/).\n",
+    "Apart from dislib, this notebook requires [PyCOMPSs](https://www.bsc.es/research-and-development/software-and-apps/software-list/comp-superscalar/).\n",
     "\n",
     "\n",
     "## Setup\n",

notebooks/tutorial.ipynb

@@ -10,7 +10,7 @@
     "\n",
     "## Requirements\n",
     "\n",
-    "Apart from dislib, this notebook requires [PyCOMPSs 2.5](https://www.bsc.es/research-and-development/software-and-apps/software-list/comp-superscalar/).\n",
+    "Apart from dislib, this notebook requires [PyCOMPSs](https://www.bsc.es/research-and-development/software-and-apps/software-list/comp-superscalar/).\n",
     "\n",
     "\n",
     "## Setup\n",

tests/performance/mn4/scripts/performance.py

@@ -1,6 +1,8 @@
+import os
 import time
 
-from pycompss.api.api import compss_barrier
+from pycompss.api.api import compss_barrier, compss_wait_on
+from pycompss.api.task import task
 
 
 def measure(name, dataset_name, func, *args, **kwargs):
@@ -10,3 +12,9 @@ def measure(name, dataset_name, func, *args, **kwargs):
     func(*args, **kwargs)
     compss_barrier()
     print("==== TIME ==== ", name, dataset_name, time.time() - s_time)
+    print("In worker_working_dir: ", compss_wait_on(get_worker_working_dir()))
+
+
+@task(returns=1)
+def get_worker_working_dir():
+    return os.getcwd()