added filters to correlation matrix and heatmap functionality

man-group · Nov 19, 2019 · 7359eb6 · 7359eb6
1 parent 42d1944
commit 7359eb6
Show file tree

Hide file tree

Showing 45 changed files with 848 additions and 130 deletions.
diff --git a/CHANGES.md b/CHANGES.md
@@ -66,3 +66,10 @@ Changelog
     * [#28](https://github.com/man-group/dtale/issues/28): "Instances" menu option will now be displayed by default
     * [#29](https://github.com/man-group/dtale/issues/29): add hints to how users can navigate the correlations popup
     * add "unicode" as a string classification for column width calculation
+
+### 1.4.0 (2019-11-19)
+
+  * Correlations Pearson Matrix filters
+  * "name" display in title tab
+  * "Heat Map" toggle
+  * dropped unused "Flask-Caching" requirement
diff --git a/README.md b/README.md
@@ -7,7 +7,7 @@
 [![CircleCI](https://circleci.com/gh/man-group/dtale.svg?style=shield&circle-token=4b67588a87157cc03b484fb96be438f70b5cd151)](https://circleci.com/gh/man-group/dtale)
 [![PyPI](https://img.shields.io/pypi/pyversions/dtale.svg)](https://pypi.python.org/pypi/dtale/)
 [![ReadTheDocs](https://readthedocs.org/projects/dtale/badge)](https://dtale.readthedocs.io)
-[![codecov](https://codecov.io/gh/man-group/dtale/branch/master/graph/badge.svg)](https://codecov.io/gh/manahl/dtale)
+[![codecov](https://codecov.io/gh/man-group/dtale/branch/master/graph/badge.svg)](https://codecov.io/gh/man-group/dtale)
 [![Downloads](https://pepy.tech/badge/dtale)](https://pepy.tech/project/dtale)
 
 ## Getting Started
@@ -34,15 +34,16 @@ $ source ~/pyenvs/dtale/bin/activate
 # install dtale egg (important to use the "--upgrade" every time you install so it will grab the latest version)
 $ pip install --upgrade dtale
 ```
-Now you will have to ability to use D-Tale from the command-line or within a python-enabled terminal
+Now you will have to ability to use D-Tale from the command-line or within a python-enabled terminal (**FYI: Due a dependency on Flasgger, jsonschema is limited to version <3.0.0**)
 
 ### Command-line
 Base CLI options (run `dtale --help` to see all options available)
 
 |Prop|Description|
-|:--------:|------|
+|:--------|------|
 |`--host`|the name of the host you would like to use (most likely not needed since `socket.gethostname()` should figure this out)|
 |`--port`|the port you would like to assign to your D-Tale instance|
+|`--name`|an optional name you can assign to your D-Tale instance (this will be displayed in the `<title>` & Instances popup)|
 |`--debug`|turn on Flask's "debug" mode for your D-Tale instance|
 |`--no-reaper`|flag to turn off auto-reaping subprocess (kill D-Tale instances after an hour of inactivity), good for long-running displays |
 |`--open-browser`|flag to automatically open up your server's default browser to your D-Tale instance|
@@ -203,7 +204,7 @@ Selecting/Deselecting Columns
 |![](https://raw.githubusercontent.com/man-group/dtale/master/docs/images/Coverage_daily.png)|![](https://raw.githubusercontent.com/man-group/dtale/master/docs/images/Coverage_daily_regions.png)|
 
 - **Correlations**: shows a pearson correlation matrix of all numeric columns against all other numeric columns
-  - By deafult, it will show a grid of pearson correlations
+  - By deafult, it will show a grid of pearson correlations (filtering available by using drop-down see 2nd table of screenshots)
   - If you have a date-type column, you can click an individual cell and see a timeseries of pearson correlations for that column combination
     - Currently if you have multiple date-type columns you will have the ability to toggle between them by way of a drop-down
   - Furthermore, you can click on individual points in the timeseries to view the scatter plot of the points going into that correlation
@@ -212,11 +213,17 @@ Selecting/Deselecting Columns
 |------|----------|-------|
 |![](https://raw.githubusercontent.com/man-group/dtale/master/docs/images/Correlations.png)|![](https://raw.githubusercontent.com/man-group/dtale/master/docs/images/Correlations_ts.png)|![](https://raw.githubusercontent.com/man-group/dtale/master/docs/images/Correlations_scatter.png)|
 
-- **About**: This will give you information about what version of D-Tale you're running as well as if its out of date to whats on PyPi.
+|Col1 Filtered|Col2 Filtered|Col1 & Col2 Filtered|
+|------|----------|-------|
+|![](https://raw.githubusercontent.com/man-group/dtale/master/docs/images/Correlations_col1.png)|![](https://raw.githubusercontent.com/man-group/dtale/master/docs/images/Correlations_col2.png)|![](https://raw.githubusercontent.com/man-group/dtale/master/docs/images/Correlations_both.png)|
 
-|Up To Date|Out Of Date|
-|--------|:------:|
-|![](https://raw.githubusercontent.com/man-group/dtale/master/docs/images/About-up-to-date.png)|![](https://raw.githubusercontent.com/man-group/dtale/master/docs/images/About-out-of-date.png)|
+- **Heat Map**: this will hide any non-float columns (with the exception of the index on the right) and apply a color to the background of each cell
+  - Each float is renormalized to be a value between 0 and 1.0
+  - Each renormalized value is passed to a color scale of red(0) - yellow(0.5) - green(1.0)
+![](https://raw.githubusercontent.com/man-group/dtale/master/docs/images/Heatmap.png)
+
+Turn off Heat Map by clicking menu option again
+![](https://raw.githubusercontent.com/man-group/dtale/master/docs/images/Heatmap_toggle.png)
 
 - **Instances**: this will give you information about other D-Tale instances are running under your current Python process.
 
@@ -251,6 +258,12 @@ Here is an example of clicking the "Preview" button:
 
 ![](https://raw.githubusercontent.com/man-group/dtale/master/docs/images/Instances_preview.png)
 
+- **About**: This will give you information about what version of D-Tale you're running as well as if its out of date to whats on PyPi.
+
+|Up To Date|Out Of Date|
+|--------|:------:|
+|![](https://raw.githubusercontent.com/man-group/dtale/master/docs/images/About-up-to-date.png)|![](https://raw.githubusercontent.com/man-group/dtale/master/docs/images/About-out-of-date.png)|
+
 - **Resize**: mostly a fail-safe in the event that your columns are no longer lining up. Click this and should fix that
 - **Shutdown**: pretty self-explanatory, kills your D-Tale session (there is also an auto-kill process that will kill your D-Tale after an hour of inactivity)
 

diff --git a/docker/2_7/Dockerfile b/docker/2_7/Dockerfile
@@ -44,4 +44,4 @@ WORKDIR /app
 
 RUN set -eux \
   ; . /root/.bashrc \
-  ; easy_install dtale-1.1.0-py2.7.egg
+  ; easy_install dtale-1.4.0-py2.7.egg
diff --git a/docker/3_6/Dockerfile b/docker/3_6/Dockerfile
@@ -44,4 +44,4 @@ WORKDIR /app
 
 RUN set -eux \
   ; . /root/.bashrc \
-  ; easy_install dtale-1.1.0-py3.7.egg
+  ; easy_install dtale-1.4.0-py3.7.egg
diff --git a/docs/images/Correlations.png b/docs/images/Correlations.png
diff --git a/docs/images/Correlations_both.png b/docs/images/Correlations_both.png
diff --git a/docs/images/Correlations_col1.png b/docs/images/Correlations_col1.png
diff --git a/docs/images/Correlations_col2.png b/docs/images/Correlations_col2.png
diff --git a/docs/images/Correlations_scatter.png b/docs/images/Correlations_scatter.png
diff --git a/docs/images/Correlations_ts.png b/docs/images/Correlations_ts.png
diff --git a/docs/images/Heatmap.png b/docs/images/Heatmap.png
diff --git a/docs/images/Heatmap_toggle.png b/docs/images/Heatmap_toggle.png
diff --git a/docs/images/Info_menu_small.png b/docs/images/Info_menu_small.png
diff --git a/docs/images/Menu_one_col.png b/docs/images/Menu_one_col.png
diff --git a/docs/source/conf.py b/docs/source/conf.py
@@ -63,9 +63,9 @@
 # built documents.
 #
 # The short X.Y version.
-version = u'1.3.7'
+version = u'1.4.0'
 # The full version, including alpha/beta/rc tags.
-release = u'1.3.7'
+release = u'1.4.0'
 
 # The language for content autogenerated by Sphinx. Refer to documentation
 # for a list of supported languages.

diff --git a/docs/source/requirements.txt b/docs/source/requirements.txt
@@ -1,4 +1,4 @@
-jsonschema<3.0.0 # needed for sphinx, flasgger, Flask-Caching collisons
+jsonschema<3.0.0 # needed for sphinx, flasgger collisons
 flasgger==0.9.3
 sphinx==1.6.7
 nbsphinx

diff --git a/dtale/cli/script.py b/dtale/cli/script.py
@@ -17,6 +17,7 @@
 @click.option('--no-reaper', is_flag=True,
               help='flag to turn off auto-reaping (process cleanup after period of inactivity)')
 @click.option('--open-browser', is_flag=True, help='flag to automatically open default web browser on startup')
+@click.option('--name', type=str, help='name to apply to your D-Tale session')
 @setup_loader_options()
 @click.option('--log', 'logfile', help='Log file name')
 @click.option('--log-level',
@@ -25,7 +26,7 @@
               default='info',
               show_default=True)
 @click.option('-v', '--verbose', help='Set the logging level to debug', is_flag=True)
-def main(host, port=None, debug=False, no_reaper=False, open_browser=False, **kwargs):
+def main(host, port=None, debug=False, no_reaper=False, open_browser=False, name=None, **kwargs):
     """
     Runs a local server for the D-Tale application.
 
@@ -38,7 +39,7 @@ def main(host, port=None, debug=False, no_reaper=False, open_browser=False, **kw
     data_loader = check_loaders(kwargs)
 
     show(host=host, port=int(port or find_free_port()), debug=debug, subprocess=False, data_loader=data_loader,
-         reaper_on=not no_reaper, open_browser=open_browser, **kwargs)
+         reaper_on=not no_reaper, open_browser=open_browser, name=name, **kwargs)
 
 
 if __name__ == '__main__':

diff --git a/dtale/static/css/main.css b/dtale/static/css/main.css
@@ -10322,3 +10322,18 @@ table tbody tr.highlight-row {
 .dropdown:hover .dropdown-content {
     display: block;
 }
+
+.flames {
+    background: url('../images/fire.jpg');
+    background-size: cover;
+    background-repeat: no-repeat;
+    background-position: center;
+    -webkit-background-clip: text;
+    -webkit-text-fill-color: transparent;
+    -webkit-animation: fire .4s linear infinite;
+}
+
+@keyframes fire {
+    0%{background-position:0% 50%;}
+    0%{background-position:100% -50%;}
+}
diff --git a/dtale/static/images/fire.jpg b/dtale/static/images/fire.jpg
diff --git a/dtale/templates/dtale/base.html b/dtale/templates/dtale/base.html
@@ -6,7 +6,7 @@
 	<meta name="viewport" content="width=device-width, initial-scale=1, user-scalable=yes">
 	<meta http-equiv="Content-Type" content="text/html; charset=utf-8" />
 	<link rel="shortcut icon" href="{{ url_for('static', filename='images/favicon.png') }}">
-    <title>D-Tale</title>
+    <title>{{ title }}</title>
     <script type="text/javascript" src="{{ url_for('static', filename='dist/base_styles_bundle.js') }}"></script>
     {% block css %}{% endblock %}
     {#

diff --git a/dtale/views.py b/dtale/views.py
@@ -80,7 +80,18 @@ def build_dtypes_state(data):
     :return: a list of dictionaries containing column names, indexes and data types
     """
     dtypes = get_dtypes(data)
-    return [dict(name=c, dtype=dtypes[c], index=i) for i, c in enumerate(data.columns)]
+    mins = data.min().to_dict()
+    maxs = data.max().to_dict()
+
+    def _format_dtype(col_index, col):
+        dtype = dtypes[col]
+        dtype_data = dict(name=col, dtype=dtype, index=col_index)
+        if classify_type(dtype) == 'F':  # floats
+            dtype_data['min'] = mins[col]
+            dtype_data['max'] = maxs[col]
+        return dtype_data
+
+    return [_format_dtype(i, c) for i, c in enumerate(data.columns)]
 
 
 def format_data(data):
@@ -104,6 +115,7 @@ def startup(data=None, data_loader=None, port=None, name=None):
     :param data: pandas.DataFrame or pandas.Series
     :param data_loader: function which returns pandas.DataFrame
     :param port: integer port for running Flask process
+    :param name: string label to apply to your session
     """
     global DATA, DTYPES, SETTINGS, METADATA
 
@@ -116,9 +128,6 @@ def startup(data=None, data_loader=None, port=None, name=None):
                 'data loaded must be one of the following types: pandas.DataFrame, pandas.Series, pandas.DatetimeIndex'
             )
 
-        if isinstance(data, (pd.DatetimeIndex, pd.MultiIndex)):
-            data = data.to_frame(index=False)
-
         logger.debug('pytest: {}, flask: {}'.format(running_with_pytest(), running_with_flask()))
         data, curr_index = format_data(data)
         port_key = str(port)
@@ -170,10 +179,15 @@ def view_main():
 
     :return: HTML
     """
-    curr_settings = SETTINGS.get(get_port(), {})
+    port = get_port()
+    curr_settings = SETTINGS.get(port, {})
+    curr_metadata = METADATA.get(port, {})
     _, version = retrieve_meta_info_and_version('dtale')
+    title = 'D-Tale'
+    if curr_metadata.get('name'):
+        title = '{} ({})'.format(title, curr_metadata['name'])
     return render_template(
-        'dtale/main.html', settings=json.dumps(curr_settings), version=str(version), processes=len(DATA)
+        'dtale/main.html', settings=json.dumps(curr_settings), version=str(version), processes=len(DATA), title=title
     )
 
 
@@ -379,7 +393,7 @@ def get_data():
             ids = json.loads(ids)
         else:
             return jsonify({})
-        col_types = grid_columns(data)
+        col_types = DTYPES[port]
 
         f = grid_formatter(col_types)
         curr_settings = SETTINGS.get(port, {})

diff --git a/package.json b/package.json
@@ -1,6 +1,6 @@
 {
   "name": "dtale",
-  "version": "1.3.7",
+  "version": "1.4.0",
   "description": "All-purpose Data Viewer",
   "main": "main.js",
   "directories": {
@@ -141,6 +141,7 @@
     "react-modal-bootstrap": "1.1.1",
     "react-motion": "0.5.2",
     "react-redux": "7.1.1",
+    "react-select": "3.0.8",
     "react-virtualized": "9.21.1",
     "redux": "4.0.4",
     "redux-thunk": "2.3.0",

diff --git a/setup.py b/setup.py
@@ -50,7 +50,7 @@ def run_tests(self):
 
 setup(
     name="dtale",
-    version="1.3.7",
+    version="1.4.0",
     author="MAN Alpha Technology",
     author_email="[email protected]",
     description="Web Client for Visualizing Pandas Objects",
@@ -63,7 +63,6 @@ def run_tests(self):
         "jsonschema<3.0.0",
         "flasgger==0.9.3",
         "Flask",
-        "Flask-Caching",
         "Flask-Compress",
         "future",
         "itsdangerous",

diff --git a/static/__tests__/dtale/DataViewer-about-expired-test.jsx b/static/__tests__/dtale/DataViewer-about-expired-test.jsx
@@ -66,7 +66,8 @@ describe("DataViewer tests", () => {
       result
         .find(DataViewerMenu)
         .find("ul li button")
-        .at(5)
+        .findWhere(b => _.includes(b.text(), "About"))
+        .first()
         .simulate("click");
       setTimeout(() => {
         result.update();

diff --git a/static/__tests__/dtale/DataViewer-about-test.jsx b/static/__tests__/dtale/DataViewer-about-test.jsx
@@ -1,4 +1,5 @@
 import { mount } from "enzyme";
+import _ from "lodash";
 import React from "react";
 import { ModalClose } from "react-modal-bootstrap";
 import { Provider } from "react-redux";
@@ -63,7 +64,8 @@ describe("DataViewer tests", () => {
       result
         .find(DataViewerMenu)
         .find("ul li button")
-        .at(5)
+        .findWhere(b => _.includes(b.text(), "About"))
+        .first()
         .simulate("click");
       setTimeout(() => {
         result.update();
@@ -76,7 +78,8 @@ describe("DataViewer tests", () => {
         result
           .find(DataViewerMenu)
           .find("ul li button")
-          .at(5)
+          .findWhere(b => _.includes(b.text(), "About"))
+          .first()
           .simulate("click");
         setTimeout(() => {
           result.update();

diff --git a/static/__tests__/dtale/DataViewer-base-test.jsx b/static/__tests__/dtale/DataViewer-base-test.jsx
@@ -15,11 +15,11 @@ const originalOffsetHeight = Object.getOwnPropertyDescriptor(HTMLElement.prototy
 const originalOffsetWidth = Object.getOwnPropertyDescriptor(HTMLElement.prototype, "offsetWidth");
 
 const COL_PROPS = [
-  { locked: true, width: 70, name: "dtale_index", dtype: "int64" },
-  { locked: false, width: 20, name: "col1", dtype: "int64" },
-  { locked: false, width: 20, name: "col2", dtype: "float64" },
-  { locked: false, width: 20, name: "col3", dtype: "object" },
-  { locked: false, width: 20, name: "col4", dtype: "datetime64[ns]" },
+  { locked: true, width: 70, name: "dtale_index", dtype: "int64", visible: true },
+  { locked: false, width: 20, name: "col1", dtype: "int64", visible: true },
+  { locked: false, width: 20, name: "col2", dtype: "float64", visible: true, min: 2.5, max: 5.5 },
+  { locked: false, width: 20, name: "col3", dtype: "object", visible: true },
+  { locked: false, width: 20, name: "col4", dtype: "datetime64[ns]", visible: true },
 ];
 
 describe("DataViewer tests", () => {
@@ -90,7 +90,7 @@ describe("DataViewer tests", () => {
           .find(DataViewerMenu)
           .find("ul li span.font-weight-bold")
           .map(s => s.text()),
-        ["Describe", "Filter", "Correlations", "Coverage", "Resize", "About", "Instances 1", "Shutdown"],
+        ["Describe", "Filter", "Correlations", "Coverage", "Resize", "Heat Map", "Instances 1", "About", "Shutdown"],
         "Should render default menu options"
       );
 
@@ -119,7 +119,7 @@ describe("DataViewer tests", () => {
           .map(s => s.text()),
         _.concat(
           ["Describe", "Move To Front", "Lock", "Sort Ascending", "Sort Descending", "Clear Sort", "Filter", "Formats"],
-          ["Histogram", "Correlations", "Coverage", "Resize", "About", "Instances 1", "Shutdown"]
+          ["Histogram", "Correlations", "Coverage", "Resize", "Heat Map", "Instances 1", "About", "Shutdown"]
         ),
         "Should render menu options associated with selected column"
       );