Merge pull request #87 from cosanlab/licensing

Licensing

Author: jcheong0428

.gitignore

@@ -14,6 +14,7 @@ feat/tests/data/fex_s20_cv_20170330.txt
 feat/tests/data/cond_pain_2017-03-30_s20_r01.txt
 notebooks/content/*.csv
 notebooks/content/*.mp4
+notebooks/content/dev_*
 
 #Ignoreresources folder.
 resources/

LICENSE

@@ -1,10 +1,17 @@
 
 MIT License
 
-Copyright (c) 2018, Jin Hyun Cheong, TianKang Xie, Sophie Byrne, Nathaniel Hanes, Luke Chang
+Copyright (c) 2018, Jin Hyun Cheong, TianKang Xie, Sophie Byrne, Luke Chang
 
 Permission is hereby granted, free of charge, to any person obtaining a copy of this software and associated documentation files (the "Software"), to deal in the Software without restriction, including without limitation the rights to use, copy, modify, merge, publish, distribute, sublicense, and/or sell copies of the Software, and to permit persons to whom the Software is furnished to do so, subject to the following conditions:
 
 The above copyright notice and this permission notice shall be included in all copies or substantial portions of the Software.
 
 THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY, FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM, OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE SOFTWARE.
+
+FaceBoxes: https://github.com/cleardusk/3DDFA_V2/blob/master/LICENSE
+RetinaFace: https://github.com/biubug6/Pytorch_Retinaface/blob/master/LICENSE.MIT
+MTCNN: https://github.com/ipazc/mtcnn/blob/master/LICENSE
+FacialLandmarks: https://github.com/cunjian/pytorch_face_landmark
+Residual Masking Network: https://github.com/phamquiluan/ResidualMaskingNetwork/issues/18
+JAA-Net: https://github.com/ZhiwenShao/PyTorch-JAANet/issues/19

README.md

@@ -1,12 +1,14 @@
-# Py-FEAT: Python Facial Expression Analysis Toolbox (FEAT)
+# Py-FEAT: Python Facial Expression Analysis Toolbox
 [![Package versioning](https://img.shields.io/pypi/v/py-feat.svg)](https://pypi.org/project/py-feat/)
 [![Build Status](https://api.travis-ci.org/cosanlab/py-feat.svg?branch=master)](https://travis-ci.org/cosanlab/py-feat/)
 [![Coverage Status](https://coveralls.io/repos/github/cosanlab/py-feat/badge.svg?branch=master)](https://coveralls.io/github/cosanlab/py-feat?branch=master)
+![Python Versions](https://img.shields.io/badge/python-3.6%20%7C%203.7%20%7C%203.8%20%7C%203.9-blue)
+[![GitHub license](https://img.shields.io/github/license/cosanlab/py-feat)](https://github.com/cosanlab/py-feat/blob/master/LICENSE)
 
 
 Py-FEAT is a suite for facial expressions (FEX) research written in Python. This package includes tools to detect faces, extract emotional facial expressions (e.g., happiness, sadness, anger), facial muscle movements (e.g., action units), and facial landmarks, from videos and images of faces, as well as methods to preprocess, analyze, and visualize FEX data. 
 
-For detailed examples, tutorials, and API please refer to the [Py-FEAT website](https://cosanlab.github.io/feat/). 
+For detailed examples, tutorials, and API please refer to the [Py-FEAT website](https://cosanlab.github.io/py-feat/). 
 
 ## Installation
 Option 1: Easy installation for quick use
@@ -41,7 +43,7 @@ out = detector.detect_image("input.png")
 out.plot_detections()
 ```
 ### 3. Preprocessing & analyzing FEX data
-See examples in our [tutorial](https://cosanlab.github.io/py-feat/content/analysis.html#). 
+We provide a number of preprocessing and analysis functionalities including baselining, feature extraction such as timeseries descriptors and wavelet decompositions, predictions, regressions, and intersubject correlations. See examples in our [tutorial](https://cosanlab.github.io/py-feat/content/analysis.html#). 
 
 ## Supported Models 
 Please respect the usage licenses for each model.
@@ -76,3 +78,5 @@ Emotion detection models
 4. Run the tests again with `pytest tests/` to make sure everything still passes, including your new feature. If you broke something, edit your feature so that it doesn't break existing code. 
 5. Create a pull request to the main repository's `master` branch.
 
+## Licenses
+Py-FEAT is provided under the MIT license. You also need to respect the licenses of each model you are using. Please see the LICENSE file for links to each model's license information. 

docs/conf.py

@@ -64,7 +64,7 @@
 
 # General information about the project.
 project = u'FEAT'
-copyright = u"2019, Jin Hyun Cheong, TianKang Xie, Sophie Byrne, Nathaniel Hanes, Luke Chang "
+copyright = u"2019, Jin Hyun Cheong, TianKang Xie, Sophie Byrne, Luke Chang "
 
 # The version info for the project you're documenting, acts as replacement
 # for |version| and |release|, also used in various other places throughout
@@ -218,7 +218,7 @@
 latex_documents = [
     ('index', 'feat.tex',
      u'FEAT Documentation',
-     u'Jin Hyun Cheong, Sophie Byrne, Nathaniel Hanes, Luke Chang ', 'manual'),
+     u'Jin Hyun Cheong, Tiankang Xie, Sophie Byrne, Luke Chang ', 'manual'),
 ]
 
 # The name of an image file (relative to this directory) to place at
@@ -249,7 +249,7 @@
 man_pages = [
     ('index', 'feat',
      u'FEAT Documentation',
-     [u'Jin Hyun Cheong, Sophie Byrne, Nathaniel Hanes, Luke Chang '], 1)
+     [u'Jin Hyun Cheong, Tiankang Xie, Sophie Byrne, Luke Chang '], 1)
 ]
 
 # If true, show URL addresses after external links.
@@ -264,7 +264,7 @@
 texinfo_documents = [
     ('index', 'feat',
      u'FEAT Documentation',
-     u'Jin Hyun Cheong, TianKang Xie, Sophie Byrne, Nathaniel Hanes, Luke Chang ',
+     u'Jin Hyun Cheong, TianKang Xie, Sophie Byrne, Luke Chang ',
      'feat',
      'One line description of project.',
      'Miscellaneous'),

feat/__init__.py

@@ -4,7 +4,7 @@
 
 from __future__ import absolute_import
 
-__author__ = """Jin Hyun Cheong, Tiankang Xie, Sophie Byrne, Nathaniel Hanes, Luke Chang """
+__author__ = """Jin Hyun Cheong, Tiankang Xie, Sophie Byrne, Luke Chang """
 __email__ = '[email protected]'
 __all__ = ['detector', 'data','utils','plotting','__version__']
 

feat/plotting.py

@@ -12,6 +12,7 @@
 import seaborn as sns
 import matplotlib.colors as colors
 from collections import OrderedDict
+from sklearn.preprocessing import minmax_scale
 
 __all__ = [
     "draw_lineface",
@@ -662,6 +663,12 @@ def draw_muscles(currx, curry, au=None, ax=None, *args, **kwargs):
         "orb_oris_u": orb_oris_u,
         "orb_oc_l": orb_oc_l,
         "cor_sup_l": cor_sup_l,
+        "pars_palp_l": orb_oc_l_inner,
+        "pars_palp_r": orb_oc_r_inner,
+        "masseter_l_rel": masseter_l,
+        "masseter_r_rel": masseter_r,
+        "temporalis_l_rel": temporalis_l,
+        "temporalis_r_rel": temporalis_r,
     }
 
     muscle_names = [
@@ -700,6 +707,12 @@ def draw_muscles(currx, curry, au=None, ax=None, *args, **kwargs):
         "orb_oris_l",
         "orb_oris_u",
         "cor_sup_l",
+        "pars_palp_l",
+        "pars_palp_r",
+        "masseter_l_rel",
+        "masseter_r_rel",
+        "temporalis_l_rel",
+        "temporalis_r_rel",
     ]
     todraw = {}
     facet = False
@@ -729,7 +742,6 @@ def draw_muscles(currx, curry, au=None, ax=None, *args, **kwargs):
                 del kwargs[muscle]
     for muscle in todraw.keys():
         if todraw[muscle] == "heatmap":
-            # muscles[muscle].set_color(get_heat(muscle, au, facet))
             muscles[muscle].set_color(get_heat(muscle, au, facet))
         else:
             muscles[muscle].set_color(todraw[muscle])
@@ -783,9 +795,9 @@ def get_heat(muscle, au, log):
     """Function to create heatmap from au vector
 
     Args:
-        au: vector of action units
-        muscle: string representation of a muscle
-        boolean: whether the action unit values are on a log scale
+        muscle (string): string representation of a muscle
+        au (list): vector of action units
+        log (boolean): whether the action unit values are on a log scale
 
 
     Returns:
@@ -828,18 +840,21 @@ def get_heat(muscle, au, log):
         "orb_oc_r_inner": 16,
         "orb_oris_l": 13,
         "orb_oris_u": 13,
+        "pars_palp_l": 19,
+        "pars_palp_r": 19,
+        "masseter_l_rel": 17,
+        "masseter_r_rel": 17,
+        "temporalis_l_rel": 17,
+        "temporalis_r_rel": 17,
     }
     if muscle in aus:
         unit = aus[muscle]
     if log:
         num = int(100 * (1.0 / (1 + 10.0 ** -(au[unit]))))
     else:
-        num = int(au[unit] * 20)
+        num = int(au[unit])
     # set alpha (opacity)
-    if au[unit] == 0:
-        alpha = 0
-    else:
-        alpha = 0.5
+    alpha = au[unit]/100
     # color = colors.to_hex(q[num])
     # return str(color)
     color = colors.to_rgba(q[num], alpha=alpha)
@@ -852,6 +867,7 @@ def plot_face(
     vectorfield=None,
     muscles=None,
     ax=None,
+    feature_range=False,
     color="k",
     linewidth=1,
     linestyle="-",
@@ -867,6 +883,7 @@ def plot_face(
         vectorfield: (dict) {'target':target_array,'reference':reference_array}
         muscles: (dict) {'muscle': color}
         ax: matplotlib axis handle
+        feature_range (tuple, default: None): If a tuple with (min, max),  scale input AU intensities to (min, max) before prediction.
         color: matplotlib color
         linewidth: matplotlib linewidth
         linestyle: matplotlib linestyle
@@ -888,14 +905,16 @@ def plot_face(
             "Don't forget to pass an 'au' vector of len(20), "
             "using neutral as default"
         )
-
-    landmarks = predict(au, model)
+        
+    landmarks = predict(au, model, feature_range = feature_range)
     currx, curry = [landmarks[x, :] for x in range(2)]
 
     if ax is None:
         ax = _create_empty_figure()
 
     if muscles is not None:
+        # Muscles are always scaled 0 - 100 b/c color palette is 0-100
+        au = minmax_scale(au, feature_range=(0,100))
         if not isinstance(muscles, dict):
             raise ValueError("muscles must be a dictionary ")
         draw_muscles(currx, curry, ax=ax, au=au, **muscles)
@@ -933,12 +952,13 @@ def plot_face(
     return ax
 
 
-def predict(au, model=None):
+def predict(au, model=None, feature_range=None):
     """Helper function to predict landmarks from au given a sklearn model
 
     Args:
         au: vector of action unit intensities
         model: sklearn pls object (uses pretrained model by default)
+        feature_range (tuple, default: None): If a tuple with (min, max),  scale input AU intensities to (min, max) before prediction.
 
     Returns:
         landmarks: Array of landmarks (2,68)
@@ -956,6 +976,9 @@ def predict(au, model=None):
     if len(au.shape) == 1:
         au = np.reshape(au, (1, -1))
 
+    if feature_range:
+        au = minmax_scale(au, feature_range=feature_range, axis=1)
+
     landmarks = np.reshape(model.predict(au), (2, 68))
     # landmarks[1, :] = -1 * landmarks[1, :]  # this might not generalize to other models
     return landmarks

feat/tests/test_plot.py

@@ -89,7 +89,7 @@ def test_plot_face():
     assert_plot_shape(plt.gca())
     plt.close()
 
-    plot_face(au=au, vectorfield={"reference": predict(au2)})
+    plot_face(au=au, vectorfield={"reference": predict(au2)}, feature_range=(0,1))
     assert_plot_shape(plt.gca())
     plt.close()
 

notebooks/README.md

@@ -23,5 +23,5 @@ git commit -m "updated jupyter book"
 
 5. Upload to gh-pages
 ```
-ghp-import -n -p -f notebooks/_build/html
+ghp-import -n -p -f -c py-feat.org notebooks/_build/html
 ```

notebooks/_build/.doctrees/api/index.doctree

@@ -10,11 +10,11 @@
   },
   {
    "cell_type": "code",
-   "execution_count": 10,
+   "execution_count": 9,
    "metadata": {
     "ExecuteTime": {
-     "end_time": "2021-03-27T00:28:05.706631Z",
-     "start_time": "2021-03-27T00:28:03.811433Z"
+     "end_time": "2021-03-29T14:46:54.002864Z",
+     "start_time": "2021-03-29T14:46:51.722215Z"
     }
    },
    "outputs": [
@@ -196,13 +196,13 @@
   },
   {
    "cell_type": "code",
-   "execution_count": 7,
+   "execution_count": 10,
    "metadata": {
     "ExecuteTime": {
-     "end_time": "2021-03-26T23:58:58.227979Z",
-     "start_time": "2021-03-26T23:58:57.624279Z"
+     "end_time": "2021-03-29T14:46:58.671126Z",
+     "start_time": "2021-03-29T14:46:58.081734Z"
     },
-    "scrolled": true
+    "scrolled": false
    },
    "outputs": [
     {
@@ -265,6 +265,28 @@
     "# hf.close()"
    ]
   },
+  {
+   "cell_type": "markdown",
+   "metadata": {},
+   "source": [
+    "Load h5 model"
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": 4,
+   "metadata": {
+    "ExecuteTime": {
+     "end_time": "2021-03-29T14:46:09.331977Z",
+     "start_time": "2021-03-29T14:46:09.327240Z"
+    }
+   },
+   "outputs": [],
+   "source": [
+    "from feat.utils import load_h5\n",
+    "clf = load_h5('../../feat/resources/pyfeat_aus_to_landmarks.h5')"
+   ]
+  },
   {
    "cell_type": "markdown",
    "metadata": {

notebooks/_build/.doctrees/content/dev_plotting.doctree

@@ -3,6 +3,7 @@ Py-Feat: Python Facial Expression Analysis Toolbox
 [![Package versioning](https://img.shields.io/pypi/v/py-feat.svg)](https://pypi.org/project/py-feat/)
 [![Build Status](https://api.travis-ci.org/cosanlab/py-feat.svg?branch=master)](https://travis-ci.org/cosanlab/py-feat/)
 [![Coverage Status](https://coveralls.io/repos/github/cosanlab/py-feat/badge.svg?branch=master)](https://coveralls.io/github/cosanlab/py-feat?branch=master)
+![Python Versions](https://img.shields.io/badge/python-3.6%20%7C%203.7%20%7C%203.8%20%7C%203.9-blue)
 [![GitHub forks](https://img.shields.io/github/forks/cosanlab/py-feat)](https://github.com/cosanlab/py-feat/network)
 [![GitHub stars](https://img.shields.io/github/stars/cosanlab/py-feat)](https://github.com/cosanlab/py-feat/stargazers)
 [![GitHub license](https://img.shields.io/github/license/cosanlab/py-feat)](https://github.com/cosanlab/py-feat/blob/master/LICENSE)
@@ -57,13 +58,15 @@ from feat import Detector
 ## Available models
 Below is a list of models implemented in Py-Feat and ready to use. The model names are in the titles followed by the reference publications.
 ### Action Unit detection
-- `rf`: Random Forest model trained on Histogram of Oriented Gradients. 
-- `svm`: SVM model trained on Histogram of Oriented Gradients. 
-- `logistic`: Logistic Classifier model trained on Histogram of Oriented Gradients. 
-- `JAANET`: Joint facial action unit detection and face alignment via adaptive attention ([Shao et al., 2020](https://arxiv.org/pdf/2003.08834v1.pdf))
+- `rf`: Random Forest model trained on Histogram of Oriented Gradients extracted from BP4D, DISFA, CK+, UNBC-McMaster shoulder pain, and AFF-Wild2 datasets
+- `svm`: SVM model trained on Histogram of Oriented Gradients extracted from BP4D, DISFA, CK+, UNBC-McMaster shoulder pain, and AFF-Wild2 datasets
+- `logistic`: Logistic Classifier model trained on Histogram of Oriented Gradients extracted from BP4D, DISFA, CK+, UNBC-McMaster shoulder pain, and AFF-Wild2 datasets
+- `JAANET`: Joint facial action unit detection and face alignment via adaptive attention trained with BP4D and BP4D+ ([Shao et al., 2020](https://arxiv.org/pdf/2003.08834v1.pdf))
 - `DRML`: Deep region and multi-label learning for facial action unit detection by ([Zhao et al., 2016](https://www.cv-foundation.org/openaccess/content_cvpr_2016/papers/Zhao_Deep_Region_and_CVPR_2016_paper.pdf))
 ###  Emotion detection
-- `FeatNet` by Tiankang Xie 
+- `rf`: Random Forest model trained on Histogram of Oriented Gradients extracted from ExpW, CK+, and JAFFE datasets
+- `svm`: SVM model trained on Histogram of Oriented Gradients extracted from ExpW, CK+, and JAFFE datasets
+- `fernet`: Deep convolutional network
 - `ResMaskNet`: Facial expression recognition using residual masking network by ([Pham et al., 2020](https://ailb-web.ing.unimore.it/icpr/author/3818))
 ###  Face detection
 - `MTCNN`: Multi-task cascaded convolutional networks by ([Zhang et al., 2016](https://arxiv.org/pdf/1604.02878.pdf); [Zhang et al., 2020](https://ieeexplore.ieee.org/document/9239720))
@@ -78,4 +81,4 @@ Below is a list of models implemented in Py-Feat and ready to use. The model nam
 We are excited for people to add new models and features to Py-Feat. Please see the [contribution guides](https://cosanlab.github.io/feat/content/contribute.html). 
 
 ## License 
-Py-Feat is under the [MIT license](https://github.com/cosanlab/feat/blob/master/LICENSE) but the provided models may have separate licenses. Please cite the corresponding references for the models you've used.
+Py-FEAT is provided under the  [MIT license](https://github.com/cosanlab/py-feat/blob/master/LICENSE). You also need to cite and respect the licenses of each model you are using. Please see the LICENSE file for links to each model's license information.