Merge pull request #2 from ZanMervic/develop

Orange3-Fairness release 0.1.6

README.md

@@ -2,20 +2,20 @@
 
 
 <p align="center">
-<a href="https://github.com/ZanMervic//orange3-fairness/actions"><img alt="Actions Status" src="https://github.com/ZanMervic/orange3-fairness/actions/workflows/test.yml/badge.svg"></a>
+<a href="https://github.com/biolab//orange3-fairness/actions"><img alt="Actions Status" src="https://github.com/biolab/orange3-fairness/actions/workflows/test.yml/badge.svg"></a>
 <a href="https://pypi.org/project/Orange3-Fairness/"><img alt="PyPI" src="https://img.shields.io/pypi/v/orange3-fairness?color=blue"></a>
 <a href="https://codecov.io/gh/ZanMervic/orange3-fairness" ><img src="https://codecov.io/gh/ZanMervic/orange3-fairness/graph/badge.svg?token=MSQ0ZUPA6B"/></a>
 <a href='https://orange3-fairness.readthedocs.io/en/latest/?badge=latest'><img src='https://readthedocs.org/projects/orange3-fairness/badge/?version=latest' alt='Documentation Status' /></a>
 </p>
 
 ![Example Workflow](doc/readme-screenshot.png)
 
-Fairness add-on for the [Orange](http://orange.biolab.si).
+Fairness add-on for the [Orange](http://orangedatamining.com/).
 
 
 # Easy installation
 
-First, [download](https://orange.biolab.si/download) the latest Orange release from
+First, [download](https://orangedatamining.com/download/) the latest Orange release from
 our website. Then, to install the fairness add-on, head to
 `Options -> Add-ons...` in the menu bar.
 

README.pypi

@@ -1,7 +1,6 @@
 Orange3 Fairness
 ================
 
-Orange3 Fairness is an add-on for the [Orange3](http://orange.biolab.si) data mining suite. It provides extensions for fairness-aware AI.
+Orange3 Fairness is an add-on for the [Orange3](http://orangedatamining.com/) data mining suite. It provides extensions for fairness-aware AI.
 
-.. image:: https://raw.githubusercontent.com/ZanMervic/orange3-fairness/main/doc/readme-screenshot.png
-   :alt: Screenshot of Orange3 Fairness add-on
+![Screenshot of Orange3 Fairness add-on](https://raw.githubusercontent.com/biolab/orange3-fairness/main/doc/readme-screenshot.png)

doc/widgets/adversarial-debiasing.md

@@ -12,8 +12,15 @@ A classification model with a debiasing option.
 - Learner: adversarial debiasing learning algorithm
 - Model: a trained adversarial debiasing model
 
-**Adversarial Debiasing** classification algorithm with or without fairness constraints.
+**Adversarial Debiasing** widget offers a in-processing type of fairness mitigation algorithm. It involves simultaneous training of a predictor and a discriminator. The goal of the predictor is to predict the target variable accurately. At the same time, the discriminator aims to predict the protected variable (such as gender or race) based on the predictor’s predictions. The main goal is to maximize the predictor’s ability to predict the target variable while reducing the discriminator’s ability to predict the protected variable based on those predictions.
+
+In short, **Adversarial Debiasing** is a classification algorithm with or without fairness constraints.
+
+![](images/adversarial-debiasing.png)
 
 Example
 -------
-TODO
+
+In this example we will try to obtain bias free predictions using the `Adversarial Debiasing` widget. First we include the widget into the canvas and tune the settings to suit our needs. What is important here is to tick the `Use debiasing` box and set the `Adversary Loss Weight` to more than 0 if we want to see the effect of the debiasing. We then connect the widget along with the dataset into the `Test & Score` widget to evaluate the performance of the model. In the evaluation results we can see the performance of the model as well as the fairness metrics for its predictions.
+
+![](images/adversarial-debiasing-example.png)

doc/widgets/as-fairness.md

@@ -8,10 +8,17 @@ Adds Fairness attributes to the dataset.
 
 **Outputs**
 
-- Data: reference dataset with selected Fairness attributes.
+- Data: reference dataset with selected Fairness meta-attributes.
 
-**As Fairness** converts a dataset to a fairness dataset with marked favorable class values, protected attributes and priviliged protected attribute values.
+**As Fairness** is a widget that allows the user to select and add Fairness meta-attributes to the dataset. These meta-attributes are required by other fairness widget to function properly. These meta-attributes are: `Favorable Class Value`, `Protected Attributes` and `Privileged Protected Attribute Values`.
+
+![](images/as-fairness-data.png)
 
 Example
 -------
-TODO
+
+This simple example shows how to use the **As Fairness** widget. First, we load the German Credit Data dataset and then we connect it to the **As Fairness** widget. We select the `Personal status and sex` attribute as the `Protected Attribute` and all values representing males as the `Privileged Protected Attribute Values`. We also select `good` as the `Favorable Class Value`. The output of the widget is a dataset with the added meta-attributes.
+
+![](images/as-fairness-data-example.png)
+
+It should be noted that datasets included in orange with the tag `fairness` already have some default fairness meta-attributes added to them and thus do not require the use of the **As Fairness** widget before using other fairness widgets. These datasets include: `Adult`, `German Credit Data` and `COMPAS Analysis`.

doc/widgets/combine-preprocessors.md

@@ -11,8 +11,9 @@ Combines multiple preprocessors into one.
 
 - Preprocessor: a combination of first and second preprocessor
 
-**Combine Preprocessors** TODO
+**Combine Preprocessors** is a widget that combines multiple preprocessors into one which can be inputed into a learner widget. This comes in handy when we want to use the reweighing widget along with some other preprocessing method. We can simply connect the reweighing widget to the first preprocessor input and the other preprocessing method to the second preprocessor input. The output of the combine preprocessors widget can then be connected to the learner widget.
 
 Example
 -------
-TODO
+
+An example of using this widget can be seen in the [Reweighing](reweighing.md) widget documentation.

doc/widgets/dataset-bias.md

@@ -7,8 +7,17 @@ Computes the bias of a dataset.
 - Data: dataset to be evaluated
 
 
-**Dataset Bias** computes the bias of a dataset. More specifically, it computes the disparate impact and statistical parity difference metrics for the dataset.
+**Dataset Bias** computes and displays the bias of a dataset. More specifically, it computes the disparate impact and statistical parity difference metrics for the dataset.
+
+![](images/dataset-bias.png)
 
 Example
 -------
-TODO
+
+This example shows a very simple use of the **Dataset Bias** widget. First we load one of the fairness datasets, in this case the Adult dataset. Then we connect the dataset to the **Dataset Bias** widget. The widget displays the disparate impact and statistical parity difference metrics for the dataset.
+
+![](images/dataset-bias-example.png)
+
+Note, we did not use the [As Fairness Data](as-fairness.md) widget before using the **Dataset Bias** widget. This is because the Adult dataset already has the required fairness meta-attributes added to it.
+
+Another thing to note is that the **Dataset Bias** widget (and other fairness widgets) do not support missing values, this is indicated by the warning icon above the widget. Any missing values in the dataset will automatically be imputed with the average or most frequent value before being used by the widget.

doc/widgets/equalized-odds-postprocessing.md

@@ -13,8 +13,11 @@ Postprocessing Fairness algorithm
 - Learner: learner with Fairness postprocessing
 - Model: a trained model with Fairness postprocessing
 
-**Equalized Odds postprocessing** Postprocessing fairness algorithm which changes the predictions of a classifier to satisfy equalized odds.
+**Equalized Odds postprocessing** Postprocessing fairness algorithm which modifies the predictions of any given classifier to meet certain fairness criteria. It works by first fitting the learner to the training data, creating a model, and using it to get the predictions. It uses these predictions to fit the Equalized Odds Postprocessing algorithm, which creates a post-processor. This post-processor is then used to adjust the model’s predictions on the test data.
 
 Example
 -------
-TODO
+
+In this example we will use the `Equalized Odds Postprocessing` to debias the predictions of a linear regression model. All we need to do is include the `Equalized Odds Postprocessing` widget in our workflow and connect any model and desired preprocessors to it. We then connect the `Equalized Odds Postprocessing` to the `Test & Score` widget along with the dataset to evaluate the performance of the model with postprocessing.
+
+![](images/equal-odds-postprocessing-example.png)

doc/widgets/reweighing.md

@@ -8,11 +8,24 @@ Applies the reweighing algorithm to the dataset.
 
 **Outputs**
 
-- Preprocessed Data:
--
+- Preprocessed Data: reference dataset with a meta-attribute `weights` added to it.
+- Preprocessor: a preprocessor trained on the reference dataset.
 
-**Reweighing** applies the reweighing algorithm to a dataset, which adjusts the weights of rows.
+**Reweighing** is a widget that mitigates bias in a dataset by assigning weights to individual instances in a way that encourages the model to prioritize learning from underrepresented groups while de-emphasizing overrepresented groups. This widget can be used in two ways:
+
+- When the user provides the data as an input this widget will apply the reweighing algorithm to the dataset and output the preprocessed dataset with the meta-attribute `weights` added to it. The user can then use the preprocessed dataset as an input to other widgets. It will also output a preprocessor that can be used to apply the same transformation to a subset of the dataset.
+
+- This widget can also be provided as an input to a learner widget. In this case the widget will be applied to any of the training datasets that are provided as an input to the learner widget.
 
 Example
 -------
-TODO
+
+The first example shows how the `Reweighing` widget can be used to preprocess a dataset. First load a fairness dataset, in this case we will use the compas analysis dataset. We than split the dataset into a training and testing set using the `Data Sampler` widget. We connect the training set to the `Reweighing` widget which will train the algorithm and create a preprocessor. The preprocessor can be connected to the `Apply Domain` widget along with the testing set to apply the same transformation to the testing set. The preprocessed testing set can then be connected to the `Dataset Bias` widget to evaluate the bias of the dataset.
+
+![](images/reweighing-dataset-example.png)
+
+The second example demonstrates how to use the `Reweighing` widget as a preprocessor for a learner widget. We use it by connecting it and any other preprocessors we want to use into the `Combine Preprocessors` widget which we connect into the `Weighted Logistic Regression` widget. We then connect a dataset with fairness attributes and the learner into the `Test & Score` widget to evaluate the performance of the learner. In the evaluation results we can see the performance of the learner as well as the fairness metrics for its predictions.
+
+![](images/reweighing-preprocessor-example.png)
+
+Note, in this example we used the [Weighted Logistic Regression](weighted-logistic-regression.md) and the [Combine preprocessors](combine-preprocessors.md) widgets which are described in their respective sections.

doc/widgets/weighted-logistic-regression.md

@@ -5,13 +5,19 @@ Logistic regression model that supports instance weights.
 **Inputs**
 
 - Data: reference dataset
+- Preprocessor: preprocessing method(s)
 
 **Outputs**
 
-- Data: reference dataset with selected Fairness attributes.
+- Learner: logistic regression learning algorithm
+- Model: trained model
+- Coefficients: logistic regression coefficients
 
-**Weighted Logistic Regression** TODO
+**Weighted Logistic Regression** this widget is a logistic regression model that supports instance weights. It is useful when we want to predict on data preprocessed with the [Reweighing](reweighing.md) widget or use the [Reweighing](reweighing.md) widget as a preprocessor for the learnes. At the time of writing it is the only learner that supports instance weights.
+
+Besides the support for weights it works in the exact same way as the normal [Logistic Regression](https://orange3.readthedocs.io/projects/orange-visual-programming/en/latest/widgets/model/logisticregression.html) widget so for more details check its documentation.
 
 Example
 -------
-TODO
+
+The example of how to use the `Weighted Logistic Regression` can be found on the [Reweighing](reweighing.md) widget documentation. An example of how to use the `Logistic Regression` widget in general can be found on the [Logistic Regression](https://orange3.readthedocs.io/projects/orange-visual-programming/en/latest/widgets/model/logisticregression.html) widget documentation.

orangecontrib/fairness/evaluation/scoring.py

@@ -1,4 +1,3 @@
-from numpy import unique
 from abc import abstractmethod
 from Orange.data import DiscreteVariable, ContinuousVariable, Domain
 from Orange.evaluation.scoring import Score