Add pre-defined cohorts in responsibleaidashboard-diabetes-decision-m…

…aking.ipynb (#1252)

* [WIP] Add pre-defined cohorts in responsibleaidashboard-diabetes-decision-making.ipynb

Signed-off-by: Gaurav Gupta <[email protected]>

* Fix the e2e test for notebook

Signed-off-by: Gaurav Gupta <[email protected]>

* Make _cohort.py module a public module (#1253)

* Make _cohort.py a public module

Signed-off-by: Gaurav Gupta <[email protected]>

* Add missing file

Signed-off-by: Gaurav Gupta <[email protected]>

* Fix cohort namespace

Signed-off-by: Gaurav Gupta <[email protected]>

* minor fix to url for responsibleai package in setup.py (#1260)

* Counterfactual Chart: Correct target description according to task_type (#1261)

* Counterfactual Chart: Correct target description according to task_type

Signed-off-by: Gaurav Gupta <[email protected]>

* Change function name

Signed-off-by: Gaurav Gupta <[email protected]>

* Make Range lowercase

Signed-off-by: Gaurav Gupta <[email protected]>

* fix whitespace in values of adult census income dataset (#1263)

* Add what-If scatter chart from highchart lib (#1262)

* add whatIf scatter chart

* widget test

* what if local importance bar chart

* fix

* widget

* fix tooltip

* refactor

* test

* test

* add a builddebug yarn command to build UX locally which can be debugged in browser e2e (#1265)

* allow rai text insights to work with RAI dashboard (#1269)

* remove duplicate code in explanation dashboard (#1266)

* Individual causal style responsive (#1268)

* add whatIf scatter chart

* widget test

* what if local importance bar chart

* fix

* widget

* fix tooltip

* refactor

* test

* test

* Causal Style

* Allow duplicating cohorts multiple times (#1274)

* allow duplicating a cohort more than once

* lintfix

* Disable column header highlighting on hover in IndividualFeatureImportanceView (#1272)

* disable column highlight on hover

* lintfix

* Rename new cohorts from "Unsaved" to "Temporary cohort" (#1273)

* rename Unsaved to Temporary cohort

* localize temp cohort

* Counterfactual style refactor (#1275)

* style refactor

* test

* test

* test

* fix

* Don't change cursor on hover over cohort name

* Fix (#1281)

* fix cohort info styling (#1277)

* fix readme link to fairness and interpretability example notebook (#1282)

* add new RAI Utils package for common utilities shared across RAI packages (#1280)

* Add ICE chart (#1283)

* Fix

* ice chart

* ic

* test

* test

* update docstring for explanation dashboard in regards to min number of rows (#1271)

* make builds more reliable by adding retry logic to urlretrieve calls in notebooks (#1218)

* upgrade pytest to 7.0.1, remove mock and updgrade pytest-mock to 3.6.1 (#1287)

* remove deprecated codecov parameter (#1293)

* Fix min/max special case in cohort filter creation with "in the range of" (#1279)

* fix logic in the case that min or max are zero

* lintfix

* Rename 'Dashboard navigation' to 'Dashboard configuration' (#1291)

* Rename 'Dashboard navigation' to 'Dashboard configuration'

Signed-off-by: Gaurav Gupta <[email protected]>

* Notebook change

Signed-off-by: Gaurav Gupta <[email protected]>

* Add raiutils to PR template (#1290)

* fix heatmap bug (#1297)

* Make "save and switch" work from cohort settings (#1276)

* make save and switch work

* fix naming

* lintfix

* adjustment according to Ilya's comment

* lintfix

* add retry logic to codecov step and only upload results for one python version (#1298)

* add github action to release raiutils to pypi (#1294)

* Add highchart for Dataset Explorer  (#1286)

* test

* style

* click

* fix test

* fix test

* test

* test

* test

* test

* Update requirements-linting.txt to add flake8-pytest-style (#1296)

* Fix sort abs (#1299)

* Rename "base cohort" to "global cohort" (#1278)

* change base cohort to global cohort

* fix spelling

* lintfix

* fix codecov comment not appearing on PRs (#1302)

* take absolute value of error calculation for regression scenario (#1301)

* Limit individual feature importance selection to up to 5 (#1305)

* update feature importance string

* limit selection to up to 5

* add group count

* remove message bar, show info icon instead

* update e2e locator

* fix E2E failure on feature importance

* add ariaLabel for expand collapse button

* add renderOnNewLayer props

* Add error message for counterfactual panel (#1310)

* add error message for counterfactual

* update error message in camel case to fix build error

* Add to_json() and from_json() methods to Cohort class (#1300)

* Add to_json() and from_json() methods to Cohort class

Signed-off-by: Gaurav Gupta <[email protected]>

* Address code review comments

Signed-off-by: Gaurav Gupta <[email protected]>

* Fix linting

Signed-off-by: Gaurav Gupta <[email protected]>

* Add a highchart heatmap helper class (#1307)

* add highchart heatmap helper class

* add erroneously deleted line back

* Fix cohort setting string (#1304)

* Fix string

* remove none

* name

* test

* Fix all data style (#1303)

* Add a feature flag for the new model overview experience (#1306)

* add feature flag for new model overview experience and turn it off by default

* remove useless constructor

* Clean up charts code (#1313)

* clean up chart code

* remove arg

* Expand the counterfactual flyout to cover the full page (#1315)

Signed-off-by: Gaurav Gupta <[email protected]>

* Bump minimist from 1.2.5 to 1.2.6 (#1292)

* Bump minimist from 1.2.5 to 1.2.6

Bumps [minimist](https://github.com/substack/minimist) from 1.2.5 to 1.2.6.
- [Release notes](https://github.com/substack/minimist/releases)
- [Commits](https://github.com/substack/minimist/compare/1.2.5...1.2.6)

---
updated-dependencies:
- dependency-name: minimist
  dependency-type: indirect
...

Signed-off-by: dependabot[bot] <[email protected]>

* minimist ^1.2.6

Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
Co-authored-by: xuke444 <[email protected]>
Co-authored-by: Roman Lutz <[email protected]>

* fix random node download failures by upgrading to latest github action with retry logic (#1317)

* Add dataset cohort table to new ModelOverview experience (#1314)

* add only dataset cohort table, update wrapping code

* lintfix

* lintfix

* build error fix

* Add installation instructions for raiwidgets to README (#1320)

* refactor RAIInsights into RAIInsightsBase class for basic functionality (#1284)

* Fix what if counterfactual header and description text misaligned (#1316)

* align

* e2e

* add clear temporary cohort button to error analysis (#1322)

* Raise UserConfigValidationException in case no model but valid model serializer (#1325)

Signed-off-by: Gaurav Gupta <[email protected]>

* Add test case for handling different types in causal (#1321)

Signed-off-by: Gaurav Gupta <[email protected]>

* show shift to an empty cohort in tree view as an empty node (#1318)

* Bug fixing (#1326)

* Move chart description up and remove scroll bar

* Change string

* Add box outlier for dataset explorer (#1323)

* add outlier for dataset explorer

* name

* update string when no datapoint selected (#1331)

* Fix Big empty space for featureImportance chart (#1328)

* legend

* removed invalid test case

* constant

* Disable save as new cohort button if nothing is selected in error tree (#1327)

* Add disaggregated analysis table/heatmap (#1332)

* disaggregated analysis changes only

* lintfix

* Change warning message to user exception for model type and task type mismatch (#1330)

* Change warning message to user exception for model type and task type mismatch

Signed-off-by: Gaurav Gupta <[email protected]>

* Fix flake8 errors

Signed-off-by: Gaurav Gupta <[email protected]>

* Change the counterfactual text color from black to grey (#1337)

Signed-off-by: Gaurav Gupta <[email protected]>

* Limit each component description width up to 750px for readability (#1336)

* limit description width up to 750px

* export maxWidth from a common place

* block empty cohort creation in RAI Dashboard (#1335)

* Add warning message in cohort editor for invalid input value; Update 'Shift cohort' to 'Switch cohort' (#1339)

* add error message for invalid value

* update shift cohort to switch cohort

* Rename counterfactual style files to confirm with *.styles.ts (#1338)

Signed-off-by: Gaurav Gupta <[email protected]>

* Add disaggregated analysis table to Model Overview (#1341)

* pull in changes for disaggregated analysis

* add styles file

* add textured NaN cells

* module import for textured cells and grid y axis

* lintfix

* use combobox for dropdown rather than dropdown

* lintfix

* Rename causal style files to confirm with *.styles.ts (#1342)

Signed-off-by: Gaurav Gupta <[email protected]>

* update responsibleai to interpret-community 0.25.0 (#1343)

* All component title and descriptions should be aligned (#1346)

* update

Signed-off-by: vinutha karanth <[email protected]>

* lintfix

Signed-off-by: vinutha karanth <[email protected]>

* Remove 5K limit blurb from local explanations tab (#1347)

Signed-off-by: Gaurav Gupta <[email protected]>

* Sort features by default in counterfactual flyout (#1312)

* Sort features by default in counterfactual flyout

Signed-off-by: Gaurav Gupta <[email protected]>

* Fix failing tests

Signed-off-by: Gaurav Gupta <[email protected]>

* attempt to fix test

Signed-off-by: Gaurav Gupta <[email protected]>

* Remove check

Signed-off-by: Gaurav Gupta <[email protected]>

* Bump moment from 2.28.0 to 2.29.2 (#1333)

Bumps [moment](https://github.com/moment/moment) from 2.28.0 to 2.29.2.
- [Release notes](https://github.com/moment/moment/releases)
- [Changelog](https://github.com/moment/moment/blob/develop/CHANGELOG.md)
- [Commits](moment/moment@2.28.0...2.29.2)

---
updated-dependencies:
- dependency-name: moment
  dependency-type: direct:production
...

Signed-off-by: dependabot[bot] <[email protected]>

Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
Co-authored-by: Roman Lutz <[email protected]>

* Counterfactual flyout top section need to be moved to left & Error analysis move side content to align with description text (#1350)

* update

Signed-off-by: vinutha karanth <[email protected]>

* lintfix

Signed-off-by: vinutha karanth <[email protected]>

* move the root all data statistics to ErrorReport and ErrorAnalysisData (#1344)

* update error analysis documentation to clarify the error tree splits on errors even when other metrics are selected (#1349)

Co-authored-by: Vinutha Karanth <[email protected]>

* update erroranalysis to 0.2.1 and remove some duplicate dependencies (#1334)

* disable turbo checking for large amount of data (#1351)

Signed-off-by: Ke Xu <[email protected]>

* force re-render when chart type changes (#1354)

Signed-off-by: Ke Xu <[email protected]>

* move the root all data statistics to ErrorReport and ErrorAnalysisData (#1352)

* Rename output column name in the counterfactual flyout (#1353)

Signed-off-by: Gaurav Gupta <[email protected]>

* Show column chart for categorical feature in data explorer (#1355)

* Show column chart for categorical feature in data explorer

* address comments

* update fluentui (#1356)

Signed-off-by: Ke Xu <[email protected]>

* update code owner (#1308)

* update code owner

Signed-off-by: Ke Xu <[email protected]>

* remove dup

Signed-off-by: Ke Xu <[email protected]>

Co-authored-by: Roman Lutz <[email protected]>

* update version to match studio (#1357)

Signed-off-by: Ke Xu <[email protected]>

* alignment fixes (#1359)

* Add charts for new model overview experience (#1348)

* rename OverallTable to DisaggregatedMetricsTable and move to core-ui

* Copy the ModelPerformanceTab into model-assessment and rename to ModelOverview

* reference OverallTable again in fairness widget

* refactor core chart component out into core-ui

* refactor out core chart component into core-ui

* lintfix

* undo tsconfig.lib.json change

* fix chartAndConfigsId in tests

* lintfix

* add table for cohort metrics and add dropdown metric selector, add new metrics

* add new metrics

* undo unwanted changes

* fix casing

* add superscript 2 for r-squared

* update tests to reflect new metrics

* lintfix

* add feature flag

* fix mae

* fix mae calculation

* first version of new model overview table

* get probability distribution box plot to work

* add feature flag for new model overview experience and turn it off by default

* add highchart heatmap helper class

* remove custom styling

* add erroneously deleted line back

* remove useless constructor

* modularize model overview

* show outliers in box plot, fix positioning

* remove showmetricsummary

* refactor heatmap code into a common class

* add featureDropdownRef to allow focusing

* add only dataset cohort table, update wrapping code

* lintfix

* lintfix

* build error fix

* add chart config flyout (in progress)

* add chart config flyout (in progress)

* address feedback, use finalized color

* adjust feature selection to disable options if limit is reached, add axis config buttons

* select all via dropdown

* lintfix

* refactor box plot calculations and rendering

* add style file

* textured NaN cells, grid labels on y axis

* standardize box plots to use fences

* fix merge issues

* unify box plot tooltip formatting code, fix bar chart sizing issue

* small fixes

* rearrange feature dropdown

* lintfix

* remove commented out code

* remove box plot tooltip customization

* lintfix

* add a few unit tests

* unit tests for smaller utilities, localization fixes, consistent flyout flow with confirm/cancel buttons

* lintfix

* fix chart config flyout update

* fix test case

* rename files for lint

* file rename for lint

* release rai-core-flask 0.3.0 (#1361)

* upgrade python version used with flask CI to fix segfault error (#1363)

* release raiwidgets and responsibleai v0.18.0 (#1360)

* fix two bugs (#1364)

* Add pre-built cohort into adult census notebook (#1243)

* [WIP] Add pre-built cohort into adult census notebook

Signed-off-by: Gaurav Gupta <[email protected]>

* erroranalysis version bump in raiwidgets to 0.1.31 (#1245)

* Make cohrtData empty list in case no pre-bdefined cohorts are injected (#1247)

Signed-off-by: Gaurav Gupta <[email protected]>

* Simplify the train pipeline responsibleaidashboard-census-classification-model-debugging.ipynb (#1195)

* Simplify the train pipeline responsibleaidashboard-census-classification-model-debugging.ipynb

Signed-off-by: Gaurav Gupta <[email protected]>

* Address code review comments

* Update notebooks/responsibleaidashboard/responsibleaidashboard-census-classification-model-debugging.ipynb

Co-authored-by: Roman Lutz <[email protected]>

Co-authored-by: Roman Lutz <[email protected]>
Signed-off-by: Gaurav Gupta <[email protected]>

* Add regression test for pre-defined cohorts in raiwidgets (#1249)

Signed-off-by: Gaurav Gupta <[email protected]>

* color (#1248)

* Add feature importance box & bar chart (#1241)

* refactor

* build

* build

* temp

* temp

* temp

* temp

* box

* cache

* e2e

* e2e

* fix

* e2e fix

* e2e

* fix e2e

* widget

* widget

* fix

* widget

* e2e

* e2e

* e2e

* test

* test

* PreBuilt cohorts UX changes (#1242)

* Intial SDK implementation cohorts

Signed-off-by: Gaurav Gupta <[email protected]>

* Add basic validationf for cohorts

Signed-off-by: Gaurav Gupta <[email protected]>

* Add serialized version of cohort config to ResponsibleAiDashboard

Signed-off-by: Gaurav Gupta <[email protected]>

* Add more tests cohorts

Signed-off-by: Gaurav Gupta <[email protected]>

* fix broken builds due to pip upgrade which broke pip-tools (#1185)

* refactor matrix filter and area state to be private static (#1179)

* Change variable name

Signed-off-by: Gaurav Gupta <[email protected]>

* Add more cohort filters

Signed-off-by: Gaurav Gupta <[email protected]>

* Add cohort data to dashboard e2e

Signed-off-by: Gaurav Gupta <[email protected]>

* Add more cohorts filters

Signed-off-by: Gaurav Gupta <[email protected]>

* Document various data validation for cohorts

Signed-off-by: Gaurav Gupta <[email protected]>

* Add new interfaces for pre-built cohort

Signed-off-by: Gaurav Gupta <[email protected]>

* Add more cohort filters

Signed-off-by: Gaurav Gupta <[email protected]>

* Add prebuilt cohort walking logic in UI and add more data validation scenarios

Signed-off-by: Gaurav Gupta <[email protected]>

* Add basic data validation checks

Signed-off-by: Gaurav Gupta <[email protected]>

* Add logic to translate the Index cohort filter

Signed-off-by: Gaurav Gupta <[email protected]>

* Remove commented out code

Signed-off-by: Gaurav Gupta <[email protected]>

* Add SDK validations for Index based cohort filter

Signed-off-by: Gaurav Gupta <[email protected]>

* Add code for validating classification outcome

Signed-off-by: Gaurav Gupta <[email protected]>

* Add error filter validations and add tests

Signed-off-by: Gaurav Gupta <[email protected]>

* Add fake cohorts for regression dataset

Signed-off-by: Gaurav Gupta <[email protected]>

* Add fake cohorts for multi-class classification dataset

Signed-off-by: Gaurav Gupta <[email protected]>

* Add handling of regression filter

Signed-off-by: Gaurav Gupta <[email protected]>

* Add support for classification outcome in UI

Signed-off-by: Gaurav Gupta <[email protected]>

* Add validations for Predicted Y and True Y cohort filters

Signed-off-by: Gaurav Gupta <[email protected]>

* Add UI code to handle prediced Y and true Y for pre-built cohort filters

Signed-off-by: Gaurav Gupta <[email protected]>

* Add cohort validation with test data to raiwidgets

Signed-off-by: Gaurav Gupta <[email protected]>

* Add tests for validating Predicted/True Y cohorts

Signed-off-by: Gaurav Gupta <[email protected]>

* Add UI support for TrueY/PredictedY for classification

Signed-off-by: Gaurav Gupta <[email protected]>

* Rename cohort_filter_list to cohort_list

Signed-off-by: Gaurav Gupta <[email protected]>

* Rename UI varibles to match SDK

Signed-off-by: Gaurav Gupta <[email protected]>

* Fix duplicate cohort name

Signed-off-by: Gaurav Gupta <[email protected]>

* Add SDK cohorts to notebook

Signed-off-by: Gaurav Gupta <[email protected]>

* Add dataset validations and add categorical features

Signed-off-by: Gaurav Gupta <[email protected]>

* Add validations for categorical_features

Signed-off-by: Gaurav Gupta <[email protected]>

* Fix sorted imports

Signed-off-by: Gaurav Gupta <[email protected]>

* Add code for translating categorical values

Signed-off-by: Gaurav Gupta <[email protected]>

* Move cohort processing to a separate file

Signed-off-by: Gaurav Gupta <[email protected]>

* Fix code review comments

Signed-off-by: Gaurav Gupta <[email protected]>

* Refactor cohort translated function into different small functions

Signed-off-by: Gaurav Gupta <[email protected]>

* Change to lowercase for outcome

Signed-off-by: Gaurav Gupta <[email protected]>

* Fix code review comments

Signed-off-by: Gaurav Gupta <[email protected]>

* Refactor cohort_list validations and converge pytest common functions into fixtures

Signed-off-by: Gaurav Gupta <[email protected]>

* Add conftest into raiwidgets tests

Signed-off-by: Gaurav Gupta <[email protected]>

* Add validations for cohort list

Signed-off-by: Gaurav Gupta <[email protected]>

* Add cohortData test

Signed-off-by: Gaurav Gupta <[email protected]>

* Fix sorted imports

Signed-off-by: Gaurav Gupta <[email protected]>

* isort fix

Signed-off-by: Gaurav Gupta <[email protected]>

* Add UI unit tests for cohort translation

Signed-off-by: Gaurav Gupta <[email protected]>

* Add more checks in UI uni test

Signed-off-by: Gaurav Gupta <[email protected]>

* Add UI tests for regression cohorts

Signed-off-by: Gaurav Gupta <[email protected]>

* REmove notebook change

Signed-off-by: Gaurav Gupta <[email protected]>

* Fix typescript build

Signed-off-by: Gaurav Gupta <[email protected]>

* Change cohort filter values so that cohort filters non-zero points

Signed-off-by: Gaurav Gupta <[email protected]>

* Fix for empty cohort list

Signed-off-by: Gaurav Gupta <[email protected]>

* Simplify the train pipeline responsibleaidashboard-census-classification-model-debugging.ipynb (#1195)

* Simplify the train pipeline responsibleaidashboard-census-classification-model-debugging.ipynb

Signed-off-by: Gaurav Gupta <[email protected]>

* Address code review comments

* Update notebooks/responsibleaidashboard/responsibleaidashboard-census-classification-model-debugging.ipynb

Co-authored-by: Roman Lutz <[email protected]>

Co-authored-by: Roman Lutz <[email protected]>

* Propagate error strings instead of raising exceptions

Signed-off-by: Gaurav Gupta <[email protected]>

* Fix code issues

Signed-off-by: Gaurav Gupta <[email protected]>

* Fix code review comments

Signed-off-by: Gaurav Gupta <[email protected]>

* Fix code review comments

Signed-off-by: Gaurav Gupta <[email protected]>

Co-authored-by: Ilya Matiach <[email protected]>
Co-authored-by: Roman Lutz <[email protected]>

* Make _cohort.py module a public module (#1253)

* Make _cohort.py a public module

Signed-off-by: Gaurav Gupta <[email protected]>

* Add missing file

Signed-off-by: Gaurav Gupta <[email protected]>

* fix notebook build failures due to pywinpty dependency release failing in python 3.6 (#1257)

* fix notebook build failures due to pywinpty dependency release failing in python 3.6

* build pywinpty from conda instead

* add lowerbound

* fixup

* fixup

* Add supported models and data types to README.md responsibleai (#1259)

Signed-off-by: Gaurav Gupta <[email protected]>

* make getting-started notebook a markdown file showing APIs (#1223)

* refactor tabs out of RAI dashboard into a separate component (#1256)

* Add individual causal scatter chart (#1258)

* temp

* refactor

* test

* style fix

* comment

* minor fix to url for responsibleai package in setup.py (#1260)

* Fix UX e2e tests and address code review comments

Signed-off-by: Gaurav Gupta <[email protected]>

* Fix eslint

Signed-off-by: Gaurav Gupta <[email protected]>

* Address review comments

Signed-off-by: Gaurav Gupta <[email protected]>

* Reset the number of samples in test dataset

Signed-off-by: Gaurav Gupta <[email protected]>

Co-authored-by: Ilya Matiach <[email protected]>
Co-authored-by: Roman Lutz <[email protected]>
Co-authored-by: Bo Zhang <[email protected]>
Signed-off-by: Gaurav Gupta <[email protected]>

* Change cohort name

Signed-off-by: Gaurav Gupta <[email protected]>

Co-authored-by: Ilya Matiach <[email protected]>
Co-authored-by: Bo Zhang <[email protected]>
Co-authored-by: Roman Lutz <[email protected]>
Co-authored-by: tongy-msft <[email protected]>
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
Co-authored-by: xuke444 <[email protected]>
Co-authored-by: Vinutha Karanth <[email protected]>

apps/widget-e2e/src/describer/modelAssessment/modelAssessmentDatasets.ts

@@ -123,7 +123,14 @@ const modelAssessmentDatasets = {
       "s6"
     ],
     modelStatisticsData: {
-      cohortDropDownValues: ["All data"],
+      cohortDropDownValues: [
+        "All data",
+        "Cohort Index",
+        "Cohort Predicted Y",
+        "Cohort True Y",
+        "Cohort Regression Error",
+        "Cohort Age and BMI"
+      ],
       defaultXAxis: "Error",
       defaultXAxisPanelValue: "Error",
       defaultYAxis: "Cohort",