notebooks/heart-surgery-demo.ipynb

{
 "cells": [
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "scrolled": false,
    "slideshow": {
     "slide_type": "skip"
    }
   },
   "outputs": [],
   "source": [
    "%load_ext autoreload\n",
    "%autoreload 2"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "scrolled": true,
    "slideshow": {
     "slide_type": "skip"
    }
   },
   "outputs": [],
   "source": [
    "import pandas as pd\n",
    "import numpy as np\n",
    "\n",
    "from fiber import Cohort\n",
    "from fiber.condition import (\n",
    "    Procedure, \n",
    "    Diagnosis,\n",
    "    VitalSign, \n",
    "    LabValue, \n",
    "    Encounter\n",
    ")\n",
    "from fiber.utils import Timer\n",
    "from fiber.storage.yaml import get_condition"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "# Cohort Definition\n",
    "\n",
    "We are considering patients that had a bypass surgery (ICD-9: `35.*`) or valve surgery (ICD-9: `36.1*`) that are at least 18 y/o.\n",
    "\n",
    "A limitation in this definition is that we cannot (as in other studies), reduce the patients to ICU patients. Our best approximation is to ensure the encounter type `Inpatient`, which we will check manually."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "scrolled": false
   },
   "outputs": [],
   "source": [
    "heart_surgery_condition = (\n",
    "    Procedure(code='35.%', context='ICD-9').age(min_age=18) | \n",
    "    Procedure(code='36.1%', context='ICD-9').age(min_age=18)\n",
    ")"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "scrolled": true
   },
   "outputs": [],
   "source": [
    "heart_surgery_cohort = Cohort(heart_surgery_condition)\n",
    "print(f'{len(heart_surgery_cohort.mrns())} patients in the cohort')"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "# Demographics\n",
    "FIBER exposes some demographics information on the cohort, such as gender and age distribution."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "demographics = heart_surgery_cohort.demographics"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "outputs": [],
   "source": [
    "demographics[\"age\"][\"figure\"]"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "slideshow": {
     "slide_type": "-"
    }
   },
   "outputs": [],
   "source": [
    "demographics[\"gender\"][\"figure\"]"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "# Onsets\n",
    "We are interested in different outcomes for the patients in the cohort for different time intervals, e.g. the (re-)hospitalization in the second week after the heart surgery.\n",
    "- Mortality 0, 7, 14 and 28 days \n",
    "    - This cannot be done on the interval level, because of missing data. The only approximation we can offer is the \n",
    "    `deceased_indicator` in the patient information.\n",
    "- Acute Kidney Injury,  ICD-9 code `584.*` or AKI phenotype\n",
    "- Stroke (Cerebrovascular event), 0, 7, 14 and 28 days\n",
    "\t- Occlusion and stenosis of precerebral arteries, ICD-9 code `433.*`\n",
    "\t- Occlusion of cerebral arteries, ICD-9 code `434.*`\n",
    "\t- Acute but ill-defined cerebrovascular disease, ICD-9 code `436.*`\n",
    "- (Re-)Hospitalization in the {first, second, third, fourth} week after the heart surgery\n",
    "- Emergency Encounter in the {first, second, third, fourth} week after the heart surgery\n",
    "\n",
    "### TODO: \n",
    "- We could also use the more complex AKI phenotype as defined by AKIN and KDIGO."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "scrolled": false,
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "outputs": [],
   "source": [
    "aki = heart_surgery_cohort.has_onset(\n",
    "    name=\"aki\",\n",
    "    condition=Diagnosis(code=\"584.%\", context=\"ICD-9\"),    \n",
    ")"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "scrolled": true
   },
   "outputs": [],
   "source": [
    "stroke = heart_surgery_cohort.has_onset(\n",
    "    name=\"stroke\",\n",
    "    condition=Diagnosis(code='433.%', context=\"ICD-9\") | Diagnosis(code='434.%', context=\"ICD-9\") | Diagnosis(code='436.%', context=\"ICD-9\"), \n",
    ")"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "hospitalization_cond = Encounter(category='Inpatient')\n",
    "hospitalization = heart_surgery_cohort.has_onset(\n",
    "    name=\"hospitalization\",\n",
    "    condition=hospitalization_cond,\n",
    "    time_windows=[(1,7),(8,14),(15,21),(22,28)]\n",
    ")"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "scrolled": true
   },
   "outputs": [],
   "source": [
    "emergency_cond = Encounter(category='Emergency')\n",
    "emergency = heart_surgery_cohort.has_onset(\n",
    "    name=\"emergency\",\n",
    "    condition=emergency_cond,\n",
    "    time_windows=[(1,7),(8,14),(15,21),(22,28)]\n",
    ")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "# Preconditions\n",
    "\n",
    "We find out which patients are affected from a list of common comorbidities or influencing factors of heart surgery outcomes.\n",
    "\n",
    "We are implicitly considering the entire medical history of a patient as known to Mt Sinai Health Systems. "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "slideshow": {
     "slide_type": "skip"
    }
   },
   "outputs": [],
   "source": [
    "preconditions = {}"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "slideshow": {
     "slide_type": "-"
    }
   },
   "outputs": [],
   "source": [
    "diagnoses = [\n",
    "    \"congestive heart failure\",\n",
    "    \"fluid and electrolyte disorders\",\n",
    "    \"liver disease\",\n",
    "    \"rheumatoid arthritis/collagen vascular diseases\",\n",
    "   # \"AIDS/HIV\",\n",
    "    \"alcohol abuse\",\n",
    "    \"blood loss anemia\",\n",
    "    \"cardiac arrhythmia\",\n",
    "    \"chronic pulmonary disease\",\n",
    "    \"coagulopathy\",\n",
    "    \"deficiency anemia\",\n",
    "    \"depression\",\n",
    "    \"diabetes complicated\",\n",
    "    \"diabetes uncomplicated\",\n",
    "    \"drug abuse\",\n",
    "    \"hypertension complicated\",\n",
    "    \"hypertension uncomplicated\",\n",
    "    \"lymphoma\",\n",
    "    \"metastatic cancer\",\n",
    "    \"obesity\",\n",
    "    \"other neurological disorders\",\n",
    "    \"paralysis\",\n",
    "    \"peptic ulcer disease excluding bleeding\",\n",
    "    \"peripheral vascular disorders\",\n",
    "    \"psychoses\",\n",
    "    \"pulmonary circulation disorders\",\n",
    "    \"renal failure\",\n",
    "    \"solid tumor without metastasis\",\n",
    "    \"valvular disease\",\n",
    "    \"weight loss\"\n",
    "]"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "scrolled": true,
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "outputs": [],
   "source": [
    "for cond in diagnoses:\n",
    "    with Timer() as t:\n",
    "        condition = get_condition(Diagnosis, cond, ['ICD-10', 'ICD-9'])\n",
    "        preconditions[condition._label] = heart_surgery_cohort.has_precondition(\n",
    "            name=condition._label,\n",
    "            condition=condition\n",
    "        )\n",
    "        break\n",
    "    print(f'{condition._label} done in {t.elapsed} s')"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "# Lab Values\n",
    "\n",
    "We are extracting different lab test results (values, abnormal and result flags) for blood samples. \n",
    "\n",
    "We aggregate them by day for the {third, second and the} day before the procedure."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "scrolled": true,
    "slideshow": {
     "slide_type": "skip"
    }
   },
   "outputs": [],
   "source": [
    "lab_values = {}"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "lv_cond = {\n",
    "    \"Blood Urea Nitrogen\": LabValue(\"UREA NITROGEN-BLD\"),\n",
    "    \"Blood Creatinine\": LabValue(\"CREATININE-SERUM\"),\n",
    "    \"Anion Gap\": LabValue(\"ANION GAP\"),\n",
    "    \"Bilirubin\": LabValue(\"BILIRUBIN TOTAL\"),\n",
    "    \"Albumin\": LabValue(\"ALBUMIN, BLD\"),\n",
    "    \"Chloride\": LabValue(\"CHLORIDE-BLD\"),\n",
    "    \"Glucose\": LabValue(\"GLUCOSE\"),\n",
    "    \"Hematocrit\": LabValue(\"HEMATOCRIT\"),\n",
    "    \"Hemoglobin\": LabValue(\"HEMOGLOBIN\"),\n",
    "    \"Platelet Count\": LabValue(\"PLATELET\"),\n",
    "    \"Potassium\": LabValue(\"POTASSIUMBLD\"),\n",
    "    \"Sodium\": LabValue(\"SODIUM-BLD\"),\n",
    "    \"White Blood Cell Count\": LabValue(\"WHITE BLOOD CELL\") | LabValue(\"WBC\"),\n",
    "    \"INR\": LabValue(\"INR\"),\n",
    "    \"PTT\": LabValue(\"APTT\"),\n",
    "    \"PT\": LabValue(\"PRO TIME\"),\n",
    "    \"Lactate\": LabValue(\"WB LACTATE-ART (POCT)\"),\n",
    "}"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "outputs": [],
   "source": [
    "for name, cond in lv_cond.items():\n",
    "    with Timer() as t:\n",
    "        lab_values[name] = heart_surgery_cohort.values_for(cond, before=heart_surgery_condition)\n",
    "    print(f'Fetching {name} done in {t.elapsed} s')"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "slideshow": {
     "slide_type": "-"
    }
   },
   "outputs": [],
   "source": [
    "pivoted_values = {}\n",
    "for name, lab_df in lab_values.items():\n",
    "    with Timer() as t:\n",
    "        pivoted_values[name] = heart_surgery_cohort.aggregate_values_in(\n",
    "            time_windows=((-3,-3), (-2,-2), (-1,0)),\n",
    "            df=lab_df, \n",
    "            aggregation_functions=lv_cond[name].default_aggregations,\n",
    "            name=name\n",
    "        )\n",
    "    print(f'{name} done in {t.elapsed} s')"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "# VitalSign\n",
    "\n",
    "To showcase FIBER's data extraction possibilities, we are also extracting oral temperature measurements for the patients in our cohort and aggregate them like the lab values. "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "scrolled": true
   },
   "outputs": [],
   "source": [
    "temp_cond = VitalSign(\"temperature, oral\")\n",
    "df_temp = heart_surgery_cohort.values_for(temp_cond, before=heart_surgery_condition)\n",
    "df_temp = df_temp[df_temp.unit_of_measure == 'oC']"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "scrolled": true
   },
   "outputs": [],
   "source": [
    "pivoted_temp = heart_surgery_cohort.aggregate_values_in(\n",
    "    time_windows=((-3,-3), (-2,-2), (-1,0)),\n",
    "    df=df_temp, \n",
    "    aggregation_functions=temp_cond.default_aggregations,\n",
    "    name=name\n",
    ")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "# Merging all information\n",
    "\n",
    "We now merge all previous intermediate results into one big dataframe, that is machine learning ready. "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "scrolled": false
   },
   "outputs": [],
   "source": [
    "with Timer() as t:\n",
    "    df = heart_surgery_cohort.merge_patient_data(\n",
    "        hospitalization,\n",
    "        emergency,\n",
    "        aki,\n",
    "        stroke,\n",
    "        *preconditions.values(),\n",
    "        *pivoted_values.values(),\n",
    "        pivoted_temp\n",
    "    )\n",
    "print(f'Merging all values done in {t.elapsed}')"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "slideshow": {
     "slide_type": "skip"
    }
   },
   "outputs": [],
   "source": [
    "df.head()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "slideshow": {
     "slide_type": "skip"
    }
   },
   "outputs": [],
   "source": [
    "print(df.shape)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "slideshow": {
     "slide_type": "skip"
    }
   },
   "outputs": [],
   "source": [
    "# How filled are the columns? \n",
    "for c in df.columns:\n",
    "    print(f'{c}: {1 - df[c].isnull().sum() / len(df[c]):.2f}')"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "scrolled": false,
    "slideshow": {
     "slide_type": "skip"
    }
   },
   "outputs": [],
   "source": [
    "df.info()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "scrolled": true
   },
   "outputs": [],
   "source": [
    "df.to_csv(\"/tmp/heart-surgery.csv\")"
   ]
  }
 ],
 "metadata": {
  "celltoolbar": "Slideshow",
  "kernelspec": {
   "display_name": "Python 3",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.6.5"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 2
}