examples/JUPYTER/Demo_yourowndata.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {
    "colab_type": "text",
    "id": "RK255E7YoEIt"
   },
   "source": [
    "# DeepLabCut Toolbox\n",
    "https://github.com/DeepLabCut/DeepLabCut\n",
    "\n",
    "This notebook demonstrates the necessary steps to use DeepLabCut for your own project.\n",
    "This shows the most simple code to do so, but many of the functions have additional features, so please check out the overview & the protocol paper!\n",
    "\n",
    "This notebook illustrates how to:\n",
    "- create a project\n",
    "- extract training frames\n",
    "- label the frames\n",
    "- plot the labeled images\n",
    "- create a training set\n",
    "- train a network\n",
    "- evaluate a network\n",
    "- analyze a novel video\n",
    "- create an automatically labeled video \n",
    "- plot the trajectories\n",
    "\n",
    "This notebook demonstrates the necessary steps to use DeepLabCut for your own project.\n",
    "\n",
    "This shows the most simple code to do so, but many of the functions have additional features, so please check out the overview & the protocol paper!\n",
    "\n",
    "Nath\\*, Mathis\\* et al.: Using DeepLabCut for markerless pose estimation during behavior across species. Nature Protocols, 2019.\n",
    "\n",
    "Paper: https://www.nature.com/articles/s41596-019-0176-0\n",
    "\n",
    "Pre-print: https://www.biorxiv.org/content/biorxiv/early/2018/11/24/476531.full.pdf"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "colab_type": "text",
    "id": "9Uoz9mdPoEIy"
   },
   "source": [
    "## Create a new project\n",
    "\n",
    "It is always good idea to keep the projects separate if you want to use different networks to analze your data. You should use one project if you are tracking similar subjects/items even if in different environments. This function creates a new project with sub-directories and a basic configuration file in the user defined directory otherwise the project is created in the current working directory.\n",
    "\n",
    "You can always add new videos (for lableing more data) to the project at any stage of the project. "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 1,
   "metadata": {
    "colab": {},
    "colab_type": "code",
    "id": "jqLZhp7EoEI0"
   },
   "outputs": [],
   "source": [
    "import deeplabcut"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "colab": {},
    "colab_type": "code",
    "id": "c9DjG55FoEI7"
   },
   "outputs": [],
   "source": [
    "task='Reaching' # Enter the name of your experiment Task\n",
    "experimenter='Mackenzie' # Enter the name of the experimenter\n",
    "video=['videos/video1.avi','videos/video2.avi'] # Enter the paths of your videos OR FOLDER you want to grab frames from.\n",
    "\n",
    "path_config_file=deeplabcut.create_new_project(task,experimenter,video,copy_videos=True) \n",
    "\n",
    "# NOTE: The function returns the path, where your project is. \n",
    "# You could also enter this manually (e.g. if the project is already created and you want to pick up, where you stopped...)\n",
    "#path_config_file = '/home/Mackenzie/Reaching/config.yaml' # Enter the path of the config file that was just created from the above step (check the folder)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Now, go edit the config.yaml file that was created! \n",
    "Add your body part labels, edit the number of frames to extract per video, etc. "
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### Note that you can see more information about ANY function by adding a ? at the end,  i.e. "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 2,
   "metadata": {},
   "outputs": [],
   "source": [
    "deeplabcut.extract_frames?"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "colab_type": "text",
    "id": "0yXW0bx1oEJA"
   },
   "source": [
    "## Extract frames from videos \n",
    "A key point for a successful feature detector is to select diverse frames, which are typical for the behavior you study that should be labeled.\n",
    "\n",
    "This function selects N frames either uniformly sampled from a particular video (or folder) ('uniform'). Note: this might not yield diverse frames, if the behavior is sparsely distributed (consider using kmeans), and/or select frames manually etc.\n",
    "\n",
    "Also make sure to get select data from different (behavioral) sessions and different animals if those vary substantially (to train an invariant feature detector).\n",
    "\n",
    "Individual images should not be too big (i.e. < 850 x 850 pixel). Although this can be taken care of later as well, it is advisable to crop the frames, to remove unnecessary parts of the frame as much as possible.\n",
    "\n",
    "Always check the output of cropping. If you are happy with the results proceed to labeling."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "colab": {},
    "colab_type": "code",
    "id": "t1ulumCuoEJC"
   },
   "outputs": [],
   "source": [
    "%matplotlib inline\n",
    "#there are other ways to grab frames, such as uniformly; please see the paper:\n",
    "\n",
    "#AUTOMATIC:\n",
    "deeplabcut.extract_frames(path_config_file) "
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "colab_type": "text",
    "id": "Gjn6ZDonoEJH"
   },
   "source": [
    "## Label the extracted frames\n",
    "\n",
    "Only videos in the config file can be used to extract the frames. Extracted labels for each video are stored in the project directory under the subdirectory **'labeled-data'**. Each subdirectory is named after the name of the video. The toolbox has a labeling toolbox which could be used for labeling. "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "colab": {},
    "colab_type": "code",
    "id": "iyROSOiEoEJI"
   },
   "outputs": [],
   "source": [
    "# napari will pop up!\n",
    "# Please go to plugin > deeplabcut to start\n",
    "# then, drag-and-drop the project configuration file into the viewer (the value of path_config_file)\n",
    "# finally, drop the folder containing the images (in 'labeled-data') in the viewer\n",
    "\n",
    "%gui qt6\n",
    "import napari\n",
    "napari.Viewer()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "colab_type": "text",
    "id": "vim95ZvkPSeN"
   },
   "source": [
    "## Check the labels\n",
    "\n",
    "[OPTIONAL] Checking if the labels were created and stored correctly is beneficial for training, since labeling is one of the most critical parts for creating the training dataset. The DeepLabCut toolbox provides a function `check\\_labels'  to do so. It is used as follows:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "colab": {},
    "colab_type": "code",
    "id": "NwvgPJouPP2O"
   },
   "outputs": [],
   "source": [
    "deeplabcut.check_labels(path_config_file) #this creates a subdirectory with the frames + your labels"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "colab_type": "text",
    "id": "of87fOjgPqzH"
   },
   "source": [
    "If the labels need adjusted, you can use relauch the labeling GUI to move them around, save, and re-plot!"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "colab_type": "text",
    "id": "xNi9s1dboEJN"
   },
   "source": [
    "## Create a training dataset\n",
    "\n",
    "This function generates the training data information for network training based on the pandas dataframes that hold label information. The user can set the fraction of the training set size (from all labeled image in the hd5 file) in the config.yaml file. While creating the dataset, the user can create multiple shuffles if they want to benchmark the performance (typcailly, 1 is what you will set, so you pass nothing!). \n",
    "\n",
    "After running this script the training dataset is created and saved in the project directory under the subdirectory **'training-datasets'**\n",
    "\n",
    "This function also creates new subdirectories under **dlc-models** and appends the project config.yaml file with the correct path to the training and testing pose configuration file. These files hold the parameters for training the network. Such an example file is provided with the toolbox and named as **pose_cfg.yaml**. For most all use cases we have seen, the defaults are perfectly fine.\n",
    "\n",
    "Now it is the time to start training the network!"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "colab": {},
    "colab_type": "code",
    "id": "eMeUwgxPoEJP",
    "scrolled": true
   },
   "outputs": [],
   "source": [
    "deeplabcut.create_training_dataset(path_config_file)\n",
    "#remember, there are several networks you can pick, the default is resnet-50!"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "colab_type": "text",
    "id": "c4FczXGDoEJU"
   },
   "source": [
    "## Start training:\n",
    "\n",
    "This function trains the network for a specific shuffle of the training dataset. "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "colab": {},
    "colab_type": "code",
    "id": "_pOvDq_2oEJW"
   },
   "outputs": [],
   "source": [
    "deeplabcut.train_network(path_config_file)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "colab_type": "text",
    "id": "xZygsb2DoEJc"
   },
   "source": [
    "## Start evaluating\n",
    "This function evaluates a trained model for a specific shuffle/shuffles at a particular state or all the states on the data set (images)\n",
    "and stores the results as .csv file in a subdirectory under **evaluation-results**"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "colab": {},
    "colab_type": "code",
    "id": "nv4zlbrnoEJg"
   },
   "outputs": [],
   "source": [
    "deeplabcut.evaluate_network(path_config_file, plotting=True)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "colab_type": "text",
    "id": "OVFLSKKfoEJk"
   },
   "source": [
    "## Start Analyzing videos\n",
    "This function analyzes the new video. The user can choose the best model from the evaluation results and specify the correct snapshot index for the variable **snapshotindex** in the **config.yaml** file. Otherwise, by default the most recent snapshot is used to analyse the video.\n",
    "\n",
    "The results are stored in hd5 file in the same directory where the video resides. "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "colab": {},
    "colab_type": "code",
    "id": "Y_LZiS_0oEJl"
   },
   "outputs": [],
   "source": [
    "videofile_path = ['videos/video3.avi','videos/video4.avi'] #Enter a folder OR a list of videos to analyze.\n",
    "\n",
    "deeplabcut.analyze_videos(path_config_file,videofile_path, videotype='.avi')"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "colab_type": "text",
    "id": "iGu_PdTWoEJr"
   },
   "source": [
    "## Extract outlier frames [optional step]\n",
    "\n",
    "This is an optional step and is used only when the evaluation results are poor i.e. the labels are incorrectly predicted. In such a case, the user can use the following function to extract frames where the labels are incorrectly predicted. This step has many options, so please look at:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 3,
   "metadata": {},
   "outputs": [],
   "source": [
    "deeplabcut.extract_outlier_frames?"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "colab": {},
    "colab_type": "code",
    "id": "gkbaBOJVoEJs"
   },
   "outputs": [],
   "source": [
    "deeplabcut.extract_outlier_frames(path_config_file,['/videos/video3.avi']) #pass a specific video"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "colab_type": "text",
    "id": "8ib0uvhaoEJx"
   },
   "source": [
    "## Refine Labels [optional step]\n",
    "Following the extraction of outlier frames, the user can use the following function to move the predicted labels to the correct location. Thus augmenting the training dataset. "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "colab": {},
    "colab_type": "code",
    "id": "n_FpEXtyoEJy"
   },
   "outputs": [],
   "source": [
    "%gui qt6\n",
    "deeplabcut.refine_labels(path_config_file)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "**NOTE:** Afterwards, if you want to look at the adjusted frames, you can load them in the main GUI by running: ``deeplabcut.label_frames(path_config_file)``\n",
    "\n",
    "(you can add a new \"cell\" below to add this code!)\n",
    "\n",
    "#### Once all folders are relabeled, check the labels again! If you are not happy, adjust them in the main GUI:\n",
    "\n",
    "``deeplabcut.label_frames(path_config_file)``\n",
    "\n",
    "Check Labels:\n",
    "\n",
    "``deeplabcut.check_labels(path_config_file)``"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "colab": {},
    "colab_type": "code",
    "id": "CHzstWr8oEJ2"
   },
   "outputs": [],
   "source": [
    "#NOW, merge this with your original data:\n",
    "\n",
    "deeplabcut.merge_datasets(path_config_file)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "colab_type": "text",
    "id": "QCHj7qyboEJ6"
   },
   "source": [
    "## Create a new iteration of training dataset [optional step]\n",
    "Following the refinement of labels and appending them to the original dataset, this creates a new iteration of training dataset. This is automatically set in the config.yaml file, so let's get training!"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "colab": {},
    "colab_type": "code",
    "id": "ytQoxIldoEJ7"
   },
   "outputs": [],
   "source": [
    "deeplabcut.create_training_dataset(path_config_file)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "colab_type": "text",
    "id": "pCrUvQIvoEKD"
   },
   "source": [
    "## Create labeled video\n",
    "This function is for visualiztion purpose and can be used to create a video in .mp4 format with labels predicted by the network. This video is saved in the same directory where the original video resides. \n",
    "\n",
    "THIS HAS MANY FUN OPTIONS! \n",
    "\n",
    "``deeplabcut.create_labeled_video(config, videos, videotype='avi', shuffle=1, trainingsetindex=0, filtered=False, save_frames=False, Frames2plot=None, delete=False, displayedbodyparts='all', codec='mp4v', outputframerate=None, destfolder=None, draw_skeleton=False, trailpoints=0, displaycropped=False)``\n",
    "\n",
    "So please check:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 4,
   "metadata": {},
   "outputs": [],
   "source": [
    "deeplabcut.create_labeled_video?"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "colab": {},
    "colab_type": "code",
    "id": "6aDF7Q7KoEKE"
   },
   "outputs": [],
   "source": [
    "deeplabcut.create_labeled_video(path_config_file,videofile_path)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "colab_type": "text",
    "id": "8GTiuJESoEKH"
   },
   "source": [
    "## Plot the trajectories of the analyzed videos\n",
    "This function plots the trajectories of all the body parts across the entire video. Each body part is identified by a unique color."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "colab": {},
    "colab_type": "code",
    "id": "gX21zZbXoEKJ"
   },
   "outputs": [],
   "source": [
    "%matplotlib notebook #for making interactive plots.\n",
    "deeplabcut.plot_trajectories(path_config_file,videofile_path)"
   ]
  }
 ],
 "metadata": {
  "accelerator": "GPU",
  "colab": {
   "name": "Demo-yourowndata.ipynb",
   "provenance": [],
   "version": "0.3.2"
  },
  "kernelspec": {
   "display_name": "Python [conda env:DLC2]",
   "language": "python",
   "name": "conda-env-DLC2-py"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.6.9"
  },
  "varInspector": {
   "cols": {
    "lenName": 16,
    "lenType": 16,
    "lenVar": 40
   },
   "kernels_config": {
    "python": {
     "delete_cmd_postfix": "",
     "delete_cmd_prefix": "del ",
     "library": "var_list.py",
     "varRefreshCmd": "print(var_dic_list())"
    },
    "r": {
     "delete_cmd_postfix": ") ",
     "delete_cmd_prefix": "rm(",
     "library": "var_list.r",
     "varRefreshCmd": "cat(var_dic_list()) "
    }
   },
   "types_to_exclude": [
    "module",
    "function",
    "builtin_function_or_method",
    "instance",
    "_Feature"
   ],
   "window_display": false
  }
 },
 "nbformat": 4,
 "nbformat_minor": 1
}