1. VAME Workflow
Thank you for working with VAME! We are excited to explain its function here and give some guiding on how to analyse your data with our newest update of VAME v.1.0-alpha. The new version introduces a lot of new functions, has a much cleaner implementation and fixes some previous bugs. This wiki page will cover the VAME workflow, which consists of four main steps and optional steps to analyse your data.
- Set up your project, get your data into the right format and create a training dataset for the VAME deep learning model
- Train a variational autoencoder which is parameterized with recurrent neural network to embed behavioural dynamics
- Evaluate the trained model based on its reconstruction capabilities
- Segment so called behavioural motifs/poses/states from the input time series
- Optional: Create motif videos to get insights about the fine grained poses
- Optional: Investigate the hierarchical order of your behavioural states by detecting communities in the resulting markov chain
- Optional: Create community videos to get more insights about behaviour on a hierarchical scale
- Optional: Visualization and projection of latent vectors onto a 2D plane via UMAP
- Optional: Use the generative model (reconstruction decoder) to sample from the learned data distribution, reconstruct random real samples or visualize the cluster centre for validation
- Optional: Create a video of an egocentrically aligned animal + path through the community space (similar to our gif on github)
- Some remarks about considerations of obtaining behavioural data can be found at the end of this wiki
The following will cover all the steps and optional choices above in detail. Note that we also have a workflow script with short explanations and from where you can call the function after installing VAME. We hope that VAME helps to accelerate your research!
Throughout VAME we assume that you either use DeepLabCut (DLC) for pose estimation or that your resulting x,y-coordinates are stored in a .csv file similar to the one of DLC.
Step 1.1: vame.init_new_project(project='Your-VAME-Project', videos = ['/directory/to/your/video-1','/directory/to/your/video-2','...'], working_directory = '/YOUR/WORKING/DIRECTORY/')
VAME starts by initializing a new project. This will create a folder with your project name + the current date. Within this folder four sub-folder gets created and a config.yaml file.
config = vame.init_new_project(project='Your-VAME-Project', videos=['/directory/to/your/video-1','/directory/to/your/video-2','...'], working_directory='/YOUR/WORKING/DIRECTORY/', videotype='.mp4')
Once you created your project you can easily address your config file by specifying the path to it
config = '/YOUR/WORKING/DIRECTORY/Your-VAME-Project-Apr30-2020/config.yaml'
The config.yaml file is essential for your VAME project as you can set all the necessary parameter in here.
In your project you find a sub-folder called data. Here, you can put your egocentric aligned pose estimation file for every video within its folder. Please make sure its named correctly, e.g. videoname-PE-seq.npy. Afterwards, you can call the function vame.create_trainset(config) which will create the training set for the recurrent neural network in "/Your-VAME-Project-Apr30-2020/data/train/...".
Note that vame.create_trainset(config) will linear interpolate every NaN in your pose estimation NumPy file and save it in your data folder as data/video-1/video-1-PE-seq-clean.npy. This file will be accessed later on from vame.behavior_segmentation() to segment your pose data.
Now you can start training your recurrent neural network.
vame.rnn_model(config, model_name='VAME', pretrained_weights=False, pretrained_model=None)
The RNN model function takes as input your config file, a string which specifies the name of your model, and in case you want to use a pretrained model you can set pretrained_weights=True and pretrained_model='VAME-pretrained'. The last parameter is a string which is the name of the pretrained model. Note that you can always train a model from scratch and use it later again as pretrained model. However, the model must be explicitly the same.
For this step you need to make sure how many input signals you have, i.e. how many virtual marker did you place on the animal. As an example, if you observe a mouse and have only markers at the nose, back and trail-root, you end up with six signals (x-,y-coordinates). Note, the better your virtual markers are learned the better results VAME will get you. It can, however, cope with some noise in your data.
Another important variable is the time window size from which VAME should make predictions about the behavioral state of the animal. Our camera was shooting with 60 frames per second and we choose the time window to be 30 time steps wide to infer the behavioral state from half a second of data. If you choose the time window too big, the RNN might have troubles with finding a good latent representation. The same goes for too short time windows. For now, we recommend a number between 20 and 40 time steps.
Once you figured out the number of signals and the time window, please specify this in your config.yaml:
num_features: 12
time_window: 30
The other variable should for the moment stay like they are.
NOTE: This step requires that you have your GPU ready and working. If VAME can not find a GPU it will write an error message.
When your model reached convergence or the maximal numbers of epochs, you can evaluate how good is able to reconstruct your input signal and to predict the future dynamics.
vame.evaluate_model(config, model_name='VAME')
The evaluation function takes as input your config file and the name of the model you want to evaluate.
If your are satisfied with the reconstruction of the input signal you can finally segment your behavioral signals into discrete blocks of behavioral motifs.
Now its time to let the model infer the underlying behavioral structure of your data. This step will load your trained model and set it to inference mode. Here, only the trained encoder is used to infer the latent state structure from which in a second step the k-Means cluster will be segmented.
vame.behavior_segmentation(config, model_name='VAME', cluster_method='kmeans', n_cluster=[30])
As usual, the function takes as input your config file as well as the name of the model. There are two more
parameters you can control. The first one is the cluster_method which is by default set to kmeans. It is however possible to use a Gaussian mixture model to cluster from the latent state vector by setting cluster_method='GMM'. We recommend to leave the kmeans setting for now.
The second parameter is a list of behavioral states you want to cluster. E.g. n_cluster=[30] or n_cluster=[15,30,45]. After inferring the latent vectors for every animal this will create a folder in the results part for every behavior video with the cluster size. In this way you can set multiple cluster sizes and later investigate which particular size brings the best results for your experiment.
In our paper we were using 30 behavioral states. This number can be higher or lower, depending on how much data you have at hand to train the model and on how coarse- or fine-grained you want to cluster your behavior. Note, that there will be a trade-off when you set the cluster number too high or too low.
A last important note is that the resulting motif labels are starting at the time_window / 2 frame. This means if you chose time_window = 30, your first motif label is the 15th frame of your video.
After segmenting your pose estimation time series into discrete behavioral motifs its time to quantify the behavior in your experiment.
vame.behavior_quantification(config, model_name='VAME', cluster_method='kmeans', n_cluster=30)
This function will give you the distribution of the motif usage and a transition matrix between behavioral motifs. The results will be stored in the results folder for every video.
From here, you can take these results and start doing your own analysis for your experiment. Enjoy!