1. VAME Workflow
The VAME workflow consists of 5 Steps. In this page we will walk you through them.
To start a VAME project you need to have your behavior and pose estimation data ready. In this beta version of VAME it is required that your pose estimation data is a priori egocentrically aligned which means that your postural (x,y)-coordinate move within the egocentric coordinates of the animal and not through the global coordinate space which the animal is exploring (Open-Field case). If you are using a restrained setup like a treadmill where your animals are head-fixed your postural coordinates a implicitly egocentric.
Note: Make sure that your data from multiple recordings/animals is captured from the same distance and angle. Skewed or scaled data can lead to poor performance, especially if you want to compare motif distributions between recordings/animals. It is, however, still possible to segment each session individually after training.
VAME starts by initializing a new project. This will create a folder with your project name + the current date. Within this folder four sub-folder gets created and a config.yaml file.
config = vame.init_new_project(project='Your-VAME-Project', videos=['/directory/to/your/video-1','/directory/to/your/video-2','...'], working_directory='/YOUR/WORKING/DIRECTORY/', videotype='.mp4')
Once you created your project you can easily address your config file by specifying the path to it
config = '/YOUR/WORKING/DIRECTORY/Your-VAME-Project-Apr30-2020/config.yaml'
The config.yaml file is essential for your VAME project as you can set all the necessary parameter in here.
Within your project you find a sub-folder called data. Here, you can put your egocentrically aligned pose estimation file for every video within its folder. Please make sure its named correctly, e.g. videoname-PE-seq.npy. Afterwards, you can call the function vame.create_trainset(config) which will create the training set for the recurrent neural network in "/Your-VAME-Project-Apr30-2020/data/train/..."
Now you can start training your recurrent neural network.
vame.rnn_model(config, model_name='VAME', pretrained_weights=False, pretrained_model=None)
The RNN model function takes as input your config file, a string which specifies the name of your model, and in case you want to use a pretrained model you can set pretrained_weights=True and pretrained_model='VAME-pretrained'. The last parameter is a string which is the name of the pretrained model. Note that you can always train a model from scratch and use it later again as pretrained model. However, the model must be explicitly the same.
For this step you need to make sure how many input signals you have, i.e. how many virtual marker did you place on the animal. As an example, if you observe a mouse and have only markers at the nose, back and trail-root, you end up with six signals (x-,y-coordinates). Note, the better your virtual markers are learned the better results VAME will get you. It can, however, cope with some noise in your data.
Another important variable is the time window size from which VAME should make predictions about the behavioral state of the animal. Our camera was shooting with 60 frames per second and we choose the time window to be 30 time steps wide to infer the behavioral state from half a second of data. If you choose the time window too big, the RNN might have troubles with finding a good latent representation. The same goes for too short time windows. For now, we recommend a number between 20 and 40 time steps.
Once you figured out the number of signals and the time window, please specify this in your config.yaml:
num_features: 12
time_window: 30
The other variable should for the moment stay like they are. If you want to fine-tune your model you can find more information here.
NOTE: This step requires that you have your GPU ready and working. If VAME can not find a GPU it will write an error message.
When your model reached convergence or the maximal numbers of epochs, you can evaluate how good is able to reconstruct your input signal and to predict the future dynamics.
vame.evaluate_model(config, model_name='VAME')
The evaluation function takes as input your config file and the name of the model you want to evaluate.
If your are satisfied with the reconstruction of the input signal you can finally segment your behavioral signals into discrete blocks of behavioral motifs.
Now its time to let the model infer the underlying behavioral structure of your data. This step will load your trained model and set it to inference mode. Here, only the trained encoder is used to infer the latent state structure from which in a second step the k-Means cluster will be segmented.
vame.behavior_segmentation(config, model_name='VAME', cluster_method='kmeans', n_cluster=[30])
As usual, the function takes as input your config file as well as the name of the model. There are two more
parameters you can control. The first one is the cluster_method which is by default set to kmeans. It is however possible to use a Gaussian mixture model to cluster from the latent state vector by setting cluster_method='GMM'. We recommend to leave the kmeans setting for now.
The second parameter is a list of behavioral states you want to cluster. E.g. n_cluster=[30] or n_cluster=[15,30,45]. After inferring the latent vectors for every animal this will create a folder in the results part for every behavior video with the cluster size. In this way you can set multiple cluster sizes and later investigate which particular size brings the best results for your experiment.
In our paper we were using 30 behavioral states. This number can be higher or lower, depending on how much data you have at hand to train the model and on how coarse- or fine-grained you want to cluster your behavior. Note, that there will be a trade-off when you set the cluster number too high or too low. You can read more about this here.
After segmenting your pose estimation time series into discrete behavioral motifs its time to quantify the behavior in your experiment.
vame.behavior_quantification(config, model_name='VAME', cluster_method='kmeans', n_cluster=30)
This function will give you the distribution of the motif usage and a transition matrix between behavioral motifs. The results will be stored in the results folder for every video.
From here, you can take these results and start doing your own analysis for your experiment. Enjoy!