GitBook: [master] 55 pages modified

Author: adamltyson

SUMMARY.md

@@ -21,7 +21,8 @@
   * [Cell candidate detection](user-guide/usage/cell-candidate-detection.md)
   * [Cell candidate classification](user-guide/usage/cell-candidate-classification.md)
   * [Historical options](user-guide/usage/historical-options.md)
-* [Training the network](user-guide/untitled-1.md)
+* [Training the network](user-guide/untitled-1/README.md)
+  * [Using supplied training data](user-guide/untitled-1/using-supplied-training-data.md)
 * [Visualisation](user-guide/visualisation.md)
 * [Group-level analysis](user-guide/group-level-analysis/README.md)
   * [Summarising multiple cell counts](user-guide/group-level-analysis/untitled-2.md)

installation/setting-up-your-gpu.md

@@ -6,7 +6,7 @@ description: How to speed up cellfinder by using your GPU
 
 ## Introduction
 
-**cellfinder** will run quite happily on your CPU, but the machine learning parts \(classifying cell candidates as cells or artefacts, and [Training the network](../user-guide/untitled-1.md)\) **run much faster** using  GPU.
+**cellfinder** will run quite happily on your CPU, but the machine learning parts \(classifying cell candidates as cells or artefacts, and [Training the network](../user-guide/untitled-1/)\) **run much faster** using  GPU.
 
 #### Requirements
 

user-guide/data-requirements.md

@@ -12,7 +12,7 @@ cellfinder was written to analyse certain kinds of whole brain microscopy datase
 
 For registration, you only need a single channel, but this is ideally a "background" channel, i.e. one with only autofluroescence, and no other strong signal. Typically we acquire the "signal" channels with red or green filters, and then the "background" channel with blue filters.
 
-For cell detection, you will need two channels, the "signal" channel, and the "background" channel. The signal channel should contain brightly labelled cells \(e.g. from staining or viral injections\). The models supplied with cellfinder were trained on whole-cell labels, so if you have e.g. a nuclear marker, they will need to be retrained \(see [Training the network](untitled-1.md)\). However, realistically, the network will need to be retrained for every new application
+For cell detection, you will need two channels, the "signal" channel, and the "background" channel. The signal channel should contain brightly labelled cells \(e.g. from staining or viral injections\). The models supplied with cellfinder were trained on whole-cell labels, so if you have e.g. a nuclear marker, they will need to be retrained \(see [Training the network](untitled-1/)\). However, realistically, the network will need to be retrained for every new application
 
 ### Image structure
 

user-guide/getting-started.md

@@ -37,5 +37,5 @@ If you have any spaces in your file-path, please enclose it in quotation marks,
 
 ### Retraining the machine learning network to classify cells
 
-The deep learning network included with cellfinder to classify cells as real cells or artefacts was trained on a very specific dataset. You will very likely need to retrain this if the classification is incorrect on your data. See [Training the network](untitled-1.md).
+The deep learning network included with cellfinder to classify cells as real cells or artefacts was trained on a very specific dataset. You will very likely need to retrain this if the classification is incorrect on your data. See [Training the network](untitled-1/).
 

user-guide/untitled-1/README.md

@@ -0,0 +1,104 @@
+# Training the network
+
+Cellfinder includes a pretrained network for cell candidate classification. This will likely need to be retrained for different applications. Rather than generate training data blindly, the aim is to reduce the amount of hands-on time by only generating training data where cellfinder classified a cell candidiate incorrectly.
+
+## Generate training data
+
+To generate training data, you will need:
+
+* The cellfinder output file, `cell_classification.xml` \(but `cells.xml` can also work\).
+* The raw data used initially for cellfinder
+
+To generate training data for a single brain, use `cellfinder_curate`:
+
+```bash
+cellfinder_curate signal_images background_images cell_classification.xml
+```
+
+### Arguments
+
+* Signal images
+* Background images
+* `cell_classification.xml` file
+
+{% hint style="info" %}
+You must also specify the pixel sizes, see [Specifying pixel size](../usage/specifying-pixel-size.md)
+{% endhint %}
+
+**Optional**
+
+* `-o` or `--output` Output directory for curation results. If this is not given, then the directory containing `cell_classification.xml` will be used.
+* `--symbol` Marker symbol \(Default: `ring`\)
+* `--marker-size` Marker size\(Default: `15`\)
+* `--opacity` Marker opacity \(Default: `0.6`\)
+
+A [napari](https://napari.org/) window will then open, showing two tabs on the left hand side:
+
+* `Image` Selecting this allows you to change the contrast limits, to better visualise cells
+* `Cell candidates` This shows the cell candidates than be curated. Cell 
+
+  candidates previously classified as cells are shown in yellow, and artifacts 
+
+  in blue. 
+
+By selecting the `Cell candidates` tab and then the cell selecting tool \(arrow at the top\), cell candidates can be selected \(either individually, or many by dragging the cursor\). There are then four keyboard commands:
+
+* `C` Confirm the classification result, and add this to the training set
+* `T` Toggle the classification result \(i.e. change the classification\), 
+
+  and add this to the training set.
+
+* `Alt+Q` Save the results to an xml file
+* `Alt+E` Finish curating the training dataset. This will carry out three operations:
+  * Extract cubes around these points, into two directories \(`cells` and `non_cells`\).
+  * Generate a yaml file pointing to these files for use with `cellfinder_train` \(see below\)
+  * Close the viewer
+
+Once a `yaml` file has been generated, you can proceed to training. However, it is likely useful to generate `yaml` files from additional datasets.
+
+## Start training
+
+You can then use these yaml files for training
+
+_N.B. If you have any yaml files from previous versions of cellfinder, they will continue to work, but are not documented here. Just use them as you would the files from `cellfinder_curate`._
+
+```bash
+cellfinder_train -y yaml_1.yml  yaml_2.yml -o /path/to/output/directory/
+```
+
+### Arguments
+
+* `-y` or `--yaml` The path to the yaml files defining training data
+* `-o` or `--output` Output directory for the trained model \(or model weights\)
+
+  results
+
+**Optional**
+
+* `--continue-training` Continue training from an existing trained model. If no model or model weights are specified, this will continue from the included model.
+* `--trained-model` Path to a trained model to continue training
+* `--model-weights` Path to existing model weights to continue training
+* `--network-depth` Resnet depth \(based on [He et al. \(2015\)](https://arxiv.org/abs/1512.03385)\). Choose from
+
+  \(18, 34, 50, 101 or 152\). In theory, a deeper network should classify better,
+
+  at the expense of a larger model, and longer training time. Default: 50
+
+* `--batch-size` Batch size for training \(how many cell candidates to process at once\). Default: 16
+* `--epochs` How many times to use each sample for training. Default: 1000
+* `--test-fraction` What fraction of data to keep for validation. Default: 0.1
+* `--learning-rate` Learning rate for training the model
+* `--no-augment` Do not use data augmentation
+* `--save-weights` Only store the model weights, and not the full model. Useful to save storage space.
+* `--no-save-checkpoints` Do not save the model after each training epoch. Useful to save storage space, if you are happy to wait for the chosen number of epochs to complete. Each model file can be large, and if you don't have much training data, they can be generated quickly.
+* `--tensorboard` Log to `output_directory/tensorboard`. Use `tensorboard --logdir outputdirectory/tensorboard` to view.
+* `--save-progress` Save training progress to a .csv file \(`output_directory/training.csv`\).
+
+### Further help
+
+All `cellfinder_train` options can be found by running:
+
+```bash
+cellfinder_train -h
+```
+

user-guide/untitled-1/using-supplied-training-data.md

@@ -0,0 +1,51 @@
+---
+description: How to retrain the network using the supplied data
+---
+
+# Using supplied training data
+
+cellfinder is released with a pre-trained cell candidate classification network, trained on approximately 100,000 manually annotated cell candidates \(with a roughly 50/50 split between cells and non-cells\).
+
+This data was acquired using [serial two-photon tomography](https://www.nature.com/articles/nmeth.1854). While you will likely need to retrain the network for your own data, we make the data available for a few reasons:
+
+* You might want to use this data to test the training, or assess how much training data you may need
+* You might want to retrain a different network \(i.e. a different ResNet depth\) than the one supplied \(50-layer\).
+* You might want to retrain the network using a mixture of this data \(of which there is a lot\) and your data \(of which you may not be able to generate as much\).
+
+The data is available [here](https://gin.g-node.org/cellfinder/training_data/raw/master/serial2p.tar.gz). To retrain the network using just this data, download the data, extract the tar archive, and then follow these steps:
+
+{% hint style="info" %}
+If you're using Windows, you will need to edit `training.yml` so that the paths \(in each `cube_dir` and `cell_def` entry\) match windows paths \(i.e. backslashes\)
+{% endhint %}
+
+* Activate your conda environment:
+
+```text
+conda activate cellfinder
+```
+
+* Navigate to the training data directory
+
+```text
+cd serial2p
+```
+
+* Start training
+
+```text
+cellfinder_train -y training.yml -o training_output
+```
+
+The training will likely take a few minutes to get going, once the network starts you should see something like this:
+
+```text
+Epoch 1/100
+   1/6050 [..............................] - ETA: 0s - loss: 0.9579 - accuracy:    
+   2/6050 [..............................] - ETA: 1:33:47 - loss: 3.1335 - accur   
+   3/6050 [..............................] - ETA: 3:10:17 - loss: 2.6173 - accur   
+   4/6050 [..............................] - ETA: 4:03:42 - loss: 2.2663 - accur   
+   5/6050 [..............................] - ETA: 4:30:16 - loss: 2.0002 - accur   
+```
+
+
+