Organize files

Author: jpablomch

BootstrapNAS/README.md

@@ -0,0 +1,52 @@
+# BootstrapNAS Jupyter Notebooks
+
+---
+
+<p align="center">
+<img src="architecture.png" alt="BootstrapNAS Architecture" width="500"/>
+</p>
+
+BootstrapNAS (1) takes as input a pre-trained model. (2) It uses this model to generate a weight-sharing super-network. (3) BootstrapNAS then applies a training strategy, and once the super-network has been trained, (4) it searches for efficient subnetworks that satisfy the user's requirements. (5) The configuration of the discovered sub-network(s) is returned to the user.
+
+## Quickstart 
+
+Please follow the instructions [here](https://github.com/jpablomch/bootstrapnas/wiki/Quickstart).
+
+If you already have a super-network trained with BootstrapNAS, please follow the instructions to search for sub-networks [here](https://github.com/jpablomch/bootstrapnas/wiki/Subnetwork_Search).
+
+More information about BootstrapNAS is available in our papers:
+
+[Automated Super-Network Generation for Scalable Neural Architecture Search](https://openreview.net/attachment?id=HK-zmbTB8gq&name=main_paper_and_supplementary_material).
+
+```bibtex
+  @inproceedings{
+    munoz2022automated,
+    title={Automated Super-Network Generation for Scalable Neural Architecture Search},
+    author={Mu{\~{n}}oz, J. Pablo and Lyalyushkin, Nikolay and Lacewell, Chaunte and Senina, Anastasia and Cummings, Daniel and Sarah, Anthony  and Kozlov, Alexander and Jain, Nilesh},
+    booktitle={First Conference on Automated Machine Learning (Main Track)},
+    year={2022},
+    url={https://openreview.net/forum?id=HK-zmbTB8gq}
+  }
+```
+[Enabling NAS with Automated Super-Network Generation](https://arxiv.org/abs/2112.10878)
+
+```BibTex
+@article{
+  bootstrapNAS,
+  author    = {Mu{\~{n}}oz, J. Pablo  and Lyalyushkin, Nikolay  and Akhauri, Yash and Senina, Anastasia and Kozlov, Alexander  and Jain, Nilesh},
+  title     = {Enabling NAS with Automated Super-Network Generation},
+  journal   = {CoRR},
+  volume    = {abs/2112.10878},
+  year      = {2021},
+  url       = {https://arxiv.org/abs/2112.10878},
+  eprinttype = {arXiv},
+  eprint    = {2112.10878},
+  timestamp = {Tue, 04 Jan 2022 15:59:27 +0100},
+  biburl    = {https://dblp.org/rec/journals/corr/abs-2112-10878.bib},
+  bibsource = {dblp computer science bibliography, https://dblp.org}
+}
+```
+
+## Contributing to BootstrapNAS
+Please follow the contribution guidelines in [NNCF](https://github.com/openvinotoolkit/nncf) and reach out to [Pablo Muñoz](mailto:[email protected]) if you have any questions. 
+

BootstrapNAS/architecture.png

@@ -552,7 +552,7 @@ def resnet50_cifar10(pretrained=False, progress=True, device="cpu", **kwargs):
 def create_folders_demo(base_model_name):
     from pathlib import Path
     # MODEL_DIR = Path("model")
-    MODEL_DIR = Path("../../models/pretrained")
+    MODEL_DIR = Path("../models/pretrained")
     OUTPUT_DIR = Path("output")
     DATA_DIR = Path("data")
     BASE_MODEL_NAME = base_model_name

examples/bootstrapNAS/.gitignore renamed to BootstrapNAS/examples/.gitignore

@@ -0,0 +1,84 @@
+### Configuration file
+
+The parameters for generating, training and searching on the super-network are defined in a configuration file within two exclusive subsets of parameters for training and search: 
+```json
+    "bootstrapNAS": {
+        "training": {
+            ...
+        },
+        "search": {
+            ...
+        }
+    }
+```
+
+In the `training` section, you specify the training algorithm, e.g., `progressive_shrinking`, schedule and elasticity parameters: 
+
+```json
+"training": {
+    "algorithm": "progressive_shrinking",   
+    "progressivity_of_elasticity": ["depth", "width"], 
+    "batchnorm_adaptation": {
+        "num_bn_adaptation_samples": 1500
+    },
+    "schedule": { 
+        "list_stage_descriptions": [
+                    {"train_dims": ["depth"], "epochs": 25, "depth_indicator": 1, "init_lr": 2.5e-6, "epochs_lr": 25},
+                    {"train_dims": ["depth"], "epochs": 40, "depth_indicator": 2, "init_lr": 2.5e-6, "epochs_lr": 40},
+                    {"train_dims": ["depth", "width"], "epochs": 50, "depth_indicator": 2, "reorg_weights": true, "width_indicator": 2, "bn_adapt": true, "init_lr": 2.5e-6, "epochs_lr": 50},
+                    {"train_dims": ["depth", "width"], "epochs": 50, "depth_indicator": 2, "reorg_weights": true, "width_indicator": 3, "bn_adapt": true, "init_lr": 2.5e-6, "epochs_lr": 50}
+                ]
+    }, 
+    "elasticity": {
+        "available_elasticity_dims": ["width", "depth"],
+        "width": {
+            "max_num_widths": 3,
+            "min_width": 32,
+            "width_step": 32, 
+            "width_multipliers": [1, 0.80, 0.60]
+    },
+    ... 
+}
+
+```
+In the search section, you specify the search algorithm, e.g., `NSGA-II` and its parameters. For example: 
+```json
+"search": {
+    "algorithm": "NSGA2",
+            "num_evals": 3000,
+            "population": 50,
+            "ref_acc": 93.65,
+}
+```
+
+By default, BootstrapNAS uses `NSGA-II` (Dev et al., 2002), an genetic algorithm that constructs a pareto front of efficient sub-networks. 
+
+List of parameters that can be used in the configuration file: 
+
+**Training:**
+
+`algorithm`: Defines training strategy for tuning supernet. By default, `progressive_shrinking`.
+
+`progressivity_of_elasticity`: Defines the order of adding a new elasticity dimension from stage to stage. examples=["width", "depth", "kernel"].
+
+`batchnorm_adaptation`: Specifies the number of samples from the training dataset to use for model inference during the BatchNorm statistics adaptation procedure for the compressed model.
+
+`schedule`: The schedule section includes a list of stage descriptors (`list_stage_descriptions`) that specify the elasticity dimensions enabled for a particular stage (`train_dims`), the number of `epochs` for the stage, the `depth_indicator` which in the case of elastic depth, restricts the maximum number of blocks in each independent group that can be skipped, the `width_indicator`, which restricts the maximum number of width values in each elastic layer. The user can also specify whether weights should be reorganized (`reorg_weights`), whether batch norm adaptation should be triggered at the beginning of the stage (`bn_adapt`), the initial learning rate for the stage (`init_lr`), and the epochs to use for adjusting the learning rate (`epochs_lr`). 
+
+`elasticity`: Currently, BootstrapNAS supports three elastic dimensions (`kernel`, `width` and `depth`). The `mode` for elastic depth can be set as `auto` or `manual`. If manual is selected, the user can specify, a list of possible `skipped_blocks` that, as the name suggest, might be skipped. In `auto` mode, the user can specify the `min_block_size`, i.e., minimal number of operations in the skipping block, and the `max_block_size`, i.e., maximal number of operations in the block. The user can also `allow_nested_blocks` or `allow_linear_combination` of blocks. In the case of elastic width, the user can specify the `min_width`, i.e., the minimal number of output channels that can be activated for each layers with elastic width. Default value is 32, the `max_num_widths`, which restricts total number of different elastic width values for each layer, a `width_step`, which defines a step size for a generation of the elastic width search space, or a `width_multiplier` to define the elastic width search space via a list of multipliers. Finally, the user can determine the type of filter importance metric: L1, L2 or geometric mean. L2 is selected by default. For elastic kernel, the user can specify the `max_num_kernels`, which restricts the total number of different elastic kernel values for each layer.
+
+`train_steps`: Defines the number of samples used for each training epoch.
+
+**Search:**
+
+`algorithm`: Defines the search algorithm. The default algorithm is NSGA-II.
+
+`num_evals`: Defines the number of evaluations that will be used by the search algorithm.
+
+`population`: Defines the population size when using an evolutionary search algorithm.
+
+`acc_delta`: Defines the absolute difference in accuracy that is tolerated when looking for a subnetwork.
+
+`ref_acc`: Defines the reference accuracy from the pre-trained model used to generate the super-network.
+
+*A full list of the possible configuration parameters can be found [here](https://github.com/jpablomch/nncf_bootstrapnas/blob/develop/nncf/config/experimental_schema.py).

examples/bootstrapNAS/BootstrapNAS.ipynb renamed to BootstrapNAS/examples/BootstrapNAS.ipynb

@@ -0,0 +1,10 @@
+### BootstrapNAS: Automated Super-Network Generation for Scalable Neural Architecture Search
+
+### [Quickstart](https://github.com/jpablomch/bootstrapnas/wiki/Quickstart) 
+
+### [Sub-network Search](https://github.com/jpablomch/bootstrapnas/wiki/Subnetwork_Search)
+
+### [Configuration](https://github.com/jpablomch/bootstrapnas/wiki/Configuration)
+
+More detailed guides coming soon.
+

examples/bootstrapNAS/README.md renamed to BootstrapNAS/examples/README.md

@@ -0,0 +1,70 @@
+# Setup
+
+### PyTorch
+Install PyTorch and Torchvision using the [PyTorch installation guide](https://pytorch.org/get-started/locally/#start-locally). NNCF currently supports PyTorch >=1.5.0, <=1.9.1 (1.8.0 not supported). For this quickstart, PyTorch 1.9.1 and Torchvision 0.10.1 with CUDA 11.1 was installed using:
+```bash
+pip install torch==1.9.1+cu111 torchvision==0.10.1+cu111 -f https://download.pytorch.org/whl/torch_stable.html
+```
+
+
+### NNCF
+There are two options for installing [***NNCF***](https://github.com/openvinotoolkit/nncf#installation):
+- Package built from NNCF repository or 
+- PyPI package.  
+
+To install NNCF and dependencies from the NNCF repository, install by running the following in the repository root directory and also set `PYTHONPATH` variable to include the root directory:
+```bash
+python setup.py develop
+export PYTHONPATH="${PYTHONPATH}:<Directory Path>/nncf"
+```
+
+To install NNCF and dependencies as a PyPI package, use the following:
+```bash
+pip install nncf
+```
+
+The ```examples``` folder from the NNCF repository ***is not*** included when you install NNCF using a package manager. To run the BootstrapNAS examples, you will need to obtain this folder from the repository and add it to your path.
+
+
+### Additional Dependencies
+The examples in the NNCF repo have additional requirements, such as EfficientNet, MLFlow, Tensorboard, etc., which are not installed with NNCF. You will need to install them using: 
+```
+pip install efficientnet_pytorch tensorboard mlflow returns
+```
+
+
+# Example
+To run an example of super-network generation and sub-network search, use the ```bootstrap_nas.py``` script located [here](https://github.com/openvinotoolkit/nncf/blob/develop/examples/experimental/torch/classification/bootstrap_nas.py) and the sample ```config.json``` from [here](https://github.com/jpablomch/bootstrapnas/blob/main/bootstrapnas_examples/config.json). 
+
+The file ```config.json``` contains a sample configuration for generating a super-network from a trained model. The sample file is configured to generate a super-network from ResNet-50 trained with CIFAR-10. The file should be modified depending on the model to be used as input for BootstrapNAS.  
+
+Weights for CIFAR10-based models can be found at: https://github.com/huyvnphan/PyTorch_CIFAR10 
+
+Use the following to test training a super-network: 
+```
+cd <path to NNCF>/examples/experimental/torch/classification
+python bootstrap_nas.py -m train \
+    -c <path to this repo>/bootstrapnas_examples/config.json \
+    --data <path to your CIFAR10 dataset> \
+    --weights <path to weights for resnet-50 trained with CIFAR10>
+```
+
+
+### Expected Output Files after executing BootstrapNAS
+The output of running ```bootstrap_nas.py``` will be a sub-network configuration that has an accuracy similar to the input model (by default a $\pm$1% absolute difference in accuracy is allowed), but with improvements in MACs. Format: ([MACs_subnet, ACC_subnet]). 
+
+Several files are saved to your `log_dir` after the training has ended: 
+
+- `compressed_graph.{dot, png}`- Dot and PNG files that describe the wrapped NNCF model. 
+- `original_graph.dot` - Dot file that describes the original model. 
+- `config.json`- A copy of your original config file. 
+- `events.*`- Tensorboard logs.
+- `last_elasticity.pth`- Super-network's elasticity information. This file can be used when loading super-networks for searching or inspection.
+- `last_model_weights.pth`- Super-network's weights after training. 
+- `snapshot.tar.gz` - Copy of the code used for this run. 
+- `subnetwork_best.pth` - Dictionary with the configuration of the best sub-network. Best defined as a sub-network that performs in the Pareto front, and that deviates a maximum `acc_delta` from original model.
+- `supernet_{best, last}.pth` - Super-network weights at its best and last state. 
+
+If the user wants to have a CSV output file of the search progression, ```search_algo.search_progression_to_csv()``` can be called after running the search step.
+
+For a visualization of the search progression please use ```search_algo.visualize_search_progression()``` after the search has concluded. A PNG file will be generated. 

examples/bootstrapNAS/bootstrapnas_utils.py renamed to BootstrapNAS/examples/bootstrapnas_utils.py

@@ -0,0 +1,23 @@
+### Search on an existing super-network
+
+If you have a trained super-network, you can start the search stage directly using the ```bootstrap_nas_search.py``` script located [here](https://github.com/openvinotoolkit/nncf/blob/develop/examples/experimental/torch/classification/bootstrap_nas_search.py).
+
+You will need to pass the path where the weights and elasticity information has been stored, which by default is your log directory. 
+
+```shell
+python bootstrap_nas_search.py -m
+train
+--config <Config path to your config.json used when training the super-network> 
+--log-dir <Path to your log dir for the search stage> 
+--dataset
+<cifar10, imagenet, or other depending on your model>
+--data <Path to your dataset>
+--elasticity-state-path
+ <Path to your last_elasticity.pth file generated when training of the super-network>
+--supernet-weights <Path to your last_model_weights.pth generated during training of the super-network> 
+--search-mode
+```
+
+#### Hardware-aware search 
+
+BootstrapNAS can be made hardware-aware when searching for efficient sub-networks. To accomplish this, you can pass your own  `eficiency evaluator` for the target hardware to the search component.

examples/bootstrapNAS/imgs/architecture.png renamed to BootstrapNAS/examples/imgs/architecture.png

@@ -1,6 +1,7 @@
 # Hardware-Aware Automated Machine Learning Research
 
-Auxiliary resources for using AutoX - BootstrapNAS solution in the Neural Network Compression Framework (NNCF).
+### [NNCF's BootstrapNAS - Notebooks and Examples](./BootstrapNAS/README.md) 
+