You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
Copy file name to clipboardexpand all lines: BootstrapNAS/README.md
+6-11
Original file line number
Diff line number
Diff line change
@@ -16,7 +16,7 @@ If you already have a super-network trained with BootstrapNAS, please follow the
16
16
17
17
More information about BootstrapNAS is available in our papers:
18
18
19
-
[Automated Super-Network Generation for Scalable Neural Architecture Search](https://openreview.net/attachment?id=HK-zmbTB8gq&name=main_paper_and_supplementary_material).
19
+
[Automated Super-Network Generation for Scalable Neural Architecture Search](https://openreview.net/pdf?id=HK-zmbTB8gq).
20
20
21
21
```bibtex
22
22
@inproceedings{
@@ -25,7 +25,7 @@ More information about BootstrapNAS is available in our papers:
25
25
author={Muñoz, J. Pablo and Lyalyushkin, Nikolay and Lacewell, Chaunte and Senina, Anastasia and Cummings, Daniel and Sarah, Anthony and Kozlov, Alexander and Jain, Nilesh},
26
26
booktitle={First Conference on Automated Machine Learning (Main Track)},
27
27
year={2022},
28
-
url={https://openreview.net/forum?id=HK-zmbTB8gq}
28
+
url={https://openreview.net/pdf?id=HK-zmbTB8gq}
29
29
}
30
30
```
31
31
[Enabling NAS with Automated Super-Network Generation](https://arxiv.org/abs/2112.10878)
@@ -35,15 +35,10 @@ More information about BootstrapNAS is available in our papers:
35
35
bootstrapNAS,
36
36
author = {Muñoz, J. Pablo and Lyalyushkin, Nikolay and Akhauri, Yash and Senina, Anastasia and Kozlov, Alexander and Jain, Nilesh},
37
37
title = {Enabling NAS with Automated Super-Network Generation},
Copy file name to clipboardexpand all lines: BootstrapNAS/instructions/Configuration.md
+7-7
Original file line number
Diff line number
Diff line change
@@ -1,6 +1,6 @@
1
1
### Configuration file
2
2
3
-
The parameters for generating, training and searching on the super-network are defined in a configuration file within two exclusive subsets of parameters for training and search:
3
+
The parameters for generating, training, and searching on the super-network are defined in a configuration file within two exclusive subsets of parameters for training and search:
4
4
```json
5
5
"bootstrapNAS": {
6
6
"training": {
@@ -12,7 +12,7 @@ The parameters for generating, training and searching on the super-network are d
12
12
}
13
13
```
14
14
15
-
In the `training` section, you specify the training algorithm, e.g., `progressive_shrinking`, schedule and elasticity parameters:
15
+
In the `training` section, you specify the training algorithm, e.g., `progressive_shrinking`, schedule, and elasticity parameters:
16
16
17
17
```json
18
18
"training": {
@@ -41,7 +41,7 @@ In the `training` section, you specify the training algorithm, e.g., `progressiv
41
41
}
42
42
43
43
```
44
-
In the search section, you specify the search algorithm, e.g., `NSGA-II` and its parameters. For example:
44
+
In the search section, you specify the search algorithm, e.g., `NSGA-II`, and its parameters. For example:
45
45
```json
46
46
"search": {
47
47
"algorithm": "NSGA2",
@@ -51,7 +51,7 @@ In the search section, you specify the search algorithm, e.g., `NSGA-II` and its
51
51
}
52
52
```
53
53
54
-
By default, BootstrapNAS uses `NSGA-II` (Dev et al., 2002), an genetic algorithm that constructs a pareto front of efficient sub-networks.
54
+
By default, BootstrapNAS uses `NSGA-II` (Dev et al., 2002), a genetic algorithm that constructs a Pareto front of efficient sub-networks.
55
55
56
56
List of parameters that can be used in the configuration file:
57
57
@@ -65,20 +65,20 @@ List of parameters that can be used in the configuration file:
65
65
66
66
`schedule`: The schedule section includes a list of stage descriptors (`list_stage_descriptions`) that specify the elasticity dimensions enabled for a particular stage (`train_dims`), the number of `epochs` for the stage, the `depth_indicator` which in the case of elastic depth, restricts the maximum number of blocks in each independent group that can be skipped, the `width_indicator`, which restricts the maximum number of width values in each elastic layer. The user can also specify whether weights should be reorganized (`reorg_weights`), whether batch norm adaptation should be triggered at the beginning of the stage (`bn_adapt`), the initial learning rate for the stage (`init_lr`), and the epochs to use for adjusting the learning rate (`epochs_lr`).
67
67
68
-
`elasticity`: Currently, BootstrapNAS supports three elastic dimensions (`kernel`, `width` and `depth`). The `mode` for elastic depth can be set as `auto` or `manual`. If manual is selected, the user can specify, a list of possible `skipped_blocks` that, as the name suggest, might be skipped. In `auto` mode, the user can specify the `min_block_size`, i.e., minimal number of operations in the skipping block, and the `max_block_size`, i.e., maximal number of operations in the block. The user can also `allow_nested_blocks` or `allow_linear_combination` of blocks. In the case of elastic width, the user can specify the `min_width`, i.e., the minimal number of output channels that can be activated for each layers with elastic width. Default value is 32, the `max_num_widths`, which restricts total number of different elastic width values for each layer, a `width_step`, which defines a step size for a generation of the elastic width search space, or a `width_multiplier` to define the elastic width search space via a list of multipliers. Finally, the user can determine the type of filter importance metric: L1, L2 or geometric mean. L2 is selected by default. For elastic kernel, the user can specify the `max_num_kernels`, which restricts the total number of different elastic kernel values for each layer.
68
+
`elasticity`: Currently, BootstrapNAS supports three elastic dimensions (`kernel`, `width`, and `depth`). The `mode` for elastic depth can be set as `auto` or `manual`. If _manual_ is selected, the user can specify a list of possible `skipped_blocks` that might be skipped, as the name suggests. In `auto` mode, the user can specify the `min_block_size`, i.e., the minimal number of operations in the skipping block, and the `max_block_size`, i.e., the maximal number of operations in the block. The user can also `allow_nested_blocks` or `allow_linear_combination` of blocks. In the case of elastic width, the user can specify the `min_width`, i.e., the minimal number of output channels that can be activated for each layer with elastic width. The default value is 32, the `max_num_widths`, which restricts the total number of different elastic width values for each layer, a `width_step`, which defines a step size for a generation of the elastic width search space, or a `width_multiplier` to define the elastic width search space via a list of multipliers. Finally, the user can determine the type of filter importance metric: L1, L2 or geometric mean. L2 is selected by default. The user can specify the `max_num_kernels` for the elastic kernel, which restricts the total number of different elastic kernel values for each layer.
69
69
70
70
`train_steps`: Defines the number of samples used for each training epoch.
71
71
72
72
**Search:**
73
73
74
74
`algorithm`: Defines the search algorithm. The default algorithm is NSGA-II.
75
75
76
-
`num_evals`: Defines the number of evaluations that will be used by the search algorithm.
76
+
`num_evals`: Defines the number of evaluations the search algorithm will use.
77
77
78
78
`population`: Defines the population size when using an evolutionary search algorithm.
79
79
80
80
`acc_delta`: Defines the absolute difference in accuracy that is tolerated when looking for a subnetwork.
81
81
82
82
`ref_acc`: Defines the reference accuracy from the pre-trained model used to generate the super-network.
83
83
84
-
*A full list of the possible configuration parameters can be found [here](https://github.com/openvinotoolkit/nncf/blob/develop/nncf/config/experimental_schema.py).
84
+
*A complete list of the possible configuration parameters can be found [here](https://github.com/openvinotoolkit/nncf/blob/develop/nncf/config/experimental_schema.py).
### Expected Output Files after executing BootstrapNAS
54
-
The output of running ```bootstrap_nas.py``` will be a sub-network configuration that has an accuracy similar to the input model (by default a $\pm$1% absolute difference in accuracy is allowed), but with improvements in MACs. Format: ([MACs_subnet, ACC_subnet]).
54
+
The output of running ```bootstrap_nas.py``` will be a sub-network configuration with an accuracy similar to the input model (by default a $\pm$1% absolute difference in accuracy is allowed), but with improvements in MACs. Format: ([MACs_subnet, ACC_subnet]).
55
55
56
56
Several files are saved to your `log_dir` after the training has ended:
57
57
@@ -62,9 +62,9 @@ Several files are saved to your `log_dir` after the training has ended:
62
62
-`last_elasticity.pth`- Super-network's elasticity information. This file can be used when loading super-networks for searching or inspection.
63
63
-`last_model_weights.pth`- Super-network's weights after training.
64
64
-`snapshot.tar.gz` - Copy of the code used for this run.
65
-
-`subnetwork_best.pth` - Dictionary with the configuration of the best sub-network. Best defined as a sub-network that performs in the Pareto front, and that deviates a maximum `acc_delta` from original model.
65
+
-`subnetwork_best.pth` - Dictionary with the configuration of the best sub-network. Best is defined as a sub-network that performs in the Pareto front and deviates a maximum `acc_delta` from the original model.
66
66
-`supernet_{best, last}.pth` - Super-network weights at its best and last state.
67
67
68
68
If the user wants to have a CSV output file of the search progression, ```search_algo.search_progression_to_csv()``` can be called after running the search step.
69
69
70
-
For a visualization of the search progression please use ```search_algo.visualize_search_progression()``` after the search has concluded. A PNG file will be generated.
70
+
For a visualization of the search progression, please use ```search_algo.visualize_search_progression()``` after the search has concluded. A PNG file will be generated.
Copy file name to clipboardexpand all lines: BootstrapNAS/instructions/Subnetwork_Search.md
+3-3
Original file line number
Diff line number
Diff line change
@@ -2,15 +2,15 @@
2
2
3
3
If you have a trained super-network, you can start the search stage directly using the ```bootstrap_nas_search.py``` script located [here](https://github.com/openvinotoolkit/nncf/blob/develop/examples/experimental/torch/classification/bootstrap_nas_search.py).
4
4
5
-
You will need to pass the path where the weights and elasticity information has been stored, which by default is your log directory.
5
+
You must pass the path where the weights and elasticity information have been stored, which is your log directory by default.
6
6
7
7
```shell
8
8
python bootstrap_nas_search.py -m
9
9
train
10
10
--config <Config path to your config.json used when training the super-network>
11
11
--log-dir <Path to your log dir for the search stage>
12
12
--dataset
13
-
<cifar10, imagenet, or other depending on your model>
13
+
<cifar10, imagenet, or other, depending on your model>
14
14
--data <Path to your dataset>
15
15
--elasticity-state-path
16
16
<Path to your last_elasticity.pth file generated when training of the super-network>
@@ -20,4 +20,4 @@ train
20
20
21
21
#### Hardware-aware search
22
22
23
-
BootstrapNAS can be made hardware-aware when searching for efficient sub-networks. To accomplish this, you can pass your own `eficiency evaluator` for the target hardware to the search component.
23
+
BootstrapNAS can be made hardware-aware when searching for efficient sub-networks. To accomplish this, you can pass your own `efficiency evaluator` for the target hardware to the search component.
The argument `--val_set_size 1000` signifies the utilization of 1k validation samples to evaluate each discovered
131
-
subnetwork. After running this command, results of the 200 identified subnetworks (`"num_evals": 200` set in `search` field of NNCF config)
132
-
can be obtained in the `--output_dir` folder, including `search_progression.png` and `search_progression.csv`.
133
-
From these results, we can select the subnetwork configurations that best meets different requirements.
131
+
subnetwork. After running this command, the results of the 200 identified subnetworks (`"num_evals": 200` set in the `search` field of NNCF config) will be placed in the `--output_dir` folder, including `search_progression.png` and `search_progression.csv`.
132
+
From these results, we can select the subnetwork configurations that best meet different requirements.
You can also skip the quantization step and adopt our released quantized models (find them in Sparse-and-Quantized Model of [this Table](#released-foundation-models-)).
120
+
You can also skip the quantization step and adopt our released quantized models (find them in the *Sparse-and-Quantized Model* column of [this Table](#released-foundation-models-)).
0 commit comments