Skip to content

Commit

Permalink
Merge pull request #306 from Drakrig/unit2-convnextr-fix
Browse files Browse the repository at this point in the history
Fixes for Unit 2 - ConvNext
  • Loading branch information
johko authored Jul 17, 2024
2 parents 4890225 + 1b395d9 commit 5b8557a
Showing 1 changed file with 4 additions and 5 deletions.
9 changes: 4 additions & 5 deletions chapters/en/unit2/cnns/convnext.mdx
Original file line number Diff line number Diff line change
Expand Up @@ -12,9 +12,9 @@ The key improvements are:
- Training techniques
- Macro design
- ResNeXt-ify
- Inverted bottleneck
- Large kernel sizes
- Micro design
- Inverted Bottleneck
- Large Kernel Sizes
- Micro Design

We will go through each of the key improvements.
These designs are not novel in itself. However, you can learn how researchers adapt and modify designs systematically to improve existing models.
Expand All @@ -28,10 +28,9 @@ The researchers first discerned that, while architectural design choices are cru
Inspired by DeiT and Swin Transformers, ConvNext closely adapts their training techniques. Some of the notable changes are:
- Epochs: Extending the epochs from the original 90 epochs to 300 epochs.
- Optimizer: Using AdamW optimizer instead of Adam optimizer, which differs in how it handles weight decay.
- Regularization: Using Stochastic Depth and Label Smoothing as regularization techniques.
- Mixup (generates a weighted combination of random image pairs), Cutmix (cuts part of an image and replace it with a patch from another image), RandAugment (applies a series of random augmentations such as rotation, translation, and shear), and Random Erasing (randomly selects a rectangle region in an image and erases its pixels with random values) to increase training data.
Modifying these training procedures has improved ResNet-50's accuracy from 76.1% to 78.8%.
- Regularization: Using Stochastic Depth and Label Smoothing as regularization techniques.

Modifying these training procedures has improved ResNet-50's accuracy from 76.1% to 78.8%.


Expand Down

0 comments on commit 5b8557a

Please sign in to comment.