Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Add NAS KWS model (trained using dynamic augmentation) #324

Merged
merged 8 commits into from
Feb 1, 2024

Conversation

alicangok
Copy link
Contributor

@alicangok alicangok commented Jan 1, 2024

The trained networks in this PR achieve the following accuracies for the current version of our KWS20 testing set1

  • v2: 92.0%
  • v3: 90.8%
  • NAS: 93.8%

Footnotes

  1. This testing set contains all 11005 test examples from the original Google Speech Commands dataset without any "background/others" class balancing, as opposed to many benchmark settings. For our dataset, the "others" class makes up 25.7% of the testing set, making it more challenging than the aforementioned benchmarks which feature a much smaller "others" class (10%), as well as an easily identifiable "silence" class (10%).

@alicangok alicangok marked this pull request as draft January 8, 2024 11:36
@alicangok
Copy link
Contributor Author

PR changed to draft mode, awaiting training completion for v2 & v3 models with dynamic augmentation.

@@ -11,6 +11,7 @@ python ai8xize.py --test-dir $TARGET --prefix cifar-100-mixed --checkpoint-file
python ai8xize.py --test-dir $TARGET --prefix cifar-100-simplewide2x-mixed --checkpoint-file trained/ai85-cifar100-simplenetwide2x-qat-mixed-q.pth.tar --config-file networks/cifar100-simplewide2x.yaml --softmax $COMMON_ARGS --boost 2.5 "$@"
python ai8xize.py --test-dir $TARGET --prefix cifar-100-residual --checkpoint-file trained/ai85-cifar100-residual-qat8-q.pth.tar --config-file networks/cifar100-ressimplenet.yaml --softmax $COMMON_ARGS --boost 2.5 "$@"
python ai8xize.py --test-dir $TARGET --prefix kws20_v3 --checkpoint-file trained/ai85-kws20_v3-qat8-q.pth.tar --config-file networks/kws20-v3-hwc.yaml --softmax $COMMON_ARGS "$@"
python ai8xize.py --test-dir $TARGET --prefix kws20_nas --checkpoint-file trained/ai85-kws20_nas-qat8-q.pth.tar --config-file networks/kws20-nas-hwc.yaml --softmax $COMMON_ARGS "$@"
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Is it possible to rename the KWS20_v3 and KWS20_nas models as kws_light and kws? We can also remove the v1 & v2 models...

Copy link
Contributor Author

@alicangok alicangok Jan 21, 2024

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

The demo, the KAT example, and arch fields in the model files and trained checkpoints) refers to the model as "v3" in multiple places, so we would need make the changes to all at the same time.

I would suggest holding off on this until for a few weeks until we finalize the frequency-domain model, then decide on long-term names.

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

That makes sense. Thanks...

Copy link
Contributor

@aniktash aniktash left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Looks good

@alicangok alicangok marked this pull request as ready for review January 21, 2024 17:35
@alicangok
Copy link
Contributor Author

alicangok commented Jan 21, 2024

Final changes:

Please see the edited PR description accuracy details.

The PR is now marked as "Ready".

Copy link
Contributor

@MaximGorkem MaximGorkem left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Looks good

@rotx-eva rotx-eva merged commit f8761c6 into analogdevicesinc:develop Feb 1, 2024
1 check passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

5 participants