Nvidia NeMo Multitalker Parakeet Streaming Benchmarks

[arda-argmax]

Model page:

https://huggingface.co/nvidia/multitalker-parakeet-streaming-0.6b-v1

How to run:

Run the following cli command:

openbench-cli evaluate -p nemo-mt-parakeet -d {callhome-english, chime-6, earnings21} --metrics cpwer

Results:

callhome-english

                       Benchmark Result                        
┏━━━━━━━━━━━━━━━━━━┳━━━━━━━━━━━━━━━━━━━━━━━━┳━━━━━━━━┳━━━━━━━━┓
┃     Dataset      ┃        Pipeline        ┃ Metric ┃ Value  ┃
┡━━━━━━━━━━━━━━━━━━╇━━━━━━━━━━━━━━━━━━━━━━━━╇━━━━━━━━╇━━━━━━━━┩
│ callhome-english │ NeMoMTParakeetPipeline │ cpwer  │ 0.1726 │
└──────────────────┴────────────────────────┴────────┴────────┘

chime-6

                   Benchmark Result                   
┏━━━━━━━━━┳━━━━━━━━━━━━━━━━━━━━━━━━┳━━━━━━━━┳━━━━━━━━┓
┃ Dataset ┃        Pipeline        ┃ Metric ┃ Value  ┃
┡━━━━━━━━━╇━━━━━━━━━━━━━━━━━━━━━━━━╇━━━━━━━━╇━━━━━━━━┩
│ chime-6 │ NeMoMTParakeetPipeline │ cpwer  │ 0.6525 │
└─────────┴────────────────────────┴────────┴────────┘

earnings21

                    Benchmark Result                     
┏━━━━━━━━━━━━┳━━━━━━━━━━━━━━━━━━━━━━━━┳━━━━━━━━┳━━━━━━━━┓
┃  Dataset   ┃        Pipeline        ┃ Metric ┃ Value  ┃
┡━━━━━━━━━━━━╇━━━━━━━━━━━━━━━━━━━━━━━━╇━━━━━━━━╇━━━━━━━━┩
│ earnings21 │ NeMoMTParakeetPipeline │ cpwer  │ 0.6912 │
└────────────┴────────────────────────┴────────┴────────┘

[EduardoPach]

Thanks!