Supports continuous speech recognition and barge-in

[compulim]

Fixes #2661. Fixes #5352.

Initial work done in #5397.

Changelog Entry

Added

• Resolved #2661 and #5352. Added speech recognition continuous mode with barge-in support, in PR #5426, by @RushikeshGavali and @compulim
  • Set styleOptions.speechRecognitionContinuous to true with a Web Speech API provider with continuous mode support

Changed

• Bumped dependencies to the latest versions, by @compulim in PR #5385, #5400, and #5426
  • Production dependencies
    • [email protected]

Description

Continuous mode is designed for hands-off/kiosk scenario. End-users can hold a speech-primary conversation with the bot, and occasionally, interact with gestures (e.g. tapping on a card). Speech recognition will be kept active as long as possible, until end-user turn off speech recognition.

Added new styleOptions.speechRecognitionContinuous to enable continuous mode for speech recognition.

Design

• Interactive mode: speech recognition is only active for minimal time, focus on privacy
• Continuous mode: speech recognition will be active for as long as possible, durable over non-speech interactions, barge-in is supported, focus on hands-off experience

Behavioral differences

• Continuous mode will not turn off microphone after speech is recognized
  • This is a behavior exhibited by the Web Speech API provider
    • Technically, Web Chat will not turn off microphone until end event is received, and not because result event is received
  • If Web Speech API provider does not support continuous mode, it should send end event after speech is recognized
• While the bot response is synthesizing and input mode is "expecting input":
  • Interactive mode:
    • While synthesis is ongoing, speech recognition is paused
    • After synthesis has completed, speech recognition will be resumed
  • Continuous mode:
    • While synthesis is ongoing, speech recognition is continue to be active
    • When interim is recognized, synthesis will be interrupted (a.k.a. barge-in)
    • Logically, "expecting input" is ignored (speech recognition is always active and not paused)
• While speech recognition is active, tap on card action
  • Interactive mode: will stop speech recognition, will not speak bot response
  • Continuous mode: will not stop speech recognition, will speak bot response
• While speech recognition is active, receiving a bot message proactively
  • Interactive mode: will not synthesize the bot message
  • Continuous mode: will synthesize the bot message

Technical details

• Web Chat relies on the correctness of the behavior of Web Speech API provider, including
  • Web Chat assume the microphone is on when start event is received
  • Web Chat assume the microphone is off when end event is received
• Web Chat do not care about the SpeechRecognition.continuous property, but depends on the event dispatched by the Web Speech API provider
  • Microphone will be turned off when receiving end event
  • Microphone will send the message when receiving a result event with resultIndex pointing to a result which its isFinal property is true
    • event.results[event.resultIndex].isFinal === true
• If interim is received, Web Chat will stop speech synthesis

Specific Changes

• Added new styleOptions.speechRecognitionContinuous

• I have added tests and executed them locally
• I have updated CHANGELOG.md
• I have updated documentation

Review Checklist

This section is for contributors to review your work.

• Accessibility reviewed (tab order, content readability, alt text, color contrast)
• Browser and platform compatibilities reviewed
• CSS styles reviewed (minimal rules, no z-index)
• Documents reviewed (docs, samples, live demo)
• Internationalization reviewed (strings, unit formatting)
• package.json and package-lock.json reviewed
• Security reviewed (no data URIs, check for nonce leak)
• Tests reviewed (coverage, legitimacy)

[OEvgeny]

Looks solid, couple of nits and questions