Customizing Audio Tools Library for Adaptive Noise Cancellation on ESP32

[cameronmattie]

Hey there,

I am currently working on a project that aims to eliminate ambient noise while wearing headphones that do not have active noise cancellation capabilities. My physical setup includes an INMP441 microphone as one of my audio inputs, and a PCM5102 DAC breakout board as my audio output. This involves using an adaptive FIR filter (LMS algorithm, such as what is done here: https://github.com/wespo/LMS/blob/master/LMS.h) to effectively cancel out the noise that the microphone is collecting by using the filter output to find the difference between it and the desired signal that will be sent to the DAC alongside the desired signal (can be music via Bluetooth, or zero if the music isn't playing). I am using an ESP32-WROOM-32E devkit and working in the Arduino IDE.

Project Overview

The project is ambitious, so the first step I took was to send the microphone audio as-is to the DAC. If I could get that audio to my ears, then I could experiment with filtering techniques to see what I could do about generating anti-noise. Using the example streams-i2s-i2s-2, I was able to successfully transfer data from the microphone to the DAC and the audio quality is great, albeit with a minor delay.

Technical Challenges and Debugging Steps
I had been trying to develop my own code to mimic the audio reading/writing functionality of the streams-i2s-i2s-2 code in the audio tools library, as seen below:

#include "driver/i2s.h"
#include "nvs.h"
#include "nvs_flash.h"

// I2S pins for INMP441 microphone audio input
#define I2S_MIC_SCK 22
#define I2S_MIC_WS 23
#define I2S_MIC_SD 3

// I2S pins for PCM5102 audio output
#define I2S_BCK_PIN  26
#define I2S_LRCK_PIN 25
#define I2S_DATA_PIN 1

int64_t sBuffer[64];

void setup() {

Serial.begin(921600);

// Initialize NVS
esp_err_t err = nvs_flash_init();

  if (err == ESP_ERR_NVS_NO_FREE_PAGES || err == ESP_ERR_NVS_NEW_VERSION_FOUND) {
    ESP_ERROR_CHECK(nvs_flash_erase());
    err = nvs_flash_init();
  }
  
  ESP_ERROR_CHECK(err);

// Microphone Input
initMicrophoneInput();

// DAC Audio Output
initAudioOutput();

i2s_start(I2S_NUM_0);
i2s_start(I2S_NUM_1);
delay(100);
}

void loop() {
  memset(sBuffer, 0, sizeof(sBuffer));
  size_t bytesIn = 0;
  size_t bytesOut = 0;
  esp_err_t result = i2s_read(I2S_NUM_0, &sBuffer, sizeof(sBuffer), &bytesIn, portMAX_DELAY);
    
    if (result == ESP_OK) {
      i2s_write(I2S_NUM_1, sBuffer, sizeof(sBuffer), &bytesOut, portMAX_DELAY);
      Serial.write((const char *)sBuffer, bytesOut);
    }
}
// Microphone Input
void initMicrophoneInput() {
  const i2s_config_t i2s_config_mic = {
     .mode = (i2s_mode_t)(I2S_MODE_MASTER | I2S_MODE_RX),
      .sample_rate = 44100,
      .bits_per_sample = I2S_BITS_PER_SAMPLE_16BIT,
      .channel_format = I2S_CHANNEL_FMT_ONLY_LEFT,
      .communication_format = I2S_COMM_FORMAT_STAND_I2S,
     .intr_alloc_flags = 0,
      .dma_buf_count = 16,
      .dma_buf_len = 64,
      .use_apll = true,
      .tx_desc_auto_clear = false,
      .fixed_mclk = 0
    };

  // Installing and configuring the I2S driver for microphone input
  i2s_driver_install(I2S_NUM_0, &i2s_config_mic, 0, NULL);
  i2s_pin_config_t pin_config_mic = {
    .bck_io_num = I2S_MIC_SCK,
    .ws_io_num = I2S_MIC_WS,
    .data_out_num = -1, // Not used
    .data_in_num = I2S_MIC_SD
    };
    i2s_set_pin(I2S_NUM_0, &pin_config_mic);
}

// DAC Audio Output
void initAudioOutput() {
  // I2S configuration for audio output
  const i2s_config_t i2s_config_op = {
    .mode = (i2s_mode_t)(I2S_MODE_MASTER | I2S_MODE_TX),
    .sample_rate = 44100,
    .bits_per_sample = I2S_BITS_PER_SAMPLE_16BIT,
    .channel_format = I2S_CHANNEL_FMT_RIGHT_LEFT,
    .communication_format = I2S_COMM_FORMAT_STAND_I2S,
    .intr_alloc_flags = 0,
    .dma_buf_count = 16,
    .dma_buf_len = 64,
    .use_apll = true,
    .tx_desc_auto_clear = true,
    .fixed_mclk = 0
  };
  
  // Installing and configuring the I2S driver for audio output
  i2s_driver_install(I2S_NUM_1, &i2s_config_op, 0, NULL);
  i2s_pin_config_t pin_config_op = {
    .bck_io_num = I2S_BCK_PIN,
    .ws_io_num = I2S_LRCK_PIN,
    .data_out_num = I2S_DATA_PIN,
    .data_in_num = -1 // Not used
  };
  i2s_set_pin(I2S_NUM_1, &pin_config_op);
}

The audio quality produced by my code is not good; it has a consistently higher pitch and exhibits static/clipping distortion, which is unusable for my application. The only benefit is that the audio is sent to the DAC instantaneously, without the small delay I notice from the streams-i2s-i2s-2 code. I have confirmed that the microphone, DAC, and Bluetooth module all work perfectly when tested separately, which leads me to believe the issue is likely due to how I'm integrating everything in my code.

Integration and Library Use
Since ESP has two I2S ports/4 channels (https://www.pschatzmann.ch/home/2024/03/20/i2s-output-of-4-channels-on-a-esp32/), incorporating two input signals (microphone/Bluetooth), and an output signal to the DAC shouldn’t be an issue. For Bluetooth audio input, I have been able to use this code separately (https://github.com/Freenove/Freenove_Ultimate_Starter_Kit_for_ESP32/blob/master/C/Sketches/Sketch_29.1_BluetoothByPCM5102A_V2.0.0_or_Later/Sketch_29.1_BluetoothByPCM5102A_V2.0.0_or_Later.ino). Due to the custom nature and size of the audio tools library, I am finding it challenging to navigate and modify it to meet the specific needs of my project. My goal is to transition completely to this audio tools library, moving away from my current separate code implementations for microphone input and Bluetooth functionality (if possible, I am aware of Phil's Bluetooth A2DP library but have yet to try it). I am interested in leveraging the full potential of the library for my noise-cancellation project but am seeking guidance on customizing its functions and integrating its features effectively.

Specific Questions
Given my current situation and the challenges I'm facing with integrating the various aspects of my project, I have some questions that I hope to gain clarity on:

1. Library Modification Advice: Given the complexity of the audio tools library, are there simplified approaches or specific sections within the library that I should focus on to meet my project's needs? I have looked into "AudioConfig.h" "I2SConfigStd.h", "StreamCopy.h", etc. but would really appreciate some more direction.

2. Customizing Audio Tools Library Functions: In the audio tools library, the example streams-i2s-i2s-2 and streams-i2s-filter-i2s use a copier.copy() function for data transfer, which works well but doesn't allow for easy manipulation of the data being copied. How can I modify or extend this functionality to apply an adaptive FIR filter with an LMS algorithm for noise cancellation? I am aware that in streams-i2s-filter-i2s, I could adjust the FIR filter coefficients and perhaps begin with implementing a low pass filter to the output for proof of concept, and apply the LMS algorithm afterwards once passing the output audio through a filter works.

3. Integration Advice: Given the complexity of integrating microphone input, Bluetooth audio, and DAC output, as well as incorporating an adaptive filtering technique to modify the output audio data, can any ideas on best practices or general advice be given for achieving successful integration using the audio tools library?

Any thoughts or comments would be greatly appreciated. Also, please let me know if there is any other information or details I may have left out that would help to provide a more accurate resolution to the issue at hand, I know there is a lot going on here.

Thanks!

[pschatzmann]

I suggest that you read the chapter about 'using Filters' in the Wiki and maybe read the 'The most important basics' as a precondition, which also answers some of your questions.

In order to decrease the delay you can decrease the I2S buffer size and count settings. You can use the same values that you used above!

The INMP441 is tricky to use, so don't ignore the advice given in the External 'ADCs and Microphones' chapter

One of the goals of my library was really, that you can do exactly the same, like in your example, but with much fewer lines of code: But the called functionality is the same.

[cameronmattie]

Hey Phil,

I appreciate the response, those resources proved to be very helpful. I was able to design low pass, high pass and band pass filters using the site you suggested and modify your code to filter the audio coming from the INMP441 before it’s sent to the DAC. I had just a few more questions I was hoping to pick your brain about.

is it possible to add bluetooth to stream-i2s-i2s-2, or more importantly, stream-i2s-filter-i2s? I would like to combine the microphone audio and bluetooth, and send it the the DAC. I started playing around with your Bluetooth-A2DP library, using the example with Bluetooth and a sine wave:

#include "AudioTools.h"
#include "BluetoothA2DPSink.h"

AudioInfo info(44100, 2, 16);
BluetoothA2DPSink a2dp_sink;
I2SStream i2s;
SineWaveGenerator<int16_t> sineWave(10000);  // max amplitude of 10000
GeneratedSoundStream<int16_t> sound(sineWave); 
OutputMixer<int16_t> mixer(i2s, 2);  
const int buffer_size = 160; 
uint8_t sound_buffer[buffer_size];

// Write data to mixer
void read_data_stream(const uint8_t *data, uint32_t length) {
  // write a2dp
  mixer.write(data, length);

  // write sine tone with identical length
  sound.readBytes(sound_buffer, length);
  mixer.write(sound_buffer, length);
}

void setup() {
  Serial.begin(115200);
  AudioLogger::instance().begin(Serial, AudioLogger::Warning);

  // setup Output mixer with min necessary memory
  mixer.begin(buffer_size);

  // Register data callback
  a2dp_sink.set_stream_reader(read_data_stream, false);

  // Start Bluetooth Audio Receiver
  a2dp_sink.set_auto_reconnect(false);
  a2dp_sink.start("a2dp-i2s");

  // Update sample rate
  info.sample_rate = a2dp_sink.sample_rate();

  // start sine wave
  sineWave.begin(info, N_B4);

  // start I2S out (THE PCM DAC)
  auto config_out = i2s.defaultConfig(TX_MODE);
  config_out.copyFrom(info); 
  config_out.i2s_format = I2S_STD_FORMAT;
  config_out.is_master = true;
  config_out.port_no = 1;
  config_out.pin_bck = 26; 
  config_out.pin_ws = 25;  //THE LCK PIN OF THE PCM DAC
  config_out.pin_data = 1;
  // GROUND THE SCK PIN OF THE PCM DAC
  i2s.begin(config_out);
}

void loop() { delay(100); }

I received the following error, which I have never seen before:

Sketch uses 1321457 bytes (100%) of program storage space. Maximum is 1310720 bytes.
Global variables use 46844 bytes (14%) of dynamic memory, leaving 280836 bytes for local variables. Maximum is 327680 bytes.
Sketch too big; see https://support.arduino.cc/hc/en-us/articles/360013825179 for tips on reducing it.
text section exceeds available space in board
Error compiling for board ESP32 Dev Module.

I do not intend on using this sine wave at all if I can get bluetooth as an i2s input, but have been generally struggling with combining the BluetoothA2DPsink and AudioTools libraries. What I would like is to be able to pass both the bluetooth audio and the microphone as inputs in streams-i2s-filter-i2s:

/**
 * @file streams-i2s-filter-i2s.ino
 * @brief Copy audio from I2S to I2S using an FIR filter
 * @author Phil Schatzmann
 * @copyright GPLv3
 */

#include "AudioTools.h"
 
AudioInfo info(44100, 2, 16);
I2SStream in;
I2SStream out; 

// copy filtered values
FilteredStream<int16_t, float> filtered(in, info.channels);  // Defiles the filter as BaseConverter
StreamCopy copier(out, filtered); // copies sound into i2s

// Low-pass, cutoff = 4kHz, bw = 2kHz
float coef[] = {-0.019375562476475785,
-0.035695650279056322,
-0.043402188283344051,
-0.037631549696442529,
-0.016059747368213580,
0.020232440398164245,
0.066676535023620767,
0.115981970772203791,
0.159601063866087578,
0.189560519361085283,
0.200224337364741239,
0.189560519361085283,
0.159601063866087578,
0.115981970772203791,
0.066676535023620767,
0.020232440398164245,
-0.016059747368213580,
-0.037631549696442529,
-0.043402188283344051,
-0.035695650279056322,
-0.019375562476475785};

// Arduino Setup
void setup(void) {  
  // Open Serial 
  Serial.begin(115200);
  // change to Warning to improve the quality
  AudioLogger::instance().begin(Serial, AudioLogger::Info); 

  // setup filters for all available channels
  filtered.setFilter(0, new FIR<float>(coef));
  filtered.setFilter(1, new FIR<float>(coef));

  // start I2S in
  Serial.println("starting I2S...");
  auto config_in = in.defaultConfig(RX_MODE);
  config_in.copyFrom(info); 
  config_in.i2s_format = I2S_STD_FORMAT;
  config_in.is_master = true;
  config_in.port_no = 0;
  config_in.pin_ws = 19;
  config_in.pin_bck = 18;
  config_in.pin_data = 3;
  // config_in.fixed_mclk = sample_rate * 256
  // config_in.pin_mck = 3

  in.begin(config_in);

  // start I2S out
  auto config_out = out.defaultConfig(TX_MODE);
  config_out.copyFrom(info); 
  config_out.i2s_format = I2S_STD_FORMAT;
  config_out.is_master = true;
  config_out.port_no = 1;
  config_out.pin_ws = 34;
  config_out.pin_bck = 35;
  config_out.pin_data = 1;
  out.begin(config_out);

  Serial.println("I2S started...");
}

// Arduino loop - copy sound to out 
void loop() {
  copier.copy();
}

Next, apply the standard FIR filter to the microphone alone, then pass the combination of bluetooth audio and microphone audio as output. If this worked, then try to implement the LMS algorithm either within the .ino file or somewhere within the header files to update the coefficients as the input signal and noise change. Do you think that it is possible to update the filter coefficients, and if so, where could this be done? I investigated AudioStreams.h, and the code you used from here (https://github.com/sebnil/FIR-filter-Arduino-Library/tree/master) to compare it with what you have in filter.h, but have been struggling to understand what you have done here to implement the filter and how I could adjust it:

 * @brief FIR Filter
 * Converted from
 * https://github.com/sebnil/FIR-filter-Arduino-Library/tree/master/src
 * You can use https://www.arc.id.au/FilterDesign.html to design the filter
 * @ingroup filter
 * @author Pieter P tttapa  / pschatzmann
 * @copyright GNU General Public License v3.0
 */
template <typename T>
class FIR : public Filter<T> {
  public:
    template  <size_t B>
    FIR(const T (&b)[B], const T factor=1.0) : lenB(B), factor(factor) {
      setValues(b);
    }

    template  <size_t B>
    void setValues(const T (&b)[B]){
      if (x!=nullptr) delete x;
      x = new T[lenB]();
      coeff_b = new T[2*lenB-1];
      for (uint16_t i = 0; i < 2*lenB-1; i++) {
        coeff_b[i] = b[(2*lenB - 1 - i)%lenB];
      } 

    }
~FIR() {
      delete[] x;
      delete[] coeff_b;
    }
    T process(T value) {
      x[i_b] = value;
      T b_terms = 0;
      T *b_shift = &coeff_b[lenB - i_b - 1];
      for (uint8_t i = 0; i < lenB; i++) {
        b_terms += b_shift[i] * x[i] ; 
      }
      i_b++;
      if(i_b == lenB)
        i_b = 0;

#ifdef USE_TYPETRAITS
      if (!(std::is_same<T, float>::value || std::is_same<T, double>::value)) {
        b_terms = b_terms / factor;
      }
#else
      if (factor!=1.0) {
        b_terms = b_terms / factor;
      }
#endif
      return b_terms;
    }
  private:
    const uint8_t lenB;
    uint8_t i_b = 0;
    T *x = nullptr;
    T *coeff_b;
    T factor;
};

To summarize:

1. Bluetooth Integration: I'm exploring how to integrate Bluetooth audio alongside microphone input using your Bluetooth-A2DP library and would appreciate any insights on combining these inputs for output through the DAC in the context of stream-i2s-i2s-2 or stream-i2s-filter-i2s.

2. Dynamic Filter Coefficient Updates: My aim is to apply a standard FIR filter to microphone input and then explore the possibility of dynamically updating filter coefficients, potentially using the LMS algorithm, to refine noise cancellation as input signals and noise levels change.

3. Filter Implementation Insights: Lastly, I'm keen to better understand your FIR filter implementation and how it might be adjusted or extended for dynamic coefficient updates.

Any suggestions on how to approach this within your framework would be very much appreciated, and again if there is anything I can clarify further please let me know.

Thank you!!

[pschatzmann]

1. Read the chapter about mixing and then just build a chain using A2DPStream like in this example on the input side, or better in the callback just write to a FilteredStream with i2s as output.

2. Just use the setValues method on the Filter to update your filter array or use the setFilter method on the FileteredStream if you prefer to have predefined different alternative Filters

3. The setValues method is already available

[cameronmattie]

Hey Phil,

Thanks again for the resources, I have been able to successfully test the streams-a2dp-serial on my board. Also, I have tried to implement the mixer, first by going Bluetooth to i2s instead of Bluetooth to Serial, then mixing it with microphone input. I have also tried out the basic api examples, basic-a2dp-i2s and basic-a2dp-mixer-i2s, however I don’t see the board available to connect to. Essentially, I am trying to combine the streams-i2s-filter-i2s with streams-a2dp-serial and for the mixer, streams-generator_inputmixer-audiokit, as seen in the code below. Currently, I have StreamCopy copier(out, inBT) to test the Bluetooth output, and eventually plan to replace inBT with mixer. However, the audio heard from inBT is shuttering, and I’m not exactly sure where I’m going wrong here. I also see that the mixer example combines two generated sine waves, not two separate i2s streams. Any thoughts are much appreciated.

#include "AudioTools.h"
#include "AudioLibs/AudioA2DP.h"
 
AudioInfo info(44100, 2, 16);
I2SStream in;
I2SStream out; 
A2DPStream inBT;
InputMixer<int16_t> mixer;

// copy filtered values
FilteredStream<int16_t, float> filtered(in, info.channels);  // Defiles the filter as BaseConverter
StreamCopy copier(out, inBT); // copies sound into i2s filtered

// define FIR filter parameters, unused right now
float coef[] = { 0.021, 0.096, 0.146, 0.096, 0.021};

// Arduino Setup
void setup(void) {  
  // Open Serial 
  Serial.begin(115200);
  // change to Warning to improve the quality
  AudioLogger::instance().begin(Serial, AudioLogger::Info); 
  
  // start the bluetooth audio receiver
  Serial.println("starting A2DP...");
  auto cfg = inBT.defaultConfig(RX_MODE);
  cfg.name = "MyReceiver";
  inBT.begin(cfg); 
  
  // setup filters for all available channels
  filtered.setFilter(0, new FIR<float>(coef));
  filtered.setFilter(1, new FIR<float>(coef));

  // start I2S in
  Serial.println("starting I2S...");
  auto config_in = in.defaultConfig(RX_MODE);
  config_in.copyFrom(info); 
  config_in.i2s_format = I2S_STD_FORMAT;
  config_in.is_master = true;
  config_in.port_no = 0;
  config_in.pin_ws = 23;
  config_in.pin_bck = 22;
  config_in.pin_data = 3;
  // config_in.fixed_mclk = sample_rate * 256
  // config_in.pin_mck = 3

  in.begin(config_in);

  // start I2S out
  auto config_out = out.defaultConfig(TX_MODE);
  config_out.copyFrom(info); 
  config_out.i2s_format = I2S_STD_FORMAT;
  config_out.is_master = true;
  config_out.port_no = 1;
  config_out.pin_ws = 25;
  config_out.pin_bck = 26;
  config_out.pin_data = 1;
  out.begin(config_out);

  Serial.println("I2S started...");

  mixer.add(filtered);
  mixer.add(inBT);
  mixer.begin(info);

  Serial.println("started...");
}

// Arduino loop - copy sound to out 
void loop() {
  copier.copy();
}