Grabbing audio (from inmp441) at specific times

[LockManipulator]

I am using an esp32S3 (specifically the lilygo-t-display-s3) with an inmp441 microphone and coding in the latest version of the Arduino IDE with the latest version of your library on Ubuntu 22.04.

Apologies if this is a dumb question, I've looked over the example codes and wiki and could not figure this out. It seems like audio data from an i2s microphone is constant and the examples for streaming the data somewhere are continous and do not have a way to start/stop the collection of data at specific times. My goal is to start an action (in my case spinning a stepper motor) and get the audio of when that action starts to when it ends. The time is usually quite short, ~0.25 seconds depnding on how fast the motor is spinning, and I require the audio to be in sync with the start/end of the action. Although it does not matter if I have extra data before or after, as long as each recording is in sync (e.g. if the same sound is produced at exactly halfway through the motor movement then each audio sample should show the sound at the same place each time). Example pseudo code to describe what I'm trying to do:

loop () {
if (start) {
start = false;
spinning = true;
spin_motor();
spinning = false;
}
}
// Save the audio collected between spinning = true and spinning = false to perform further analysis. The "spin_motor()" function IS blocking so I'm not sure if that would cause an issue with this library.

It does not matter whether I grab samples based on time or number of samples (I do always know exactly how long the motor will spin for so either grabbing all audio from the past x seconds or most recent x number of samples would be fine). Your example code for streaming to serial to show on the serial plotter works wonderfully and I have confirmed that it works. I have previously tried without this library and pass a semaphore to a secondary record task to record only when I want using i2s_read(). But it seems like the DMA buffer constantly gets audio from the microphone even without calling i2s_read(). So when I call i2s_read() at non regular intervals I get audio data that contains garbage/repeat audio data.

[pschatzmann]

https://github.com/pschatzmann/arduino-audio-tools/wiki/Introduction#additional-processing-in-the-loop

Redesign your spin_motor() method to make it non blocking or try to run things in parallel with a task

[LockManipulator]

Thanks for the response! I have things running in parallel with a task and getting an error from StreamCopy.h somewhere in the record function. My motor will move left (left(6) in test() function is the last line to run) and then the error gets printed and all execution is stopped.

[E] StreamCopy.h : 355 - write to target has failed after 22 retries! (1024 bytes)

This is my task that records

void record(void *arg) {
  auto cfg = i2sStream.defaultConfig(RX_MODE);
  cfg.i2s_format = I2S_STD_FORMAT; // optional because default setting
  cfg.bits_per_sample = 32;
  cfg.channels = 1; // optional because default setting
  cfg.sample_rate = 8000;
  cfg.is_master = true; // optional because default setting
  cfg.pin_bck = 1;
  cfg.pin_ws = 2;
  cfg.pin_data = 3;
  while(1) {
    xSemaphoreTake(i2sSemaphore, portMAX_DELAY); // Signal to start
    i2sStream.begin(cfg);
    while (1) {
      if (!copier.copy()){
        i2sStream.end();
        stop();
        break;
      }
    }
    xSemaphoreGive(i2sSemaphore);  // Done recording
    vTaskDelay(10); // Allow time for semaphore to be taken back just in case but recording always takes longer than spinning the motor anyways
  }
}

And I'm starting to record with:

void test() {
  Rrotate(3);
  Rgoto(5);
  Lgoto(0);

  speed = cp_speed;

  // Repeat motor movements 5x to check for consistency in timing of audio
  for (int x = 0; x < 5; x++) {
    vTaskDelay(200); // Wait for noise to quiet down
    xSemaphoreGive(i2sSemaphore); // Signal to start recording
    left(6); // Move motor left
    xSemaphoreTake(i2sSemaphore, portMAX_DELAY); // Take back semaphore until ready to record again
    right(6); // Reset motor position
    auto *pdata = (int32_t *)&raw_data[0];
    for (int x = 0; x < SAMPLE_COUNT; x++) {
      Serial.printf("%ld,", pdata[x]);
    }
    Serial.println("");
  }
  speed = reg_speed;
}

These are my variables:

#define SAMPLE_COUNT 2048
#define SAMPLE_BUFFER_SIZE sizeof(int32_t) * SAMPLE_COUNT
unsigned char raw_data[SAMPLE_BUFFER_SIZE];
AudioInfo info(8000, 1, 32);
I2SStream i2sStream; // Access I2S as stream
MemoryStream raw_samples(raw_data, SAMPLE_BUFFER_SIZE, true, PS_RAM);
StreamCopy copier(raw_samples, i2sStream); // copies sound

[pschatzmann]

I think you forgot to open the output with begin() or you try to write more data then you have reserved.
If you want to use PSRAM just use
MemoryStream raw_samples(SAMPLE_BUFFER_SIZE, PS_RAM);
or better use a DynamicMemoryStream

[LockManipulator]

I see now, silly mistake by me sorry. I fixed that and added

raw_samples.begin();

in my record function so it's now:

void record(void *arg) {
  auto cfg = i2sStream.defaultConfig(RX_MODE);
  cfg.i2s_format = I2S_STD_FORMAT; // optional because default setting
  cfg.bits_per_sample = 32;
  cfg.channels = 1; // optional because default setting
  cfg.sample_rate = 8000;
  cfg.is_master = true; // optional because default setting
  cfg.pin_bck = 1;
  cfg.pin_ws = 2;
  cfg.pin_data = 3;
  while(1) {
    lockVariable();
    raw_samples.begin();
    i2sStream.begin(cfg);
    while (1) {
      if (!copier.copy()){
        i2sStream.end();
        stop();
        break;
      }
    }
    unlockVariable();
    vTaskDelay(10);
  }
}

However the output error and results have not changed whether with or without PS_RAM. I tried "MemoryStream raw_samples(raw_data, SAMPLE_BUFFER_SIZE, true, PS_RAM);" not really knowing what it does, I simply forgot to take that out before posting. It sholud be just "MemoryStream raw_samples(raw_data, SAMPLE_BUFFER_SIZE);".

The part about trying to write more data than I have reserved sounds likely. Is it possible to see how much data I'm trying to write? I created variables to hold 2048 samples, does the write not stop after filling what it's supposed to?

[pschatzmann]

Did you activate PSRAM in the Arduino Menu ?
I am also not sure about your logic: calling stop() just leads to an endess loop, so your unlockVariable is never reached.
I suggest you make a simple test case with just writing and reading

If you double check in the code a write is failing

• when object has not been opened
• memory type is flash
• when it is at the end of memory

You can always call available() to check how many bytes have been written and availableForWrite() to check how many bytes you can still write.
When setting the log level to info you should see each 1024 bytes being written

Here is an the example that shows how to record and playback sound

[pschatzmann]

Actually, there was a bug in the calculation of the number of written bytes. I committed a correction and now your initial logic should work. Here is the test case:

#include "AudioTools.h"

#define SAMPLE_COUNT 2048
#define SAMPLE_BUFFER_SIZE sizeof(int32_t) * SAMPLE_COUNT
unsigned char raw_data[SAMPLE_BUFFER_SIZE];
AudioInfo info(8000, 1, 32);
I2SStream i2sStream;  // Access I2S as stream
MemoryStream raw_samples(raw_data, SAMPLE_BUFFER_SIZE, true, RAM);
StreamCopy copier(raw_samples, i2sStream, 1024);  // copies sound

void setup() {
  Serial.begin(115200);
  AudioLogger::instance().begin(Serial, AudioLogger::Info);

  auto cfg = i2sStream.defaultConfig(RX_MODE);
  cfg.copyFrom(info);
  i2sStream.begin(cfg);

  raw_samples.begin();

  copier.copy();

  assert(raw_samples.available() == 1024);
  assert(raw_samples.availableForWrite() == 7168);

}

void loop() {
  // put your main code here, to run repeatedly:
}

[LockManipulator]

Thank you! That update fixed my issue. I appreciate the help and support!