Using MultiOutput from SD to Kit and FFT-LED

[jaapnoordzij]

In my idea it is a trivial thing to do: combine several examples to stream from SD file to AudioKit and FFT Led. I think I need decoded MP3 for both sinks so I have the following setup:

                            | -> Kit
SD -> Helix -> MultiOutput -|
                            | -> FFT -> FFTDis -> Led

However I cannot get this to work as intended.

• I have confirmed that the hardware for the 32 x 8 WS2812B is working fine with the example sketch using the kit microphone and FFTLED output.

• When I add only the "kit" to the MultiOutput is works fine. I hear the sound.

• When I add only a "csv" to the MultiOutput is works fine (as seen on the plotter)

• When I add both "kit" and "csv" to the MultiOuput I hear only some stuttering fragments.

• When I add only the FFT to the MultiOutput nothing happens. In the serial monitor It looks like the process is stalled on a lock. When I set the loggin level to Debug the last lines are :

[D] LockGuard.h : 70 - virtual void audio_tools::Mutex::lock()
[D] LEDOutput.h : 259 - x: 0, y: 2147483647

I have also tried a SineGenerator as input to the MultiOut but that does not make any difference to the above (Hearing OK, seeing OK, combined None and FFT Nothing).
I have studied the logger output both in Info and Debug settings but could not see a solution.

What am I doing wrong in this sketch ?
Probably some beginner mistake but what ?

Even more interesting to me: What else could I have done to solve this. Do I need to buy a debugger, something like an ESP-PROG or so ?

Thanks for the great libraries and for the fact that most of the examples always work !

/**
 * @file streams-audiokit-fft-led.ino
 * @author Phil Schatzmann
 * @brief  ... read MP3 from SD, stream to AudioKit and display the
 * result on a 32*8 LED matrix
 * @version 0.1
 * @date 2022-10-14
 *
 * @copyright Copyright (c) 2022
 *
 */
#include "AudioTools.h"
#include "AudioLibs/AudioBoardStream.h"
#include "AudioCodecs/CodecMP3Helix.h"
#include "AudioLibs/AudioRealFFT.h"  // or AudioKissFFT
#include "AudioLibs/LEDOutput.h"
#include <SD.h>
#include <SPI.h>

#define PIN_LEDS 22
#define LED_X 32
#define LED_Y 8

const int chipSelect = PIN_AUDIO_KIT_SD_CARD_CS;

AudioInfo info(44100, 2, 16);
File audioFile;
AudioBoardStream kit(AudioKitEs8388V1);   // Audio source SD
CsvOutput<int16_t> csv(Serial);           // Use for plotter testing
MultiOutput mout;
EncodedAudioStream decoder(&mout, new MP3DecoderHelix()); // Decoding stream to multi
AudioRealFFT fft;    // or AudioKissFFT
FFTDisplay fft_dis(fft);
LEDOutput led(fft_dis);       // output to LED matrix

// StreamCopy copier(fft, in);          // Test with Sine input
StreamCopy copier(decoder,audioFile); // Audio to MultiOutput

void setup() {
  Serial.begin(115200);
  AudioLogger::instance().begin(Serial, AudioLogger::Info);

  // Setup MulitOutput
  // mout.add(kit); // if the only one: sound OK
  // mout.add(csv); // if the only one: display OK
  mout.add(fft); // No output
  mout.begin(info);

  // setup Audiokit as input device
  auto kcfg = kit.defaultConfig(TX_MODE);
  kcfg.sd_active = true;
  kcfg.i2s_format = I2S_STD_FORMAT; // or try with I2S_LSB_FORMAT
  kcfg.copyFrom(info);
  kit.begin(kcfg);
  kit.setVolume(0.5);

  // setup file
  SD.begin(chipSelect);
  audioFile = SD.open("/mp3/miss-marple.mp3");

  // setup I2S based on sampling rate provided by decoder
  decoder.begin(info);
  // CSV
  csv.begin(info);

  // Setup FFT output
  auto tcfg = fft.defaultConfig();
  tcfg.length = 1024;
  tcfg.copyFrom(info);
  fft.begin(tcfg);

  // Setup FFT Display
  fft_dis.fft_group_bin = 3;
  fft_dis.fft_start_bin = 0;
  fft_dis.fft_max_magnitude = 40000;
  fft_dis.begin();

  // Setup LED matrix output
  auto lcfg = led.defaultConfig();
  lcfg.x = LED_X;
  lcfg.y = LED_Y;
  led.begin(lcfg);

  // add LEDs
  FastLED.addLeds<WS2812B, PIN_LEDS, GRB>(led.ledData(), led.ledCount());
}

void loop() {
  // update FFT
  copier.copy();
  // update LEDs
  led.update();
}

[pschatzmann]

In audio it is all about processing time and that you provide the data fast enugh to the output.
Any small delay will disturb the audio, so you can't do a Serial output and listen to the music: Serial is just too slow!

I am afraid, a debugger will not help you you at all in this case.

The best way to deal with this is by measuring, doing some calculations and optimizing the bottleneck or moving them out of the audio output chain and processing the slow activities in a separate task.

[jaapnoordzij]

Thank you for the answer and I understand this.

However this does not explain why the FFT / LED matrix does not show anything and locking even when it is the only addition to the MultiOutput. I also tested passing the MulitOutput using the Copier directly from AudioFile to the FFT Stream through the Decoder.

In the Showcase section of the discussion there is someone providing music from a LineInput on the AudioKit through a MultiOutput stream both to Audiokit listening (Earphone) and FFT processing - LED DIsplay (sending the data with ESP-NOW).

[pschatzmann]

Try to find out why fftLEDOutput is not returning: I guess it has something to do with the very big y value

[jaapnoordzij]

My max X = 32 and Y = 8
The very big Y value generated on line 259 in LEDOutput.h comes from the statement:

    int currY = matrix->fftDisplay().getMagnitudeScaled(x, cfg->y);
    LOGD("x: %d, y: %d", x, currY);

This function is defined in FFTDisplay.h:

  int getMagnitudeScaled(int x, int max) {
    return mapFloat(getMagnitude(x), 0, fft_max_magnitude, 0.0f,
                    static_cast<float>(max));
  }

the function mapFloat is returning a float however getMagnitudeScaled returns this value as an int.
Could that be the reason ?
If not I will look further in more detail (I need more experience with this so maybe I'll buy my debugger anyway ;-)

[pschatzmann]

I think the issue is that for your use case you are getting much higher fft values, so you need to adapt the fft_dis.fft_max_magnitude = 40000; in your sketch.
If the values get really that high, it would also make sense to change the data type from int to float.

To limit the impact of this, I have changed the following

  int getMagnitudeScaled(int x, int max) {
    int result = mapFloat(getMagnitude(x), 0, fft_max_magnitude, 0.0f,
                    static_cast<float>(max));
    if (result > max){
      LOGD("fft_max_magnitude too small: current value is %f", getMagnitude(x))
    }                
    return min(result, max); 
  }

[jaapnoordzij]

You are truly great Phil, thanks a lot. Because of the log I can see that the max value for MP3 music from the SD is about 600000 and take that in to account. The effect is that my setup works even with the MultiOutput to I2C and FFT-LED !

[pschatzmann]

Cool