StreamCopy for multy clients

[josef2600]

so i have changed the StreamCopy.h codes to a new file StreamCopymulty.h . the purpose is this:
StreamCopy copier(outTimed, client);
to

#define MAX_CLIENTS 3
WiFiClient my_clients[MAX_CLIENTS];
StreamCopy3multi copiers(decoders,my_clients[0],my_clients[1],my_clients[2],DEFAULT_BUFFER_SIZE);

i have created a new file StreamCopymulty.h so i wouldn't change anything in the source code. this is the file:
10.zip
in it i created a new class class StreamCopy3multi : public StreamCopyTmulti<uint8_t> . and my basic thoughts where to read from 3 clients and put them all in the main buffer that sends the data to output to be decoded and played.
i check all 3 clients and if they have buffer, i read them and send them to buffer. but the problem was it wasn't playing anything unless it receives 3 clients, then it plays only the 3th one and doesn't play the others. so i tried to always read 3 times, like, if there is only 1 client, i read that 3 times, but to no success. this is how i did it:

            if ((bytes_to_read > 0))
            {
              result = from->readBytes((uint8_t*)&buffer[0], bytes_to_read);
            }
            else if ((bytes_to_read2 > 0)) result += from2->readBytes((uint8_t*)&buffer[result], bytes_to_read2);
            else if ((bytes_to_read1 > 0)) result += from1->readBytes((uint8_t*)&buffer[result], bytes_to_read1);
                  
            //Serial.println("238");
            if ((bytes_to_read1 > 0))
            {
              result += from1->readBytes((uint8_t*)&buffer[result], bytes_to_read1);
            }
            else if ((bytes_to_read > 0)) result += from->readBytes((uint8_t*)&buffer[result], bytes_to_read);
            else if ((bytes_to_read2 > 0)) result += from2->readBytes((uint8_t*)&buffer[result], bytes_to_read2);
                  
            //Serial.println("240");
            if ((bytes_to_read2 > 0))
            {
              result += from2->readBytes((uint8_t*)&buffer[result], bytes_to_read2);
            }
            else if ((bytes_to_read > 0)) result += from->readBytes((uint8_t*)&buffer[result], bytes_to_read);
            else if ((bytes_to_read1 > 0)) result += from1->readBytes((uint8_t*)&buffer[result], bytes_to_read1);

any thing result to only 1 stream playing and only the last one. even if i do this way for all of the code :

        void begin(Print &to, AudioStream &from)
        {
            this->from = &from;
            this->to = &to;
            is_first = true;
            LOGI("buffer_size=%d",buffer_size);    
        }

i do return the size of all 3 reading to coverter_ptr->convert((uint8_t*)buffer.data(), result ); . the result is sum of all bytes readied. and i do change the start address of reading of buffer in the code. eg result += from->readBytes((uint8_t*)&buffer[result], bytes_to_read); . i dont underestand how the code does the decoding, beacause it is too large for me!
can you help? please read the full code i sent as a zip file.
and this is the code i am using in arduino:

#define ARDUINO_LOOP_STACK_SIZE (6 * 1024)

#define DEFAULT_BUFFER_SIZE  256
#define DEFAULT_SAMPLE_RATE 16000 // 44100
#define DEFAULT_CHANNELS 1 // 2
#define DEFAULT_BITS_PER_SAMPLE 16
#define I2S_BUFFER_SIZE DEFAULT_BUFFER_SIZE
#define I2S_BUFFER_COUNT 3  //6
#define ANALOG_BUFFER_SIZE 256 // 1024
#define ANALOG_BUFFER_COUNT 2 // 6
#define CODEC_DELAY_MS 1 // 10
#define COPY_DELAY_ON_NODATA 5

//#define OPUS_DEC_MAX_BUFFER_SIZE 20*1024  //4*1024

#include <Arduino.h>

#include "AudioTools.h"
#include "AudioTools/StreamCopymulty.h"
#include "AudioCodecs/CodecMP3Helix.h"

#define MAX_CLIENTS 3

#define my_codec_is_new   MP3DecoderHelix()

#define my_pin_pwm1_is 25
#define my_pin_pwm2_is 26


const char *mssid = "xxx";
const char *mpassword = "xxx";
uint16_t me_port = 7000;
WiFiServer server(me_port);

WiFiClient my_clients[MAX_CLIENTS];

AudioInfo info(8000, 1, 16);

PWMAudioOutput pwm1;
InputMixer<int16_t> mixer;

//EncodedAudioStream* decoders[MAX_CLIENTS];
//StreamCopy* copiers[MAX_CLIENTS];

EncodedAudioStream decoders(&pwm1, new my_codec_is_new);
StreamCopy3multi copiers(decoders,my_clients[0],my_clients[1],my_clients[2],DEFAULT_BUFFER_SIZE);





unsigned long lastCheckedTime = 0;

bool is_this_client_connected[MAX_CLIENTS+2] = {};



void connectWifi()
{
  // connect to WIFI
  WiFi.begin(mssid, mpassword);
  while (WiFi.status() != WL_CONNECTED)
  {
      delay(500);
      Serial.print(".");
  }
  Serial.println();
  Serial.println(WiFi. localIP());

  // Performance Hack              
  //esp_wifi_set_ps(WIFI_PS_NONE);
  WiFi.setSleep(false);
}


void setup()
{
  Serial.begin(230400);
  delay(500);
  AudioLogger::instance().begin(Serial, AudioLogger::Info);//Warning  Info
  delay(100);

  connectWifi();
  Serial.println("");

  Serial.print("IP address: ");
  Serial.println(WiFi.localIP());
  server.begin();

  // setup pwm1 output
  auto config1 = pwm1.defaultConfig();
  config1.copyFrom(info);
  config1.resolution = 8;  // must be between 8 and 11 -> drives pwm frequency (8 is default)
  config1.start_pin = my_pin_pwm1_is;
  
  pwm1.begin(config1);
  decoders.begin();
  
  //pwm2.begin(config2);

  //for (int i = 0; i < MAX_CLIENTS; i++)
  //{
  //  decoders[i] = new EncodedAudioStream(&pwm1, new my_code_is_new);
   // copiers[i] = new StreamCopy(*decoders[i], my_clients[i]);
   // decoders[i]->begin();
    //mixer.add(*decoders[i]);

  //}
  //mixer.begin(info);
}

void loop()
{
  WiFiClient newClient = server.available();
  if (newClient)
  {
    for (uint8_t i = 0; i < MAX_CLIENTS+1; i++)
    {
      if (!(my_clients[i]) || !(my_clients[i].connected()) || (i >= MAX_CLIENTS))
      {
        if (i >= MAX_CLIENTS) 
        {
          i = 0;
          my_clients[i].stop();
        }
        if (my_clients[i]) my_clients[i].stop();
        is_this_client_connected[i] = 1;
        my_clients[i] = newClient;
        break;
      }
    }
  }

  for (uint8_t i = 0; i < MAX_CLIENTS; i++)
  {
    if (my_clients[i] && !(my_clients[i].connected()))
    {
      my_clients[i].stop();
      is_this_client_connected[i] = 0;
      //mixer.end(*decoders[i]);
      //delete decoders[i];
      //decoders[i] = new EncodedAudioStream(&me_I2S, new my_code_is_new);
      //decoders[i]->begin();
      //mixer.add(*decoders[i]);
    }
  }

  for (uint8_t i = 0; i < MAX_CLIENTS; i++)
  {
    if (my_clients[i] && (my_clients[i].connected()))
    {
      copiers.copy();
    }
  }
  
  //copiers.copy();
  delay(1);
}

oh, and i am using PWM for output. the quality for me doesn't matter. i just need to play 3 clients at the same time. and all the audio's are the same mp3 codec and info. like i am sending 1 mp3 file in 3 different times to it.
i did everything i could fined, but to no success. as i understand, the reason is how the decoder code works. it is using delay_ms in it and when a data is sent to it, it is waiting for timing with delay_ms and that causes the jittery audio when i create multiple decoders or creating multi input or output. it has to read at least 1ms of data to play it then goes to the next, and it will cause the jittery, it is like the audio only plays 1ms of 1th then 1ms of 2th then 1ms of 3th audio and by this point, nothing can be heard! it can be fixed if instead of using delay, you used timers for audio timing. but i cannot do that and i think it is very hard for you too, so i propose the use of multi stream copy so we send 1 total buffer for decoding.

thanks.

[pschatzmann]

Read the chaper about mixing! There are not enough resources for multiple mp3. So you will need to replace the mp3 with something more efficient: wav will work great...

Subsequent mp3 segments depend on their predecessors, so you can't just do what you're doing.

[josef2600]

if you check the Arduino code, i have don it too. the mixer didn't do it ether. it was jittery too.
about the wav, i don't know why i cant use it correctly. i am using curl to send the file like:
curl -X POST -H "Content-Type: audio/L16" --data-binary @D:\k\22.wav http://192.168.1.10:7000/audio
or:
curl -X POST -H "Content-Type: audio/wav" --data-binary @D:\k\14.wav http://192.168.1.10:7000/audio
but none of them are good. sometimes it gets the audio, next time it plays only noise. i don't know why. like this: i send it, it plays noise, i close curl window and open it again and send it again, and it works. i cant figure it out what is my problem. that is why i haven't worked with wav. technically, except from mp3 (and wav), i cant use any other decoders. all of them report errors.
did you fined out what is the problem with StreamCopy3multi ? why is it need all 3 clients to be connected and then it plays only 1 of them?
if not, i think i have don enough time for it. thank you. and get better.

[pschatzmann]

You could try to change the architecture:
Solution 1: post an aac file with 3 channels
Solution 2: use htttp post just to submit the command and not the audio data. Get the data to be mixed via an urlstream

[SiWoC]

I don't mean to hijack this post, but Solution 2 sounds like something I was thinking to use to solve my "ESP32-S3 only has BLE" problem.

• The client would send: "get my MP3 on this url http://192.168.1.1/comeAndGetIt" over either BLE or WiFi (webserver?)
• the ESP32 would use an urlStream to get data from the client and play it

that's what you're suggesting right?

[pschatzmann]

Right: ideally it could be just a protocol to drive the audioplayer

[josef2600]

please hijack away! i love that.
but i think you need to understand how multi receiving needs to work. i am going to tell every one who is interested in this, what i understand so far. please feel free to object or right me in any way you think is correct.
first i like you to know i am a weird person, so don't be shocked if i go to a wrong way or what ever! i am one of the old programmers that started programming in 90's when we worked with MS.DOS ! now day's, no body knows what a floppy disk is or was! i started with qbasic then went to C. i love C. mainly probably because i am 95% hardware programmer (micro controller - microprocessor). as i know, i have worked with all mainstream controller, but my first love was MC8051. so much fun memory with that. although, i started with PIC, i was working with 8051 and AVR too. but assembly language was something else! then i went to arm and stm a little, but i didn't like them much, so i went all the way in to pic32. love the MIPS architecture. just love it. many years ago, i left AVR then ARM and solely stuck to PIC. about 5 years ago, i found ESP and i got very interested! then i started C++ for ESP32. i am always fighting with coders about using c++. not to be disrespect to any one, its just the way i think. if you can use C, you should never use C++ instructions. i know i know C++ is mandatory, that is not what i said! and always know your architecture. just because you can use a code, doesn't mean you should. it is very dangerous and sluggish. the reason for using odd codes is the majority of coders do not have hardware knowledge. usually they are from PC and want to do the same to a RISK architecture. that is very very bad. for example, ARM by default, doesn't even has divide instruction. no floating point at all. but coders want to use 1000 instruction that x86 has! it's ARM , it only has 47 instruction.
any way, that is why i always get in fight with coders.
now the main problem is timing. either for decoder or coder or network. then we have a problem with mixing. but the biggest problem would be with the decoder (in this code). the main reason is i cant understand any of the source code. it is copied from Linux driver and it is very complex and it is not wise to change it if you don't know everything about it. and i, personally, have had enough of it! the arduino-audio-tools is a very good and strong library, i can read and understand most of the source code, but it just uses the decoder from someone else and that is were it goes wrong. pschatzmann has converted the main code to be usable in here, but he didn't write that.
but my solution would be 3 things:

1. leave it the hell alone!
2. use 3 or more ESP32 for it. that will causes lot of eeeeeeee
3. the crazy way! go and write everything from 0 ! inventing the wheel again!

as for now, all of this is too crazy for me. maybe if i cool off i will return.
but i always check to see what is going on in this library, at least until i get too bored!
tanks, best regards, Josef.