Combining MultiStreamingDecoder and InputMixer?

[z-l-p]

I currently have a signal flow like this working great:

URLStream > MultiStreamingDecoder > FormatConverterStream > VolumeStream > I2SStream

URLStream url;
I2SStream i2s;
VolumeStream vol(i2s);
FormatConverterStream conv(vol);
MultiStreamingDecoder multi; 

(using the MultiStreamingDecoder copier to drive the computation)

But I'd like to add another sound source and swap the VolumeStream for a mixer like this:

URLStream > MultiStreamingDecoder > FormatConverterStream (44100, 2, 16) > InputMixer
                                                                                     > I2SStream
               WhiteNoiseGenerator (44100, 2, 16) > GeneratedSoundStream > InputMixer

My new stream setup looks like this (only showing the structure here to keep it simple):

WhiteNoiseGenerator noiseGenerator(600);
GeneratedSoundStream noise(noiseGenerator);
URLStream url;
I2SStream i2s;
InputMixer<int16_t> mixer;
StreamCopy copier(i2s, mixer);
FormatConverterStream conv(mixer);
MultiStreamingDecoder multi; 

multi.setInput(url);
multi.setOutput(conv);

mixer.add(conv);
mixer.add(noise);  

with copiers like this:

multi.copy(); // decoder
copier.copy(); // mixer

This doesn't look right to me (and causes panics when a URL loads) so I think I'm misunderstanding something fundamental. Is there a way to do this with only 1 copier? Since the InputMixer's inputs aren't explicitly named streams, how can I tell FormatConverterStream to wrap a specific input of InputMixer? Or do I need a queue in between?
(I can put together a simplified test sketch, but in the meantime maybe I could get some feedback on the structure?)

[pschatzmann]

You can only read from an InputMixer: so you should rather use the OutputMixer!
Alternatively you can write to a Queue and use this a source to the InputMixer.

[z-l-p]

I'm trying both options but I can't quite figure out the architecture. I'm sure it's simple, but it's driving me a bit crazy!
I tried this for the outputMixer:

WhiteNoiseGenerator noiseGenerator(600);
GeneratedSoundStream noise(noiseGenerator);
URLStream url;
MultiStreamingDecoder multi;
I2SStream i2s;
OutputMixer<int16_t> mixer(i2s, 2);
FormatConverterStream conv(multi);

StreamCopy copier1(mixer, conv); 
StreamCopy copier2(mixer, noise); 

 multi.copy(); // decoder
 copier1.copy(); // mixer
 copier2.copy(); // mixer

But it doesn't compile:

error: no matching function for call to 'audio_tools::FormatConverterStream::FormatConverterStream(audio_tools::MultiStreamingDecoder&)'
   92 | FormatConverterStream conv(multi);
      |                                 ^

[pschatzmann]

No, you need to make your MultiStreamingDecoder chain output to OutputMixer:

FormatConverterStream conv(mixer);
MultiStreamingDecoder multi(conv); 

so multi.copy() is filling it with the first part. I think this step might do multiple writes to the OutputMixer, so you need to switch the automaitic index management off!

For the second copy part pay attantion that the decoder might fill the OutputMixer in irregular intervalls with some big chunks.
So make sure that the buffer is big enough and check for the amout of data provided by thefirst copy step (described above) with mixer.available(0) and only provide any data if required and the required amount of data with your copyier2 with copyBytes()

// support multipe writes 
mixer.setAutoIndex (false);

// fist copy step
mixer.setIndex(0);
multi.copy();
// second copy step 
mixer.setIndex(1);
auto bytes = mixer.available(0); // check for amount of data from first step 
if (bytes>0) copier2.copyBytes(bytes);
// force output
mixer.flushMixer();

copier1 is not needed!

chain1: URLStream -MultiStreamingDecoder,copy-> FormatConverterStream -> OutputMixer
chain2: GeneratedSoundStream - StreamCopy.copy-> OutputMixer

[pschatzmann]

For the InputQueue, just use a QueueStream with a proper Buffer implementation: see Readme.md!
A dynamic buffer implementation might be a good choice if you don't want to manage the buffer size yourself...

[z-l-p]

Thanks for the thorough explanation. I'm getting closer! I'm implementing the OutputMixer technique you described.
(FYI MultiStreamingDecoder multi(conv) won't compile but I already had multi.setInput(url) and multi.setOutput(conv) so I think it does the same thing.)
I think the chain is correct, but it won't decode anything from valid URLs. I get this in the log after the URLStream negotiates the connection:

I] URLStream.h : 105 - ==> http status: 200
[I] StreamingDecoder.h : 264 - copy: success
[I] AudioTypes.h : 124 - MP3DecoderHelix sample_rate: 44100 / channels: 2 / bits_per_sample: 16
[W] AudioOutput.h : 389 - Available Buffer 0 too small 0: requested: 4608 -> increase the buffer size
[W] AudioOutput.h : 389 - Available Buffer 0 too small 0: requested: 4608 -> increase the buffer size
[I] StreamingDecoder.h : 264 - copy: success
[I] StreamingDecoder.h : 264 - copy: success
[I] StreamingDecoder.h : 264 - copy: success
[I] StreamingDecoder.h : 264 - copy: success
[I] StreamingDecoder.h : 264 - copy: success
[I] StreamingDecoder.h : 264 - copy: failure
[I] StreamingDecoder.h : 264 - copy: failure
[I] StreamingDecoder.h : 264 - copy: failure
[I] StreamingDecoder.h : 264 - copy: failure

or sometimes simply

[I] URLStream.h : 105 - ==> http status: 200
[E] HeaderParserMP3.h : 204 - Could not find FrameSync
[I] StreamingDecoder.h : 797 - mime from detector: 
[E] StreamingDecoder.h : 811 - Could not determine mime type

Certainly 0 bytes in the buffer is bad. But which buffer and how to set it? Or is it really just a symptom of something else? If it's not obvious, I can post a simplified code example later today. Thanks for all your help so far.

[z-l-p]

I tried larger mixer buffer sizes (1024 x 8 up to 1024 x 32). The error pattern is similar:
It usually fails on the mime type detection, sometimes it starts the streams but shows the buffer problems. Here's what it looks like with a 1024 * 10: (note the 4 byte increase in the available buffer with each message, until it hits 152 and falls to 0. Maybe a clue?)

[W] AudioOutput.h : 389 - Available Buffer 0 too small 20: requested: 1020 -> increase the buffer size
[E] ResampleStream.h : 191 - write error 0 vs 1020
[W] AudioOutput.h : 389 - Available Buffer 0 too small 20: requested: 156 -> increase the buffer size
[E] ResampleStream.h : 191 - write error 0 vs 156
[I] StreamingDecoder.h : 264 - copy: success
[I] StreamCopy.h : 173 - StreamCopy::copy  1024 -> 1024 -> 1024 bytes - in 1 hops
[W] AudioOutput.h : 389 - Available Buffer 0 too small 24: requested: 1020 -> increase the buffer size
[E] ResampleStream.h : 191 - write error 0 vs 1020
[W] AudioOutput.h : 389 - Available Buffer 0 too small 24: requested: 1020 -> increase the buffer size
[E] ResampleStream.h : 191 - write error 0 vs 1020
[W] AudioOutput.h : 389 - Available Buffer 0 too small 24: requested: 1020 -> increase the buffer size
[E] ResampleStream.h : 191 - write error 0 vs 1020
[W] AudioOutput.h : 389 - Available Buffer 0 too small 24: requested: 152 -> increase the buffer size
[E] ResampleStream.h : 191 - write error 0 vs 152
[I] StreamingDecoder.h : 264 - copy: success
[I] StreamCopy.h : 173 - StreamCopy::copy  1024 -> 1024 -> 1024 bytes - in 1 hops
[W] AudioOutput.h : 389 - Available Buffer 0 too small 28: requested: 1020 -> increase the buffer size
[E] ResampleStream.h : 191 - write error 0 vs 1020
[W] AudioOutput.h : 389 - Available Buffer 0 too small 28: requested: 1020 -> increase the buffer size
[E] ResampleStream.h : 191 - write error 0 vs 1020
[W] AudioOutput.h : 389 - Available Buffer 0 too small 28: requested: 1020 -> increase the buffer size
[E] ResampleStream.h : 191 - write error 0 vs 1020
[W] AudioOutput.h : 389 - Available Buffer 0 too small 28: requested: 152 -> increase the buffer size
[E] ResampleStream.h : 191 - write error 0 vs 152

[pschatzmann]

Just test with one copy pass and double check the situation.
Maybe you don't provide enough white noise data to completely mix all the data.
After calling mixer.flush(), mixer.available(0); and mixer.available(1); should be 0 and all buffers processed and cleared...

The amount of data provided by FormatConverterStream and by the GeneratedSoundStream must be identical after each step!

[z-l-p]

OK, I changed my loop to this for debugging:

void loop() {
  mixer.setIndex(0);
  multi.copy();
  // second copy step 
  mixer.setIndex(1);
  auto bytes = mixer.available(0); // check for amount of data from first step 
  if (bytes>0) copier2.copyBytes(bytes);
  
  Serial.print("Before Flush: ");
  Serial.print("buf0: ");
  Serial.print(mixer.available(0));
  Serial.print(", buf1: ");
  Serial.println(mixer.available(1));
  mixer.flushMixer();
  Serial.print("After Flush: ");
  Serial.print("buf0: ");
  Serial.print(mixer.available(0));
  Serial.print(", buf1: ");
  Serial.println(mixer.available(1));
}

And got results like this:

Before Flush: buf0: 12284, buf1: 1024
After Flush: buf0: 11260, buf1: 0

If I understand correctly, we see that buffer 0 still has 12284 bytes left to process after multi.copy(). We need to keep the buffers balanced, so we need more data in buffer 1 to allow another copy, so we ask for 12284 bytes from the noise generator. But the debug shows that there are only 1024 bytes left in buf 1, so after the flush there is still more work to do in buf 0 but no data in buf 1 to keep it balanced. I don't see a .resize method for the GeneratedSoundStream so how do I enlarge its buffer to keep up?
For now I altered the loop like this, so the GeneratedSoundStream copies as many times as needed to keep up. This is probably a misunderstanding, but it does help.

void loop() {
  mixer.setIndex(0);
  multi.copy();
  // second copy step 
  mixer.setIndex(1);
  auto bytes = mixer.available(0); // check for amount of data from first step 
  if (bytes > 0) {
    int remainingBytes = bytes;
    while (remainingBytes > 0) {
      copier2.copyBytes(1024);
      remainingBytes = remainingBytes - 1024;
    }
  }
  Serial.print("Before Flush: ");
  Serial.print("buf0: ");
  Serial.print(mixer.available(0));
  Serial.print(", buf1: ");
  Serial.println(mixer.available(1));
  mixer.flushMixer();
  Serial.print("After Flush: ");
  Serial.print("buf0: ");
  Serial.print(mixer.available(0));
  Serial.print(", buf1: ");
  Serial.println(mixer.available(1));
}

With this loop, I do get something from both streams. I still have much trouble starting the stream. It usually fails with mime type errors, the sound is choppy, and the noise generator is only playing on 1 channel. (I removed the print statements and set logging to Warning) So... progress but still very confused how to mix effectively.

[pschatzmann]

Hmm, the issue seems to be the big write sizes that are really unsuited to do some mixing! Maybe the following approach might work around this:

• Send the output to a BufferedStream which forwards to a CallbackStream: this way you should be able to handle a more manageble write size of 1k
• In the callback you can do the output mixing

Usually I don't recommend to mix data coming from a decoder because of this issue and to use WAV, but the approach above should circumvent this.

[pschatzmann]

The above described approach works like a charm:

#include "AudioTools.h"
#include "AudioTools/AudioCodecs/CodecMP3Helix.h"
#include "AudioTools/AudioLibs/AudioBoardStream.h"

WhiteNoiseGenerator<int16_t> noiseGenerator(600);
GeneratedSoundStream<int16_t> noise(noiseGenerator);
URLStream url("ssid", "password");
//I2SStream i2s;
AudioBoardStream i2s(AudioKitEs8388V1); // Testing this on an AudioKit
OutputMixer<int16_t> mixer(i2s, 2);
CallbackStream cb;
BufferedStream buffered(cb);
MultiStreamingDecoder multi;
MP3DecoderHelix mp3;

// with each write: mix the data with white noise
size_t onWrite(const uint8_t *data, size_t len){
  //LOGI("=>  onWrite: %u", len);
  uint8_t noise_data[len];
  size_t read = noise.readBytes(noise_data, len);
  assert(read == len);
  mixer.write(0, data, len);
  mixer.write(1, noise_data, len);
  mixer.flushMixer();
  return len;
}

void setup(){
  Serial.begin(115200);
  AudioToolsLogger.begin(Serial, AudioToolsLogLevel::Info);  

  // setup i2s
  auto config = i2s.defaultConfig(TX_MODE);
  // // Custom I2S pins for my Xaio board
  // config.pin_ws = 9; // (DAC "LRCK")
  // config.pin_bck = 7; // (DAC "BCK")
  // config.pin_data = 8; // (DAC "DIN")
  i2s.begin(config);

  // setup mixer
  mixer.begin();

  // register supported codecs with their mime type
  multi.addDecoder(mp3, "audio/mpeg");
  multi.setInput(url);
  multi.setOutput(buffered);
  multi.setMimeSource(url); // use mime provided by url
  // multi.addNotifyAudioSource(i2s); // update i2s from decoder
  multi.begin();

  // start UrL
  url.begin("https://locus.creacast.com:9443/acra_wave_farm.mp3"); 

  // Start white noise
  noiseGenerator.begin();

  // Do the mixing in smaller steps
  cb.setWriteCallback (onWrite);
}

void loop() {
  multi.copy();
}

ps. the WhiteNoiseGenerator does not depend on AudioInfo, so nothing needs to be done there. For a SineWaveGenerator you could do multi.addNotifiyAudioInfo(sine) to handle mono and stereo and sampling rate changes properly.

[z-l-p]

Wow, I was just pouring over the class docs for BufferedStream and CallbackStream, and I was about to ask for help with writing the callback function. I refreshed this discussion and found that you answered my question before I even asked. I can't thank you enough!

[z-l-p]

One more complication. In my application I need to keep the white noise running during the loading of a new URLstream (there are several URLs for the user to choose). The url.begin process blocks while the the connection happens, so I moved it into a task. Now I've confirmed that the loop (including copier) keeps running during the url loading, but the noise falls silent anyway.
Is there something like silence_on_nodata for the MultiStreamingDecoder so it can output silence while waiting for the url to connect?

[pschatzmann]

I guess there must be something wrong with your program logic: you must make sure that the copying of the silence to i2s is still going on while waiting for the url stream to be open

[z-l-p]

Actually I want to copy silence to the mixer while waiting for the URL (so the noise generator can continue to be audible with no gaps). The copier is running, but I wonder how to keep sending silence from the MultiStreamingDecoder when there's nothing to decode?

[pschatzmann]

You don't: use a separate copy path with a copier that you start to use before loading the url and you switch back to the regular process when the data is available: so the copying should be in it's own task...

[z-l-p]

That makes sense. Thanks!