Mix MP3 and generatedSoundStream

[schromp]

Did anyone find a way to mix a generatedSoundStream and a decoded mp3 file?
I have tried to achieve this but have failed...

Firstly, when using a output mixer together with the encoded mp3 the playback speed gets too fast.
Secondly when combining the two audiostreams in the mixer i get a buffer too small error, even though it is set bigger.

I2SStream i2s; OutputMixer<int16_t> mixer(i2s, 2); VolumeStream volumeSong(mixer); VolumeStream volumeWhiteNoise(mixer); WhiteNoiseGenerator<int16_t> whiteNoise; GeneratedSoundStream<int16_t> whiteNoiseIn(whiteNoise); EncodedAudioStream decoder(&volumeSong, new MP3DecoderHelix()); StreamCopy copierSong; StreamCopy copierWhiteNoise(volumeWhiteNoise, whiteNoiseIn); void setup() { auto config = i2s.defaultConfig(TX_MODE); whiteNoiseIn.begin(config); whiteNoise.begin(); i2s.begin(config); volumeSong.setVolume(0.3); decoder.begin(); mixer.begin(4608); currentSong = SD.open("/001.mp3"); copierSong.begin(decoder, currentSong); } void loop() { copierWhiteNoise.copy(); copierSong.copy(); }

Error:
[W] AudioOutput.h : 400 - Available Buffer 0 too small 1024: requested: 4608 -> increase the buffer size

Is this even possible with the resources of an esp32?

The end goal is to have have a analog radio like audio where i can fade out the white noise and fade in the mp3 file.
Is there a simpler way to do this or any way at all?

[pschatzmann]

You have 2 major problems to solve:

• Your speed issue: If you are mixing, you should to make sure that the output info (sample rate, channels, bit_per_sample) of both mixed streams are identical and matches to the one used for i2s: you would usually do this by resampling to a defined target. However I think for white noise you could get away by making sure that you use the right sample rate (from mp3) when you set up i2s or you synchronize them by calling decoder.addNotifyAudioInfo(i2s)
• you need to feed the mixer with the same data amount and here is where the major challenge lies. You never now how much pcm data a single copierSong.copy(); is producing and in average it is about 4x the amount of copierWhiteNoise.copy(); As you can see in the log above the decoder is providing the data in batches of 4608 bytes.
  • With the OutputMixer, try in increase the buffer size to 2 x 4608 bytes. Decode the mp3 data first and generate the same amount of white noise that has been written to the mixer by using the available (int idx) method
  • Try using the InputMixer instead.
  • If any of the above approaches fail: the following should work in any case: In a separate task, try to fill a queue with decoded pcm data and use this queue is input source for your mixing.

[schromp]

Firstly thanks for your answer :)

Using the input mixer would be preferable for me. I understand how to wire up the white noise generator to it but im struggling with connecting the encodedaudiostream.
The example on the merging audio wiki page only shows this using a MemoryStream.
I did not manage to find a way to convert the File object i get from SD.open() nor did i find a way to directly wire up the encoded audio stream.
When searching this discussion forum the question has been asked but not answered yet.

Thing i have tried:

MemoryStream memStream(currentSong, currentSong.size()); // Doesnt match the constructor

 EncodedAudioStream dec(&currentSong, new MP3DecoderHelix()); i2s.begin(config); dec.addNotifyAudioChange(i2s); dec.begin(); mixer.add(dec); mixer.begin(config);

Can you point me into a direction or give me some keywords so i can figure this out?

Thanks a lot :)

[pschatzmann]

I don't quite understand your question: you can use both the whiteNoiseIn and decoder from your original sketch as components for your InputMixer.

The Mixing Wiki describes how to do mixing on the input side and the Codec Wiki has a chapter Decoding on the Input Side which is just showing that you can read PCM data from a EncodedAudioStream

[schromp]

Original decoding example:

#include "AudioTools.h" #include "AudioTools/AudioCodecs/CodecMP3Helix.h" #include "BabyElephantWalk60_mp3.h" MemoryStream mp3(BabyElephantWalk60_mp3, BabyElephantWalk60_mp3_len); EncodedAudioStream dec(&mp3, new MP3DecoderHelix()); I2SStream i2s; StreamCopy copier(i2s, mp3); void setup(){ Serial.begin(115200); i2s.begin(); dec.addNotifyAudioChange(i2s); dec.begin(); } void loop(){ copier.copy(); }

After thinking about it some more my confusion comes from this i think:
We give the decoder the mp3 memorystream. But then we pass the copier the original mp3 again?
So in my mind there would be two pipelines:
mp3 -> decoder
mp3->copier->i2s

How does the copier get the decoded data then?

When wanting to change the MemoryStream to a File input with the following sketch i just get white noise. (Im guessing because im passing in non-decoded data and thats just what it sounds like)

#include "AudioTools.h" #include "AudioTools/AudioCodecs/CodecMP3Helix.h" #include <SD.h> File mp3File; EncodedAudioStream dec(&mp3File, new MP3DecoderHelix()); I2SStream i2s; StreamCopy copier(i2s, mp3File); void setup() { Serial.begin(115200); if (!SD.begin()) { Serial.println("SD card initialization failed!"); stop(); } mp3File = SD.open("/90.0/001.mp3"); if (!mp3File) { Serial.println("Failed to open MP3 file!"); return; } else { Serial.println("Opened the MP3 file"); } dec.begin(); AudioInfo info = dec.audioInfo(); auto config = i2s.defaultConfig(TX_MODE); config.sample_rate = info.sample_rate; config.channels = info.channels; i2s.begin(config); delay(100); dec.addNotifyAudioChange(i2s); } void loop() { copier.copy(); }

[pschatzmann]

It should be StreamCopy copier(i2s, dec); copy want to copy the decoded data to i2s

[schromp]

Alright.
This seems to be working, and makes way more sense, however i get a different error which i just cant seem to debug.
Can you point me to some help again maybe? If not, thats okay then i will change the projects requirements to something simpler. I dont want to annoy with the frequent questions.

It seems like like the program crashes because the amount of data written changes (always) after the 3rd loop.
This is tested using a mp3 file and hardware setup which worked when i was using the decoder on the output side instead of the input side but it doesnt matter which file i test, its always the 3rd loop that drops from 4608 to 512

Code:

#include "AudioTools.h" #include "AudioTools/AudioCodecs/CodecMP3Helix.h" #include <SD.h> File mp3File; EncodedAudioStream dec(&mp3File, new MP3DecoderHelix); I2SStream i2s; StreamCopy copier(i2s, dec); void setup() { // LOGLEVEL_HELIX = LogLevelHelix::Debug; AudioLogger::instance().begin(Serial, AudioLogger::Debug); Serial.begin(115200); if (!SD.begin()) { Serial.println("SD card initialization failed!"); stop(); } mp3File = SD.open("/90.0/001.mp3"); if (!mp3File) { Serial.println("Failed to open MP3 file!"); stop(); return; } else { Serial.println("Opened the MP3 file"); } auto config = i2s.defaultConfig(TX_MODE); // config.sample_rate = 48000; i2s.begin(config); dec.addNotifyAudioChange(i2s); dec.begin(); } void loop() { copier.copy(); }

Error Log:

rst:0xc (SW_CPU_RESET),boot:0x13 (SPI_FAST_FLASH_BOOT) configsip: 0, SPIWP:0xee clk_drv:0x00,q_drv:0x00,d_drv:0x00,cs0_drv:0x00,hd_drv:0x00,wp_drv:0x00 mode:DIO, clock div:2 load:0x3fff0030,len:1184 load:0x40078000,len:13232 load:0x40080400,len:3028 entry 0x400805e4 [ 22][D][esp32-hal-cpu.c:244] setCpuFrequencyMhz(): PLL: 480 / 2 = 240 Mhz, APB: 80000000 Hz [ 39][V][esp32-hal-uart.c:330] uartBegin(): UART0 baud(115200) Mode(800001c) rxPin(3) txPin(1) [ 48][V][esp32-hal-uart.c:416] uartBegin(): UART0 not installed. Starting installation [ 59][V][esp32-hal-uart.c:463] uartBegin(): UART0 initialization done. Opened the MP3 file [D] I2SStream.h : 71 - bool audio_tools::I2SStream::begin(audio_tools::I2SConfig) [D] BaseStream.h : 118 - virtual void audio_tools::AudioStream::setAudioInfo(audio_tools::AudioInfo) [I] AudioTypes.h : 128 - out: sample_rate: 44100 / channels: 2 / bits_per_sample: 16 [D] I2SESP32.h : 63 - bool audio_tools::I2SDriverESP32::begin(audio_tools::I2SConfigESP32) [D] I2SESP32.h : 182 - bool audio_tools::I2SDriverESP32::begin(audio_tools::I2SConfigESP32, int, int) [I] AudioTypes.h : 128 - sample_rate: 44100 / channels: 2 / bits_per_sample: 16 [I] I2SConfigESP32.h : 80 - rx/tx mode: TX_MODE [I] I2SConfigESP32.h : 81 - port_no: 0 [I] I2SConfigESP32.h : 82 - is_master: Master [I] I2SConfigESP32.h : 83 - sample rate: 44100 [I] I2SConfigESP32.h : 84 - bits per sample: 16 [I] I2SConfigESP32.h : 85 - number of channels: 2 [I] I2SConfigESP32.h : 86 - signal_type: Digital [I] I2SConfigESP32.h : 88 - i2s_format: I2S_STD_FORMAT [I] I2SConfigESP32.h : 90 - auto_clear: true [I] I2SConfigESP32.h : 92 - use_apll: true [I] I2SConfigESP32.h : 97 - buffer_count:6 [I] I2SConfigESP32.h : 98 - buffer_size:512 [I] I2SConfigESP32.h : 103 - pin_bck: 14 [I] I2SConfigESP32.h : 105 - pin_ws: 15 [I] I2SConfigESP32.h : 107 - pin_data: 22 [D] I2SESP32.h : 214 - i2s_driver_install [D] I2SESP32.h : 230 - i2s_set_pin [D] I2SESP32.h : 241 - i2s_zero_dma_buffer [D] I2SESP32.h : 245 - begin - started [I] AudioEncoded.h : 75 - virtual void audio_tools::EncodedAudioOutput::addNotifyAudioChange(audio_tools::AudioInfoSupport&) [D] CodecMP3Helix.h : 128 - virtual void audio_tools::MP3DecoderHelix::addNotifyAudioChange(audio_tools::AudioInfoSupport&) [D] Allocator.h : 68 - Allocated 4 [D] AudioIO.h : 30 - void audio_tools::TransformationReader<T>::begin(T*, Stream*) [with T = audio_tools::ReformatBaseStream] [D] AudioEncoded.h : 140 - virtual bool audio_tools::EncodedAudioOutput::begin() [D] AudioEncoded.h : 142 - virtual bool audio_tools::EncodedAudioOutput::begin() [D] AudioCodecsBase.h : 27 - virtual void audio_tools::AudioDecoder::setAudioInfo(audio_tools::AudioInfo) [D] CodecMP3Helix.h : 82 - virtual bool audio_tools::MP3DecoderHelix::begin() [D] MetaDataFilter.h : 30 - void audio_tools::MetaDataFilter<Decoder>::begin() [with Decoder = libhelix::MP3DecoderHelix] [D] StreamCopy.h : 100 - copy 1024 bytes [D] StreamCopy.h : 243 - available: 1024 [D] AudioIO.h : 45 - TransformationReader::readBytes: 1024 [I] AudioIO.h : 60 - read size: 512 [D] Allocator.h : 68 - Allocated 512 [I] AudioIO.h : 67 - buffer size: 10240 [I] Buffers.h : 376 - resize: 10240 [D] Allocator.h : 68 - Allocated 10240 [D] BaseStream.h : 319 - bool audio_tools::QueueStream<T>::begin() [with T = unsigned char] [D] AudioIO.h : 164 - virtual void audio_tools::ReformatBaseStream::setOutput(Print&) [D] AudioEncoded.h : 187 - EncodedAudioOutput::write: 512 [D] CodecMP3Helix.h : 112 - virtual size_t audio_tools::MP3DecoderHelix::write(const uint8_t*, size_t): 512 [D] AudioEncoded.h : 199 - EncodedAudioOutput::write: 512 -> 512 [D] AudioEncoded.h : 187 - EncodedAudioOutput::write: 512 [D] CodecMP3Helix.h : 112 - virtual size_t audio_tools::MP3DecoderHelix::write(const uint8_t*, size_t): 512 [D] CodecMP3Helix.h : 137 - static void audio_tools::MP3DecoderHelix::infoCallback(MP3FrameInfo&, void*) [I] AudioTypes.h : 128 - MP3DecoderHelix sample_rate: 48000 / channels: 2 / bits_per_sample: 16 [I] I2SStream.h : 93 - virtual void audio_tools::I2SStream::setAudioInfo(audio_tools::AudioInfo) [D] BaseStream.h : 118 - virtual void audio_tools::AudioStream::setAudioInfo(audio_tools::AudioInfo) [I] AudioTypes.h : 128 - in: sample_rate: 48000 / channels: 2 / bits_per_sample: 16 [I] AudioTypes.h : 128 - out: sample_rate: 48000 / channels: 2 / bits_per_sample: 16 [I] I2SESP32.h : 45 - i2s_set_sample_rates: 48000 [D] Buffers.h : 77 - writeArray 4608 -> 4608 [D] AudioEncoded.h : 199 - EncodedAudioOutput::write: 512 -> 512 [D] AudioIO.h : 164 - virtual void audio_tools::ReformatBaseStream::setOutput(Print&) [D] Buffers.h : 50 - readArray 1024 -> 1024 [D] AudioIO.h : 99 - TransformationReader::readBytes: 1024 -> 1024 [D] StreamCopy.h : 390 - write: 1024 [D] I2SStream.h : 116 - I2SStream::write: 1024 [D] I2SESP32.h : 98 - size_t audio_tools::I2SDriverESP32::writeBytes(const void*, size_t) [D] I2SESP32.h : 106 - i2s_write 1024 -> 1024 bytes [D] StreamCopy.h : 396 - write: 1024 -> 1024 [I] StreamCopy.h : 163 - StreamCopy::copy 1024 -> 1024 -> 1024 bytes - in 1 hops [D] StreamCopy.h : 100 - copy 1024 bytes [D] StreamCopy.h : 243 - available: 1024 [D] AudioIO.h : 45 - TransformationReader::readBytes: 1024 [D] Buffers.h : 50 - readArray 1024 -> 1024 [D] AudioIO.h : 99 - TransformationReader::readBytes: 1024 -> 1024 [D] StreamCopy.h : 390 - write: 1024 [D] I2SStream.h : 116 - I2SStream::write: 1024 [D] I2SESP32.h : 98 - size_t audio_tools::I2SDriverESP32::writeBytes(const void*, size_t) [D] I2SESP32.h : 106 - i2s_write 1024 -> 1024 bytes [D] StreamCopy.h : 396 - write: 1024 -> 1024 [I] StreamCopy.h : 163 - StreamCopy::copy 1024 -> 1024 -> 1024 bytes - in 1 hops [D] StreamCopy.h : 100 - copy 1024 bytes [D] StreamCopy.h : 243 - available: 1024 [D] AudioIO.h : 45 - TransformationReader::readBytes: 1024 [D] Buffers.h : 50 - readArray 1024 -> 1024 [D] AudioIO.h : 99 - TransformationReader::readBytes: 1024 -> 1024 [D] StreamCopy.h : 390 - write: 1024 [D] I2SStream.h : 116 - I2SStream::write: 1024 [D] I2SESP32.h : 98 - size_t audio_tools::I2SDriverESP32::writeBytes(const void*, size_t) [D] I2SESP32.h : 106 - i2s_write 1024 -> 1024 bytes [D] StreamCopy.h : 396 - write: 1024 -> 1024 [I] StreamCopy.h : 163 - StreamCopy::copy 1024 -> 1024 -> 1024 bytes - in 1 hops [D] StreamCopy.h : 100 - copy 1024 bytes [D] StreamCopy.h : 243 - available: 1024 [D] AudioIO.h : 45 - TransformationReader::readBytes: 1024 [D] Buffers.h : 50 - readArray 1024 -> 1024 [D] AudioIO.h : 99 - TransformationReader::readBytes: 1024 -> 1024 [D] StreamCopy.h : 390 - write: 1024 [D] I2SStream.h : 116 - I2SStream::write: 1024 [D] I2SESP32.h : 98 - size_t audio_tools::I2SDriverESP32::writeBytes(const void*, size_t) [D] I2SESP32.h : 106 - i2s_write 1024 -> 1024 bytes [D] StreamCopy.h : 396 - write: 1024 -> 1024 [I] StreamCopy.h : 163 - StreamCopy::copy 1024 -> 1024 -> 1024 bytes - in 1 hops [D] StreamCopy.h : 100 - copy 1024 bytes [D] StreamCopy.h : 243 - available: 1024 [D] AudioIO.h : 45 - TransformationReader::readBytes: 1024 [D] AudioIO.h : 164 - virtual void audio_tools::ReformatBaseStream::setOutput(Print&) [D] AudioEncoded.h : 187 - EncodedAudioOutput::write: 512 [D] CodecMP3Helix.h : 112 - virtual size_t audio_tools::MP3DecoderHelix::write(const uint8_t*, size_t): 512 [D] Buffers.h : 77 - writeArray 4608 -> 4608 [D] Buffers.h : 77 - writeArray 4608 -> 4608 [D] Buffers.h : 77 - writeArray 4608 -> 512 assert failed: void libhelix::MP3DecoderHelix::provideResult(MP3FrameInfo&) MP3DecoderHelix.h:154 (written == toWrite) 

[pschatzmann]

Not sure if this will help, did you try to decrease the copy size ? e.g. with StreamCopy copier(i2s, dec, 256);

[schromp]

I did now, and i tried some more values but all that achieved was just that it happened later in the program because it takes longer to reach that part in the mp3 file i guess.

I have also checked the mp3 file using ffmpeg and like i said it has worked when using on the ouput side.
Could this be a memory limitation because were doing this on the input side? (Im just guessing, i have no real understanding of the internals of this library)

[pschatzmann]

The EncodedAudioStream read is decoding the data into a queue and it seems that the queue was not big enough for the big decoded arrays created by the mp3 decoder.

I committed a correction which adjusted the queue size to make it work with libhelix.

If this is not working for you you can try to incread the buffer size: with a copy size of 1024 bytes 5 means 5 * 1024 bytes:
//dec.transformationReader().setResultQueueFactor(5);

[schromp]

Yes this worked. Thank you for the fix :)

FYI i had to increase the resultQueueFactor by 14 to get it working.

[pschatzmann]

Was this with the StreamCopy default size or with a decreased value ?

[schromp]

Default size:

#include "AudioTools.h" #include "AudioTools/AudioCodecs/CodecMP3Helix.h" #include <SD.h> File mp3File; EncodedAudioStream dec(&mp3File, new MP3DecoderHelix); I2SStream i2s; StreamCopy copier(i2s, dec); void setup() { // LOGLEVEL_HELIX = LogLevelHelix::Debug; // AudioLogger::instance().begin(Serial, AudioLogger::Debug); Serial.begin(115200); if (!SD.begin()) { Serial.println("SD card initialization failed!"); stop(); } mp3File = SD.open("/95.0/001.mp3"); if (!mp3File) { Serial.println("Failed to open MP3 file!"); stop(); return; } else { Serial.println("Opened the MP3 file"); } auto config = i2s.defaultConfig(TX_MODE); // config.sample_rate = 48000; i2s.begin(config); dec.addNotifyAudioChange(i2s); dec.transformationReader().setResultQueueFactor(14); dec.begin(); } void loop() { copier.copy(); }

[schromp]

Interesting thing i have noticed:
When using the new fix the VolumeStream has no effect anymore.

#include "AudioTools.h" #include "AudioTools/AudioCodecs/CodecMP3Helix.h" #include <SD.h> File mp3File; EncodedAudioStream dec(&mp3File, new MP3DecoderHelix); I2SStream i2s; VolumeStream volume(i2s); StreamCopy copier(volume, dec); void setup() { // LOGLEVEL_HELIX = LogLevelHelix::Debug; // AudioLogger::instance().begin(Serial, AudioLogger::Debug); Serial.begin(115200); if (!SD.begin()) { Serial.println("SD card initialization failed!"); stop(); } mp3File = SD.open("/95.0/001.mp3"); if (!mp3File) { Serial.println("Failed to open MP3 file!"); stop(); return; } else { Serial.println("Opened the MP3 file"); } volume.setVolume(0.1); auto config = i2s.defaultConfig(TX_MODE); // config.sample_rate = 48000; i2s.begin(config); dec.addNotifyAudioChange(i2s); dec.transformationReader().setResultQueueFactor(14); dec.begin(); } void loop() { copier.copy(); Serial.printf( "Current volume: %f\n", volume.volume()); // this prints correct as: Current volume: 0.100000 }

This also does not work if you try to use the volumeStream on the input side:

#include "AudioTools.h" #include "AudioTools/AudioCodecs/CodecMP3Helix.h" #include <SD.h> File mp3File; EncodedAudioStream dec(&mp3File, new MP3DecoderHelix); I2SStream i2s; VolumeStream volume(dec); StreamCopy copier(i2s, volume); void setup() { // LOGLEVEL_HELIX = LogLevelHelix::Debug; // AudioLogger::instance().begin(Serial, AudioLogger::Debug); Serial.begin(115200); if (!SD.begin()) { Serial.println("SD card initialization failed!"); stop(); } mp3File = SD.open("/95.0/001.mp3"); if (!mp3File) { Serial.println("Failed to open MP3 file!"); stop(); return; } else { Serial.println("Opened the MP3 file"); } volume.setVolume(0.1); auto config = i2s.defaultConfig(TX_MODE); // config.sample_rate = 48000; i2s.begin(config); dec.addNotifyAudioChange(i2s); dec.transformationReader().setResultQueueFactor(14); dec.begin(); } void loop() { copier.copy(); Serial.printf( "Current volume: %f\n", volume.volume()); // this prints correct as: Current volume: 0.100000 }

Both examples compile and play the file but do not change the volume

[pschatzmann]

you forgot to call volume,begin()!

[schromp]

it has been a long day i guess...
thank you very much.
ill post a resolution to my original question shortly and then this discussion will be done

[schromp]

The original question was about mixing a mp3 stream and a generated sound.
Here is how i got it to work:

File audioFile; I2SStream i2s; InputMixer<int16_t> mixer; WhiteNoiseGenerator<int16_t> whiteNoise; GeneratedSoundStream<int16_t> whiteNoiseIn(whiteNoise); EncodedAudioStream decoder(&audioFile, new MP3DecoderHelix()); VolumeStream volumeSong(decoder); VolumeStream volumeWhiteNoise(whiteNoiseIn); StreamCopy copier(i2s, mixer); // Arduino Setup void setup(void) { Serial.begin(115200); while (!Serial) ; if (!SD.begin()) { Serial.println("SD card initialization failed!"); stop(); } audioFile = SD.open("/95.0/001.mp3"); if (!audioFile) { Serial.println("Failed to open MP3 file!"); stop(); return; } else { Serial.println("Opened the MP3 file"); } Serial.println("Card Mount succeeded"); auto config = i2s.defaultConfig(TX_MODE); whiteNoiseIn.begin(config); whiteNoise.begin(); i2s.begin(config); volumeSong.setVolume(0.1); volumeSong.begin(); volumeWhiteNoise.setVolume(0.1); volumeWhiteNoise.begin(); decoder.setStream(currentSong); decoder.transformationReader().setResultQueueFactor(14); decoder.begin(); mixer.add(volumeSong); mixer.add(volumeWhiteNoise); mixer.begin(config); copier.begin(); } void loop() { copier.copy(); }

[level0dev]

I'm trying to adapt your solution to use with the audiokit, but it seems this code is missing a declaration for currentSong.

What should be passed to decoder.setStream ?

edit: For anyone wondering the same - it needs audioFile as the argument.

Also, for some reason my audiokit fails to initialize the SD card with the code above. Removing the SD if statement and simply calling SD.begin() in its place works fine. Not sure if this is just a problem with my board or something specific to all audiokit boards.

[level0dev]

I'm a newbie to both ESP32 and C. I'm trying to achieve exact same effect that @schromp was asking about (basically add radio tuning noise to mp3.

The end effect I want to achieve is similar to @alastaira's London Calling Prop - tune a radio and the closer you get to the station value the clearer the mix becomes (between the generated noise and mp3).

I would really love a proper example for mixing an mp3 and generated audio (even better if it was written for the audiokit). @pschatzmann - would this request be served better by a new discussion topic?