PDM microphone input corrupted when DFS is enabled while using APLL (IDFGH-13860)

[therealergo]

Answers checklist.

• I have read the documentation ESP-IDF Programming Guide and the issue is not addressed there.
• I have updated my IDF branch (master or release) to the latest version and checked that the issue is present there.
• I have searched the issue tracker for a similar issue and not found a similar issue.

IDF version.

v5.3.1

Espressif SoC revision.

ESP32

Operating System used.

Windows

How did you build your project?

Command line with idf.py

If you are using Windows, please specify command line type.

PowerShell

Development Kit.

Custom Board

Power Supply used.

USB

What is the expected behavior?

Clear audio to be recorded from a pair of PDM microphones.

What is the actual behavior?

Audio contains random crackles and pops.

Steps to reproduce.

1. Download the attached minimal test code test_fw_audio_wav_sdcard_pdm.c onto a board with PDM microphones and an SD card. This code will save 1MB chunks of stereo audio from the microphones to the SD card while it runs.
2. Run the code with DFS disabled. Listen to the recorded audio, which is clear.
3. Run the code with DFS enabled (any CPU frequency range). Listen to the recorded audio, which contains periodic crackles and pops.

Debug Logs.

No response

More Information.

As far as I can tell from the ESP-IDF documentation, datasheet, and reference manual, this configuration should be fully supported. The documentation claims that the I2S peripheral should be able to run from the APLL while only holding the NO_LIGHT_SLEEP power management lock, and the datasheet and reference manual do not suggest that changing the CPU or APB bus speed in this configuration will cause any issues.

[therealergo]

@L-KAYA Reading through this commit, as far as I can tell this completely removes the ability to use DFS frequencies below APB_MAX with a number of different peripherals and modes (DAC DMA, I2S PDM, I2S STD, I2S TDM, and I2S LCD).

I can't speak to all of these modes. That said, we currently use DFS to switch between 10MHz run and 80MHz run with I2S STD mode. This is already deployed on a production device, and it works well. The reported issue is only with I2S PDM mode. I suspect there are other teams which would have similar issues, since this is a breaking change that removes functionality.

Additionally, as noted in the original report, this issue occurs whenever the CPU frequency changes. Holding the ESP_PM_APB_FREQ_MAX is not sufficient to prevent this issue when I2S PDM mode is used, because e.g. switching from 80MHz run to 240MHz run also causes the issue.

[L-KAYA]

Hi @therealergo, thanks for your feedback!

I'm sorry that the fix doesn't solve the issue, and seems I misunderstood your problem. Let me confirm again, and please correct me if I have something wrong:

1. DFS + I2S STD + APLL can work correctly, but DFS + I2S PDM + APLL data corrupted;
2. Have you also tried to capture the clock on the pin, and check if its frequency is correct during the CPU frequency scaling? (I captured locally, the clock is stable)
3. Will the audio speed up or lower down after the frequency scaling? (I monitored the interrupt, the interrupt interval jitter increased about 10%, but as you said STD mode works well, it might not an issue)
4. Do you also want to keep APB frequency scalable when DFS on? (Yes, this fix limited the minimum scalable frequency, I'm considering to revert the commit since it does not help but might break other things)
5. If possible, could you provide the extra sdkconfigs in your case? (It's helpful to reproduce and align the sdk environment)

If the audio speed and the clock are both correct but only the audio corrupted when CPU frequency changed, it might probably an issue in the PDM2PCM converter.

Anyway, I'll have a further investigation and will post here if there are any new questions or findings.

Sorry again for the inconvenience, please feel free to re-open the issue~

[therealergo]

@L-KAYA Thanks for looking into it! To answer each of these:

1. That is correct. DFS + I2S STD + APLL works correctly, but DFS + I2S PDM + APLL does not work correctly. I have seen this at every DFS frequency range, so DFS + I2S PDM + APLL does not work correctly even when DFS is scaling from 160MHz to 240MHz (and the APB speed is constant).
2. I have captured the PDM clock pin, and did not see anything odd. The frequency does not change up or down when the frequency changes, and I do not see any obvious glitches or issues.
3. The speed of the audio may not change, but some audio samples are corrupted. I've attached sample audio recorded using the test code attached to the original report. The "TEST_DFS_OFF.WAV" audio is recorded with DFS disabled, the "TEST_DFS_ON_160_240.WAV" audio is recorded with DFS between 160MHz and 240MHz enabled. With larger DFS frequency ranges, the driver will also record fewer samples than expected, and sometimes even timeout while waiting for samples.
4. Yes, for DFS + I2S STD + APLL, APB frequency should remain scalable. That case works just fine. I recommend reverting it so that it doesn't break that case.
5. I've attached the sdkconfig used with the previously-attached test code.

Based on the situations that I've seen fail, my assumption is roughly the same. It's either an issue with interrupt handling, or an issue with the PDM2PCM converter itself.

Please do let me know what you find on further investigation! Please do re-open the issue.

attachment.zip

[L-KAYA]

Hi @therealergo I have reproduced the issue on my side!

Conclusions

1. The noise should be related to both CPU and APB frequency;
2. You said Holding the ESP_PM_APB_FREQ_MAX is not sufficient but holding it seems work for me. I also tested the PLL clock source, it can also work, I guess it also due to the APB frequency is held for PLL clock source by default . So, the commit should work and may not need to be reverted;
3. Different minimum CPU frequency can also influence the final audio SNR.

Generally, it probably a hardware issue (just a speculation with no evidence, maybe some i2s registers react too slow to latch the correct data when APB is at low frequency).

It is not likely to be fixed directly, but can be bypassed by holding the APB frequency (which has been fixed in the commit).

Test Results

DFS off

DFS on (APLL)

min_freq = 10MHz

min_freq = 20MHz

(No idea why 20MHz get such a bad performance 😂 )

min_freq = 40MHz

(Seem to be the best when APB and CPU lower than 80MHz)

min_freq = 80MHz

min_freq = 10MHz + APB freq lock

DFS on (PLL)

[L-KAYA]

Hi @therealergo, I did some further tests for the STD mode (with ESP32-LyraT board, which carries es8388 codec). Here is the waveform (16 KHz, 16-bit):

Although they look pretty same, and seem no glitches in the audio, it sounds different actually. The music sounds like bouncing and contains short-period echo.

After some discussion, it is confirmed that it's a timing issue that caused by the cross-clock-domain data transmission. The DMA and I2S module are in different clock domains, when the DFS is on, the I2S module stay fixed as it is clocked by APLL, but DMA source clock is dynamic, which makes it lost synchronization, and the data/bit sequence might go wrong. Such disorder won't affect too much to the STD mode, because it is PCM format data already, switching some data can only make the audio sounds weird. But the will increase the noise to the PDM mode, because the PDM2PCM converter filters the raw PDM data based on a correct sequence, otherwise there will be glitches and noise.

This cross-clock-domain issue should only exist on ESP32 and ESP32-S2, so ESP32-P4 can still use APLL + DFS with APB frequency scalable.

This issue will be closed, since now we have to acquire the APB lock to ensure the DMA clock domain won't be dynamic when using APLL + DFS feature.

[therealergo]

Hi @L-KAYA, thanks for looking into this further.

To be sure, it sounds like a silicon issue in the ESP32 prevents this feature from functioning as intended, correct? That is unfortunate, since it was a documented feature that worked well enough that we have already deployed a significant number of devices which use it. After doing extensive testing and optimization on our end, we can work around it with a 20-40% battery life penalty in our application. Is it possible for you to better quantify what "switching some data" means in the I2S STD case? None of our testing found any "echo" or "bouncing" in the I2S STD audio, but we may have seen one case where the two I2S channels seemed to randomly swap. Could this bug cause that?

With this change in place, there is other documentation that needs to be changed to remove references to this feature. For example, the pictured "Power Management" section of the programming guide also needs to be updated:

After re-testing on my end, I see why I didn't think that it was related to APB frequency. I assumed that setting the CPU minimum frequency to 80MHz would make the APB always remain at 80MHz too, since the APB_CLK is derived from CPU_CLK. However, digging through rtc_clk.c, it seems like when the CPU has to switch between 80MHz and 240MHz, it first has to switch to XTAL_CLK in between. This causes a brief glitch in APB_CLK, which seems to correspond with each pop in the I2S PDM audio. This is also why I wasn't hearing any issues when the CPU was only configured for 80Mhz <-> 160MHz switching.

Interestingly, if I remove the check in rtc_clk.c on line 539 that enforces that switch to XTAL during the 80MHz to 240MHz switch process, everything still appears to operate correctly. However, that change fixes the audio corruption when the CPU is configured for 80Mhz <-> 240MHz switching. Is that switch back to XTAL to change PLL modes a strict hardware requirement?