Fluidsynth testing suite

[spessasus]

Hi,

I'm planning to add a test suite similar to fluidsynth to my own synth. My question is, how is testing of the various SF2/MIDI file combinations implemented? Does it render every file to wav and report if its checksum has changed for manual listening test?

Thanks in advance!

[derselbst]

Currently, it's a purely manual test, i.e. render to wav and rehearse the result. Not satisfying. Checksum is too waggly. Some kind of audio fingerprint or a simple maximum permitted RMS difference would be preferable. I was playing around with sox a bit, but could find a pleasing solution yet. If you do, I'd be curious to know!

[spessasus]

Thanks for the answer. Though I'm not sure what you meant by the checksum being too "waggly". Isn't fluidsynth's output deterministic?

[derselbst]

Fluidsynth renders with floats. Floating point arithmetic is not associative. I expect tiny difference whether you're using x87 vs. SSE2 FPU, whether or not multi threaded rendering is used, whether you're using Linux Math or Windows Math library, Release vs. Debug builds, etc. But what I'm really afraid of is that minor code changes will have side effects causing the checksum of all audio to change, causing me to rehearse all the audio and listen for differences, which would however defeat the purpose of having automated tests backed by checksum.

[spessasus]

Could downsampling the wav files to 16 or even 8 bits possibly resolve the issue? If the changes are as significant as making a change in a final uin8 sample, then it could be worth investigating. And the hardware running the tests is the same when rendering with github actions, no?

Or a simple diff, as in taking the previous render, inverting the waveform (by multiplying by -1) and mixing the two? If the max sample of a resulting waveform is larger than, let's say, 100 for an s16 wav file, then it's worth investigating?

What do you think?

[derselbst]

Could downsampling the wav files to 16 or even 8 bits possibly resolve the issue?

Nope, downsampling introduces even more quantization errors. That's why there is dithering when float -> s16 conversion.

Or a simple diff, as in taking the previous render, inverting the waveform (by multiplying by -1) and mixing the two? If the max sample of a resulting waveform is larger than, let's say, 100 for an s16 wav file, then it's worth investigating?

Checking-in prerendered files (and diffing them somehow) is indeed the best approach I can think of. But also the worst: Where to save all that data? Usually I'd say, that's what git lfs is for. However here on Github, there's a monthly LFS transfer quota of 1GB. In the past we've easily hit that limit just because I put like 60 MB into LFS, since all those CI builds are causing a lot of traffic. So LFS is not a real option. Storing them in a submodule is not an option either, because fluidsynth already requires checking out submodules, and if every user checks out the prerendered files, this would cause a lot of traffic as well, and unhappy users, who would checkout a bunch of files they don't need.

So my best guess is to store them in a completely separate repo. Which can then be cloned as part of the CI build and compared against. Which raises the problem of how to keep those testdata reproducible. I.e. the prerendered testdata must match with the MIDI-testdata, which is bound to the fluidsynth repo via submodule. Ideally, the prerendered testdata would be a submodule. But hang on, we can't use a submodule here...

So, I don't see it's a low hanging fruit.

[derselbst]

Or a simple diff, as in taking the previous render, inverting the waveform (by multiplying by -1) and mixing the two? If the max sample of a resulting waveform is larger than, let's say, 100 for an s16 wav file, then it's worth investigating?

Checking-in prerendered files (and diffing them somehow) is indeed the best approach I can think of. But also the worst: Where to save all that data?

Not saving the audio files is the answer! Instead regenerate them every time! Once with the current HEAD revision and once with some older revision of fluidsynth.

This "older revision" could be a precompiled docker container image, containing fluidsynth's latest stable. One could also simply recompile a different branch / different release / previous revision and re-execute the rendering tests.

Ultimately, invert both results, compare SNR / RMS / abs difference and fail the CI build if it exceeds some threshold.

... so many ideas and so little time...

[spessasus]

This "older revision" could be a precompiled docker container image, containing fluidsynth's latest stable. One could also simply recompile a different branch / different release / previous revision and re-execute the rendering tests.

How about a check for every commit on master: render the tests, then do git reset --hard HEAD~1 and render them again, then diffing them?

So the final sequence would be: On every commit to master run:

1. Clone the master repo
2. Render all test files
3. Copy the output wav files to somewhere else, like a dir called current, so the next run won't overwrite them
4. git reset --hard HEAD~1
5. Render all test files
6. Invert renders of the previous commit and mix them down (diff)
7. Check for the threshold
8. And instead of failing the build, wouldn't it be better if some sort of a bot could open an issue? It would be like "Commit caused differences in the audio output" and the description would specifically list the files that have changed (for example only chorus changed so files that don't use it don't need to be checked manually). Though I don't know if an automated issue opener like that is possible.

What do you think?

[derselbst]

It would be very nice if that "workflow" allowed me to compare arbitrary builds of fluidsynth. Ideally even against those old versions that didn't even knew about the test suite. It's really just about executing the fluidsynth binary. ${PATH_TO_REF_FLUIDSYNTH_BINARY} should do.

git reset --hard HEAD~1

Where this reference comes from, can be part of the CI build logic. So HEAD~1 is possible, but doesn't need to.

How about a check for every commit on master

It should not be limited to master. PRs are equally important.

wouldn't it be better if some sort of a bot could open an issue?

Sound like quite a lot effort to get that right. Opening issues, after the commit was made feels... wrong. Also, how to prevent a million issues being opened? My time is limited, I can already see myself simply closing them all because I have no time to investigate. So better keep it simple and fail the build, IMO.

Rest sounds good to me.