Should WASAPI exclusive mode support 24-bit PCM when audio.sample-format is set to "float"?

[jimhen3ry]

I’d like to get feedback on a possible enhancement of the fluid_wasapi.c Windows driver to support 24-bit PCM in WASAPI exclusive mode when audio.sample-format is set to "float".

Background

The Miditzer is a theatre pipe organ emulation that supports real-time performance. In this context, latency—the time from pressing a key on a MIDI keyboard to hearing a corresponding sound—is very important. The addition of the Windows WASAPI driver is a very welcome improvement in FluidSynth for reducing latency.

WASAPI supports two modes, shared and exclusive. Shared mode presents the Windows Audio Engine (WAE) as a virtual audio device. It is an “omnivorous” device that accepts many audio formats from multiple applications and then resamples and mixes as necessary to deliver audio samples to the real audio DAC hardware. While this abstraction is convenient, it comes at a cost in latency. In a few tests on one mid-grade consumer laptop, latency in WASAPI shared mode was around 40 milliseconds, which is enough to be perceived as lag by a performer.

WASAPI exclusive mode talks directly to the audio hardware. This is a brittle interface: there is no translation of the audio samples provided by the driver, and the driver must supply samples in a format accepted by the hardware. The set of formats accepted by audio hardware is usually very limited. The fluidsynth -Q command can be used on a Windows machine to enumerate supported formats.

Issue

The issue we encountered when implementing FluidSynth’s WASAPI exclusive mode in the Miditzer is that, on most audio hardware, we are effectively limited to using "16bits" as the audio.sample-format setting. In shared mode we can use "float" to provide the full precision of FluidSynth’s synthesizer output to the WAE, which can then resample and deliver 24-bit samples to the audio hardware if configured appropriately. With the current limitations of the FluidSynth WASAPI driver, we can reduce latency by using exclusive mode, but at the potential cost of reduced output precision. As it stands, we can offer two options in the Miditzer:

• Higher latency, higher precision with WASAPI shared mode
• Lower latency, reduced output precision relative to the internal float path with WASAPI exclusive mode
  (high and low are used here in a relative sense; both latency and quality are good in absolute terms)

At a hardware level, DACs are fixed-point devices. Any “float” audio path must be converted to fixed-point somewhere before being presented to the analog conversion hardware. From that perspective, 24-in-32 PCM is a hardware realization of “float” intent: full-scale linear PCM with more resolution than the analog noise floor. Recognition of IEEE float format by audio hardware has no direct analog meaning at the DAC interface.

In exclusive mode, many audio devices reject IEEE float formats but accept high-resolution PCM (commonly 24-bit PCM in a 32-bit container). When audio.sample-format=float is requested, this can cause exclusive-mode initialization to fail, even though the hardware is capable of full-resolution, low-latency output.

Proposal

I propose enhancing fluid_wasapi.c so that, in WASAPI exclusive mode, a request for audio.sample-format="float" may be realized using a 24-in-32 PCM format when IEEE float is not accepted but higher-resolution PCM is supported by the selected device. This behavior would apply only to WASAPI exclusive mode and would not affect shared mode behavior.

Questions

Would such an enhancement align with FluidSynth’s design goals?

Are there other commonly supported hardware formats that should be considered in addition to 24-in-32 PCM for realizing float-quality output?

Should fluidsynth -Q report exclusive-mode float support realized via 24-bit PCM simply as float support (as it does now), or should this distinction be reported explicitly?

---

I believe I can provide an implementation of the enhancement I’m proposing.

Thanks for any thoughts.

[derselbst]

I propose enhancing fluid_wasapi.c so that, in WASAPI exclusive mode, a request for audio.sample-format="float" may be realized using a 24-in-32 PCM format

Generally, I'm open to support this use-case. However, I would refrain from abusing audio.sample-format="float" for this. I'd rather prefer to introduce a new option, like 24bits, or 32bits.

Are there other commonly supported hardware formats that should be considered in addition to 24-in-32 PCM for realizing float-quality output?

Not to my knowledge.

Should fluidsynth -Q report exclusive-mode float support realized via 24-bit PCM simply as float support (as it does now), or should this distinction be reported explicitly?

fluidsynth -Q should report it and having a dedicated option will make it clearer, what's actually being used.

---

Regarding the implementation of "24bit audio": There are three options that come to my mind:

1. 24bit "packed" integer (i.e. only using 3-bytes, seems uncommon).
2. Wrapping 24bit samples in a 32bit signed int, but keeping the original 24bit range of [-1^23 ; 1^23 -1], i.e. the most significant byte effectively stays zero.
3. Promoting 24bit samples to 32bit signed int, i.e. left shift by 8bits to utilize the full [-1^31 ; 1^31 -1] range, effectively having the least significant byte zero (this is the convention used by libsndfile, which I think is a good one).

It's not yet entirely clear to me which of the two latter options you would be using in the end. I looked up the Microsoft docs, but couldn't find the answer either.

If you would be using option 3. the name of the new option should be 32bits. If you're using option 2. the name of the new option should be 24bits.

Other things to consider:

• Adding a rendering function to the synth that support 24 / 32 bit int samples.
• Adding support to file-renderer.

[jimhen3ry]

I’m open to adding an explicit "24bits" option if that’s the preferred direction, but I am concerned about the blast radius: audio.sample-format is global, and we’d need to ensure that all existing drivers either handle the new value sensibly or fail in a clearly defined way.

As a smaller-scope alternative that keeps the change tightly confined to the exclusive mode of the WASAPI driver, we could make the effective transport explicit in the -Q output. Concretely, the WASAPI exclusive path could emit a single FLUID_INFO log message when an integer PCM transport format is used. This would cause -Q to report the mode as OK(I), making it visible that the requested format was realized using an integer transport. Using FLUID_INFO logging in this way parallels what the driver is already doing to report supported modes with resampling in shared mode.

The corresponding footnote could be updated to:

OK(I) : Mode is supported; Shared mode may involve resampling; Exclusive mode may use an integer transport format.

This keeps the behavior explicit and discoverable (via -Q) without expanding audio.sample-format semantics beyond what’s already documented. Would this address your concern about overloading audio.sample-format="float"?

---

The transport format is dictated by what the audio hardware will accept. On Windows, in exclusive mode, the audio hardware either accepts or rejects an exact WAVEFORMAT* that we propose via IAudioClient::IsFormatSupported / Initialize, so the driver can explicitly probe whichever representations we care about.

For the common “24-bit PCM in a 32-bit container” case, the Windows expectation (per WAVEFORMATEXTENSIBLE) is that the valid bits are left-aligned in the container (wBitsPerSample = 32, wValidBitsPerSample = 24, least-significant byte zero). That corresponds to your option (3), and it’s also the convention used by libsndfile.

A 32-bit container with only the lower 24 bits used (option 2) doesn’t appear to be a meaningful or commonly accepted PCM representation on WASAPI, so we likely wouldn’t probe that format.

Packed 24-bit (3-byte) PCM can be probed, but it is only occasionally supported by endpoints.

A possible probing order for the transport format, when the intent is to transmit the greatest available resolution, could be:

• IEEE float
• 32-bit integer PCM (32 valid bits)
• 24-bit PCM left-aligned in a 32-bit container
• packed 24-bit PCM (3-byte samples)

[derselbst]

I’m open to adding an explicit "24bits" option if that’s the preferred direction, but I am concerned about the blast radius: audio.sample-format is global, and we’d need to ensure that all existing drivers either handle the new value sensibly or fail in a clearly defined way.

Adding 24bit support to all the other drivers is out of scope. They should simply fail. This table could be extended to include the supported options.

This keeps the behavior explicit and discoverable (via -Q) without expanding audio.sample-format semantics beyond what’s already documented. Would this address your concern about overloading audio.sample-format="float"?

No. I expect that other audio drivers might follow in the future with 24bit support (ALSA for instance), and I don't want to live with that silent float-to-int hack forever.

For the common “24-bit PCM in a 32-bit container” case, the Windows expectation (per WAVEFORMATEXTENSIBLE) is that the valid bits are left-aligned in the container (wBitsPerSample = 32, wValidBitsPerSample = 24, least-significant byte zero). That corresponds to your option (3), and it’s also the convention used by libsndfile.

Thanks for the clarification! In this case however, I'd recommend the new option setting to be called 32bits, as it would align with a new rendering function fluid_synth_write_s32. If the hardware discards the least-significant-byte, well then that's beyond our control. But these are implementation details. Feel free to draft an implementation for the WASAPI driver. I'd only probe for 32bit int and 24bit left-aligned int32 PCM, but feel free.

I could support in adjusting the other audio drivers to fail with the new option, document it, or inventing the new synth rendering function.

[jimhen3ry]

Adding 32bits as an additional option for audio.sample-format seems like a good path forward. Perhaps this would be a roadmap for implementation:

1. Standardize audio drivers to fail on unsupported or unrecognized audio.sample-format values.
2. Add fluid_synth_write_s32 to the public synth rendering API.
3. Add support for audio.sample-format="32bits" in fluid_wasapi.c.
4. Extend fluid_wasapi_device_enumerate.c to probe and report 32bits support.

Would you like me to open Issues for these four tasks?

[derselbst]

Better: A single issue with a task list ;)

Regarding fluid_synth_write_s32: I'm thinking of porting the current fluid_synth_write_s16 to C++, templating it, and then call it like:

FLUIDSYNTH_API int fluid_synth_write_s32(...)
{
  return fluid_synth_write_int<int32_t>();
}


FLUIDSYNTH_API int fluid_synth_write_s16(...)
{
  return fluid_synth_write_int<int16_t>();
}

Would you like to do that or should I? (I'll probably find time after Christmas).

[jimhen3ry]

I'll take a stab at the port. At worst, you can have a good laugh and throw away my attempt. Should I make that a task in the "32bits" issue, probably with a broader issue name, or open the port as a separate issue?

[derselbst]

Ok. Make it a task in the same issue. I'll adjust it if need be.