IOStream's method append(_ sampleBuffer:) not work as expected

[RainyTunes]

Describe the bug

As you suggested, I tried to use append(_ sampleBuffer:) to play audio during the stream.

However, I can only hear some noise which lasts half a second from audience playback after playing my 30-second audio (extracted from a video).

Here is my code

public func testPushAudio_CMSampleBuffer() {
        guard let fileURL = self.getURL() else { return }
        let asset = AVAsset(url: fileURL)
        let assetReader = try? AVAssetReader(asset: asset)
        do {
            
        } catch {
            print("Error creating asset reader: \(error)")
            return
        }

        guard let audioTrack = asset.tracks(withMediaType: .audio).first else {
            print("Audio track not found.")
            return
        }

        let audioOutputSettings: [String: Any] = [
            AVFormatIDKey: Int(kAudioFormatLinearPCM),
            AVLinearPCMIsBigEndianKey: false,
            AVLinearPCMIsFloatKey: false,
            AVLinearPCMBitDepthKey: 16
        ]

        let audioInput = AVAssetReaderTrackOutput(track: audioTrack, outputSettings: audioOutputSettings)
        if let reader = assetReader {
            reader.add(audioInput)
            reader.startReading()

            mixer.audioIO.lockQueue.async { [weak self] in
                guard let self else {
                    NSLog("IOStream self error")
                    return
                }
                while reader.status == .reading {
                    if let sampleBuffer = audioInput.copyNextSampleBuffer() {
                        let targetSampleTime = getTargetSampleTime(sampleBuffer)
                        NSLog("IOStream append sampleBuffer targetSampleTime = \(targetSampleTime)")
                        self.append(sampleBuffer, track: 1)
                    }
                }
            }

            if reader.status == .completed {
                print("Audio processing completed.")
            }
        }
        
    }

I am a rookie of AVFoundation, but after debugging, I think the following log might need attention.
It seems that the sample time log of my audio indicates it indeed is regarded as a 30-second audio and seems correct up to now, but the sampleTime of the two IOAudioRingBuffers are not consecutive. In addition, whenever I trigger playing my audio, its sampleTime always starts from 0.

I highly appreciate your work and understand you are busy. It would be really helpful if you could give me further suggestions. I will continue to work on this feature, which is an urgent development for us, so your response would be greatly appreciated.

To Reproduce

Please see above.

Expected behavior

can play normally

Version

1.9.1

Smartphone info.

iPhone 14 pro
iPad Air3

Additional context

append(audioBuffer, when: audioTime, track: 1)

This method has the same behavior as above, although the audioTime parameter is set.

Screenshots

No response

Relevant log output

swift
// built-in mic audio 
IOAudioRingBuffer=0x0000000302dda260 sampleTime=3473176256 skip = 0  length=1882
IOAudioRingBuffer=0x0000000302dda260 sampleTime=3473177197 skip = 0  length=1882
IOAudioRingBuffer=0x0000000302dda260 sampleTime=3473178138 skip = 0  length=1880
IOAudioRingBuffer=0x0000000302dda260 sampleTime=3473179078 skip = 0  length=1882
IOAudioRingBuffer=0x0000000302dda260 sampleTime=3473180019 skip = 0  length=1882
IOAudioRingBuffer=0x0000000302dda260 sampleTime=3473180960 skip = 0  length=1882
// ----------my audio starts----------(sampleTime starts from 0)
IOAudioRingBuffer=0x0000000302dc5630 sampleTime=0 skip = 0  length=12288
IOAudioRingBuffer=0x0000000302dc5630 sampleTime=3072 skip = 0  length=16384
IOAudioRingBuffer=0x0000000302dc5630 sampleTime=7168 skip = 0  length=32768
IOAudioRingBuffer=0x0000000302dc5630 sampleTime=15360 skip = 0  length=32768
IOAudioRingBuffer=0x0000000302dc5630 sampleTime=23552 skip = 0  length=32768
IOAudioRingBuffer=0x0000000302dc5630 sampleTime=31744 skip = 0  length=16384
......
......
IOAudioRingBuffer=0x0000000302dc5630 sampleTime=1289216 skip = 0  length=12288
IOAudioRingBuffer=0x0000000302dc5630 sampleTime=1292288 skip = 0  length=16384
IOAudioRingBuffer=0x0000000302dc5630 sampleTime=1296384 skip = 0  length=16384
IOAudioRingBuffer=0x0000000302dc5630 sampleTime=1300480 skip = 0  length=32768
IOAudioRingBuffer=0x0000000302dc5630 sampleTime=1308672 skip = 0  length=16384
IOAudioRingBuffer=0x0000000302dc5630 sampleTime=1312768 skip = 0  length=16384
IOAudioRingBuffer=0x0000000302dc5630 sampleTime=1316864 skip = 0  length=32484
// ----------my audio ends----------(sampleTime revert back to 34731xxxxx)
IOAudioRingBuffer=0x0000000302dda260 sampleTime=3473181901 skip = 0  length=1882
IOAudioRingBuffer=0x0000000302dda260 sampleTime=3473182842 skip = 0  length=1880
IOAudioRingBuffer=0x0000000302dda260 sampleTime=3473183782 skip = 0  length=1882
IOAudioRingBuffer=0x0000000302dda260 sampleTime=3473184723 skip = 0  length=1882
IOAudioRingBuffer=0x0000000302dda260 sampleTime=3473185664 skip = 0  length=1882
IOAudioRingBuffer=0x0000000302dda260 sampleTime=3473186605 skip = 0  length=1882
IOAudioRingBuffer=0x0000000302dda260 sampleTime=3473187546 skip = 0  length=1880
IOAudioRingBuffer=0x0000000302dda260 sampleTime=3473188486 skip = 0  length=1882
IOAudioRingBuffer=0x0000000302dda260 sampleTime=3473189427 skip = 0  length=1882
IOAudioRingBuffer=0x0000000302dda260 sampleTime=3473190368 skip = 0  length=1882

[shogo4405]

It is not designed to append all the data in a while loop like in the sample code, right?

[shogo4405]

If it exceeds approximately 0.371 seconds (1024 * 16 / 44100), it will exceed the capacity of the circular buffer and not work as intended. It might be possible to achieve what you want by appending in increments of 0.1 seconds.

In addition to CMSampleBuffer, you can also use AVAudioBuffer. With AVAudioEngine, there are methods to mix audio. Please consider this for your needs as well.

[RainyTunes]

Thank you for your detailed explanation. I now understand how it works.

I tried your method of pushing a 0.1 buffer one by one using thread sleep. It works, but the audio is not smooth.

Alternatively, I solved it by allocating a larger, flexible circular buffer and pushing with the same loop. I know this is not the best solution, but it works perfectly for me.

I would greatly appreciate it if you have a better solution for this.

[shogo4405]

I heard that the requirement was to play the video while extracting about 0.1 seconds of audio at the timing of the draw. Since this seems like a patch solution, I think it's good to address it by increasing the size of the RingBuffer.
https://github.com/shogo4405/HaishinKit.swift/blob/main/Sources/Screen/ScreenObject.swift#L458-L465