PylonCapture lags with PredictPose--part II

[edmundchong]

Hi, I'm having the same issue as #1233.
I think the solution there was to reduce the image resolution / size, or alternatively, I possibly can upgrade my GPU.

I am unable to do either in the near term, and in exploring workarounds, I had several questions + observations:

Firstly, I managed to avoid the lags for PylonCapture-->PredictPose, by changing GrabStrategy from OneByOne to LatestImages. However, while I do not need every frame for my online SLEAP tracking, I do need to save all frames for offline processing later. I was hoping that OneByOne could still work, but by pushing it to a PublishSubject which would then be separately piped into i) VideoWriter, ii) PredictPose:

I used TimeInterval to measure how fast each chain is--and I was surprised that PredictPose basically slows down whatever VideoWriter is doing, or essentially, the output of the PublishSubject itself. In other words, VideoWriter is forced to wait for frames at the same speed that PredictPose is operating on, and lag accumulates.

Alternatively, Dammy O. suggested this hack:

Where the timer just grabs a single frame periodically. This works--but only if the frequency of the timer is slower than the upper limit of what my PredictPose can do. If I'm trying to grab single frames faster than what PredictPose can handle, then I see again the same bottleneck issue as above--the chain going to VideoWriter is slowed down.

Is there a more correct way of achieving this split between VideoWriter (where I want all frames), and PredictPose (where I'm ok with dropping a few frames here and there), and not having any lag?

[bruno-f-cruz]

You can decouple the bottleneck by using ObserveOn. This way the videowriter/acquisition should not have to wait on the thread from the pose estimation.

Alternatively (which is a simpler solution) simply use a Slice operator to downsample the input to the inference operator.

[edmundchong]

Slice works great, thanks!