iOS publisher/subscriber video elements always letterboxed or pillarboxed

[aoberoi]

Perhaps because of limitations in the MediaStreamRenderer from com.eface2face.iosrtc, publisher video elements, when not at the exact aspect ratio of the device's camera's video frame, always scale the image to fit inside the element (using letterboxing and pillarboxing). This is similar to the fitMode: 'contain' behavior, but cannot be turned off.

[aoberoi]

This happens to the subscriber video element as well.

[ibc]

Hi, I'm 100% interested and open to discuss the proper behavior of the video size in order to mimic the HTML <video> element behavior as much as possible. Indeed this may require some changes in the cordova-plugin-iosrtc.

May you please open a bug in the tracker so we can discuss about it?

[aullman]

@ibc this is actually the same behaviour as the HTML <video> element. It's just different than the OpenTok platform. opentok.js does some sizing and masking of the video element to enable 2 different fit modes, contain and cover. cover is the default for us which crops the video rather than letter/pillar boxing it. I'm not sure how we could get the same effect in iosrtc.

[aoberoi]

@adam: even if this was about the fact that the DOM cannot place anything over the <video> elements, shouldn't the <video> element itself be scaled up to the size we logically want the frame before cropping? i'd understand if it wouldn't get masked off at the boundaries of the publisher or subscriber <div>, but thats not what i'm seeing here.

@ibc i think iosrtc MediaStreamRenderer is doing the right thing as per the spec and the default styles of the browser (the default stylesheet for the browser has object-fit: contain). but if you were to apply object-fit: cover on the <video> element, it would scale the video up so that the entire element had some part of the video frame in it, and there would be no letter/pillar boxing. this is the part that i think you might be able to help implement, and also i think the part that accounts for my comment above.

[aoberoi]

in summary i think the iosrtc MediaStreamRenderer has to read the object-fit property from the video element, and if it is set to cover then it also has to inspect the width and height so that it can effectively compute which parts of the video frame to render and how big to scale it.

[aoberoi]

ah crap, sorry for the frequent updates.

even if iosrtc implements what i said above, this would be different from the standard, because the standard implementation allows the video frame to be larger than its container and the video pokes out of the edges. that would be a bad implementation for opentok to work with because on iOS we cannot crop those parts out by wrapping it inside a parent with overflow: hidden.

hence the standard implementation is what i described above but only the scaling and no cropping. i'd actually prefer a slightly non-standard implementation so we can address this use case.

[aullman]

Basically this is what we want to be able to do: http://jsbin.com/yaheka/edit