Skip to content

Commit 2a561e8

Browse files
author
Miguel Ponce de Leon
committed
Building initial draft of note for #1 
1 parent 8d90216 commit 2a561e8

10 files changed

+329
-0
lines changed

‎docs/research-notes/webrtc-getUserMedia/.gitignore

+5
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,5 @@
1+
build.properties
2+
3+
html/
4+
5+
pdf/

‎docs/research-notes/webrtc-getUserMedia/build.properties.sample

+8
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,8 @@
1+
name.project=webrtc-getUserMedia
2+
pdf.builder.path=../../../../../github/openRMC/asciidoc2html-pdf/asciidoc2html-pdf/asciidoc2html-pdf.sh
3+
pdf.builder.output=pdf
4+
html.builder.path=../../../../../github/openRMC/asciidoc2html-pdf/asciidoc2html-pdf/asciidoc2html-pdf.sh
5+
html.builder.output=html
6+
html.builder.publish=/Volumes/docs-dav/mca/research-notes
7+
perforce.workingdir=/Users/epower/Documents/perforce/projects/MCA-EI-CF-2011-1303/wip/research-notes
8+

‎docs/research-notes/webrtc-getUserMedia/build.xml

+65
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,65 @@
1+
<project>
2+
<!-- load based properties from home dir -->
3+
<property file="${user.home}/doc-build.properties" />
4+
5+
<!-- load based properties from local dir -->
6+
<property file="build.properties" />
7+
8+
<dirname property="name.dir" file="build.xml"/>
9+
10+
<target name="clean-all" depends="clean-html,clean-pdf" />
11+
12+
<target name="clean-html">
13+
<delete dir="${html.builder.output}" />
14+
</target>
15+
16+
<target name="clean-pdf">
17+
<delete dir="${pdf.builder.output}" />
18+
</target>
19+
20+
21+
<target name="html" depends="clean-html" description="makes HTML draft of document">
22+
<mkdir dir="${html.builder.output}" />
23+
<copy todir="${html.builder.output}/images" failonerror="false" >
24+
<fileset dir="src/images" />
25+
</copy>
26+
<exec executable="/bin/sh">
27+
<arg line='-c "${html.builder.path} html ${name.dir}/src/${name.project}.asciidoc ${name.dir}/${html.builder.output}"'/>
28+
</exec>
29+
<echo message="${name.dir}/src/${name.project}.asciidoc"/>
30+
</target>
31+
32+
<target name="pdf" depends="clean-pdf" description="makes a PDF draft of the document">
33+
<mkdir dir="${pdf.builder.output}" />
34+
<exec executable="/bin/sh">
35+
<arg line='-c "${pdf.builder.path} pdf ${name.dir}/src/${name.project}.asciidoc ${name.dir}/${pdf.builder.output}"'/>
36+
</exec>
37+
</target>
38+
39+
<target name="web-deploy" depends="html" description="deploys the HTML document to the configured webdav share">
40+
<mkdir dir="${html.builder.publish}/${name.project}" />
41+
<copy todir="${html.builder.publish}/${name.project}" >
42+
<fileset dir="${html.builder.output}" />
43+
</copy>
44+
</target>
45+
46+
<target name="p4-deploy" depends="pdf" description="deploys the PDF document to the configured Perforce directory">
47+
<copy file="${pdf.builder.output}/${name.project}.pdf" todir="${perforce.workingdir}"/>
48+
49+
<exec executable="p4">
50+
<arg value="add"/>
51+
<arg value="${perforce.workingdir}/${name.project}.pdf"/>
52+
</exec>
53+
54+
<exec executable="p4">
55+
<arg value="submit"/>
56+
<arg value="-d"/>
57+
<arg value="added ${name.project}.pdf"/>
58+
</exec>
59+
60+
</target>
61+
62+
<target name="deploy" depends="web-deploy,p4-deploy" description="builds and deploys the documents">
63+
64+
</target>
65+
</project>

‎docs/research-notes/webrtc-getUserMedia/src/bib.asciidoc

+10
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,10 @@
1+
// bibliogrpahy section is a standard asciidoc bibliography list
2+
// http://www.methods.co.nz/asciidoc/userguide.html#_bibliography_lists
3+
4+
[bibliography]
5+
.References
6+
- [[[gum]]] Burnett, C. et al; "Media Capture And Streams"; W3C 2012; http://www.w3.org/TR/mediacapture-streams/ - Retrieved 28/6/2013
7+
- [[[gum-status]]] Power E, McLaughlin J; "WebRTC getUserMedia - What happens when there’re missing media sources?"; TSSG 2013; https://projects.lab.tssg.org/docs/mca/research-notes/gum-report/report.html - Retrieved 28/6/2013
8+
- [[[webrtc-context]]] McLaughlin J; "openrmc.webrtc.Context"; TSSG 2013; https://projects.lab.tssg.org/docs/mca/dev-notes/js-openrmc-api/js-openrmc-api.html#openrmc.webrtc.Context
9+
- [[[webrtc-init]]] McLaughlin J; "openrmc.webrtc.api.init"; TSSG 2013; https://projects.lab.tssg.org/docs/mca/dev-notes/js-openrmc-api/js-openrmc-api.html#openrmc.webrtc.api.methods
10+

‎docs/research-notes/webrtc-getUserMedia/src/feature9930-background.asciidoc

+14
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,14 @@
1+
+getUserMedia()+ as specified in <<gum>> is somewhat limited in the diagnostics
2+
that are given to the user in the event of an error occurring. In an
3+
experiment <<gum-status>> we mapped the results of calling +getUserMedia()+
4+
requesting various combinations of media against the presence or otherwise
5+
of the relevant input devices.
6+
7+
This is further complicated by the cross browser differences in the type
8+
and structure of errors returned (at the time of writing neither Chrome
9+
nor Firefox conform to the W3C specification of +NavigatorUserMediaError+
10+
as laid out in <<gum>>. And that's even before considering browsers which
11+
don't support +getUserMedia()+ in the first place.
12+
13+
The goal of this task is to provide better, uniform, diagnostics for the
14+
application developer, so they can better provide feedback to the user.

‎docs/research-notes/webrtc-getUserMedia/src/feature9930-implementation.asciidoc

+93
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,93 @@
1+
Depsite the lack of granularity in the error responses returned by
2+
+getUserMedia()+, with a little forensic work it is possible to derive a
3+
better picture of the WebRTC media capabilities of the host browser.
4+
5+
In the first instance, support of any kind for +getUserMedia()+ is tested
6+
and if not present, an error is raised.
7+
8+
Where +getUserMedia()+ is supported, a method can then be used to check
9+
whether or not multimedia input devices are available to the application.
10+
Key to this is accessing the MediaStream API methods +getAudioTracks()+ and
11+
+getVideoTracks()+, both returning arrays, on the stream passed into the
12+
+success+ callback supplied to +getUserMedia()+.
13+
14+
The required contraints parameter for +getUserMedia()+ has two boolean
15+
attributes +audio+ and +video+ allowing the developer to request access to
16+
audio and video input respectively.
17+
18+
If audio is requested, and +getAudioTracks()+ has a length of 0 then it can
19+
be inferred that no microphone is available. It may be physically not present
20+
or blocked by another process, but in any case it is not available to the
21+
requesting application. The same applies to video and +getVideoTracks()+.
22+
23+
In the case of the entire call to +getUserMedia()+ failing for some reason,
24+
the +failure+ callback is called with an error.
25+
26+
=== Errors ===
27+
28+
The +getUserMedia()+ errors that can be raised by openRMC are laid out below.
29+
There are two contexts in which errors are raised, both outlined in the
30+
following sections, which raise a different subset of errors.
31+
32+
* *+NOT_SUPPORTED+* - WebRTC is not supported on the host browser
33+
* *+PERMISSION_DENIED+* - Access to multimedia input devices is denied
34+
* *+CONSTRAINT_NOT_SATISFIED+* - A mandatory constraint was requested, and
35+
could not be satisfied
36+
* *+CONSTRAINTS_REQUIRED+* - At least one of audio or video must be requested
37+
* *+AUDIO_NOT_AVAILABLE+* - Raised when audio is requested, but no audio
38+
input is available
39+
* *+VIDEO_NOT_AVAILABLE+* - Raised when video is requested, but no video
40+
input is available
41+
* *+AV_NOT_AVAILABLE+* - Raised when both audio and video are requested,
42+
but neither audio nor video inputs are available
43+
44+
45+
=== Error Checking Scenarios ===
46+
47+
In the openRMC scenario, there are two instances where errors are returned,
48+
the exact subset of errors depending on the calling context:
49+
50+
==== +openrmc.webrtc.api.init()+ ====
51+
52+
+getUserMedia()+ is called with optional audio/video media constraints as
53+
part of the WebRTC API initialisation. The capabilities available to the
54+
app are then tested, and a context created.
55+
56+
The +failure(errCxt)+ callback is called with a context +errCxt+ with an
57+
+error+ attribute corresponding to one of the openRMC media errors outlined
58+
above. The possible error returned are +NOT_SUPPORTED+, +PERMISSION_DENIED+,
59+
and +CONSTRAINTS_REQUIRED+
60+
61+
==== +openrmc.webrtc.api.getUserMedia() ====
62+
63+
+openrmc.webrtc.api.getUserMedia()+ wraps the native +getUserMedia()+ and
64+
provides the enhanced diagnostics described in this document.
65+
The +failure(errCxt)+ callback is called with a context +errCxt+ with an
66+
+error+ attribute corresponding to one of the openRMC media errors outlined
67+
above. This can contain any of the error conditions described above.
68+
69+
70+
=== Application Checking Of Capabilities ===
71+
72+
The WebRTC API must be properly initialised by calling
73+
+openrmc.webrtc.api.init()+. This created a WebRTC context awhich can be
74+
queried for capabilities. This is available as an attribute +baseContext+
75+
76+
+baseContext+ (and indeed any context passed to the +success(cxt)+ callback)
77+
has a methods to check availablity of a particular resource:
78+
79+
* +isrtcavailable()+ - +true+ if WebRTC is available, +false+ otherwise
80+
* +isrtcaudioavailable()+ - Returns +true+ if an audio input is available,
81+
otherwise +false+
82+
* +isrtcvideoavailable()+ - Returns +true+ if a video input is available,
83+
otherwise +false+
84+
85+
With respect to +baseContext+, these methods are are all wrapped in
86+
convenience methods with the same signature in +openrmc.webrtc.api+
87+
88+
It is the application's responsibility to query +baseContext+ for host
89+
browser capabilities following a call to +openrmc.webrtc.api.init()+.
90+
Should a subsequent call to +openrmc.webrtc.api.getUserMedia()+ be
91+
made requesting a resource or resources not detected on init, one of the
92+
+XXX_NOT_AVAILABLE+ errors will be raised.
93+

‎docs/research-notes/webrtc-getUserMedia/src/feature9930-risks.asciidoc

+12
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,12 @@
1+
The primary risk at the time of writing is a combination of lack of
2+
cross-browser consistency in terms of the native errors returned by calls to
3+
+getUserMedia()+. As the spec is evolving, the browser implementations will
4+
also evolve, and this required tracking.
5+
6+
A further complications is lack of support for +getAudioTracks()+ and
7+
+getVideoTracks()+ is streams returned by +getUserMedia()+. Chrome supports
8+
these calls, and behaves as per the spec. Firefox however as of version 22
9+
does not. The are present and correct in Firefox 23 and behave similarly to
10+
Chrome, but in any earlier Firefox version, there is simply no way for
11+
an application to access the media streams. This is considered a low risk
12+
as Firefox 23 is due to move to "RELEASE" status on 6/8/2013

‎docs/research-notes/webrtc-getUserMedia/src/gum-report.asciidoc

+86
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,86 @@
1+
:reporttype: Research Note TSSG-2013
2+
:reporttitle: WebRTC getUserMedia - What happens when there's missing media sources?
3+
:author: Eamonn Power
4+
:email: [email protected]
5+
:group: Telecommunications Software and Systems Group (TSSG)
6+
:address: Waterford Institute of Technology, West Campus, Carriganore, Waterford, Ireland
7+
:revdate: May 17, 2013
8+
:revnumber: Issue 4190
9+
:docdate: May 17, 2013
10+
:description: JavaScript, getUserMedia
11+
:legal: (C) Waterford Institute of Technology
12+
:encoding: iso-8859-1
13+
:toc:
14+
We wanted to know what should and would happen, from the perspective of a JavaScript app, if a browser makes a getUserMedia request on a system where either the camera, mic or both are inactive or missing. This question arose from functional testing with our EFM contact, Phelim. We set up a quick client-side only test to try it out on Chrome 26.0.1410.65. The HTML document is compiled Jade and a small bit of JavaScript.
15+
16+
-----
17+
18+
navigator.getMedia = ( navigator.getUserMedia ||
19+
navigator.webkitGetUserMedia ||
20+
navigator.mozGetUserMedia ||
21+
navigator.msGetUserMedia);
22+
23+
24+
function test(audioreq, videoreq){
25+
navigator.getMedia (
26+
27+
// constraints
28+
{
29+
video: videoreq,
30+
audio: audioreq
31+
},
32+
33+
// successCallback
34+
function(localMediaStream) {
35+
var video = document.querySelector('video');
36+
video.src = window.URL.createObjectURL(localMediaStream);
37+
console.log('| Contraints: audioreq: ' + audioreq +
38+
', videoreq: ' + videoreq +
39+
' - Audio Tracks:' + localMediaStream.getAudioTracks().length +
40+
', Video Tracks: ' + localMediaStream.getVideoTracks().length);
41+
},
42+
43+
// errorCallback
44+
function(err) {
45+
console.log('Contraints: audioreq: ' + audioreq +
46+
', videoreq: ' + videoreq + ' - err:' + err.code);
47+
}
48+
49+
);
50+
};
51+
52+
// test(false, false); - not included as it errors out
53+
test(true, false);
54+
test(false, true);
55+
test(true, true);
56+
57+
-----
58+
59+
The source code for the JavaScript is based on an example provided by Mozilla with additional results rendering for our purposes.
60+
61+
-----
62+
<!DOCTYPE html>
63+
<html>
64+
<head>
65+
<title>GUM demo</title>
66+
</head>
67+
<body>
68+
<video></video>
69+
<script src="gum.js"></script>
70+
</body>
71+
</html>
72+
73+
-----
74+
75+
We wanted to see if an error appeared on the console (something we could handle in the page JS) and what addtional information we could infer the getUserMedia results to enhance user experience.
76+
77+
Here are the results of obervations (where A indicates an available audio track and V indicates an available video track):
78+
79+
[width="100%",options="header"]
80+
|=====
81+
| | None | Mic Only | Camera Only | Camera and Mic
82+
|audio:false, video: false|NOT SUPPD ERROR|NOT SUPPD ERROR|NOT SUPPD ERROR|NOT SUPPD ERROR
83+
|audio:true, video: false|PERMISS DENIED| A | PERMISS DENIED| A
84+
|audio:false, video: true |PERMISS DENIED| PERMISS DENIED| V | V
85+
|audio:true, video: true |PERMISS DENIED| A | V | AV
86+
|=====

‎docs/research-notes/webrtc-getUserMedia/src/images/mediastream.jpg

34.2 KB
Loading

‎docs/research-notes/webrtc-getUserMedia/src/webrtc-getUserMedia.asciidoc

+36
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,36 @@
1+
:reporttype: Research Note openRMC-2013
2+
:reporttitle: WebRTC What happens when there's missing media sources?
3+
:author: John McLaughlin, Eamonn Power, Miguel Ponce de Leon
4+
:email: [email protected], [email protected], [email protected]
5+
:group:
6+
:address:
7+
:revdate: September 09, 2013
8+
:revnumber: N/A
9+
:docdate: September 12, 2013
10+
:description: We wanted to know what should and would happen, from the perspective of a JavaScript app, if a browser makes a getUserMedia request on a system where either the camera, mic or both are inactive or missing.
11+
:legal: LICENSE.txt
12+
:encoding: iso-8859-1
13+
:toc:
14+
15+
== WebRTC getUserMedia ==
16+
The getUsermedia API is one of the 3 key features of WebRTC, as it is used to gain access to the users camera and microphone.
17+
18+
include::../../webrtc-key-features-content/src/getUsermedia.asciidoc[]
19+
20+
== What happens when there's missing media source ==
21+
22+
include::gum-report.asciidoc[]
23+
24+
== Specifying a comprehensive diagnostics for WebRTC initialisation ==
25+
26+
include::feature9930-background.asciidoc[]
27+
28+
=== Implementation Strategy ===
29+
30+
include::feature9930-implementation.asciidoc[]
31+
32+
== Risks ==
33+
34+
include::feature9930-risks.asciidoc[]
35+
36+
include::bib.asciidoc[]

0 commit comments

Comments
 (0)