WEBGL_texture_from_depth_video WebGL working group (public_webgl 'at' khronos.org) Ningxin Hu, Intel Anssi Kostiainen, Intel Rob Manson, buildAR Members of the WebGL working group NN

This extension provides support for the Media Capture Depth Stream Extensions. Specifically, it supports uploading to a WebGL texture a video element whose source is a MediaStream object containing a depth track.

A video element whose source is a MediaStream object containing a depth track may be uploaded to a WebGL texture of format RGB and type UNSIGNED_SHORT_5_6_5.
[NoInterfaceObject] interface WEBGL_texture_from_depth_video { };
This code sets up a video element from a depth stream, uploads it to a WebGL texture, and samples that texture in the fragment shader, reconstructing the 16-bit depth values from the red, green and blue channels.
var ext = gl.getExtension("WEBGL_texture_from_depth_video");
if (ext) {
  navigator.getUserMedia({ video: true }, successVideo, failureVideo);
}

var depthVideo;

function successVideo(s) {
  // wire the stream into a <video> element for playback
  depthVideo = document.querySelector('#video');
  depthVideo.src = URL.createObjectURL(s);
  depthVideo.play();
}

// ... later, in the rendering loop ...

if (ext) {
 gl.texImage2D(
     gl.TEXTURE_2D,
     0,
     gl.RGB,
     gl.RGB,
     gl.UNSIGNED_SHORT_5_6_5,
     depthVideo
   );  
}

<script id="fragment-shader" type="x-shader/x-fragment">
  varying vec2 v_texCoord;
  // u_tex points to the texture unit containing the depth texture.
  uniform sampler2D u_tex;
  void main() {
    vec4 floatColor = texture2D(u_tex, v_texCoord);
    vec3 rgb = floatColor.rgb;
    ...
    float depth = 63488. * rgb.r + 2016. * rgb.g + 31. * rgb.b;
    ...
  }
</script>
Initial revision. Moved to rejected on the grounds that it can be done without an extension.