MediaStream API should support pan, zoom and tilt

[T1mmos]

⚙ Compilation target

ES2022

⚙ Library

TypeScript 5.3.2

Missing / Incorrect Definition

The interfaces MediaTrackCapabilities and MediaTrackConstraints (MediaTrackConstraintSet) should include the zoom, tilt and pan properties, so control of the camera is accessible via TypeScript.

In pure JavaScript, this works: https://googlechrome.github.io/samples/image-capture/update-camera-zoom.html

In TypeScript, the necessary properties for camera control are unavailable: https://github.com/microsoft/TypeScript/blob/main/src/lib/dom.generated.d.ts (e.g. search for "zoom" -> no results).

Context: a use case where (optical) zoom is required is where a user needs to scan a very tiny QR code (1 cm2). If you move your phone too far away, it's too small to scan; if you move too close, the camera cannot focus on it. A solution here is to move your phone further away but allow the user to zoom which would 1. enable the camera to focus and 2. make the QR code big enough in screen space. I noticed that on my phone, the native photo app zooms by default, no troubles scanning the QR code here.

Sample Code

interface MediaTrackCapabilities {
    aspectRatio?: DoubleRange;
    autoGainControl?: boolean[];
    channelCount?: ULongRange;
    deviceId?: string;
    displaySurface?: string;
    echoCancellation?: boolean[];
    facingMode?: string[];
    frameRate?: DoubleRange;
    groupId?: string;
    height?: ULongRange;
    noiseSuppression?: boolean[];
    sampleRate?: ULongRange;
    sampleSize?: ULongRange;
    width?: ULongRange;
}

interface MediaTrackConstraintSet {
    aspectRatio?: ConstrainDouble;
    autoGainControl?: ConstrainBoolean;
    channelCount?: ConstrainULong;
    deviceId?: ConstrainDOMString;
    displaySurface?: ConstrainDOMString;
    echoCancellation?: ConstrainBoolean;
    facingMode?: ConstrainDOMString;
    frameRate?: ConstrainDouble;
    groupId?: ConstrainDOMString;
    height?: ConstrainULong;
    noiseSuppression?: ConstrainBoolean;
    sampleRate?: ConstrainULong;
    sampleSize?: ConstrainULong;
    width?: ConstrainULong;
}

interface MediaTrackConstraints extends MediaTrackConstraintSet {
    advanced?: MediaTrackConstraintSet[];
}

Documentation Link

No response

[IllusionMH]

Is it supported anywhere except Chrome?
See https://github.com/microsoft/TypeScript-DOM-lib-generator?tab=readme-ov-file#why-is-my-fancy-api-still-not-available-here which notes that at least 2 different engines should support it to be included.

You can expand those types for your project using declaration emerging.

[RyanCavanaugh]

This API appears to be quite old so I'd assume it's well-supported by now https://github.com/webrtc/samples/blob/gh-pages/src/content/getusermedia/pan-tilt-zoom/js/main.js#L29

[fatcerberus]

declaration emerging

Is this the new TS feature where types for stuff just miraculously appear from the aether?

[ivancuric]

Hey, noticed the same thing for other advanced properties such as focusMode and torch. Currently using type augmentation:

declare global {
  interface MediaTrackSettings {
    torch?: boolean;
    focusMode?: "none" | "manual" | "single-shot" | "continuous";
  }

  interface MediaTrackCapabilities {
    torch?: MediaTrackSettings["torch"];
    // TODO: check if this is always an array
    focusMode?: MediaTrackSettings["focusMode"][];
  }

  interface MediaTrackConstraintSet {
    torch?: MediaTrackSettings["torch"];
    focusMode?: MediaTrackSettings["focusMode"];
  }
}

[RyanCavanaugh]

The camera-control constraints are supported by the platform but are still absent from the DOM declarations. TypeScript 5.3.2 and the current native compiler both report TS2339 for zoom, pan, and tilt on MediaTrackCapabilities and MediaTrackConstraintSet.

declare const capabilities: MediaTrackCapabilities;
declare const constraints: MediaTrackConstraintSet;

// Expected: The default DOM library declares zoom, pan, and tilt on both media-track types.
// Actual: Each access reports TS2339.
capabilities.zoom;
capabilities.pan;
capabilities.tilt;
constraints.zoom;
constraints.pan;
constraints.tilt;

A current secure browser context reports all three as supported media constraints.