Desktop app · Windows · proof of concept

The desktop, Minority Report style

A webcam tracks both hands: point, click, carry and snap windows, scroll, close, zoom the whole screen and turn the volume, with a transparent overlay showing where each hand is and what it is about to do.

It is a proof of concept, built to test whether webcam gesture control of a desktop is usable. It grew from Hand Volume Control, rebuilt on the current MediaPipe Tasks API.

Source code

Gestures

Wake it up

Nothing happens until you hold an open palm to the camera for half a second; it stops two seconds after your hands leave view.

Point and click

Move your open hand; pinch and let go to click, twice to double-click. The ring snaps to buttons, links and tabs nearby.

Carry and snap

Pinch on a window and move to carry it; take it to an edge to maximise, snap to a half or minimise.

Scroll

Two fingers up, moved up and down, scroll the window under the ring.

Zoom the screen

Pinch with both hands and spread them to zoom the whole screen; close them to return to 1x.

Volume

Other hand: pinch and slide up or down, 2 % a step. A fist held for 1.5 s closes a window.

Why these gestures

Each reuses a habit you already have: pinch and drag copy the mouse, two-finger scroll is the trackpad's, spreading two hands is pinch-to-zoom. Closing a window is deliberately slow, because it is the one destructive action.

A raised hand shakes, so it never asks you to hit a pixel: it snaps to the nearest target (a bubble cursor, using Windows UI Automation), zoom makes pointing finer, and a One Euro filter smooths the hand.

Run it

Needs Windows 10 or 11, a webcam, and uv, which installs Python 3.12 and the dependencies. Calibrate once for your distance and reach; --swap-hands is for left-handed use.

uv sync
uv run gesture-cameras            # list cameras
uv run gesture-control --calibrate
uv run gesture-control --outline hand

Known limits

  1. Windows running as administrator cannot be moved, clicked or scrolled from it.
  2. Fullscreen windows are not carried; pointer and zoom cover the primary monitor only.
  3. Snapping finds only what an app exposes to UI Automation.

Privacy

Camera frames are processed in memory and never saved or sent. One network request, on the first run only, fetches the MediaPipe gesture model from storage.googleapis.com; after that it works offline. It stores only your calibration, and reads only the kind and position of controls near the pointer, never their text.

This page stores only your light or dark choice, whether you hid the intro, and your answer to the notice, all on this device.