How to Build Motion Controls in the Browser Using JavaScript and MediaPipe

How to Build Motion Controls in the Browser Using JavaScript and MediaPipe

Have you ever wanted to control a web app with your hands — like a wizard waving commands into reality? With modern web APIs and machine learning, you can! In this tutorial, you’ll learn how to use JavaScript, HTML5, and MediaPipe to detect hand gestures in real time and interact with the browser using motion controls.

Step 1: Get the Video Data

To start, you’ll need access to the user’s webcam. You can do this with the navigator.mediaDevices.getUserMedia() API, which captures a live video stream.

const constraints = { audio: false, video: { width, height } };

navigator.mediaDevices.getUserMedia(constraints)
  .then(function(mediaStream) {
    video.srcObject = mediaStream;
    video.onloadedmetadata = function() {
      video.play();
      setInterval(drawVideoFrame, 100);
    };
  })
  .catch(function(err) { console.log(err); });

function drawVideoFrame() {
  context.drawImage(video, 0, 0, width, height);
}

This simple snippet captures video and draws each frame onto a <canvas> every 100 milliseconds.

Step 2: Track Hand Motions with MediaPipe

Next, import Google’s MediaPipe Hands library to detect and visualize hand landmarks in the video feed.

<script src="https://cdn.jsdelivr.net/npm/@mediapipe/hands/hands.js"></script>
<video class="input_video"></video>
<canvas class="output_canvas" width="1280" height="720"></canvas>

<script>
const hands = new Hands({locateFile: (file) => {
  return `https://cdn.jsdelivr.net/npm/@mediapipe/hands/${file}`;
}});
hands.setOptions({
  maxNumHands: 1,
  modelComplexity: 1,
  minDetectionConfidence: 0.5,
  minTrackingConfidence: 0.5
});
</script>

The library provides a multiHandLandmarks array of coordinates for each hand. You can then map these points to a custom cursor that follows your hand in real time.

Step 3: Detect Gestures (Pinch Detection)

To add interactions, let’s detect when the user performs a “pinch” gesture — when the thumb and index finger come close together. By comparing their coordinates, we can determine whether a pinch has occurred.

function isPinched(handData) {
  const fingerTip = handData.multiHandLandmarks[0][8];
  const thumbTip = handData.multiHandLandmarks[0][4];
  const dx = Math.abs(fingerTip.x - thumbTip.x);
  const dy = Math.abs(fingerTip.y - thumbTip.y);
  const dz = Math.abs(fingerTip.z - thumbTip.z);
  return (dx < 0.08 && dy < 0.08 && dz < 0.11);
}

With debounce logic and custom events (pinch_start, pinch_move, pinch_stop), you can make your web elements respond naturally to real-world hand movements.

Interactive Demos

• Beam Sword — Fun with Motion Controls
• Magic Quill — Air Writing

Conclusion

And that’s it! You’ve just built a basic framework for gesture-controlled web applications. With webcam data, MediaPipe tracking, and JavaScript gesture logic, you can create immersive, touchless experiences. From interactive art to AR-style interfaces, the possibilities are endless.

What kind of motion-controlled apps will you create? Share your experiments in the comments!