Everyone loves emojis 👻

Everyone loves video 🎥

Here you get both at once 🤯

Learn how you can combine the webcam API in the browser with emojis in a creative way.

This is the sequel to the blog post about ASCII. There you can read about how web technology can convert pixels in an image to ASCII and emojis.

Now we're taking it a step further, and using the webcam API in the browser to transform a stream of images into a stream of emojis 👻👻👻

Demo time

Before I explain how it works, let's look at the finished product.

How does it work?

A video is just a sequence of images. From the previous blog post, we know how to convert pixels in a single image to emojis. All we need to do is:

1. Request access to the webcam
2. At regular intervals, take a picture from the webcam stream
3. Draw that picture to a canvas element
4. Use the canvas API to read pixel data, exactly as in the previous blog post

Access to webcam

To read a video stream from a webcam, we use the mediaDevices API in the browser. It is well supported across all browsers and gives us the ability to request access to the user's audio and video devices.

Starting a video stream can be done as follows:

const stream = await navigator.mediaDevices.getUserMedia({
  audio: false,
  video: true
});

The user will then be asked to approve that the page gets access to a camera. The API offers several options to customize which device you want access to, e.g., you can easily request the camera that points forward or backward on a mobile phone. For our purposes in this blog post, we'll keep it simple, but there's plenty of room for customization here.

To start the video, we need a video element on the page.

<video style="display:none"
       playsinline 
       muted />

We don't care about seeing the actual video stream, so we hide it with display:none. The playsinline attribute means that the video plays in the page, instead of fullscreen. Note that on Safari, the video won't play unless the video element is within the screen, so there you need to cheat a bit with CSS to force the element to always be within the viewport.

Then we take the camera stream and send it to a video element on the page:

videoElement = document.querySelector('video');
videoElement.srcObject = stream;
await videoElement.play();

Capturing an image from the stream

Here we use the canvas API to read an image. We can use context.drawImage() which takes a CanvasImageSource as input. This can be an image, but it can also be a video element.

type CanvasImageSource = 
  HTMLOrSVGImageElement | 
  HTMLVideoElement | 
  HTMLCanvasElement | 
  ImageBitmap;

So we simply do:

const canvas = document.createElement('canvas');
const context = canvas.getContext('2d');
context.drawImage(videoElement, 0, 0, width, height)

Extracting pixel data from the video image

To extract all the pixels in the image:

const imageData = context.getImageData(0, 0, width, height)

We can then convert this to emojis in exactly the same way as I showed in the previous blog post.

It's as simple as that. Web technology never ceases to amaze!

Part 3?

Videomoji is fun, but it can get even better. The next thing I'm going to do is map all the existing emojis to the colors they best match. Then we'll get a video stream with colors!

If you want to try this out on your own, you can read the source code here