Demo: https://editor.p5js.org/heyozramos/sketches/LVsjLSPt7
A while back I came across a really fun tweet by @cassidoo on Twitter
At the end of the clip, there's a scene where a million copies of her are singing together. It's a really cool effect and in this first tutorial in a series on Segmentation Effects, I'll show you how to recreate it with JavaScript!
Our tools
We'll be using:
- p5.js in global mode. p5.js is a beginner-friendly creative coding library
- Google's Bodypix. Bodypix is a segmentation and pose estimation computer vision library powered by TensorFlow. We'll initialize BodyPix through Handsfree.js
- Handsfree.js is my library for interacting with web pages handsfree
<head> <script src="https://cdn.jsdelivr.net/npm/p5@0.10.2/lib/p5.js"></script> <script src="https://unpkg.com/handsfree@6.1.7/dist/handsfree.js"></script> <link rel="stylesheet" type="text/css" href="https://unpkg.com/handsfree@6.1.7/dist/handsfree.css"> </head> Instantiating Handsfree.js
Handsfree.js is a library that wraps popular computer vision models like Jeeliz Weboji (head tracking), Google's BodyPix (segmentation and pose estimation), and soon, eye tracking through WebGazer and Handtracking through Handtrack.js. I originally started it as an assistive tech tool to help people browse the web handsfree through face gestures, but it also doubles as a controller for games, Algoraves, and other creative coding projects!
Oz Ramos@heyozramos
Day 1 of #100DaysofCode and #100DaysofMLCode
My goal with Handsfree.js will be to wrap JavaScipt based computer vision libraries under a single API for the purpose of interacting with web pages (with head, body, hand tracking)
Here's a preview (link at end is broken, will fix!)00:36 AM - 05 Nov 2019
12
42
So to begin, we need to instantiate Handsfree.js. We instantiate once for every camera that we want to use (if you have a multi-cam setup or using a mobile device with front/back webcams). Instantiation looks like this:
/** * This method gets called once automatically by p5.js */ function setup () { // Instantiate Handsfree. "H"andsfree is the class, "h"andsfree is the instance handsfree = new Handsfree({ autostart: true, models: { head: {enabled: false}, bodypix: { enabled: true, // Uncomment this if it runs slow for you // modelConfig: { // architecture: 'MobileNetV1', // outputStride: 8, // multiplier: 0.75, // quantBytes: 2 // } } } }) // Disable default Handsfree.js plugins Handsfree.disableAll() } Creating a buffer canvas
The next thing we'll need to do is create a "buffer" canvas. This canvas will contain the cutout, which we'll then paste into the p5 canvas a bunch of times:
function setup () { // ... // Create a canvas that'll contain our segmentation $buffer = document.createElement('canvas') $buffer.style.display = 'none' bufferCtx = $buffer.getContext('2d') document.body.appendChild($buffer) // This part's important, match the buffer dimensions to the webcam feed $buffer.width = handsfree.debugger.video.width $buffer.height = handsfree.debugger.video.height widthHeightRatio = $buffer.width / $buffer.height // Create a slider to adjust the number of clones slider = createSlider(3, 12, 4, 1) // Finally, create the p5 canvas renderer = createCanvas(600, 600) $canvas = renderer.canvas canvasCtx = $canvas.getContext('2d') } Creating a mask
(GIF source: https://blog.tensorflow.org/2019/11/updated-bodypix-2.html)
We'll be using BodyPix as a kind of "smart green screen" that's able to remove all the background pixels. We'll do this in a method called draw that is run automatically by p5 every few milliseconds:
function draw() { background(220); // Just for convenience const model = handsfree.model const body = handsfree.body // Only paint when we have data if (body.data) { // Create a mask, with all non segmented pixels as magenta mask = model.bodypix.sdk.toMask( body.data, {r: 0, g: 0, b: 0, a: 0}, // foreground {r: 255, g: 0, b: 255, a: 255} // background ) // Paint the mask into a buffer canvas model.bodypix.sdk.drawMask( $buffer, handsfree.debugger.video, mask, 1, 0, 0 ) // Make all magenta pixels in the buffer mask transparent let imageData = bufferCtx.getImageData(0, 0, $buffer.width, $buffer.height) for (let i = 0; i < imageData.data.length; i += 4) { if (imageData.data[i] === 255 && imageData.data[i + 1] === 0 && imageData.data[i + 2] === 255) { imageData.data[i + 3] = 0 } } bufferCtx.putImageData(imageData, 0, 0) // Dimensions of each mask w = $buffer.width / slider.value() h = w * widthHeightRatio // Paste the mask a bunch of times for (row = 0; row < slider.value() * 3; row++) { for (col = 0; col < slider.value(); col++) { // Stagger every other row x = col * w - (w / 2) * (row % 2) y = row * h / 3 - h / 2 * (1 + row / 10) canvasCtx.drawImage($buffer, x, y, w, h) } } } } On each webcam frame, Handsfree.js captures segmentation data into handsfree.body.data which we'll use to create the mask.
// Create a mask, with all background pixels as magenta (255, 0, 255) mask = model.bodypix.sdk.toMask( body.data, {r: 0, g: 0, b: 0, a: 0}, // foreground {r: 255, g: 0, b: 255, a: 255} // background ) // Paint the mask into a buffer canvas model.bodypix.sdk.drawMask( $buffer, handsfree.debugger.video, mask, 1, 0, 0 ) 
Next, we'll go through each pixel and make any magenta pixels transparent:
// Make all magenta pixels in the buffer mask transparent let imageData = bufferCtx.getImageData(0, 0, $buffer.width, $buffer.height) for (let i = 0; i < imageData.data.length; i += 4) { if (imageData.data[i] === 255 && imageData.data[i + 1] === 0 && imageData.data[i + 2] === 255) { imageData.data[i + 3] = 0 } } bufferCtx.putImageData(imageData, 0, 0) Finally, we paste the buffer into the p5 canvas a bunch of times:
// Dimensions of each mask w = $buffer.width / slider.value() h = w * widthHeightRatio // Paste the mask a bunch of times for (row = 0; row < slider.value() * 3; row++) { for (col = 0; col < slider.value(); col++) { // Stagger every other row x = col * w - (w / 2) * (row % 2) y = row * h / 3 - h / 2 * (1 + row / 10) canvasCtx.drawImage($buffer, x, y, w, h) } } And that's all there is to it!
Let me know in the comments if anything is unclear. I hope you enjoy playing around with this demo, it's the first in a set of tutorials on creating Instagram SparkAR-like filters for the web with p5.js.
Top comments (0)