For over five years, we've partnered with a leading UK media agency, to deliver cutting-edge digital solutions that blend physical spaces with interactive experiences. In this post, we’ll explore the technical solutions behind their popular photo and video processing application, diving into how background removal technology enhances user experiences.

Background Replacement feature with Green Screen

One of the most requested photo/video effects is background replacement using a green screen. In our app, there are two places where you’ll find this effect: the live preview and the final digital/analog product generated by the app. While these two use cases share a lot of similarities, they also have key differences.
For the live preview, we capture the video stream, break it down into individual frames, apply the background replacement, and then send the updated frames back to the live preview canvas. The trick is to do all this as quickly as possible, so it looks seamless. This process is similar to how we handle video processing, just at a faster pace. Replacing the background on a single image, on the other hand, is way less resource-intensive and much simpler to pull off.
The real question isn't about whether we can handle photo/video processing in-house; it’s more about what kind of external services we should tap into during the process to optimize performance.
Three Methods for Background replacement

Seriously.js by Brian Chirls

Seriously.js is a real-time, node-based video compositor for the web, inspired by tools like After Effects. It enables dynamic video effects with GPU acceleration, supporting various image inputs such as video, canvas, and Three.js. With built-in 2D transformations and a plugin system, it offers flexibility for interactive, high-quality video compositing directly in the browser.

Implementation
The chroma effect utilizes the principles of chroma keying, often referred to as "green screen" or "blue screen" techniques, to seamlessly blend images by rendering specific colors transparent. This method allows users to effectively remove backgrounds by selecting a range of colors from the image. With adjustable settings for the target color, weight, and balance, users can fine-tune the effect to suit different backgrounds. The real-time application of this effect makes it particularly useful for dynamic scenarios like video editing and live streaming.

Remove.bg API

This API automatically removes backgrounds from images, enhancing image processing efficiency. It offers a reliable and accurate solution that simplifies editing for users. With its advanced background removal capabilities, it saves time compared to manual editing. Additionally, the API is easy to integrate into various apps and websites, making it a versatile choice for developers. Here’s a straightforward example from the API documentation demonstrating how to use it in practice.

TensorFlow - Body segmentation API

The TensorFlow Body Segmentation API allows developers to segment human bodies in images or video using machine learning models. It identifies different parts of the human body, such as arms, legs, and torso, creating detailed masks for each region. This API is useful for applications in fitness, virtual backgrounds, augmented reality, and more. It provides real-time processing and it can be easily integrated into applications for tasks like human pose estimation or activity recognition.

Background replacement with the Selfie Segmentation model

We leveraged the body segmentation functionality of the API to identify the background and individuals in an image using masks. After segmentation, we can replace the background with any image of our choice, using any of two different models with the API. The BodyPix model helps identify and track a person's body parts, while the MediaPipe Selfie Segmentation model is designed to detect the background and individuals in a selfie using masks.

API integration
The implementation is swift and straightforward. We simply need to instantiate the model and integrate it into our processing workflow.

Now that we've built our segmentation model, it's time to put it to work. The function takes a canvas with our image on it as input, and the output is a binary mask showing the segments created by the model.

The output binary mask visually looks something like this. Now, we can use the background mask to cut out the background image and layer it on top of the original as an overlay.

https://github.com/tensorflow/tfjs-models/blob/master/body-segmentation/images/drawMask.jpg

Hope you enjoyed our comparison post about background removal technologies. Open for some opinions and suggestions based on your experience, let us know in the comments.

Author: Norbert Juhász, Software Developer @Tappointment
Image credit: Generated by DALL-E and Canva

Code images: codesnap.dev, github

Links:
Seriously.js - https://github.com/brianchirls/Seriously.js
Remove.bg API - https://www.remove.bg/
TensorFlow Body segmentation API - https://github.com/tensorflow/tfjs-models/blob/master/body-segmentation/README.md