MileciaAdding virtual things to the real world is something that's made a huge leap in popularity, especially with QR codes. With augmented reality (AR), we can take images in the real world and display digit things on top of them, kind of like the Nintendo DS used to do with those game cards.
In this tutorial, we'll make a browser-based AR app that displays interesting things on top of images. By the end of this, you'll know how to work with AR in a Redwood app.
Set up the Redwood app
We'll start by creating a new Redwood app. In a terminal, run the following command.
$ yarn create redwood-app ar-demo
This will generate a lot of files you can work with on both the front-end and back-end, but our focus will be on the front-end. You can find all of the code for the React front-end in the web directory.
There are a few libraries we'll need to add to our app in order to get the AR functionality we need and we'll be doing that in a slightly different way.
Add the libraries you need
There aren't a lot of pre-built JavaScript AR libraries out there and the ones available have their own unique set of challenges, but we'll use AR.js since many of the others are built on top of this.
You'll also find that JavaScript AR systems commonly use Three.js or A-frame to handle rendering objects. The app we're making is a little unique in that it's built around the browser. Most AR apps are built for use on mobile devices, but this is a way that all JavaScript developers can play with AR without needing mobile development knowledge.
The tricky part about working with AR in the browser is that the libraries we have available want to be in the <body> of the app, which is hard to do with any of the frameworks. So we'll start by adding the libraries we're working with to the <head> of our index.html file in the web > src directory.
<script src="https://aframe.io/releases/1.0.4/aframe.min.js"></script>
<!-- we import arjs version without NFT but with marker + location based support -->
<script src="https://raw.githack.com/AR-js-org/AR.js/master/aframe/build/aframe-ar.js"></script>
We're using the A-frame version of the AR.js library. Now that we have the libraries imported, let's add the elements we need inside the <body>.
Setting up the AR elements
This will feel odd to anyone who's been working in a framework because we typically don't add anything to the DOM like this. It's a quirk to this AR library because it requires the elements to be directly inside the <body>, not in a component. I did some digging trying to find ways to make this into a component and took a look at some other libraries, but couldn't find anything ready to use.
There are ways to componentize this, but it will take some time and craftiness! For now, it's important to see how this works so that you at least know how to work with AR.
So inside the <body>, add the following code.
<a-scene embedded arjs>
<a-marker type="pattern" url="https://raw.githubusercontent.com/flippedcoder/blog-examples/main/clear-world/pattern-check.patt">
<a-sphere radius="1" color="#EF2D5E"></a-sphere>
</a-marker>
<a-entity camera></a-entity>
</a-scene>
This gives you an AR environment that will let you detect when a certain picture is displayed in the camera. These pictures are called markers. So anytime you are running this app and you point to a marker, you'll get some kind of interaction in the browser.
Since we're using A-frame, that interaction could be any number of things from a simple object being displayed on top of the marker to an entire game being created on top of the marker. You can have a lot of fun playing around with A-frame and what you want to display on your marker.
There are a few things to note in the code. Everything is contained within the <a-scence> which is an A-frame tag that creates the virtual environment. The <a-marker> tag is how we enable AR functionality in the app. We tell the app to look for an image that has a pattern type and give it an absolute link to the pattern file.
Whenever the correct marker image is found and it matches the pattern we have, then the <a-sphere> is displayed on top of the marker. Anything that you build inside of the <a-marker> tag will display on top of the marker image so feel free to get more creative!
Lastly, we have the <a-camera> tag. This is what opens the webcam or the camera on your phone so that you can target the marker image in real life.
That's actually all for the code part! Now we need to make a marker image. There's a tool available that lets us do that easily.
Make a marker image
To make a marker image and the associated pattern file, go to this site: https://jeromeetienne.github.io/AR.js/three.js/examples/marker-training/examples/generator.html. You can upload an image of your choosing, but keep in mind that there needs to be high contrast between the background and foreground and simple images work the best. You can also use the example image that's displayed.
You'll need to download the image and the marker. The marker is your pattern file and the image is the file that you will point your camera at. When you have the marker file, you can upload it to your own GitHub repo and replace the link in the code above.
Now that you have your marker file and the image ready, we can start the app and see what this looks like!
Running the AR app
In a terminal, run the following command.
$ yarn redwood dev
This will start up the Redwood app and you should see something like this in your browser.
Now if you open the image on your mobile device, you should see a pink ball on top of it in the browser.
Now you're working with AR directly in your browser and if you scroll down the page you'll see the rest of the app at the bottom. This is super useful if the AR portion of your app is just a small part of what a user needs.
Finished code
You can check out the full code in the clear-world folder of this repo or you can take a look in this Code Sandbox.
Conclusion
Adding AR to the web is actually still a pretty tricky thing to do considering most libraries have been built around the mobile experience. There's still a lot of room to grow and create web-based AR apps for people projects that might need more than just the AR interface and they need a full-stack web app supporting it.
