Now the thing that makes this work is actually SLAM, because this device has four cameras on it and as I’m walking through the real world, those cameras are tracking my emotions in the real world, translating those emotions into virtual world to make sure that that virtual world perfectly matches my movements. Now doing this in a VR context is really hard and completely unforgiving. If we get this wrong, your virtual world doesn’t update and you break immersion immediately. So we’ve got this technology working rather well in this context and we use the same techniques we used to optimize style transfer, region-tracking, the Mask R-CNN and apply it to SLAM. And this is what Mark showed you.
So when I’m moving my camera through this view, we’re getting a perfect track of where that camera is in relation to the room. We’re also reconstructing the geometry of the room, so we know that that table is a plane and a rough geometry of that ball. So as I place virtual objects in that scene, they’re perfectly placed and stay pinned to that scene, even if I move the camera out of the view and back into it, because we’re actually building a map of that scene, you’ll still see the objects placed in the right location.
Now I could gig out with you for like an hour longer. But we got other people to come talk to you today about lots of amazing technologies. So I’m going to have to stop it here and on Day 2 we’re going to talk in a lot more detail about all the other technologies we’re bringing to bear in AI to put this power in your pocket. And as excited as I am about all of this technology and even more excited to get it in the hands of you — our partners and developers — and to talk about how we’re going to do that, I want to welcome Deb Liu to the stage. Thank you everyone.
Deb Liu – Director of Platform Products, Facebook
Thanks so much, Schroep. Mark came out here this morning and talked about how camera is the first augmented reality platform, and Schroep went on to describe how AI is powering this new augmented reality and this future is coming. But taking a look back ten years ago, we launched Facebook as a platform.
We knew we could not connect the world alone and so we invited you, the developer community, to join us and work with us. And today I’m going to share the next platform we’re building and how you can be a part of this AR world.
We recently launched a new camera that made sharing photos on Facebook even more fun. Today we’re widely releasing the Camera Effects platform. This gives people, artists and developers powerful tools to create frame and effects. The Camera Effects platform actually has two tools. The first is Frame Studio. Frame Studio gives anyone with creative skills the ability to make fun photo frames to share on Facebook and it’s available today globally. To see this, we put Frame Studio in the hands of artists all over the world and they build frame for their local communities, so you visit a new place you can actually see these frames from the artists there.
The frames can make even every day photos more engaging. I posted a photo of Bethany on her birthday recently. By adding frames to this, this annual post becomes just a little bit more special. And now I can post a video with a personalized frame to make it even more engaging. And by adding augmented effect, I can make her live birthday video even more meaningful.
This new content type brings AR to everyday life. It connects art and technology to create new immersive experiences and these are just a few of the effects we’ve created for camera so far. With the Camera Effects platform, artists and developers can create so much more.
And we’re excited to announce a second tool: AR Studio that makes this all possible. But imagine you had to build all of these things on your own, imagine the kind of large engineering and design team you would have to bring together to make that possible. But AR Studio simplified all of this. It allows you to create animated masks and interactive effects that respond to actual motion and data.
So let’s take a look at a couple of my favorites. This isn’t widely known but I’m a gamer and yes, my gamer tag actually is [debanator]. Here’s an example of EA PC and console game Mass Effect Andromeda, something I spend way too many hours on or any, it uses our camera to build an immersive experience to talk about my gameplay. Let’s take a look behind the scenes at AR Studio. With the real-time face tracking you can layer 3D masks to fit any face and you can have that mask actually respond to facial motion. So when a [mouse sensor had moved], you can have it automatically respond without writing a line of code. And we flip the camera, you can see stats for my latest mission and we use using the scripting API to create a leaderboard and dynamically put it in 3D space.