Our computer vision systems have automatically applied over 2 trillion labels. This is what allows us, when you search for Pomeranian, to find the right picture. And by the way, over 24 billion of those labels are for selfies. We even have Pomeranian selfies. Google Photos shows what’s possible when you approach an existing area from a new perspective.
Another core use case on users’ phones is communications. It’s an exciting area, and there is a lot of innovation. But given our advancements in machine learning, we wanted to approach this core use case with a new perspective. Erik Kay is going to join to talk to you more about it.
Erik Kay – Engineering Director at Google
Thanks, Sundar. You know, communications is all about sharing life’s moments. That great restaurant I found, the winning shot in overtime, my daughter’s recital. Today I can share moments like these the second they happen, just by pulling out my phone and sending a message. Communications is such an important part of our lives, and it’s an important focus for Google as well.
What makes me personally excited about communications is the potential for innovation when you combine the power of mobile with advancements in machine learning. So today we’re giving you a look at what we’ve been up to with two new communication apps that show what’s possible when we bring Google technology to this essential human activity.
The first is a new messaging app called Allo. Thank you.
Allo is a smart messaging app. It learns over time to make conversations easier, more expressive, and more productive by bringing all the richness of Google right into your chat. Allo is based on your phone number, so you can easily get in touch with anyone in your phone book.
This morning, we’re going to walk you through three areas that make Allo really special. First are some ways to express yourself and keep the conversation going. Then, we’ll talk about what it means to have the Google Assistant built right into your messaging app. And, finally, we’ll tell you how Allo keeps your conversations private and secure.
So let’s get started. On the stage with me to help demo is Amit. He leads product management. And on the screen behind me you can see Amit’s in a chat — in a conversation with his friend Joy. So we designed Allo to help you express yourself and keep the conversation going. So, of course, there’s a great selection of stickers which we’ve sourced from independent artists and content producers from around the world. But expression is more than just emojis and stickers. So we’ve added more features that help you say what you mean.
First, let’s look at whisper shout. We wanted to give you a way to add more emotion to your words. Sometimes you want to get your point across big and bold. Other times you want to say things a bit more softly. Whisper shout lets you express how you really feel by making your replies very big or very small.
Amit, can you show everyone how that works? So Amit has typed “yay” and throws a smiley face in there. Now watch, rather than tapping the send button, he slides it down to whisper and slides it up again to shout. Down to whisper and up again to shout. I think it’s pretty cool, too. No more yelling in all caps to get your point across.
It’s also fun to add emotion to your photos. And Ink lets you get creative with the photos you send. Amit’s picked a photo of his adorable baby girl and wrote ahoy on the little sailor. Posted it. It’s that simple. I use Ink all the time. It’s really fun.
Another way Allo helps you express yourself is by letting you type less, a lot less. We’ve taken a page from the inbox playbook and built Smart Reply right into your chat conversations. This is especially powerful in messaging when you’re on the go. So now you can keep your conversations going with just a quick tap.
So you can see when Joy asks: dinner later? That Amit has offered two Smart Reply suggestions: I’m busy and I’m in! Okay. I promise I won’t act out any more emojis. So Allo uses machine learning to suggest replies on the fly, anticipating what you might want to say next. Now, these aren’t just canned replies. Allo learns over time and will suggest responses based on how you like to express yourself. The more you use Allo the better the suggestions will become. So the suggestions you see are unique to you. You can say the things you want without having to type a single word. And since messaging isn’t just about texts, smart replies contain stickers and emojis, too, because as they say, an emoji’s worth a thousand words. Do they say that?
Now I want to show you something really cool. Allo even offers smart replies when people send photos to you. This works because in addition to understanding text, Allo builds on Google’s computer vision capabilities to understand of the content and the context of images. In this case, Allo understood that the picture was of a dog, that it was a cute dog, and even the breed of the dog. In our internal testing, we found that Allo is 90% accurate in determining whether a dog deserves the “cute dog” response.
So let’s try something even harder. When Joy sends a photo of pasta, we’re able to identify the precise details of the image and create smart replies mentioning both linguini and the clams. What’s really cool here is that we don’t just identify what’s in the image. Smart Reply actually creates a conversational response, something like “yummy,” the kind of thing you would actually say in response to a photo of food. This is only possible because we’ve married our strengths in computer vision and natural language processing. And if you think about it, there’s a lot of complex technology at work here just to help you say something as fun — as simple and natural as “I love linguini.”
So, that’s a little how Allo helps you express yourself and keep your conversations going. The intelligence behind Smart Reply also gives you a taste at how assistive technology can make your message experience simpler and more productive. The Google Assistant built right into Allo takes it even farther, so I’m pleased to introduce one of our leads, Rebecca, to tell you more about the Assistant in Allo.
Rebecca Michael – Head of Marketing, Communication Products at Google
Thanks, Erik. As you heard earlier, the Google Assistant is an ongoing dialogue between you and Google that helps you get things done in your world. It’s also designed as an ambient experience. It’s there for you whenever you need it. And in messaging that really means bringing the Google Assistant right into your conversation with friends. So I’m going to show you how the Assistant can help in Amit’s and Joy’s conversation.
So they’re planning a dinner and Joy now says she would like Italian food. The Assistant intelligently recognizes that they could use some tips for Italian restaurants nearby and you can see its proactive suggestions at the bottom of the screen there. Tapping this brings up restaurant cards that everyone in the chat can see. These are powered by Google’s Knowledge Graph which means that Allo can help with all kinds of information in the real world. So there’s some back and forth about which restaurant to go to. And it looks like they’re leaning towards Cucina at 7 o’clock.
So Amit taps on the restaurant card to bring up more details. Looks good. Serves pasta which we know Joy fancied. So Amit can now go ahead and make a reservation right there through OpenTable. The Assistant prompts him to confirm the number of diners and what time they want to eat. And now they’re booked and ready to go.
So what we’re seeing here — what we’re seeing here is completely new. In the past, Amit would have had to leave the chat to do a Google search, return with some restaurant options, switch back again to share the options, go out again to make the reservation at OpenTable and then come back in to share the details with the rest of the group.
In Allo, Amit and Joy can choose and reserve a restaurant right there in the chat and in a natural and seamless way.
So OpenTable will be just one of the many partners that will be part of the Google Assistant ecosystem, and we can’t wait to share more soon with the developer community about how to get started.
Cool. So we saw some proactive suggestions from Google there. You can also call on the Google Assistant directly at any time just by typing @Google as Amit is doing now. And he’s going to ask for funny cat pics. Really, Amit? Okay. Cool. So Google obliges, of course, with a lovely lineup of cats from Google Image search. Wow, that chin.
Okay. So you just saw how the Google Assistant can be really helpful in groups. You can also have a one-on-one chat with Google. What we’re seeing now is Amit’s contact list and Google’s appearing at the top there. So let’s jump in and have a chat.
Just like with any other conversation, this one picks up right where you left off and the Assistant will remember things like your name and even tell you how it’s feeling. So let’s try something more interesting. Amit’s a big Real Madrid fan and he wants to know how they got on in their last match. So he asks the Assistant: did my team win? It looks like they did. They won their — yeah. Some Real Madrid fans out there. Cool. And so they won their last match on Saturday. Let’s see when they are playing next. That’s pretty cool. They are through the Champion’s League final at the end of the month. We can keep going like this and find more news about the team just by tapping on the suggestions there.