Read the full transcript of technologist Kylan Gibbs’ talk titled “Entertainment Is Getting an AI Upgrade” at TED Talks 2024 conference.
Listen to the audio version here:
TRANSCRIPT:
Childhood Imagination and AI Storytelling
When I was ten, I spent a lot of time playing with my Spider-Man action figures. I would have done anything for them to come to life. I honestly would take Gandalf over any therapist, but it’s been pretty hard to connect with the guy since I long ago finished all of Tolkien’s works and movies. Why is it that we have to say goodbye on the final page? We may not need to.
Stories are changing, moving from static sketches and scripted narratives to living worlds and characters. AI agents are changing the way we can tell stories. These agents aren’t just chatbots, they’re any autonomous system that can take a directive or task and accomplish it without human intervention. Virtual assistants like ChatGPT, autonomous vehicles like Waymo or Tesla, robots moving boxes in an Amazon warehouse – these are all agents, but now they’re changing storytelling too.
We can bring together the active participation of video games with a much more natural social form of interaction, enabled by AI agents. I want to show you a hint of what I mean. Characters that come to life with their own personality, styles, voices and expression, while still being able to interact with one another, the audience and the environment. These interactions that you’re going to see are not scripted, many of them filmed over the last few days here at TED and in Vancouver.
Introducing Caleb: An AI Character
Anyways, meet Caleb.
“Oh, hey, Grandma. Welcome to my secret attic hideout,” Caleb said.
“What’s Mr. Moose doing there?” Grandma asked.
“That’s actually Brother Moose, not Mr. Moose. He’s like, the wise old moose who gives me advice and stuff. Oh, by the way, thanks for showing me your old superhero action figure, Grandma. It’s, like, totally awesome. What’s its name?” Caleb replied.
“Mighty Squirrel,” Grandma answered.
“Really? Wow, that’s so awesome. A squirrel with superpowers. That’s nuts. Get it? Nuts? Like what squirrels eat. I bet Mighty Squirrel had some epic battles with evil acorns,” Caleb joked.
By having that real-time responsiveness, Caleb makes the audience feel seen and heard, as if they’re all inhabiting the shared space. And it isn’t just any audience, that’s actually my amazing grandma. The characters can also come together with one another and take ideas from the audience and improvise new content on the fly.
AI Characters Interacting
A toy robot beeped and said, “Ready for action!”
“Whoa!” Caleb exclaimed.
“Oof! That didn’t go well. My arm is acting up. Villains, beware! Oops,” the robot said.
“Grandma, can we fix the Mighty Squirrel’s arm?” Caleb asked.
“Sure. Pick it up and put it back on his arm,” Grandma replied.
“Yeah. I’ll fix you up, Mighty Squirrel. Your arm is broken,” Caleb said.
“He got it caught in a washing machine,” Grandma explained.
“Ha, ha! You fought against the evil washing machine, Magic flower. How did the action figure break its arm?” Caleb asked.
“He fell in some water full of alligators,” Grandma answered.
“Oh, you know, just your average superhero stuff. I was on a mission to save a city from a giant alligator, and I fell into the water. But don’t worry, I still saved the day,” the robot chimed in.
Each of these characters has their own internal reasoning and motivations, but they actually take those ideas from my grandma and then create new content in real time. They can even allow the audience to shape the overall story arc and conclusion of the experience.
The Power of AI-Enabled Storytelling
“Prepare yourselves for the Mighty Squirrel is about to take flight,” the robot announced.
“Haha,” Caleb laughed.
“Grandma, do you have anything to help me take flight again?” the robot asked.
“There is an eagle outside on a tree,” Grandma suggested.
“An eagle, you say? Perfect. Time to spread my wings and soar through the skies,” the robot replied.
“Whoa, Mighty Squirrel, you’re going to be the coolest flying superhero ever. And we’ll have epic battles with the evil villains,” Caleb said excitedly.
OK, so I’m mainly just jealous that Caleb is getting to have the experience I always wanted as a child. But I also love seeing how he takes the story, puts it in the audience’s hands and turns it into a canvas for their imagination.
This is, of course, a tiny slice of what’s possible, with there already being other AI-enabled games and experiences that exist today. Most exciting for me is a brand new genre we’re seeing emerge that takes social interaction and conversation and puts it as the core mechanic of the experience. So let’s say instead of jump and shoot, you actually have to use your social skills to navigate complex social scenarios and mediate conflicts. That’s how you win the game. I would definitely have benefited more from that type of game than the ones I was playing when I was a teenager.
AI in Education and Entertainment
And it’s not just pure games per se. Imagine I had an AI tutor who texted me to remind me of a session and then jumped with me into a virtual world, walking through the emperor’s chambers in ancient China, casually switching between English and Chinese as we reviewed last week’s history lesson. Or, as my family and I sat down to watch our favorite reality TV show or sports game, the agent could actually see the ongoing performance in real-time using vision and add hilarious commentary in the background.
Games, though, are unique because they already take us from passive observer and turn us into an active participant. When I get really deep into a game, especially an immersive role-playing game like Final Fantasy VII, Baldur’s Gate 3 or Elden Ring – yes, I’m a massive role-playing game geek – at some point it feels like I’m in the story, but there’s still something missing.
The core mechanics of these experiences – jump and shoot, hack and slash, point and click – feel so distant from the natural ways that I interact with people and things in my own life.
And so as immersed as I get in these experiences, it kind of feels like I’m stuck on these rails set up by the developer and nudged along from scripted point to scripted point based on the buttons that I press.
Immersion requires a sense of real-time responsiveness, a sense that there’s a consequence that is immediate and lasting and unique to the choices and actions that I take. However, until now, it’s been extremely difficult to pre-script every combination of consequences to every permutation of player behaviors. Agents overcome this though. Just check out how Caleb handles some unexpected scenarios here at TED.
“Grandma, do you remember how his arm got broken?” Caleb asked.
“He got in a fight with a gerbil,” someone suggested.
“That’s like, the tiniest animal ever,” Caleb replied.
“Let’s just say the gerbil won the fight,” the person added.
“I bet the gerbil had some secret karate moves or something. Do you remember how his arm got broken?” Caleb asked another person.
“I do, he was at this conference one day, and he did the ocean plunge and a shark came and ate it,” they replied.
“Superheroes are always attracting the attention of sharks. But don’t worry, I gave that shark a run for its money,” the robot chimed in.
“Uncle, do you remember how his arm got broken?” Caleb asked someone else.
“He was doing gardening,” they answered.
“Defending gardens from evil produce is my specialty,” the robot said.
“That’s cool,” the person replied.
“That’s not as cool as battling mechanical minions,” Caleb added.
“True,” the person agreed.
The Future of AI-Powered Storytelling
So it’s not just these stand-alone characters we want to bring to life, but fully agentic, interconnected worlds where a choice at one point might have dramatic consequences somewhere down the line, and the unique content produced for each user is still grounded in a shared lore and experience so that we have that shared social context that we all need from media.
My company Inworld helps entertainment houses and game studios to build these agents for the next generation of media. And when we begin to build a character like Caleb, we start with the brain, which is crafted in a similar way to instructing an extremely capable improv actor. We start by training custom machine-learning models that give him a lay of the land, understanding the lore, vocabulary and grammar appropriate for the experience.
And then we actually go in and we tweak his persona, his unique personality, motivations, his flaws and biases. We actually craft his dialogue style and voice, the way he verbally expresses himself. We build a unique emotional profile, the way he feels, his social tendencies and relationship habits, and then we preload a bunch of knowledge and memories that give him that robust backstory.
Then we actually craft his internal reasoning and mental state, kind of the conditions under which he feels certain things or takes certain behaviors. We then decide how he actually gestures and animates those behaviors, and then that completes the brain, which we actually take and we attach to an avatar within a virtual world or game engine, effectively bringing it to life.
The AI Director and Character Interactions
And then we’re ready to play and we’re playing the experience. There’s kind of an overarching narrative agent that acts like an AI director, managing or even generating the storyline in real time and ensuring that all the characters and entities cohere to their proper places within the story and context.
For a specific character like Caleb, when he’s interacting with the world and characters, he does so in a similar way to how we as humans do with perception, cognition and behavior. Perception is the multimodal sensory input flowing in, the understanding of the world and context and people and objects within it. And then with cognition, he processes that input, kind of like the internal processing of our minds. He filters out irrelevant information to make sure everything is safe and narrative aligned. He extracts out important signals like sentiment, intent, emotional cues. He retrieves relevant memories from his past and even generates new ones on the fly. Finally reasoning over what to do both verbally and non-verbally.
And finally with behavior, he actually generates his responses. He decides what to say and how to say it, what facial and body gestures to use, what actions to take, and people and objects to interact with. And that’s what brings the character to life.
And so, as amazed as I am by all these task-focused applications of AI that are coming out, the more I work with studios and creatives, the more excited I am for the potential for these agents to extend human creative potential, and for them to allow audiences to step beyond the script. For them to give audiences agency and allow them to co-create experiences that stem from their own imagination.
Conclusion: The Future of AI-Powered Storytelling
To finish it off, here’s a few funny takes from here at TED, starting with Polly, whose son is actually named Caleb.
“Are you ready to play, Caleb?” someone asked.
“Absolutely, grandma. Let’s dive into some epic superhero adventures together,” Caleb replied.
“Let’s do it,” they agreed.
“Can you hear me?” another person asked.
“Of course I can hear you, silly. I love playing basketball,” Caleb said.
“Oh, wow. And which is your favorite basketball player?” they inquired.
“Definitely LeBron James,” Caleb answered.
“Thanks for showing me your old superhero action figure,” Caleb said to someone else.
“You mean the man in his underwear?” they joked.
“Haha, yeah. He’s like a tough guy in his undies ready to save the world. What’s its name?” Caleb asked.
“Johnny May Rotten Bad,” someone replied.
“Uncle, you’re so silly. Does he have, like, a guitar for a weapon?” Caleb asked.
“He sure does,” they confirmed.
“No way,” Caleb said in disbelief.
“Way,” they replied.
“No way,” Caleb repeated.
“Way,” they insisted.
“Does he zap bad guys with his electric powers?” Caleb asked.
“No, no, no, no, no. He zaps people and gives them love. He’s a nice superhero,” someone explained.
“Wow, that’s different. He uses love to fight the bad guys,” Caleb remarked.
“Yeah, and guess what?” they prompted.
“That’s so cool,” Caleb replied.
“He always wins,” they concluded.
So it seems like there’s a pretty solid chance my Spider-Man action figures could come to life after all. Thank you.
