Jensen Huang’s Speech At Caltech 2024 Commencement (Transcript)

If we are transforming the computer so profoundly, it will, of course, have implications in every industry. And I’ll talk about that in just a little bit.

And as you enter industry, it’s important you know what’s happening. Modern computing traces back to the IBM System 360. That was the architecture manual that I learned from. It’s an architecture manual that you don’t need to learn from. A lot better documentation and better descriptions of computers and architecture has been presented since.

The Legacy of the IBM System 360

But the System 360 was incredibly important at its time, and in fact, the basic ideas of the System 360, the architecture of it, the principal ideas and architecture and strategy of the System 360 are still governing the computer industry today. And it was introduced a year after my birth. In the 80s, I was among the first generation of VLSI engineers who learned to design chips from Mead and Conway’s landmark textbook.

And I’m not sure if it’s still being taught here. It should be in the introduction of VLSI systems. Based on Carver Mead’s pioneering work here at Caltech on chip design methodologies and textbook that revolutionized IC design. And it enabled our generation to design supergiant chips and ultimately the CPU.

The Rise of the IT Industry

The CPU led to exponential growth in computing. The performance, the incredible technology advances, that’s called Moore’s Law, fueled the information technology revolution. The industrial revolution that we are part of, that my generation was part of, saw the mass production of something the world had never seen before. The mass production of something that was invisible, easy to copy, the mass production of software.

And it led to a $3 trillion industry. When I sat where you sat, the IT industry was minuscule. And the concept that you could make money selling software was a fantasy. And yet today, it’s one of the most important commodities, most important technologies and product creations that our industry produces.

The Limits of CPU Scaling

However, the limits of the NARD scaling, of transistor scaling, and instruction level parallelism have slowed CPU performance. And the slowed CPU performance gains is happening at a time when computing demand continues to grow exponentially. This exponentially growing gap between demand of computing and the capabilities of computers, if not addressed, computing energy consumption and cost, inflation, would eventually stifle every industry. We see very clear signs of computing inflation as we speak.

And after two decades of advancing NVIDIA’s CUDA, NVIDIA’s accelerated computing offers a path forward. That’s the reason why I’m here. Because finally, the industry realized of the incredible effectiveness of accelerated computing at precisely the time that we’re witnessing computing inflation after several decades. By offloading time-consuming algorithms to a GPU that specializes in parallel processing, we routinely achieve 10, 100, sometimes 1,000-fold speedups, saving money, cost, and energy.

The Reach of Accelerated Computing

We now accelerate application domains from computer graphics, ray tracing, of course, to gene sequencing, scientific computing, astronomy, quantum circuit simulations, SQL data processing, and even pandas, data science. Accelerated computing has reached a tipping point. That is our first great contribution to the computer industry, our first great contribution to society, accelerated computing.

It now gives us a path forward for sustainable computing where cost will continue to decline as computing requirement continues to grow. A hundred-fold, a hundred-fold of anything in time or cost or energy savings that accelerated computing opened surely would trigger a new development somewhere else. We just didn’t know what it was until deep learning came to our consciousness.

The Big Bang of Deep Learning

A whole new world of computing emerged. Jeff Hinton, Alex Krzyzewski, and Ilya Sutskever used NVIDIA CUDA GPUs to train AlexNet and shocked the computer vision community by winning the 2012 ImageNet Challenge. This was the big moment, the big bang of deep learning, a pivotal moment that marked the beginning of the AI revolution.

Our decisions after AlexNet transformed our company is something that’s worth taking note of. Our decisions after AlexNet transformed our company and likely everything else. We saw the potential of deep learning and believed, just believed through principle thinking, believed through our own analysis of the scalability of deep learning. We believed the approach could learn other valuable functions. That maybe deep learning is a universal function learner and how many problems are difficult or impossible to express using fundamental first principles.

Exploring the Limits of Deep Learning

And so when we saw this, when we saw this, we thought this is a technology we really have to pay attention to because its limits are potentially only limited by model and data scale. However, there were challenges at the time. This is 2012, shortly after 2012.

How could we explore the limits of deep learning without having to build these massive GPU clusters? At the time we were a rather small company, building these massive GPU clusters could cost hundreds and hundreds of millions of dollars. And if we didn’t though, there was no assurance that it would be effective if we scaled.

If You Build It, Will They Come?

However, no one knew how far deep learning could scale. And if we didn’t build it, we’d never know. This is one of those, if you build it, will they come? Our logic is if we don’t build it, they can’t come.

And so we dedicated ourselves based on our first principled beliefs and our analysis. And we got ourselves to the point where we believed this was going to be so effective and when the company believes something, we should go act on it.

Reinventing Computing for Deep Learning

So we dove deep into deep learning, and over the next decade, systematically reinvented everything. We reinvented every computing layer, starting with the GPU itself. The invention of the modern GPU, which is very different than the GPU of the past that we invented in the first place, and we went on to invent just about every other aspect of computing, the interconnects, the systems, the networking, and of course, software.

We invested billions. We invested billions into the unknown. Thousands of engineers for a decade worked on deep learning and advancing and scaling deep learning without really knowing how far we could really take the technology. We invested billions. And we designed and built supercomputers to explore the limits of deep learning and AI.

The Rise of AI Supercomputers

And in 2016, we announced DGX-1, our first AI supercomputer, and I delivered the first one to a startup in San Francisco, a startup nobody knew anything about, a group of friends of mine who were working on artificial intelligence, a company called OpenAI. In 2022, 10 years after AlexNet, and about a million-fold increase in computing later, a million-fold. If you could just imagine, what would it be like if your laptop was a million-fold more capable?

A million-fold later, OpenAI launched ChatGPT, and AI went mainstream. During this decade, NVIDIA transformed ourselves from a graphics company that many of you probably first knew us as, that builds GPUs, to now an AI company that builds massive data center scale supercomputers. We transformed our company completely. We also transformed computing completely.

The New Computing Paradigm

The fundamental way of doing computing today has been radically changed. The computing stack now uses GPU to process large language models that are trained on supercomputers rather than CPUs that are processing instructions written by programmers. We are now creating software that no humans can write. We are now creating software that does things that no humans can imagine, even just 10 years ago.

Computers are now intention-driven rather than instruction-driven. Tell a computer what you want, and it will figure out how. And like humans, AI applications will understand the mission, reason, plan, and orchestrate a team of large language models to perform tasks. Future applications will do and perform very similar to the way we do things, assemble teams of experts, use tools, reason and plan, and execute our mission.

The Emergence of the AI Industry

Software and what software can do has been completely changed. Even our industry, as it’s being changed and transformed, created yet another industry, an industry the world’s never seen before. An industry is forming right in front of our eyes. AI’s input and output are tokens. For all the engineers in the room, you know what I mean.

These are floating-point numbers that embed intelligence. Companies are now building a new type of data center that didn’t exist before that specialize in producing intelligence tokens. Essentially, AI factories. Like AC generators that Nikola Tesla invented of the past industrial revolution, we now have AI token generators, and they will be the factories of a new industrial revolution.

AI Factories and Robotics

There’s large industries producing energy, electricity. We now have a large industry producing something invisible called software. In the future, in the very near future, we’ll have industries that are producing, manufacturing intelligence tokens, AI generators. A new computing model has emerged, and a new industry has emerged, all because we reasoned from first principles, formed our belief about the future, and we acted on them.

The next wave of AI is robotics, where AI, in addition to a language model, also has a physical world model. We work with hundreds of companies building robots, robotic vehicles, pick-and-place arms, humanoid robots, and even entire gigantic warehouses that are robotic. But unlike our AI factory strategy and our experience there, which was really formed through reasoning and deliberate action, our robotics journey resulted from a series of setbacks.

The Invention of the GPU

As you know, NVIDIA invented the GPU. This was before we invented AI factories. Our first great contribution to the computer industry was reinventing computer graphics through programmable shaders. We invented the GPU and programmable shading in 2000. We wanted to integrate GPUs into every computer, and so we started to combine our GPUs with motherboard chips, and we launched a fabulous integrated graphics chip at the time for AMD CPUs.

Our chipset business was an instant success. I think it went from zero to a billion dollars practically overnight. But then all of a sudden, AMD wanted to control all of the technology in the PC, and we wanted to stay independent, so they purchased ATI and no longer needed us.

Setbacks and Pivots

We turned to Intel. That probably wasn’t a great idea, but we turned to Intel and negotiated a license to connect to Intel CPUs. Apple was excited by what we were building and asked us to work on a new computer with them, which became the first MacBook Air. Well, Intel saw what happened and decided they didn’t want us to do that anymore, and so they terminated our agreement.

Well, we pivoted again, and this time we went and licensed ARM, and we built a low-power SoC, a mobile SoC, the world’s first SoC that was essentially a computer, a full operating computer, and it was incredible. Our chip excited Google, and they asked us to work on a new device, which turned out to have been the Android mobile device.

Building for a $0 Billion Market

Well, Qualcomm decided they didn’t want us to do that, and so they didn’t want us to connect to their modems, and it’s hard to build a mobile device without being connected to a modem. And there were no other LTE modem companies, so we had to exit the mobile device market. Well, this happened practically on a year rhythm, and we would build something, it would be incredibly successful, generate enormous amounts of excitement, and then one year later, we were kicked out of those markets.

Well, with no more markets to turn to, we decided to build something where we are sure there are no customers, because one of the things you can definitely guarantee is where there are no customers, there are also no competitors, and nobody cares about you. And so we chose a market with no customers, a $0 billion market, and it was robotics.

The World’s First Robotics Computer

We built the world’s first robotics computer processing an algorithm nobody understood at the time called deep learning. This is over 10 years ago now. Ten years later, I can’t be happier with what we’ve built and the opportunity to create the next wave of AI. More importantly, we developed agility and a culture of resilience.

One setback after another, we shook it off and skated to the next opportunity. Each time, we gained skills and strengthened our character. We strengthened our corporate character. Our company is really hard to distract and really hard to discourage, and no setback that comes our way doesn’t look like an opportunity these days.

The Unexpected Journey

Ironically, the robotics computer that we built today doesn’t even need graphics, which is why our journey started in the first place. So where we are today tells us something and teaches us something. The world is uncertain, as Richard Feynman would say, and the world can be unfair and deal you with tough cards. Swiftly, shake it off.

You’ve apparently been paying too much attention to your books. Swiftly, shake it off. Come on, that’s pretty clever. I made myself laugh. There’s another opportunity out there, or create one. Let me tell you one more story.

The Lesson from the Moss Garden

I used to work from one of our international sites for one month each summer. When our kids were in their teens, we spent a summer in Japan. For a weekend, we visited Kyoto and the Silver Temple. If you haven’t had a chance to go, you must. It’s renowned for its exquisite moss garden.

The day we visited was quintessential Kyoto summer day, suffocatingly hot and humid, sticky. Heat is radiating from the ground. The air was thick, still. Along with the other tourists, we wandered through the meticulously groomed moss garden. And I noticed the lone gardener.

Dedication to Craft

Now, remember, the moss garden, this is the Silver Temple, the moss garden is gigantic. It’s about the size of this courtyard. And it has the collection, the largest collection of just about apparently every species of moss in the world. And it’s just exquisitely maintained. I noticed the lone gardener squatting, carefully picking at the moss with a bamboo tweezer and putting it in the bamboo basket.

And you have to, it’s a bamboo tweezer, you know, and it’s just this one gardener. And the basket looked empty. Well, for a moment there, I thought he was picking imaginary moss into a pile of imaginary dead moss. And so I walked up to him and I said, what are you doing? And in his English, he said, “I’m picking dead moss. I’m taking care of my garden.”

Plenty of Time

And I said, “But your garden is so big.” And he responded, “I have cared for my garden for 25 years. I have plenty of time.” Well, that was one of the most profound learnings in my life. And it really taught me something. This gardener has dedicated himself to his craft and doing his life’s work. And when you do that, you have plenty of time.

I begin each morning, I do every single morning exactly the same way, I begin each morning by doing my highest priority work first. I have a very clear priority list and I start from the highest priority work first. Before I even get to work, my day is already a success. I’ve already completed my most important work and can dedicate my day to helping others.

Advice for the Graduates

And when people apologize for interrupting me, I always say I have plenty of time and I do. Graduates of the class of 2024, I can hardly imagine anyone more prepared for the future than you. You dedicated yourself, you worked hard, you earned a world-class education from one of the most prestigious schools in the world. And as you commence into the next stage, take my learnings and hopefully they’ll help you along the way.

I hope you believe in something, something unconventional, something unexplored, but let it be informed and let it be reasoned. Then dedicate yourself to making it happen. You may find your GPU, you may find your CUDA, you may find your generative AI, you may find your NVIDIA.

Setbacks and Superpowers

I hope you will see setbacks as new opportunities. Your pain and suffering will strengthen your character, your resilience and agility, and they are the ultimate superpowers. Of all of the things that I value most about my abilities, intelligence is not top of that list. My ability to endure pain and suffering, my ability to work on something for a very, very long period of time, my ability to handle setbacks and see the opportunity just around the corner I consider to be my superpowers, and I hope they’re yours.

And I hope you find a craft. I hope you find a craft. It’s not important to decide on day one, it’s not even important to decide any time soon, but I hope you do find a craft, that you want to dedicate your lifetime to perfecting, to hone the skills of, and let it be your life’s work.

Prioritize Your Life

And then lastly, prioritize your life. There’s so many things going on, there’s so many things to do, but prioritize your life and you will have plenty of time to do the important things. Congratulations, class of 2024, go get them.

Related Posts

• Making Sense #469: w/ Tristan Harris on Escaping an Anti-Human Future (Transcript)
• HUGE* Conversations: w/ Google DeepMind CEO Demis Hassabis (Transcript)
• Figure AI CEO Brett Adcock’s Interview @ Shawn Ryan Show (Transcript)
• Moonshots #241: w/ Eric Schmidt – Singularity’s Arrival, 92-Gigawatt Problem & More (Transcript)
• “We Are Being Gaslit By The AI Companies!” – Karen Hao on DOAC Podcast (Transcript)