Interviews · Interview
Token Efficiency: Cut AI Costs with Databricks + NVIDIA
Key takeaways
- Physical AI is intelligence about the physical world. Real-world data for it is too costly, slow or dangerous to collect, so simulation becomes the data generator.
- "Compute is the new data." The goal is to make compute the only bottleneck, using it to turn one real example into thousands or millions of variations where coverage is weak.
- Edge cases, such as a child running into the road, are trained in simulation, the same way pilots train in a flight simulator before flying a plane.
- The demo fine-tunes a small open-source model on two years of chat history using serverless NVIDIA GPUs on Databricks, then evaluates it against a proprietary model with custom LLM judges.
- An AI factory differs from a data center. A data center stores and moves data like a warehouse; an AI factory turns data and energy into tokens, so the metric is tokens per watt.
- NVIDIA is improving token efficiency by more than 10 times a year: work that costs $1 this year costs under 10 cents next year.
- Physical AI will reach scale first in factories, warehouses and supply chains over the next two years, then move into everyday spaces.
- Jensen Huang's advice to Lebaredian: "As much as necessary, as little as possible." Focus on what only you can do well.
- For practitioners: learn to orchestrate many long-running agents. The bottleneck is becoming the rate at which we produce good ideas.
About this video
What comes after agentic AI? In this episode of The Data + AI Exchange, Viktoria Semaan talks with Rev Lebaredian, VP of Physical AI Simulation at NVIDIA, aboutthe future of AI: why simulation generates the training data, what "compute is the new data" means.
Includes a demo of fine-tuning open-source models on Databricks with serverless NVIDIA GPU compute, plus AI factories vs data centers, cutting costs through token efficiency, and Rev's number one piece of advice from Jensen Huang.
Chapters
- 00:55What is Physical AI?
- 03:20Compute is The New Data
- 07:57Demo: Fine-tuning Models on Databricks
- 10:47AI Factories vs Data Centers
- 14:00Increasing Token Efficiency and Reducing Costs
- 16:55Personal Productivity: Using Coding Agents
- 17:48Number 1 Advice from Jensen Huang
- 18:54Advice for AI Practitioners: Mastering Agentic AI
Links and resources
Transcript
Generated from the video's captions and lightly edited.
We would like our cars, self-driving cars, to understand what it's like when a child runs into the middle of the road.
NVIDIA is introducing a new kind of a measure of productivity intelligence, tokens per what?
Every year, we are increasing our token efficiency by more than 10 times.
What's one advice that Jensen gave you?
Jensen has this one saying,
AI is evolving so fast that it's really hard to keep up. Think about this. Just 5 years ago, we talked about reinforcement learning, deep learning. It's kind of like perception AI. Then when ChatGPT got launched, we got exposed to generative AI. And this year, we all talk about agentic AI. But today, I want to talk about the next phase, physical AI. AI that doesn't just reason, but interacts, touches, and feel. Let's go.
So exciting to be here with VP of Omniverse and simulation at NVIDIA. I heard that at NVIDIA, you are really the force behind Omniverse, behind physical AI. And I think for people that are just getting started, physical AI feels like almost a science fiction, right? So if we think about AI, we think about first we had a perception AI, where we had inputs, and then we got inference as a result. Then we moved to generative AI. From generative AI, we moved to agentic AI, which is now everybody is talking about.
But you are in charge of physical AI. For somebody who is just maybe utilizing right now ChatGPT, can you tell us what it takes to move to those new levels, to agentic AI and physical AI?
Yeah, first, maybe we should define what physical AI is a bit in the context of what you just said. The modern AI that we have is built by taking information, taking data, pushing it through a supercomputer, and out comes an algorithm that essentially encodes knowledge about the data that you put into it. The data that we've had most access to, the one that's mostly available to us, is abstract information, information like language. It's the easiest to digitize. What we've seen the past few years with this explosion of LLMs is natural because we can gather all this knowledge information from the internet and produce the compression or encoding of that knowledge in the form of LLMs.
The AI that we would like to have, the one that will probably be most valuable to us, is AI, intelligence, knowledge about the physical world. But unfortunately, this is much harder for us to build because gathering information about the physical world is in many cases too costly, too time-consuming, too dangerous, or just impossible in many cases to sense it from the the physical world. So, the way we can get this data is by using a trick. We can take our understanding of the physical world, how the physical world works in terms of physics, the rules of the physical world, we represent it in a computer, essentially create a simulation of the physical world, and that simulator becomes a data generator that we could then use to feed into the AI supercomputers that will produce the physical AI.
I watched one of your talk where you said the compute is the new data. So, in the past everybody was saying you really have to have a data so that you have strong data foundation to then build agentic AI on top of it. But you said something different. You said, "Yes, but it's so hard to collect this data." Like imagine if you are building autonomous vehicle. I just got the chance actually to drive one at NVIDIA GTC, and we got stuck behind a Waymo that had a glitch. Like how would you create this edge case?
You would never get there. So, you said that you can now simulate it, right? And you can create this So, I would love to understand more because if the companies wants to get started, so yes, they have to have a data, then they pre-train the model, they have one type of model, then they will utilize simulation data to post-train the model to get final model. So, from your perspective, then where is becoming a bottleneck? Is it compute and data simulation? Is it still the original data? Is it something else?
Ultimately, what we'd like to do is make the bottleneck only be compute. Because if you can do that, then you can solve almost all problems. We can increase the amount of compute if we can increase the amount of energy that we produce. That's That'll be the only limiter. We're already seeing this idea of synthetic data generation happening in the knowledge space, in the non-physical space. The modern LLMs that we're training, a large portion of the training data is actually generated from other LLMs. Which kind of makes sense if you think about it.
I mean, how do we train humans? We actually create data, we create textbooks, we create curriculums, we create the information that we use to train the minds of our youth. That came from somewhere, it was synthetically generated from the minds of other humans. So, we're already doing this in the knowledge space. Doing it in the physical space makes perfect sense. You gave an example of an edge case that might not occur given that it's unlikely in real life. You might never capture some of those edge cases in the AV scenario.
There's other cases where even if you could capture it in the real world, you don't want to because it's dangerous, it's immoral to do it. We would like our self-driving cars to understand what it's like when a child runs into the middle of the road. But, we don't want to actually create that scenario in the real world. When we train our pilots to go fly, before we put them in an actual airplane to fly, we put them in a simulator. They are trained in these edge cases for hours and hours inside a flight simulator before you actually put them in in a plane because you would like them to have had that experience and to have the muscle memory to do this.
That's right. But we don't want to actually create that first time in the real world. You do that in a simulation first. It's exactly the same for the AIs. We train them in exactly the same way. So the key is to represent the real world as accurately possible inside our simulators so that what we train these AIs on matches the real world close enough so that what they learn applies in the real world.
Now, during NVIDIA GTC, NVIDIA partnered with Databricks and they launched the GPU compute now natively available on Databricks. Actually exactly solves this problem that yes, only a lot of customers have on Databricks data. But then now with NVIDIA GPUs, you can simulate data. You can then fine tune the models. You can post trade with a simulated data. What are some of the interesting
And a and a key thing here in in part of that process is taking your original data which might be scarce. You don't have enough of that. And you can use the compute to amplify that data. You take the data and you can generate new data that's reconfigurations of of the little bit of data you have. For example, for the case of robotics or self-driving cars, you may have a limited amount of data of an actual robot in operation, of an actual car driving, but you can take the data that you have, reconstruct it in simulation, break down the scene, the environment, the particular scenario, and then make many modifications, permutations of the same scene, generate new data, so you go from one instance to thousands or millions of instances and you can focus it in the areas where your data distribution is weak, where you have very little coverage.
You can increase the coverage in that area.
Databricks Auto Dust Compute is a new data that we use compute to amplify the limited data we have. And I want to show you exactly what that looks like in practice. Databricks just announced something that puts this directly in the hands of every data and AI practitioner, AI runtime. Serverless NVIDIA GPUs on Databricks. You can use it to train and fine-tune custom models using your favorite frameworks and get state-of-the-art efficiency, performance, and quality. Here is the use case. Nova Brew is a coffee company. They had a chatbot on their homepage for 2 years.
It works, but they paid premium API rates to a proprietary model for every conversation. So, they have 2 years of chat history. So, they decided to fine-tune a small open-source model. To do that, they need three things: the data, an open-source model, and historically the hardest part, GPUs. With AI runtime, that GPU bottleneck now just disappeared. Two or three clicks and I can attach notebook to serverless compute. Then, you can open accelerator drop-down and pick what you need: 1 A10, 1 H100, all the way up to 8 H100s. It's serverless and it scales down to zero.
So, no infrastructure to manage. I ran a lot of fine-tune on Qwen model using Nova Brew's chat traces as training data. While it runs, MLflow gives me live GPU utilization, memory usage, even temperature. And when the run finishes, the model lands directly in Unity Catalog, versioned, governed, and ready to deploy. Then, I can serve it through a Databricks GPU model serving endpoint. This one also scales to zero, so I'm only paying for compute when someone is actually using my chatbot. Now, does my 3 billion parameter open-source model actually hold up?
We can run it through evaluation set against proprietary model with custom LLM judges. Here, we can create specifically the metrics to measure our brand voice. And look at this. While Claude is technically correct, it's a very good model, but it fails the voice judge. It leads with phrases that Nova Brew doesn't use. For example, it includes bullet point list. And our fine-tuned model? Well, it passed. It sounds like a Nova Brew because we trained it on Nova Brew data, and it costs orders of magnitude less to run. We had limited data, just 2 years of conversations, and we used compute to turn that into a model that does the job even better than the foundation model for our specific use case.
If you want to try it yourself, I will link the AI runtime links in the resources. And now, let's go back to the conversation with Rev.
Yeah, you know, it reminds me when I was driving the Mercedes, the product manager said, "You know, we generated a snow in San Francisco." I mean, like, what are the chances, right, to get snow in San Francisco? But, as you said, you can now get this perfect sunny weather, make it rainy, make it snow, make all
Exactly. And it might actually happen. It might snow occasionally.
But, no, it's so You touched today on this briefly, and Jensen talked about this recently, AI factories. This new term. And in addition, NVIDIA is introducing a new kind of a measure of productivity and intelligence, tokens per watt. So, now we are talking about how do we generate more tokens with a lower cost, and that this KPI is becoming almost like our productivity metric. So, Jensen said that perhaps an engineer that is making $500,000 as a salary should be utilizing AI that, you know, like generating $250,000 with tokens. What I would like to understand from you, what does this really mean, these AI factories, and also how to utilize this metric and put it in practice?
It's very, very good question. You know, with hyperscalers, mostly what we've been doing over the last few decades is building these data centers. They're called data centers because they're about storing data and moving data around. The AI factories we're building are very, very different from a traditional data center. The analog equivalent to a data center is like a warehouse in your logistics supply chain. You store your products and goods and components in inside a warehouse. Mostly, there's no work being done there. Occasionally, you go move things around inside the warehouse or between warehouses in your supply chain.
It's exactly what data centers were before. We store our digital data in there, and occasionally we go serve that data over a web server and a web page or to some application. The amount of energy required to run a data center is very little because you're not actually doing much work. An AI factory is fundamentally different because with AI, what we're doing is we're taking raw material, which is our raw data that comes in as input, these input tokens. We're applying energy, then we're taking that data and we're transforming it.
We're turning it into a more valuable product. We're refining it into intelligence. And the output, the measure of that intelligence, the unit, is called a token. If you have a machine that can take these raw inputs and energy as input and output something that's more valuable than the inputs, then obviously you want to do more and more of that. That's That's essentially what a factory is. We you take raw materials, if you're going to produce a car, you take your raw materials, your metals and plastics and all of the other materials along with energy, you do a bunch of work, and you transform it, and outcomes this refined product that is much more valuable than the inputs.
And name of the game is to maximize efficiency. You want for the inputs that you put in to get as many cars as you can out of it as fast as possible with as little downtime as possible, maximally efficient. So, this is what we are producing now. We're producing AI factories, and the name of the game is having maximum up time, maximum number of tokens per unit of energy that we put in, and with minimal downtime. We want to scale this up as much as possible, and we're on this continual push to make it more and more efficient.
NVIDIA is doing this by essentially working at a super Moore's law curve. Every year, we are increasing our token efficiency by more than 10 times. So, if it cost you $1 to do a certain unit of work this year, next year it will be less than 10 cents, and the year after it'll be less than 1 cent for the same unit of work. This is an incredible achievement. It's an incredible force that drives all innovation and drives all productivity in every sector.
When do you think physical AI will become mainstream? Is it like in 3 years, 5 years, just as common as now people are utilizing ChatGPT? When will those see those robots doing surgeries and then self-driving cars? When does this all become a really mainstream?
Well, it's already becoming mainstream. You got to drive in a self-driving car at GTC in Silicon Valley and San Francisco, you know, where where I live. It's become It's been mainstream for a while. You can get Waymos anywhere all the time. They're all over the place. It's become commonplace. The thing about technology is that the future with technology is already here. It's just unevenly distributed geographically. With physical AI, it's harder to have a ChatGPT moment that everybody experiences at the same time. With ChatGPT, because it's ultimately a simple webpage that anybody can get access to just by going to a website, everybody could experience it immediately.
With physical AI, we actually need to manufacture the physical systems. We need the robots. We need We need all these components. But my expectation is that over the course of the next 2 years, we're going to start seeing it become commonplace to have advanced AI-native robots inside our factories and inside our warehouses and supply chains. That's where we're going to see see the application of it first at scale, because that's where it's going to provide the most value immediately. We'll see the same robots become more consumerized. They'll become cheaper to to produce, more robust and safer.
So, we'll be able to take them out in to common spaces with normal people, and we'll see them elsewhere.
What's one clever way that you are utilizing AI now?
Well, I think the most clever way to use it now is with these new coding agents and the agents that we're building in general. We now have an amplifier of humans that is orders of magnitude. I'm personally programming again. I'm coding without actually coding. I've been coding since I was 6 years old back in the early 80s, but at some point I became a manager and lead a large organization of other engineers and coders. But now with these coding agents, with the agentic AI, I'm able to now produce more code than I was back when I was a coder myself, all by myself, by having a team of AI coders working for me.
That's been pretty amazing.
Yes, I think that I can totally relate to this. And the last question, I know you report directly to Jensen. So, what's one advice that Jensen gave you that really stuck with you?
Jensen has this one saying, it's as much as necessary, as little as possible, which is something that's now been burned into my psyche. It's a great wisdom that I think everybody should take. When you grow in your capabilities and your resources, it's very easy to start thinking about doing everything. you're like, it might it's it'll be easier for me to go produce this thing instead of going and buying it from a partner, partnering with somebody else to do it. But that's almost always the wrong thing to do. We all have limited resources, we have limited time, limited number of people to go execute on something.
So, you should do as much as necessary, but as little as possible. If somebody else is already doing it, then let them do that. Go focus on the things that only you can do well. Go focus and do it really well and contribute something unique to the world instead of trying to take something away from others.
Yeah, so I have a lot of AI practitioners in my audience. What would you recommend them to start right now?
For all of the AI practitioners, they need to jump into this whole agentic thing. Go learn how to work with agents. Go use Open Claw. Go use Hermes. Go use all the different coding agents, but truly autonomous long-running agents, the name of the game now is orchestrating and managing large numbers of agents that will go do the majority of the work for us. we are getting to the point where our only bottleneck is the rate at which we can produce good ideas, good ideas worth applying our resources to. So, we have to train ourselves to work in this new world where we have an abundance of workers, but a scarcity of good ideas.
If you want to get started with a gentle AI, all you open source models and fine-tune them with Cerebras GPU. Find the resources in the caption and I will see you in the next episode.