Physical AI: The 3 pillars of the new Robotics

In 2009, a US Airways pilot named Chesley Sullenberger, aka Captain Sully pulled an incredible move. His plane had just taken off from La Guardia and got hit by birds, destroying both his motors. With no thrust, no altitude, and only a handful of seconds to decide, he made the call to land the plane and its 155 passengers on the Hudson. By a miracle, all passengers were saved, and only the plane got damaged.

Came the investigations, and Sully faces trial for destroying the plane on the Hudson instead of returning it safely to the airport. The board ran simulations proving he could have turned back and landed safely at the airport, which meant he’d destroyed a perfectly good plane for nothing. If they were right, his career was finished.

The excellent movie “Sully” covers this story, and shows an incredible moment when his lawyer asked the accusation: “How many simuations did you run before succeeding the landing?” To which they replied: “17”. They therefore exposed that 16 other times, they would have crashed the plane on Manhattan buildings.

The problem of simulations vs reality is overly present; and this especially in Robotics. In the past few months, you’ve probably been shown videos of humanoid robots calmly folding t-shirts, or playing ping pong, or dealing poker cards with a 100% success rate… but is this really 100%? How many failed attempts occurred?

The answer has a nuance: if you’re doing pure Robotics, the number may be incredibly consistent. When you’re doing “Physical AI”, the number can vary immensely. The difference and definition of “Physical AI” is exactly what I’d like to cover in this article.

So here is our plan:

1. What is Physical AI? And how is it different from Robotics?
2. The Ladder of Physical AI: 6 Stages
3. The Components of Physical AI: World Models, Foundation Models, Reinforcement Learning, and so on…

Let’s begin:

What is Physical AI? And how is it different from Robotics?

Back in January 2026, I travelled to CES and took a picture of me with a humanoid robot. I added a caption mentioning “Physical AI: the theme of this year CES”. To which, a robotics engineer replied that Physical AI was a marketing term for something we already have available: a robot.

Was he true?

At the time, I thought he was, because Physical AI really is a fancy word for robotics… But the more I learn about it, the more I realize, there may actually be more to it than just marketing.

Let’s review the first difference:

1) Rule-Based Traditional Robotics

Back in 2018, I was working on an autonomous shuttle demo for the mayor of a city we were trying to win over. I remember this scene vividly; I was in the shuttle, and we were driving on a street that had a vehicle parked half on the sidewalk, half on the street. Suddenly, a surge of panic happened…

“How will the autonomous shuttle do?”

We had not programmed the “overtake vehicle” function, and we therefore saw only 2 possibilities: The shuttle could detect a vehicle in its lane, and get stuck… or it could detect it, and decide to pass extremely close to it; which could be considered somewhat dangerous.

The shuttle took the second option. To the mayor, it looked like a success. To us, it was actually an incredible edge case, and failure we didn’t anticipate. So what happened? We were working with a robot, and implementing what I named in my Autonomous Vehicle Architecture article “The 4 pillars of Robotics” (Perception, Localization, Planning and Control):

Perception saw the vehicle. Localization positioned ourself in the lane. Planning made the call to continue driving. Control actuated the vehicle. These are the 4 pillars of autonomous vehicles.

In Robotics, these 4 pillars are called differently: Sense, Plan, Act.

• Sense is the Perception layer, identifying what’s around you
• Plan is the Decision layer, planning the motion your robotic arm, dog, or robot must take
• Act is the Control layer, making the robot move with joints, and control algorithms

Physical AI is ALSO doing this, but differently; rather than rules, we use environments, data, Reinforcement Learning, and other ideas…

2) New-Gen Robotics: Physical AI

The Sense Plan Act scheme works everywhere, and the objective is the same: to generate a “motion” trajectory for our robot. Here is how it differ:

• In Traditional Robotics, we implement the motion via rules. We can learn that rule with Deep Learning, but overall, we have a modular infrastructure, with separate nodes communicating to eachother.
• In Physical AI, we are purely in “learn” mode. It’s the “End-To-End” version. All the keywords you see, such as imitation learning, reinforcement learning, data generation, 3D Reconstruction, World Models, Foundation Models, VLA, and so on… are part of it.

So… Physical AI is End-To-End Learning? Not quite. Back in 2016, Nvidia released an End-To-End model called PilotNet. It was predicting the steering angle of a vehicle based on its input image. Were people calling it Physical AI? Would they do it today? Absolutely not.

So here is how we can define Physical AI:

• It’s generalist — based on Foundation models (not specific to one task)
• It works cross bodies — the same network can work across a humanoid, a robotic arm, an AMR, even a car. (you can see Tesla doing this with Optimus & FSD sharing the same network)
• It uses world models — it doesn’t stop at mapping pixels to controls, it learns how the scene evolves so it can predict rollouts and plan against them.
• It’s simulation-fed — while End-To-End can work with just a few dashcam samples, Physical AI manufactures its own data through 3D reconstruction, simulators like Omniverse, and generative world models like Cosmos.
• It’s language-conditioned — it can explain itself. A VLA is a pretrained VLM with an action head, it reads the scene and the instruction, reasons over both, then emits actions. PilotNet had no semantics and no instruction to follow, it only steered. This is the wave filling CVPR 2026, a language model that reasons and then hands down the trajectory.

We’ll get in the details later — but I think you now have a good idea of it. So let’s continue:

The 6 Stages of the Physical AI Ladder (Sense & Plan/Act)

How do we know we’ve reaches Physical AI? We have discussed the idea of Sense, Plan, Act. So let’s see in more depth where each idea holds:

The Computer Vision Ladder (6 Stages)

If you’re working on the Sense part, for example in Computer Vision, you’ve probably been taught ideas, for example:

• Level 1 is image classification
• The next level is to be able to do a bounding box around it, which is object detection
• And the ultimate level is to classify not an entire image, not a box, but every single pixel of the image, which is segmentation

This paradigm has been completely DESTROYED by the new world.

Here is what we have now:

• Level 1 remains Image Classification is the most fundamental thing you can do
• Level 2 is Object Detection & Segmentation (what we just discussed)
• Level 3 is Visual Question Answering: the ability to ask a model to describe a scene or answer questions (ChatGPT does that)
• Level 4 is Scene & Depth understanding: More than just a depth map; a true 3D understanding of the scene and events
• Level 5 is Video & Temporal Reasoning: I talked about it a lot in my Video Segmentation article. The next-gen of Computer Vision is video.
• Level 6, the ultimate one (for now) is Reasoning. Beyond just seeing objects, being able to reason on what to do is the ultimate step of the Sense part.

Can you see the difference? If your model is able to do segmentation, and visual QA, it’s a great one, but it lacks the physics/3D, as well as the video/temporal, and reasoning block. If it can do depth, and represent the world in 3D, but not on videos, and not with reasoning, then it’s also not complete.

The Plan & Act ladder has similar stages

So what about Plan & Act? It turns out, there is also a ladder to it.

How to read it:

• Level 1 is purely scripted and hardcoded motion. Pilots & protos do that. Small DYI robots that you program to “move forward 10m” are exactly level 1.
• Level 2 is what everybody does: Classical Motion Planning. The algorithms like A* or RRT* do it.

So far, you can notice that Level 2 of Sense (Object Detection + Segmentation) and Level 2 of Plan (Classical Planning) is what 90% of robotics startups have been doing and still do today.

• Level 3 introduces Reinforcement Learning in Simulation. Many robotics lab have that.
• Level 4 brings Imitation Learning. In particular, we are now learning a policy from examples [Tesla, Nvidia, … do that].
• Level 5 is about Language Conditioned Manipulation. It’s a hot frontier that companies like Wayve are really good at.
• Level 6 is the ultimate step: Generalist Robot Policies that can drive anywhere.

Hopefully, you can start seeing the difference between a “learned” level 2, and a Level 5 or 6. So now, what ‘defines’ this further?

The Ingredients of Physical AI

In the first part we defined Physical AI as being generalist, cross-bodies, world-model driven, simulation-fed, and language-conditioned. In the second part, we said its level 6 were reasoning & generalist robot policies. How do we reach that? There are a few solid ingredients for this, let’s take a look.

Ingredient 1: Foundation Models

Foundation Models are the backbone of Physical AI. Ever heard of a backbone? This network that processes input images, and holds the learning of your data? The VGG Net and ResNet? That’s the same idea, but an extremely “generalist” networks, that didn’t just learn for one dataset, or one task, but holds a general understanding of the world.

Usually, Foundation Models are trained on un-labeled dataset. They themselves are: unlabeled, multi-modal (they can usually process ANY input data like text, audio, images, point clouds, …) and multi-task. It can also hold representation of objects, spatial relationships, and physical dynamics (motion, weight, pressure, …).

An example would be Nvidia Cosmos; the foundation model can process multiple input types, can be prompted, and answer questions, and so on… So that’s one element. Next one is World-Models.

Ingredient 2: World Models

Did you ever play Red Dead Redemption? If you haven’t, or if video games aren’t your style, you should definitely take a look at it. The folks from Rockstar Games have recreated the old west, and have you play as a cow-boy there. One of my all-time favourites. Can you see what this is about? 

Rockstar Games built a digital-twin of the old west. They recreated a world… and not just a standard-world, but an open-world, where you can interact in it; and where random events can happen.

World models follow that idea:

If we want to solve the long tail problem, we need to stop feeding AI billions of images, and start feeding it EXPERIENCES. If your vehicle has seen 100s of hours of driving, but most of it is a straight road, then it didn’t learn much. 

In robotics, drones, and self-driving cars, anywhere, we want to manufacture experiences. There are 2 ways to do this:

• You ‘reconstruct’ it using 3D Reconstruction, Gaussian Splatting, etc…
• You ‘generate’ it using World Models

If you’re using 3D Reconstruction, you build a very accurate model of the existing world. You recorded the street you went in, turned that to a 3D point cloud, added textures, and then plugged it to a simulator to retrain on it. The problem is, you’re limited to what you recorded. You can probably change the weather, but that’s about it.

A World Model is a generator of experiences. You start with a scene, let’s say a car driving in a road, and then you’re thinking… what if that road was now a desert? The World Model is the AI that is going to generate that desert scene. Because it’s generative AI, it can do an infinite variation; and because it’s a world model, it also defines the future world based on your actions. If you decide to drive on that sand block, what does the scene look like now?

An example:

Nvidia recently released OmniDreams. It’s a World Model that can “dream” any scene you prompt. One example here with a car driving in a tornado environment:

That’s the spirit. By definition, a World Model is:

1. Generative — you can prompt what you want, it does it
2. Closed-Loop — you can put your car models inside, let it make decisions (turn right, don’t yield to pedestrian, …) and the environment will adapt and change
3. Sim-To-Real — it can be used as training data
4. Language Editing — You can change the traffic lights, time of day, weather, road users … using a single prompt
5. (Optional) Multi-Modal — it also generates LiDAR point clouds corresponding to the scene

So, we’ve seen that:

Physical AI = World Models + Foundation Models.

Are there any other ingredients? Yes:

Ingredient 3: Vision Language Action Models

The final part is Vision Language Action Models. Once we are in a world, how do we make an action into it? The Foundation Model helps us reason, but it doesn’t help us drive autonomously. VLMs are here for that. As a standard definition, we can say that:

Vision Language Action Models = Vision Language Models + Action Models

The first part, Vision Language Model, can be any AI that can process “vision”. GPT5 for example is part of it. The second part, Action Model, can be any AI that performs an action as a result. For example, a Diffusion Transformer that generates the next trajectory to take.

Here is an example with an Nvidia model called GR00T:

All of these models are trained with the last piece, Reinforcement Learning & Slow/Fast Networks…

Alright, so we’ve seen quite a lot today. Your brain must be EXPLODING. Ready for a recap?

Summary & Next Steps

• In self-driving cars, we have 4 pillars called Perception, Localization, Planning, and Control. In Robotics & Physical AI, we have the “Sense, Plan, Act” framework.
• In Robotics, the Sense Plan Act is mostly modular and built with rules. You’re using a mix of classical algorithms and Deep Learning.
• Sense: The first part is Sense, which is built upon the 6 stages of Computer Vision. We no longer just do segmentation & detection, these are now Level 2 tasks. New tasks are Video Perception, Reasoning, Depth, and Visual QA.
• Plan/Act: Plan & Act also evolved. It’s no longer about classical A* planning algorithms, but about Foundation Models trained with Reinforcement Learning, Action Models, and World Models.
• A Foundation Model is the new backbone in Deep Learning. It learns from millions of data points in all possible format (text, sound, images, …) and can then be adapted to a downstream task
• A World Model is the environment where the Physical AI device is trained and tested. 3D Reconstruction has done a big part here, but the evolution is to simply “generate” new scenes.
• Vision Language Action Model is the final piece. It’s doing the “Act” part by merging Vision Language Models and Action Models.

Next Steps

Congratulations on reaching this far! You did an amazing job. Now here is where you can take it from here:

• Read more articles from the blog (I have dozens of these, written the same way)