In 2009, a US Airways pilot named Chesley Sullenberger, aka Captain Sully pulled an incredible move. His plane had just taken off from La Guardia and got hit by birds, destroying both his motors. With no thrust, no altitude, and only a handful of seconds to decide, he made the call to land the plane and its 155 passengers on the Hudson. By a miracle, all passengers were saved, and only the plane got damaged.

Came the investigations, and Sully faces trial for destroying the plane on the Hudson instead of returning it safely to the airport. The board ran simulations proving he could have turned back and landed safely at the airport, which meant he’d destroyed a perfectly good plane for nothing. If they were right, his career was finished.

The excellent movie “Sully” covers this story, and shows an incredible moment when his lawyer asked the accusation: “How many simuations did you run before succeeding the landing?” To which they replied: “17”. They therefore exposed that 16 other times, they would have crashed the plane on Manhattan buildings.

The problem of simulations vs reality is overly present; and this especially in Robotics. In the past few months, you’ve probably been shown videos of humanoid robots calmly folding t-shirts, or playing ping pong, or dealing poker cards with a 100% success rate… but is this really 100%? How many failed attempts occurred?

The answer has a nuance: if you’re doing pure Robotics, the number may be incredibly consistent. When you’re doing “Physical AI”, the number can vary immensely. The difference and definition of “Physical AI” is exactly what I’d like to cover in this article.

So here is our plan:

1. What is Physical AI? And how is it different from Robotics?
2. The Ladder of Physical AI: 6 Stages
3. The Components of Physical AI: World Models, Foundation Models, Reinforcement Learning, and so on…

Let’s begin:

What is Physical AI? And how is it different from Robotics?

Back in January 2026, I travelled to CES and took a picture of me with a humanoid robot. I added a caption mentioning “Physical AI: the theme of this year CES”. To which, a robotics engineer replied that Physical AI was a marketing term for something we already have available: a robot.

Was he true?

At the time, I thought he was, because Physical AI really is a fancy word for robotics… But the more I learn about it, the more I realize, there may actually be more to it than just marketing.

Let’s review the first difference:

1) Rule-Based Traditional Robotics

Back in 2018, I was working on an autonomous shuttle demo for the mayor of a city we were trying to win over. I remember this scene vividly; I was in the shuttle, and we were driving on a street that had a vehicle parked half on the sidewalk, half on the street. Suddenly, a surge of panic happened…

“How will the autonomous shuttle do?”

We had not programmed the “overtake vehicle” function, and we therefore saw only 2 possibilities: The shuttle could detect a vehicle in its lane, and get stuck… or it could detect it, and decide to pass extremely close to it; which could be considered somewhat dangerous.

The shuttle took the second option. To the mayor, it looked like a success. To us, it was actually an incredible edge case, and failure we didn’t anticipate. So what happened? We were working with a robot, and implementing what I named in my Autonomous Vehicle Architecture article “The 4 pillars of Robotics” (Perception, Localization, Planning and Control):

Perception saw the vehicle. Localization positioned ourself in the lane. Planning made the call to continue driving. Control actuated the vehicle. These are the 4 pillars of autonomous vehicles.

In Robotics, these 4 pillars are called differently: Sense, Plan, Act.

• Sense is the Perception layer, identifying what’s around you
• Plan is the Decision layer, planning the motion your robotic arm, dog, or robot must take
• Act is the Control layer, making the robot move with joints, and control algorithms

Physical AI is ALSO doing this, but differently; rather than rules, we use environments, data, Reinforcement Learning, and other ideas…

2) New-Gen Robotics: Physical AI

The Sense Plan Act scheme works everywhere, and the objective is the same: to generate a “motion” trajectory for our robot. Here is how it differ:

• In Traditional Robotics, we implement the motion via rules. We can learn that rule with Deep Learning, but overall, we have a modular infrastructure, with separate nodes communicating to eachother.
• In Physical AI, we are purely in “learn” mode. It’s the “End-To-End” version. All the keywords you see, such as imitation learning, reinforcement learning, data generation, 3D Reconstruction, World Models, Foundation Models, VLA, and so on… are part of it.

So… Physical AI is End-To-End Learning? Not quite. Back in 2016, Nvidia released an End-To-End model called PilotNet. It was predicting the steering angle of a vehicle based on its input image. Were people calling it Physical AI? Would they do it today? Absolutely not.

So here is how we can define Physical AI:

• It’s generalist — based on Foundation models (not specific to one task)
• It works cross bodies — the same network can work across a humanoid, a robotic arm, an AMR, even a car. (you can see Tesla doing this with Optimus & FSD sharing the same network)
• It uses world models — it doesn’t stop at mapping pixels to controls, it learns how the scene evolves so it can predict rollouts and plan against them.
• It’s simulation-fed — while End-To-End can work with just a few dashcam samples, Physical AI manufactures its own data through 3D reconstruction, simulators like Omniverse, and generative world models like Cosmos.
• It’s language-conditioned — it can explain itself. A VLA is a pretrained VLM with an action head, it reads the scene and the instruction, reasons over both, then emits actions. PilotNet had no semantics and no instruction to follow, it only steered. This is the wave filling CVPR 2026, a language model that reasons and then hands down the trajectory.

We’ll get in the details later — but I think you now have a good idea of it. So let’s continue:

The 6 Stages of the Physical AI Ladder (Sense & Plan/Act)

How do we know we’ve reaches Physical AI? We have discussed the idea of Sense, Plan, Act. So let’s see in more depth where each idea holds:

The Computer Vision Ladder (6 Stages)

If you’re working on the Sense part, for example in Computer Vision, you’ve probably been taught ideas, for example:

• Level 1 is image classification
• The next level is to be able to do a bounding box around it, which is object detection
• And the ultimate level is to classify not an entire image, not a box, but every single pixel of the image, which is segmentation

This paradigm has been completely DESTROYED by the new world.

Here is what we have now:

• Level 1 remains Image Classification is the most fundamental thing you can do
• Level 2 is Object Detection & Segmentation (what we just discussed)
• Level 3 is Visual Question Answering: the ability to ask a model to describe a scene or answer questions (ChatGPT does that)
• Level 4 is Scene & Depth understanding: More than just a depth map; a true 3D understanding of the scene and events
• Level 5 is Video & Temporal Reasoning: I talked about it a lot in my Video Segmentation article. The next-gen of Computer Vision is video.
• Level 6, the ultimate one (for now) is Reasoning. Beyond just seeing objects, being able to reason on what to do is the ultimate step of the Sense part.

Can you see the difference? If your model is able to do segmentation, and visual QA, it’s a great one, but it lacks the physics/3D, as well as the video/temporal, and reasoning block. If it can do depth, and represent the world in 3D, but not on videos, and not with reasoning, then it’s also not complete.

The Plan & Act ladder has similar stages

So what about Plan & Act? It turns out, there is also a ladder to it.

How to read it:

• Level 1 is purely scripted and hardcoded motion. Pilots & protos do that. Small DYI robots that you program to “move forward 10m” are exactly level 1.
• Level 2 is what everybody does: Classical Motion Planning. The algorithms like A* or RRT* do it.

So far, you can notice that Level 2 of Sense (Object Detection + Segmentation) and Level 2 of Plan (Classical Planning) is what 90% of robotics startups have been doing and still do today.

• Level 3 introduces Reinforcement Learning in Simulation. Many robotics lab have that.
• Level 4 brings Imitation Learning. In particular, we are now learning a policy from examples [Tesla, Nvidia, … do that].
• Level 5 is about Language Conditioned Manipulation. It’s a hot frontier that companies like Wayve are really good at.
• Level 6 is the ultimate step: Generalist Robot Policies that can drive anywhere.

Hopefully, you can start seeing the difference between a “learned” level 2, and a Level 5 or 6. So now, what ‘defines’ this further?

The layers of Physical AI

In the first part we defined Physical AI as being generalist, cross-bodies, world-model driven, simulation-fed, and language-conditioned. In the second part, we said its level 6 were reasoning & generalist robot policies. How do we reach that? There are a few solid layers for this, let’s take a look.

Layer 1: Foundation Models answers "How is it trained?"

Foundation Models are the backbone of Physical AI. Ever heard of a backbone? This network that processes input images, and holds the learning of your data? The VGG Net and ResNet? That’s the same idea, but an extremely “generalist” networks, that didn’t just learn for one dataset, or one task, but holds a general understanding of the world.

Usually, Foundation Models are trained on un-labeled dataset. They themselves are: unlabeled, multi-modal (they can usually process ANY input data like text, audio, images, point clouds, …) and multi-task. It can also hold representation of objects, spatial relationships, and physical dynamics (motion, weight, pressure, …).

An example would be Nvidia Cosmos; the foundation model can process multiple input types, can be prompted, and answer questions, and so on… So that’s one element. Next one is World-Models.

In my VIDEO PERCEPTION course, I teach engineers to build a Video Transformer that can (1) process videos and (2) output a class. The model is trained on supervised dataset. Then I ask them: "Is this a Foundation Model?". The answer is no, even if a Transformer, because (1) it's supervised and (2) single-task.

Layer 2: World Models answers "What does it produce?"

Did you ever play Red Dead Redemption? If you haven’t, or if video games aren’t your style, you should definitely take a look at it. The folks from Rockstar Games have recreated the old west, and have you play as a cow-boy there. One of my all-time favourites. Can you see what this is about? 

Rockstar Games built a digital-twin of the old west. They recreated a world… and not just a standard-world, but an open-world, where you can interact in it; and where random events can happen.

World models follow that idea:

If we want to solve the long tail problem, we need to stop feeding AI billions of images, and start feeding it EXPERIENCES. If your vehicle has seen 100s of hours of driving, but most of it is a straight road, then it didn’t learn much. 

In robotics, drones, and self-driving cars, anywhere, we want to manufacture experiences. There are 2 ways to do this:

• You ‘reconstruct’ it using 3D Reconstruction, Gaussian Splatting, etc…
• You ‘generate’ it using World Models

If you’re using 3D Reconstruction, you build a very accurate model of the existing world. You recorded the street you went in, turned that to a 3D point cloud, added textures, and then plugged it to a simulator to retrain on it. The problem is, you’re limited to what you recorded. You can probably change the weather, but that’s about it. It's a closed-loop simulator.

A World Model is a generator of experiences. You start with a scene, let’s say a car driving in a road, and then you’re thinking… what if that road was now a desert? The World Model is the AI that is going to generate that desert scene. Because it’s generative AI, it can do an infinite variation; and because it’s a world model, it also defines the future world based on your actions. If you decide to drive on that sand block, what does the scene look like now?

An example:

Nvidia recently released OmniDreams. It’s a World Model that can “dream” any scene you prompt. One example here with a car driving in a tornado environment:

That’s the spirit. By definition, a World Model is:

1. Generative — you can prompt what you want, it does it
2. Closed-Loop — you can put your car models inside, let it make decisions (turn right, don’t yield to pedestrian, …) and the environment will adapt and change
3. Sim-To-Real — it can be used as training data
4. Language Editing — You can change the traffic lights, time of day, weather, road users … using a single prompt
5. (Optional) Multi-Modal — it also generates LiDAR point clouds corresponding to the scene

So, we’ve seen that:

Physical AI = World Models + Foundation Models.

And these don't have to be 2 separate models. A World Model CAN BE a Foundation Model. In fact, it often is, hence the term World Foundation Model (WFM).

Are there any other layer? Yes:

Layer 3: Vision Language Action Models answer "What does it do?"

The final part is Vision Language Action Models. Once we are in a world, how do we make an action into it? The Foundation Model helps us reason, but it doesn’t help us drive autonomously. VLMs are here for that. As a standard definition, we can say that:

Vision Language Action Models = Vision Language Models + Action Models

The first part, Vision Language Model, can be any AI that can process “vision”. GPT5 for example is part of it. The second part, Action Model, can be any AI that performs an action as a result. For example, a Diffusion Transformer that generates the next trajectory to take.

Here is an example with an Nvidia model called GR00T:

Here again, a VLA can be a Foundation Model and a World Model together. These aren't separate blocks, and these can be combined.

Alright, so we’ve seen quite a lot today. Your brain must be EXPLODING. Ready for a recap?

Summary & Next Steps

• In self-driving cars, we have 4 pillars called Perception, Localization, Planning, and Control. In Robotics & Physical AI, we have the “Sense, Plan, Act” framework.
• In Robotics, the Sense Plan Act is mostly modular and built with rules. You’re using a mix of classical algorithms and Deep Learning.
• Sense: The first part is Sense, which is built upon the 6 stages of Computer Vision. We no longer just do segmentation & detection, these are now Level 2 tasks. New tasks are Video Perception, Reasoning, Depth, and Visual QA.
• Plan/Act: Plan & Act also evolved. It’s no longer about classical A* planning algorithms, but about Foundation Models trained with Reinforcement Learning, Action Models, and World Models.
• A Foundation Model is the new backbone in Deep Learning. It learns from millions of data points in all possible format (text, sound, images, …) and can then be adapted to a downstream task
• A World Model is the environment where the Physical AI device is trained and tested. 3D Reconstruction has done a big part here, but the evolution is to simply “generate” new scenes.
• Vision Language Action Model is the final piece. It’s doing the “Act” part by merging Vision Language Models and Action Models.

Next Steps

Congratulations on reaching this far! You did an amazing job. Now here is where you can take it from here:

• Read more articles from the blog (I have dozens of these, written the same way)

Several years ago, I was in Cannes with my wife and in-law family, when a terrible event happened. We were getting out of the restaurant, walking on the street down towards the famous "Croisette", when suddenly... **BANG**. A loud bang got heard. Seconds later, hundreds, if not thousands of people were rushing towards me.

I had very little information to process what was happening, but movement alone told me I had to RUN IN THE OPPOSITE DIRECTION. Unfortunately, not all of us processed it the same way. My father in law stayed static. I was there "We gotta move! NOW!", and he was standing still, forcing all of us to face the huge human wave coming towards us.

Nobody got hurt. We managed to get inside a building, let the crowd pass. Then, 2 hours of extremely intense hiding. I remember it, we thought terrorists were shooting at us. Twitter was reporting the Carlton hotel being hijacked, hundreds of death. Police sent messages telling everyone to avoid the streets and lock the doors.

All of that was nothing. 2 hours later, we got out. And the story never made the day. We never learned what truly happened. It surely wasn't terrorists like Twitter was saying.

In this story though, something interesting happened: we all analyzed danger because of movement. Had the people been static, even if everybody yelled, nobody would have run.

RADARs work the same way. They aren't the sensors you want to use on static scenes, but they are EXCELLENT at catching movement in a scene, identifying the speed of every moving object, and even, in some case, going through objects.

In this post, I want to dive into how RADARs work, help you understand why and when to use it, and help you understand the core ideas of RADAR Detection And Ranging.

Let's begin...

Why RADAR is useful

If you've been following the self-driving car space for a while, you've probably noticed something. Everyone talks about LiDARs. Everyone talks about cameras. Almost nobody talks about RADARs.

And it's not because RADARs don't matter. It's because they're hard. Cameras give you an image you can look at. LiDARs give you a point cloud you can visualize. A RADAR gives you a signal, and unless you know signal processing, that signal means nothing to you.

Andras Palffy, who I interviewed in Rotterdam for my episode on RADAR Deep Learning, summarized it better than I ever could:

LiDAR: easy to understand, easy to work with, super expensive. RADAR: hard to understand, hard to work with, and actually quite cheap.

That last part is what makes it worth your time. A RADAR costs a few hundred dollars, not tens of thousands. It fits behind the front bumper of a car you already own. And it does things no other sensor on the vehicle can do.

It works in rain, fog, snow, and total darkness, when cameras are blind and LiDARs start to struggle. It measures the velocity of a moving object directly, instead of inferring it across multiple frames. And it can even see underneath and around obstacles, which means it can warn you about an accident happening a few cars ahead, before you can physically see it.

So yes, RADAR is harder to learn than the rest of the sensor stack. But that's exactly why understanding it makes you valuable.

To sum up the picture, RADARs can measure the velocity of an object directly and can work during different scenarios such as day, night, rain, fog, snow… while other sensors may be troubled.

How RADARs work

Introduction

RADAR stands for Radio Detection And Ranging. It works by emitting electromagnetic waves that bounce back when they hit an obstacle. Since these are radio waves and not light, they go through rain, fog, and darkness without caring.

But here is the part that confused me for a long time. A RADAR doesn't "see" anything. It sends a wave out, it gets a wave back, and that returning wave is all it ever gets. So the entire job of a RADAR is to look at how the wave came back, and deduce what happened to it out there.

And this is where it clicks. Every property of that returning wave tells you something specific about the object it hit:

• The time it took to come back tells you how far the object is. This is the range.
• The change in frequency tells you how fast the object is moving toward you or away from you. This is the velocity, and it comes from the Doppler Effect.
• The strength of the returning signal tells you something about the object itself, its size, its shape, and what it's made of. This is the RADAR Cross Section.
• The direction the wave came back from tells you where the object is positioned, left or right of the sensor.

That is the whole mental model. A wave leaves, a wave returns, and by measuring 4 things about that returning wave, you reconstruct where an object is, how fast it's going, and roughly what it might be. Everything else in this article is just the details of how we extract those 4 measurements.

So before we go further, keep this in mind. Every wave has a wavelength, a frequency, an amplitude, and a phase. These aren't abstract physics terms here, they are literally the raw material we use to build a perception system.

Visually, it looks like this:

And a quick reminder on the vocabulary, because these two words come back constantly. The frequency is how many times the wave oscillates per second, and the wavelength is the physical distance between two peaks. They're inversely linked: high frequency means short wavelength, low frequency means long wavelength.

Now, let's look at the hardware actually sending those waves...

The wave itself (FMCW)

One quick detail before we measure anything. The RADAR in a car doesn't fire one short pulse and wait. It emits a continuous wave whose frequency constantly ramps up, then resets. That ramp is called a chirp, and on a graph it looks like a sawtooth.

Why does this matter? Because the frequency is always changing in a known pattern, the wave coming back is a slightly older version of the wave going out right now. Compare the two, and the gap between them hands you the round-trip time and the frequency shift at once. That's how one sensor measures distance and speed simultaneously.

The hardware doing this is simple enough: a synthesizer creates the chirp, an amplifier pushes it out to 300 meters, an antenna sends and receives, a mixer compares the two frequencies, and a processor runs everything on top.

Now let's extract our 4 measurements.

Measurement 1: How far is it? (Range)

This one is pure timing. The wave travels at the speed of light, so if you know how long it took to come back, you know how far it went.

The signal makes a round trip, so you halve the time. Range = c × T / 2. That's it. No neural network, no training data, just a stopwatch and a multiplication.

Measurement 2: How fast is it going? (Doppler Effect)

You already know this one, it's what gets you speeding tickets. It's called the Doppler Effect.

The middle wave is what the RADAR sends out. The top one bounced off a car coming toward us, and got squeezed. The bottom one bounced off a car driving away, and got stretched.

That squeeze or stretch is a measurable shift in frequency. Read the shift, and you get the speed of the vehicle and whether it's approaching or leaving.

This is RADAR's superpower. A camera or a LiDAR compares several frames to guess a speed. A RADAR reads it off a single measurement, instantly.

Measurement 3: What kind of object is it? (RADAR Cross Section)

Now we use how strong the returning signal is. Because not every object bounces a wave back the same way, and that difference is captured by a metric called the RADAR Cross Section, or RCS.

It depends on 4 things: the geometry of the target, the direction of the RADAR, the frequency, and the material the object is made of. A big flat metal truck reflects a lot. A pedestrian reflects very little.

The extreme example is the stealth aircraft. Angled surfaces deflect the waves away, a smooth body avoids scattering them back, and the paint absorbs whatever is left. The goal is to push the RCS to zero, so the plane never shows up at all.

Measurement 4: Where is it, left or right? (Angle of Arrival)

The last measurement is the simplest. A RADAR doesn't have one antenna, it has several. When a wave comes back, it hits those antennas at very slightly different moments.

That tiny difference tells you the direction the echo came from. Combine it with the range, and you no longer have just "something 30 meters away". You have something 30 meters away, 20 degrees to the left.

Extracting all of this from the raw signal (Fast Fourier Transform)

There's one step I've been skipping. The RADAR doesn't hand you range and velocity on a plate. It hands you a wave, and a raw wave is unreadable. All you see is an amplitude going up and down over time.

So we convert it with a Fast Fourier Transform, or FFT. It moves the signal from the time domain into the frequency domain. Most people just call an fft() function and move on.

This is the raw signal. Amplitude on the vertical axis, time on the horizontal one. Somewhere inside those oscillations there are obstacles, but you cannot point at any of them. Now watch what the FFT does to it.

Look at what comes out. Instead of a messy wave, you get a set of clean peaks. And each peak is an obstacle, carrying its own range, its own Doppler shift, and its own power.

This is the moment physics becomes perception. Before the FFT, you have a signal. After it, you have detections.

From measurements to a point cloud

Here is where everything connects. Every peak that comes out of the FFT arrives with our 4 measurements attached: a range, a Doppler velocity, a power, and an angle.

And range plus angle is a position. If something reflects at 30 meters, 20 degrees left, you can place it as a dot in space. Do that for every peak, and you stop having a signal. You have a point cloud, exactly like the one a LiDAR gives you, except each point also carries a speed and a reflectivity.

But notice the problem in this image. A single car doesn't produce one point, it produces a whole cluster of them scattered across its surface. The RADAR doesn't know which points belong together, it just sees dots.

So we group them with clustering algorithms. Euclidean Clustering or K-Means measure distances between points and merge the ones that clearly belong to the same object. One group becomes a car, another becomes a cyclist.

And that's the full chain. A wave goes out, comes back, gets transformed by the FFT, becomes points in space, and those points get grouped into a car with a position and a speed.

But isn't RADAR data too noisy to be useful?

This is the objection I hear the most, and it's a fair one. RADAR is genuinely noisy. You see one obvious obstacle and it reports 5 detections. You see an empty road and it reports something anyway. A can of soda on the asphalt can bounce a wave back and look like an object.

The first defence is the Signal to Noise Ratio. The stronger a detection is compared to the background, the more likely it's real.

Then comes thresholding. You set a cutoff level, keep every detection that rises above it, and throw away everything underneath. Simple in principle, but the exact value you pick decides what your RADAR can and cannot see.

And this graph shows the whole dilemma. Set the threshold too low and noise sneaks through as a false alarm. Set it too high and you miss a weak but real target. There is no perfect value, which is why professionals use an adaptive version of this called CFAR, for Constant False Alarm Rate.

So historically, the criticism was justified. For decades RADAR was treated as a secondary sensor, too noisy and too low-resolution, useful only as auxiliary input under the assumption that noisy measurements beat no measurements.

But even then, it earned its seat. No other sensor gives you the velocity of a moving object directly. That alone was worth the noise.

And today, that assumption is breaking. High-resolution imaging RADARs now produce point clouds far closer to LiDAR than to the noisy blobs we used to deal with, while keeping the cost, the weather resilience, and the direct velocity. I covered this shift in my article on the Imaging RADAR, and you can see Deep Learning applied to RADAR point clouds in my interview with Perciv AI.

How a RADAR Perception Engineer puts it together

Everything above is the theory, so let me show you it in production. A few months ago I hosted a live session with Isaac Berrios, a RADAR Perception Engineer at Anduril, in the defense space, and he walked us through the exact stack he uses every day.

You already know all three boxes. The FFT turns the raw signal into a Range-Doppler map. CFAR pulls the real targets out of the noise. And the RADAR Cross Section tells him what those targets actually are, which matters enormously in defense, where some drones are painted specifically to push their RCS toward zero and disappear.

What struck me most was how manual it still is. Those CFAR thresholds are tuned by hand, and set them slightly too high and you simply stop seeing real objects. Same three ideas you just learned, except the consequences are a little heavier.

Summary & Next Steps

• A RADAR never sees anything, it sends a wave out and reads what comes back.
• That returning wave carries 4 measurements: time gives you range, frequency shift gives you velocity, signal strength gives you the RADAR Cross Section, and the angle of arrival gives you direction.
• The FMCW chirp is what makes this possible, by letting the sensor measure distance and speed from the same continuous signal.
• The Fast Fourier Transform turns the raw wave into detections, each peak carrying its own range, speed, and power.
• Range plus angle turns those detections into a point cloud, and clustering turns that point cloud into actual cars and pedestrians.
• RADAR is noisy by nature, so engineers filter it with the Signal to Noise Ratio and adaptive thresholding (CFAR).
• It works where other sensors fail, in rain, fog, snow, darkness, and even underneath or around obstacles.
• Imaging RADAR is changing the game, producing point clouds close to LiDAR quality while keeping RADAR's cost and weather advantages.

If you want to keep going, the natural next step is sensor fusion. I'd read Introduction to RADAR Camera Fusion next, then Sensor Fusion: LiDARs & RADARs in Self-Driving Cars for the full picture.

Ever heard of Spider-Man's "Guy in the Chair"?. He's called Ned, and he's Peter Parker's best friend, but he's also the one who helps Spider-Man and whispers to him what to do when he's on a mission against the Vulture and the other bad guys. Batman has Alfred. Iron Man has Jarvis. All superheroes have someone or something that supports them while they're busy fighting the villains. Don't they?

In Robotics, we may be fighting the Perception & Planning villains, but what is our support system helping us achieve that?

Often, it's tools.

Every time I go visit a company, for example for a seminar, they show me some of the internal tools they've built. Most of the time, it's things they use every day and have invested a lot in. Many times, I've noticed companies were putting their entire robotic infrastructure on a particular tool provider: Foxglove.

Foxglove is your guy in the chair. It's a platform you use for visualization, testing, data management, and robotic workshops. Companies like Wayve, Shield AI, Gatik, and NVIDIA use Foxglove every day, especially because of a specific feature I will call "Multi-Player Mode".

Before we get to this, I'd like to give you a quick intro to the platform, so let's see:

There are a few things that I learned from Roman that I'd like to share with you:

• Can Foxglove replace RViz?
• Why Multi-Player Robotics is the future of Physical AI
• What being a CTO really is about

I. Can Foxglove replace RViz?

Traditionally, companies have used ROS (Robot Operating System) to build their self-driving cars, along with its free visualizer, RViz. As great as RViz can be, it has many drawbacks that a free and open product can have:

• Very simple viewer (only one 3D view, no side-by-side viewer with other plots etc.)
• Only on ROS with Linux (can't share a problem with managers who don't have ROS on their computer and are comfortable with it)
• Doesn't evolve (AI, robots, software)
• Single player mode (no collaboration, insight sharing, comments)
• No new or advanced features (data retrieval, querying, marking)

Looking at it this way, RViz feels very 2017-like; but these days. Wouldn't it feel normal to simply "ask" your data for some events and be able to retrieve them?

When working with free tools like RViz, you get a free visualizer, but you never get access to much. Foxglove, on the other hand, has a team funded and working every day on making these tools better.

The industry has a lot of discussions on "Saas vs Vibecoded"; I think when you experience Foxglove, there is no comparison to be made. A tool like RViz, or a custom visualizer, can work for one task, but the depth and scope of what you can do (and what you will be able to do in the future) is unmatched.

A note on MCAP Files

Foxglove has built something called MCAP (em-cap). It's a file format that replaces the previous multimodal log formats used by ROS. They benchmarked it against what ROS 2 was using at the time (SQLite-based bag files) and took the results directly to the ROS technical governance groups.

Results were so good that MCAP became the default recording format in ROS 2. Run ros2 bag record today, and you're using a Foxglove-designed format. This means when you go with a platform like Foxglove, you aren't following something; you're part of the leadership.

This brings me to where they lead too: Multi-Player Robotics.

II. Why Multi-Player Robotics is the future of Physical AI

Sebastian Thrun, known as the godfather of self-driving cars, once shared a story about a major problem his team faced during the DARPA Grand Challenge in 2005. Their self-driving car was able to drive through the desert for hours, perfectly following the destination. But during the long drives, the car sometimes detected a gigantic obstacle getting closer and closer and slammed the brakes.

Unexplainable.

Yet, it happened a certain number of times. So they ran the car again and again for hours to see the bug of the giant obstacle appearing in the middle of the desert. At some point, someone noticed this problem always happened after exactly a specific number of minutes, which led them to the solution (something about a timestamp refresh, from memory).

This would never happen with a tool like Foxglove, because we wouldn't need to re-run the car physically for hours and hours. What would happen is:

• We'd record the events to a file
• We'd replay the recording in the browser
• We'd share the problem with other engineers, some watching from other countries
• We'd simultaneously plot certain information, mark interesting comments until we debug the problem

Put differently, we wouldn't be in "Single-Player" mode, but in Multi-Player Robotics Mode.

Foxglove's real strength is collaboration. They enable Multi-Player Mode for you. You can share a recording with someone else who works on a Windows computer and has no idea or desire to install your exact version of ROS and Ubuntu. They can collaborate with you, tag important moments, and do a range of tasks so you jointly visualize and solve the problem.

Roman talked the most about events. While reviewing data, an engineer drops a timestamped annotation right on the timeline. "The foot was out of alignment" to use my own example from the demo.

Teams that never move past a single laptop and local files don't hit this wall at all. It shows up the moment there's more than one person who needs to look at the same robot's brain.

III. What being a CTO is really about

As I run monthly interviews with self-driving car and physical AI companies, I sometimes get to speak with executives, CEOs, CTOs, and Directors of Engineering. You can learn a lot from them.

One question I asked Roman was "What is different in a CTO rather than an engineer?"

He gave me a lot of interesting answers, but one quote in particular stuck with me.

He then went on to explain how he's spending A LOT of his time in Slack/Discord groups of Foxglove users, just seeing what people struggle with, need help with, and ask questions about... to shape the product!

This is something engineers rarely do. A typical engineer, especially when young and inexperienced, would see tasks like support and validation as "beneath them". When you're in startup mode, you quickly learn that nothing is beneath you. I have changed tires, fixed windshields, and done all sorts of silly tasks an engineer isn't supposed to do because I was in a startup.

As a CTO, you need to care extremely about the market and can't work in a silo. You need people to get value out of it.

On that last "value" point, there is a complete video in the Behind-The-Scenes of our interview that I absolutely recommend for figuring out where the value is in a product.

Summary & Next Steps

• Physical AI isn't just algorithms; it's also tools. Countless companies develop tools internally for tasks like visualization or data management. Foxglove developed them for you (and keeps on improving them so you can focus on the algorithms)
• RViz is a great visualization tool, but it has the drawbacks of a free visualizer. It's never improved, hasn't been enhanced with new features, and is stuck in a 2017-like world. On the other hand, a platform like Foxglove is continuously improved to stay current.
• Foxglove pioneered the MCAP format, which is now the default file format in ROS 2. When you record a scene, everything is stored in an MCAP file, which Foxglove can visualize directly.
• The future of Physical AI and Robotics happens in Multi-Player Mode. When using a tool like Foxglove, MPM lets you share recordings with others in their browsers, mark events, and collaborate on a problem rather than fixing it alone.
• A CTO tends to care about customers in ways other roles may not. They spend a lot of time reading where users struggle to anticipate the next features. They are constantly in touch with their market; it's about the market first and the features second.

Interested in the full workshop?

Here's what you get in this training:

• You will understand how Foxglove works (you can add that to your resume) and have the skill of turning everything into a Foxglove file (so any project, portfolio demo, dataset, anything… thanks to our bonus workshop).
• You'll be introduced to MCAP files, the now-default format used in the industry, and have been through the SDK to build your own
• You will understand how CTOs think, and the 2/3 critical mind-shifts engineers lack that CTOs have
• You'll get valuable career and hiring insight from a working CTO (what it actually looks like to lead technically while staying hands-on, what he screens for when hiring engineers at an early-stage startup, and how he thinks about fundraising a $40M Series B)
• Plus a lot of other bonuses included…

Get access to this, PLUS my 2 past fragments here:

https://www.thinkautonomous.ai/the-edgeneers-land

(P.S: This offer may not be available by the time you read this, make sure to join the membership while it's live)

Have you ever doubted your loving partner? In 1945, a Canadian spy named Max Vatan just received an intel that changed everything. His wife Marianne was suspected of being a German spy. His orders were now to find out if it was true, within 72 hours. If she was guilty, he then had to kill her. This is NOT a true story, but the story of the movie Allied, with Brad Pitt and Marion Cotillard.

So how do you find out if your wife is a german spy? Max had easy options and complex ones. He could have her followed. He could interrogate her friends, search the house, pull her files... but that was complex. So instead, he made a simpler choice; let her overhear a phone conversation sharing false intelligence, and see if the enemy acted on it. Unfortunately for him, they did.

This technique is called the canary trap. It's used to identify leaks by feeding slightly different versions of the same false information to different suspects. Whoever's version shows up in the enemy's hands is the traitor. There is something I love about it: it's extremely simple. While many techniques are based on heavy profiling, and probabilities... this one is just relying one one piece of information...

That simplicity, I think Occupancy Grid Mapping in robotics shares it. You could build a world with a complex 3D representation, bounding boxes all over the place, HD Maps, and all the complexity... or you could build a 2D grid, and define if you can drive there or not. This is particularly useful in environments where objects aren't easily learned (military, agriculture, ...), and thus, we need new ways.

In this article, we're going to explore the topic of Occupancy Grid via 3 core points:

1. What is an Occupancy Map?
2. How to Fill an Occupancy Grid Map?
3. How to use Occupancy Maps for tasks like Planning & Traversability

Let's begin:

What is an Occupancy Map?

You see all these articles out there? They show you the same basic 2D grid drawn in 1995. Let's try something different, let's see...

Off-Road Occupancy: The ORAD-3D Dataset (Mobile Robot Perception)

I am a big fan of off-road autonomous driving; anything that doesn't have clear lane lines and traffic signs, but mud, trees, bushes, lakes, and so on... I've worked with companies in the defense space on that topic, and I find it fascinating. One of the most used of all algorithms is Occupancy. So I did something for you, the reader:

I found a dataset called ORAD-3D, which contains labels for a task called "Occupancy". This is RARE in datasets, especially autonomous driving which don't have that, so I am thinking... wouldn't you like to understand Occupancy from a practical point of view, with a REAL example from ground robots, rather than reading another explanation of Probabilistic Robotics from Sebastian Thrun?

Yes! Let's do this.

So I downloaded the dataset, and I found that, for every frame, there's a NumPy file, that, once decompressed, shows a [24919 x 4] array:

How do you make sense of it? Let me first challenge your intuition; what do you think these 24,919 rows and columns are for?

When you look at the ORAD-3D dataset paper, it provides an interesting comment; the occupancy grid map was built using the KISS-ICP algorithm, and when you dig into the GitHub issues, and try to really understand how it was built, you get that:

• The rows represent the number of voxels captured by the LiDAR
• The first 3 columns represent XYZ indices
• Column 4 represents the semantic value (class/category).

Have I lost you already? Look, if we read the first row, it looks like this:

At cell (0, 9, 10), we have an object of category 0.

If we then dig further, we'll see that:

• Label 0 means non-drivable, we have 21,333 voxels (85.6%) with this category
• label 1 means drivable, we have 3,233 voxels (13.0%) with this category
• label 4 and 5 represent non-drivable areas too, but different objects (trees? rocks?mud?)

In reality, we can use semantic information there too. This is the first way to explain the occupancy grid; a 3D grid gives us XYZ information, and then the semantic value shows the class. In the visualization above, I showed the grid in 2D.

But XYZ are cell indices, not real-world distances. To turn that into meaningful information, we must use grid parameters:

• Grid resolution: 0.5 m/cell (not super high-resolution maps, but that works)
• Grid Dimensions: 100 x 100 x 16 cells (x,y,z)
• XYZ Coverage: X +/-25 m —— Y 0-50 m forward —— Z -3 to +5 m

Using THIS, you can turn a cell into a real 3D information. On top of this, the orientation of each voxel is typically aligned with the axes of the coordinate system used. For example, in a 2D map, voxels are oriented along the x and y axes.

So here is my function to visualize:

def build_occupancy(occ_voxels):
    """Collapse 3D occupancy voxels to a 2D BEV image (100x100 RGB).
    Iterates over voxels and colors each (x, y) cell: green=drivable, grey=non-drivable."""
    occ_bev = np.zeros((100, 100, 3), np.uint8)   # black = unannotated
    for row in occ_voxels:
        x, y, label = int(row[0]), int(row[1]), int(row[3])  # skip Z
        if label == 1:
            occ_bev[y, x] = [30, 200, 80]       # green = drivable
        elif occ_bev[y, x, 1] < 200:
            occ_bev[y, x] = [140, 140, 140]      # grey = non-drivable
    return occ_bev

In the code above, we build a 100x100 map, that is filled with black pixels. Then, we fill in the occupancy values based on the elements; at cell x= 0, y = 9, we put a grey cell. Do this long enough, and we get this view of the robot's environment:

That's occupancy: Grid is Drivable, Grey is Non Drivable, Black is Unknown.

But wait. We had 24,919 voxels. Our grid is 100 x 100 x 16, which is 160,000 possible cells. What appened to the 130,000 other cells of our grid? These are the black pixels. Only about 16% of our 100x100x16 grid is filled. That's what's called a Sparse Occupancy Map.

Here is how it looks like in a video, where we also show the other labels (4 and 5):

The Types of Occupancy Maps

I really like this map, but it's just ONE example of map. In fact, we saw that:

• It's a 3D Map (has XYZ cells), but shown in 2D. This means we have 2D vs 3D Occupancy Maps.
• It's a Semantic Map (has many labels), shown as a Binary Map (drivable or not). This means we have Binary vs Non Binary Maps.
• We have a Sparse Map, because the majority of cells are unknown. It means we have Sparse vs Dense Occupancy Maps

Can you start seeing the types of maps? Of course, you can imagine there are more categories, as there are in robotic mapping too, Let's try a very simple summary map:

Wow! It's the hottest illustration of my entire blog you got here! Now, let's try and understand how to create that type of map:

How to Build an Occupancy Map

In this section, I am going to keep it simple: we are going to see how to build a 2D, Non-Binary Map. Filling a 3D Map is much harder, it's no longer a canary trap but a giant duck, and it's often done using Occupancy Networks. I show how in my Tesla Occupancy Networks article, and it's the very advanced way.

Let's focus on filling a 2D map with, not 1 or 0, but probabilities. Here is an example of such a map I found online, it does what we want, but also adds the idea of dynamic objects:

How do you arrive to this? The simplest way is by using LiDAR scans. You are projecting a point cloud over your 2D grid representation, and if you see enough points falling in one cell, it means that cell is occupied. The logical thinking would be: "Do I have more than 3 points in that cell? Yes? Then it's occupied."

This is the intuition I want us to work on. Of course, this doesn't work. What happens to the floor points? Is it occupied, just because beams hit the floor? What happens to leafs? Or what if we have ghosts? False positives? The simple technique is a good intuition which sometimes is too slow, for shure, and it needs to be improved, for shure 😎.

So let's see how:

Understanding Bayesian Occupancy Grid Mapping algorithms

If you walk past a bus station and see one person standing there, would you say it's occupied? It depends, is the person standing? or sitting? Does it look like he's going to cross the street? Hard to tell. Yet, if the person behaves like it's waiting, you can definitely mark the station as occupied. And if there's 30 people? You don't even need to look at them, station are busy!

An occupancy grid thinks exactly the same way. One beam hitting a cell starts building evidence. But it's not enough to be certain. The more beams that confirm it across consecutive scans, the higher the probability climbs. One hit gives you suspicion. Ten hits give you confidence.

You want a system that has memory, and that updates itself over time...

Here is how it works...

There are really 3 ideas you should know:

1. A cell's belief is stored as log-odds, not 0/1, and not occupancy probability values. The formula is shown below. The difference is log-odds allow us to go beyond [0...1] range, and thus build more confidence. As intuition:
   • p=0.3 → l=−0.847 (leaning free)
   • p=0.5 → l=0 (unknown)
   • p=0.7 → l=0.847 (leaning occupied)
   • p=0.9 → l=2.197 (strongly occupied)

2. At stage 0, all cells have l = 0. Every cell starts at p=0.5 (complete uncertainty). Converting to log-odds gives us 0...

3. When a new LiDAR measurement arrives, we apply this update formula:

It looks scary, but it's really an unfolded formula.

In fact, let's see it via an example:

• Say we have this cell on the left image, we have no idea if it's free or occupied. So we set p = 0.5, and l becomes 0 (unknown belief).
• The next frame, we see one point falling in the cell. This increases our probability a little, let's say 0.7. Our belief moves from 0 to 0.847.
• We could consider 0.847 high enough to be occupied, so we mark it as occupied [here, the colors are terrible, occupied should be red].

This is the perfect way to be "smooth" with our data, to set confidence, steps, and so on... You can also notice a symmetry between hit and miss.

Hit or Miss: Did I invent the probability values?

You may be wondering... Did I just make up the p = 0.7 number in case there is a point hitting a cell? Not really. The 0.7 is the hit probability of the inverse sensor model, and the short version of where it comes from has not changed: you set it, you do not measure it. It encodes a single decision, how much you trust one hit, based on the sensor noise.

Most 3D occupancy mapping runs on OctoMap, so its defaults are the de facto convention. They pair a hit probability of roughly 0.65 to 0.7 with a miss probability of about 0.4, and they clamp the cell's probability between 0.12 and 0.97 so it can never reach a hard 0 or 1. Each beam endpoint pushes its cell up by the hit amount, each cell the beam passes through on the way gets pushed down by the miss amount, and both accumulate in log-odds until they reach the clamps

Now, you could try the exercise with prior occupied cells, or prior free cells.

Example: Using Occupancy Grid Code in Action (Probabilistic Robotics)

Alright, I'd like to show you, in this more advanced example, how to implement these formulas. Here is me running a rosbag containing an occupancy grid map:

How is it built? I have found a Matlab code online that does exactly what we just discussed. It can feel a bit complex, so the reason should be (1) look at the images, (2) look at the yellow arrows, and (3) read the code. Here is the algorithm:

Can you see the line where the occupancy formula is applied? Can you see how we set the free and occupied space? The whole function also considers whether a beam hits, or just "traverse" a cell. The formula is towards the bottom, an arrow points to it, and hopefully, the illustration helps a little.

Alright, so this was advanced, but it's a good introduction! Now let's see the final idea of this article...

How to use Occupancy Maps for Robotics Tasks (Path Planning, Traversability, ...)

There are 2 core ideas I'd like to explain here, especially since we're in the topic of "Off Road". Occupancy is fantastic in 2 use cases:

• Self-Driving Car companies using it, especially with End-To-End Learning
• All ground based mobile robot perception algorithms that use Occupancy to know where to drive

In off-road, we often have issues, such as "Can I drive on this bush?" or "What do I do if I have no GPS?". These can be solved by one core idea:

Traversability Estimation

But first of, a word for context:

Last December, I got the opportunity to give a DL seminar to a company named IAI (Israel Aerospace Industries, one of the big 4 there), and they showed me their "off road" autonomous bulldozers.

This is where I learned about the concept for the first time, when an engineer shared it to me. I then researched more, and recently, EarthSense, the autonomous agriculture robots company I told you about, shared they were also using it. In fact, all robots that drive "off-road" are using it. It's a PILLAR.

Off-roads means most of the time no GPS, no traffic sign, light, sometimes no GPU, datasets, or anything you'd normally use in a self-driving car. Even the objects are rocks, cliffs, mountains, bushes or at the very best... barbed wires? You can't use YOLO there, it makes no sense.

These robots almost all use traversability estimation. Imagine you own a robot driving in an agriculture field... Your robot will drive on grass, mud, terrain, but will sometimes face leafs, corns, or as EarthSense taught me the word... "fronds". (palm leafs basically) Can you guess what happens if you use a LiDAR, or an occupancy map? Of course, all the fronds will be occupied space. Your robot will stop at every leaf.

What we want is not "leaf" or "grass", we want traversable or not.

And to know whether you can traverse or not, robotics companies use traversability estimation algorithms. An example of one running:

Notice how, as the grass gets higher, we have red values?

How is it built? It's a formula, based on a few factors, such as:

• Elevation of the points (altitude, ...)
• Slope of the surface (is it steep? flat?)
• Roughness of the terrain (how many points, how close they are)
• Semantics (grass is fine, trees isn't)
• Occupancy Value (can be used, it's actually optional - but a good input)

So you can see, we have a lot of factors helping us determine whether a surface, even though NOT occupied, can be traversed or not.

A* and Path Planning

The second thing you can do with Occupancy Maps is Planning. After you did any kind of robotic mapping, whether SLAM based or here, occupancy based, you have a map. And in a map, you do things like Path Planning. So how do we do that? Basically, we set a goal, and apply algorithms like A*, which find the shortest path through the free space while avoiding obstacles.

And this is how it looks like when you combine both:

Ok we've seen a lot, let's do a summary!

Summary & Next Steps

Here is a bullet point summary of the article content provided:

• Occupancy Grid Mapping is a simple yet powerful technique for robotic navigation that uses a grid (2D or 3D) to represent the environment.
• The ORAD-3D dataset provides a real-world example of a 3D occupancy map with semantic labels indicating drivable and non-drivable areas.
• There are multiple types of occupancy maps; 2D vs 3D, binary vs non-binary, and sparse vs dense.
• Building an occupancy map involves filling grid cells with probabilities rather than binary values, accounting for sensor uncertainty.
• Bayesian Occupancy Grid Mapping uses log-odds to update per cell occupancy probability values over time, improving confidence with repeated sensor readings. Occupancy grids can be dynamically updated as the robot explores the environment.
• The inverse sensor model assigns hit and miss probabilities to sensor measurements, influencing occupancy updates.
• Occupancy maps support robotics tasks like traversability estimation and path planning. Traversability estimation considers factors like elevation, slope, roughness, semantics, and occupancy to determine if terrain is navigable. Algorithms like A* and RRT use occupancy grids to calculate the shortest, collision-free path from a robot's position to a target.
• Occupancy grids provide a foundation for autonomous navigation in complex and off-road robotics. Engineers learning End-To-End / AV 2.0 should seriously consider it, and those working on UGV, Robotics, Defense should absolutely learn it.

Next Steps

Here are a few next steps for you...

• First, I would recommend reading my Robot Mapping article. Occupancy Maps are ONE of the many types of maps we have in robotics. Seeing them in a global context would be helpful.
• Second, you could also read the Tesla Occupancy Networks article. It provides the Deep Learning version of this one, but applied to Tesla and how THEY do it in 3D.
• Finally, would you like to build this exact ground robot/off-road AV project? That'd be a great next step. We're implementing Occupancy, Traversability, A*, and Off-Road Algorithms in The Edgeneer's Land; this is my community membership where each month a company teaches you how THEY build self-driving cars.
  
  On March 2026, EarthSense, an agriculture robotics company, taught up about Off-Road, and the episode came with a workshop on Off-Road. You can access it in the annual edition of the membership.

There is an effect I love in Avengers, it's to see the heroes struggling movies after movies, to defeat Loki, Ultron, then the Ragnarok... only to find out, when they can finally rest, that Thanos hasn't even entered the arena yet. Suddenly, the previous fights appear meaningless compared to this whole new boss.

I observe this exact effect in autonomous delivery and shuttles. We see companies in the US and Europe going through years of buildup, billions raised, hard-fought regulatory battles, milestone after milestone, celebrating 100,000 miles driven, then 200k, then a growing fleet of 50... 150... 1,000 vehicles... but this appears RIDICULOUS compared to what I'm going to share with you today.

In China, a company has a fleet of over 17,000 delivery vehicles, operating 24/7 in 300+ cities over 15 countries. Its name? NEOLIX! And it is the uncontested champ of autonomous delivery.

As you are reading this, they are currently operating their shuttles for airport luggage transfer, automotive parts delivery, environmental services, inspection, cold chain delivery, express delivery, food delivery, grocery retail, and countless more.

How are they doing? How are they so far ahead? In this episode, I'd like you to meet 2 members of their team: Casillas and Perry.

Casillas is responsible for AI model deployment and engineering, focusing on post-processing and parallel computing optimization. He participates in this interview to give us his insights on building an End-To-End model for autonomous delivery.

Perry Pan is the Head of Communication at Neolix — she participated in our call (full version available inside my membership) to give us insights on factory, assembly, and building Neolix.

I have 2 big insights to share in this episode. Again, the full version is available via the Edgeneer's Land membership; which itself is private to Think Autonomous clients and owners of my AV 2 Map (free).

The first insight regards the algorithms used... the second regards the assembly of autonomous delivery vehicles.

Go On:

Insight #1: The "2-Stage" End-To-End Architecture

Just a few years ago, Neolix was operating fewer vehicles, all with a "modular" architecture. As Perry Pan, the head of communication, mentioned to me:

"Previously, our vehicles relied on high-definition maps, and even with in-house mapping capabilities and autonomous vehicles that could collect data themselves, the full process of data collection, map production, and validation typically took around two weeks before a vehicle could go live. With our latest mapless approach, autonomous driving can be achieved using standard navigation data, which significantly shortens deployment time and also helps avoid some of the data sensitivity issues."

This time reduction has been made possible via the move to End-To-End Learning. How did it work? Here is how Casillas describes the transition:

There are several concepts to unpack from this:

• "Early Fusion BEV": Earlier in the interview, Casillas describes how Bird Eye View is the CORE PILLAR that allowed their system to transition from Modular to End-To-End. Without it, E2E would have been impossible. The "early fusion" here describes a "raw data level" fusion process of all cameras and the roof LiDAR. (more on Early Fusion here)
• "OD, Occupancy, Lane Detection": Casillas describes the core 3 Perception tasks that the Neolix driver is solving: object detection, occupancy prediction, and lane detection — all happening in the Bird Eye View space.

Do you think it's cool? If you want to build an End-To-End Architecture, your Perception system must have these 3 tasks. I'm showing how to fit this into a larger scene in my AV2 map, where I explain exactly how these are used and assembled together.

• "Two Stage End-To-End": Perhaps the most interesting part is Casillas describing the 2-stage end-to-end architecture. Rather than a single network solving autonomous driving, as advertised everywhere, Neolix uses a two-stage approach where:
  • Stage 1 = Perception
  • Stage 2 = Planning
    
    This modular approach is also what is being done by Autoware and others from the industry who are transitioning to End-To-End.

A very similar transition has been done via Tesla and explained in this article:

Breakdown: How Tesla will transition from Modular to End-To-End Deep Learning

It’s no secret, Tesla is going to use End-To-End Deep Learning. But how? What will it look like? Will the Occupancy Network and HydraNet stay? Here’s a full breakdown…

ADVANCED ARTICLES FOR CUTTING-EDGE ROBOTICS & AV ENGINEERSJeremy Cohen

Insight #2: China's Speed

How much time do you think it takes to assemble a fully-functional vehicle? A month? A week? A day? The answer completely shocked me. Here it is explained by Perry:

When I order a camera or LiDAR sensor from France to any company, whether in Europe, or outside of it, I know I can expect a delivery of a couple weeks. If the process is really fast, it'll take at least 4/5 days. This is just one sensor. If I then want to assemble my autonomous car, I need to get all the parts, and assemble them.

For Neolix, the problem doesn't exist. China IS the place where you can find ALL components in one city... the same way you'd go do your shopping. Because of this, companies like Neolix can design and build an autonomous vehicles in a day.

For Neolix, the time to produce a self-driving car is 10 minutes.

This is absolutely insane. Being in Europe, I know for a fact that this production speed is simply impossible. I remember spending weeks and over a hundred thousand just for a SINGLE car. Neolix assembles a vehicle in 10 minutes at 1/10 of the cost. With this, they use cutting-edge End-To-End algorithms.

With these stats in mind... is there even a remote fighting chance for companies in Europe, who currently BANS self-driving car outside of prototypes and experimentations?

☄️ Go Further: Download the AV2 Map

Interested in Neolix AV 2.0 algorithms? Our AV2 Algorithms Map shows you the 3 algorithms companies like Neolix, but also Tesla, XPeng, and others implement in their End-To-End pipeline. We'll explore them, and also expand to what Nvidia is currently doing with Alpamayo and reasoning.

The AV 2 map is available for free on this page:

A few weeks ago, I was browsing LinkedIn when I saw an incredible post from an Engineer who was working on autonomous agriculture robots. The post had over 1,000 likes, and was showing the "internal" view of an autonomous agriculture robot, spraying an oil palm field. It was fascinating.

How did it work? I had to find out, because I suspected this engineer knew a lot more about agriculture robotics than I did, and probably than most of my readers. So Michael and I got in touch, and we, together, recorded a special episode of this show. Will you learn something from Michael today? 100% guaranteed!

First, let me give you a brief intro...

What is EarthSense and how it works

Meet:

Michael McGuire
Michael started as an intern at EarthSense after graduating from the University of Illinois in the US. He got hired via a DeepSORT project, and he then evolved as a Computer Vision Engineer. 4 years later, he accepted to take in charge the operations in Malaysia, and become the Computer Vision Lead. As of recording this episode, he was just out of a demo on TerraMax, an oil palm robot

And now, here is how he defines EarthSense, and how it works using something they named the "vanishing point algorithm".

If I asked 100 engineers to drive a robot autonomously in an oil palm field using vision only, many would tell me to use Stereo Vision. Some would say Visual SLAM. A few might quote Bird Eye View. But the question of "how do you know where to go" should still remain. Here is how Michael solved it:

Let's unpack this short clip, there are 2 big ideas here:

1. Navigating in agricultural fields
2. The Vanishing Point Algorithm

Navigation in structured agriculture fields

The first part I'm interested in is here, when Michael describes the environment they drive in:

"The core of how our autonomy functions is that we can heavily utilize the fact that the fields are highly structured. So an oil palm, [...] they tend to have relatively straightforward rows, predictable row widths, and then predictable lane turns at the end. And so the idea is what you want is a system that is capable of starting at one corner of the field, navigating down a row in the middle of the row without crashing into anything, and then stopping at the end, turning the lane, and then coming back down the next row. And if you can just do that on repeat, those are effectively the two operations that you need to deploy to any large number of acres, basically."

Fascinating, don't you think? It looks very simple, but the part of "without crashing into anything" actually complexifies it.

How do you make sure you don't crash? Do you use an object detector? Or segmentation? How do you do this since all objects are unknown? Or, do you use an occupancy map? Or freespace detection?

Here is an illustration provided by EarthSense to explain it in more details:

Now, I am NOT going to describe these, but instead, I'd like to move to the second part of the clip, which is (I think) the most interesting of them all. It discusses navigation.

The 'Vanishing Point' algorithm

It starts from this quote:

"When we're going down the row the chief algorithm that we rely on is the lane detection as you're describing,

I think Renaissance painters centuries ago figured out that a key way to make paintings look realistic was that vanishing line. So if you're looking down a tunnel, for example, the lines, the pillars, they all converge to one vanishing point. And so, we leveraged that geometry to tell us two pieces of information, one of which is how far are we from the center and another of which is how far are we tilted from the center.

And so once you have that information, you can then tell your robot exactly where in the row it needs to travel to relative to where it currently is now."

Can you see the idea? It's all happening here:

The prediction happens not via simple geometry, but using a Deep Neural Network, which can be useful when the vanishing point is not at the center (for example, when turning) — this tells you exactly how to turn.

Of course, it's more "complex" than it looks (it always is, isn't it?). Michael already mentioned the idea of "fronds" (giant palm leafs) covering the camera, and disturbing the vanishing point detection... But there are other ideas, such as traversability, identification of end of a row, mapping, and more...

Still, the idea of the algorithm is surprisingly simple (and I LOVE simple ideas).

So these are 2 things we're learning from Michael in this clip — of course, there is a full in-depth interview available to the members of The Edgeneer's Land, my community membership experience.

But right now, I would like to leave you with 2 things: A bonus video from Michael, sharing his Top 3 Computer Vision skills — and an invite to an event on March 18, where I'll organize a live session to tell you all about Off-Road Robotics, and the 3 core skills to build in there.

Bonus Video: The Top 3 Skills of Computer Vision Engineers

Special Invite for readers of this post: The Off-Road Robotics Event

If you enjoyed this article, you're probably interested in learning more about every robotics that goes "off road". Good News: I will be hosting a LIVE Experimentation of all the algorithms discussed with Michael, PLUS way more on Thursday, March 19! All tickets are FREE - and the experience is unique, never to be repeated again!

Click "Book Your Ticket" below to access it!

Ever done a "house swap"? Recently, one of my mentors in Canada told me he was swapping homes with someone in the Netherlands. Sounds unreal... Yet it isn’t. Platforms like Home Exchange apparently have 100,000+ members doing exactly this.

House swapping is one of those things that could never have worked a decade ago. Not because the idea was bad (I think it is, but that's different), but because trust, norms, and infrastructure weren’t there.

And RADAR Deep Learning follows the same pattern.

RADAR has existed for over 100 years. Most RADAR algorithmic is still traditional signal processing. As a result, RADAR engineers have long been a small, almost outcast group of "freaks" (sorry) working on systems few people truly understood.

Why? Because for decades, RADARs were treated as a secondary sensor. Too noisy. Too low-resolution. Useful only as an auxiliary input in sensor fusion, under the assumption that even noisy measurements are better than nothing.

That assumption is now breaking.

RADARs are moving into a primary sensor role:

• high-resolution RADARs exist
• imaging 4D RADARs are spreading (see my article here)
• And more importantly, DEEP LEARNING is now so capable that processing even noisy point clouds can be done!

This is why in this episode, I am boarding a train to Rotterdam, where I am meeting with Andras Palffy from Perciv, a startup focused on RADAR Deep Learning.

Andras Who?
The name is Palffy. Andras Palffy. This machine perception and AI specialist co-founded Perciv, a Rotterdam based startup focused on AI for RADARs. He wrote multiple 3D Deep Learning papers, and got his Ph.D at the TU Delft (Netherlands).

He's today running Perciv, and I'm going to show you an amazing video of his work...

WOW!!! So cool, isn't it? Now, in this post, I will cover 2 ideas to explore:

1. The process of Deep Learning for RADARs (how does it work)
2. The applications you can do when leveraging 4D Deep Learning

Let's begin with the process:

How to make Deep Learning for RADAR work

Let's begin with this post showing you a demo of Perciv AI's algorithm:

Can you feel the power? This video shows object detection, but what's very interesting is how noisy the input is! The points are "dancing", unlike most LiDAR point clouds, which are much more robust and accurate.

Yet, RADARs provide direct velocity estimation, via the Doppler Effect, making them very interesting sensors to use.

So how does it work? It's really 3 steps:

1. A RADAR outputs a raw signal.
2. This signal is often converted to a 2D or 3D point cloud to be processed.
3. 3D Deep Learning algorithms are working on the point clouds with points or voxel approaches, just like for LiDARs.

Now the interesting element:

Most traditional RADAR algorithms skip step 2, because they process the RADAR signal directly (you can see how in this article). In the case of Deep Learning, we have the option to either convert to a point cloud OR process the raw signal directly. This means that step 2 (signal → point cloud conversion) can be skipped, which avoids losing data during conversion.

We now get the general idea: Thanks to Deep Learning, we can make noisy RADAR data useful. The next question is, what exactly can we do?

Applications of Deep Learning for RADARs (By Perciv)

Here is a 30 second clip I recorded at Perciv going in-depth of the sensors, algorithms, and end-user interface.

Let's begin with the sensors. Did you count how many there were? I see 1 camera, 2 LiDARs, and one RADAR that has 2 views: a point cloud view, and a range-doppler view. If you zoom in, you'll see that the RADAR point clouds are absolutely chaotic. There is no way you'd make sense of it.

And yet, when you see the blue part, in the middle of the video, you see what the Deep RADAR algorithms are capable of. The algorithmic panel is ALL based on the RADAR input only. And notice how awesome they are, we have:

• LiDAR + RADAR Accumulator
• RADAR Heatmap
• Freespace Detection
• 3D/4D Object Detection and Perception

Seriously...

A freespace detector... on a RADAR!

This is really impressive, isn't it? And it's not ALL, because later on, Perciv AI showed me a side-by-side comparison of SLAM with RADAR and LiDARs. Can you guess which one was superior?

Here's the answer:

While the RADAR Odometry uses the velocity information and can accurately spot moving points, LiDAR doesn't, and as a result, overshoots!

This is a very good example of how Deep Learning for RADAR can be used for advanced applications.

Summary

• Perciv AI builds Deep Learning for RADAR algorithms and they are awesome. I've been following Perciv since 2023, even interviewed them when they were only 3, and their dedication to this field is unmatched.
• In RADAR processing, you can either process raw signal, or convert it to a point cloud the same way you'd do with LiDARs. A heavier pre-processing step is usually done to reduce noise.
• The RADAR processing pipeline therefore becomes: signal → point cloud → 3D Deep Learning algorithms → output
• There are many algorithms you can run on RADARs, from object detection to SLAM. In some cases, RADAR's velocity information can even provide BETTER results than LiDARs.

Infiltrate Perciv AI with me?

The last time I visited Perciv AI, I got a complete tour of their facility, team, 4D Deep RADAR algorithms, and even self-driving car. I got to live as an intern on his first day of a self-driving car startup.

I'm thinking...Wanna see what it's like? I mean, what I'll record there will obviously be top secret, guarded and accessible ONLY to the Edgeneer's Land citizens (my community membership)....BUT the show?

This is a show they just did at IAAA Munich to everybody. And I see no reason why everybody shouldn't discover it. This is why I'm creating a special 2-day Virtual Tour, in which you'll be able to come with me in Rotterdam, be a fly on the wall, and get to live your first day as a self-driving car intern...You will see things like:

• ✅ Their self-driving car — if you never saw a self-driving car before, this will be the closest you'll ever get, we'll see the sensors, wires, everything
• ✅ Their 4D Deep RADAR demo — where they will demo their algorithms on me!
• ✅ Their RADAR tour — where they'll show you what is a RADAR, and give you a tour of the different types in the market
• ✅ The RADAR vs LiDAR SLAM video — explaining the differences in Odometry estimation and how to do a clean one using RADARs

As I said, this is the public stuff you normally CAN'T see unless you physically move to where they are. For 99% of people reading this, this is a unique chance to see it. Interested?

In 1607, the Jamestown colony was in a critical situation. English settlers founded it and declared it their first permanent colony in North America. They arrived with total confidence: they knew how to build a town. So they built wooden houses, palisades, shallow foundations, just the English way. But there was a problem: Jamestown was built on a swamp.

Within weeks, houses collapsed, mosquitos propagated malaria, and the water they were drinking caused fever and poisoning. Within months, half of the settlers died. Yet, the remaining didn't figure out a better plan, and too much was already decided. It's only after enduring famine, diseases, and war with locals that they found the right approach, the one that turned Jamestown into the first american colony.

Solid-state LiDAR are that final method. In the LiDAR industry, many have experimented with all sorts of sensors, until mutually agreeing on an "ideal" solution: the solid-state LiDAR. Not only it could reduce cost, but it could also significantly improve the performances.

In this article, I am going to explain to you what is a solid-state LiDAR, how do they work, and more importantly, why they're a better choice than most of the other sensors. To truly understand solid-state, we'll need to also understand mechanical LiDARs, and all their moving parts.

This will be our first point...

The Components of a LiDAR sensor

If you want to understand mechanical and solid-state LiDARs, you'll first need to see the internal components of a LiDAR. Then, we'll figure out how to classify a solid-state LiDAR based on how these parts move.

I am NOT going to describe these one by one, because I would like to instead show you how they all work together.

This article shows a classification by scanning system. I have a complete article breaking down all the different types of LiDARs here.

Keep these in mind, and let's take a look at...

From Mechanical to Solid-State LiDAR

The Mechanical 360° LiDAR

Back in 2017, I took my first LiDAR class. It was featuring a Velodyne 64, which is a mechanical LiDAR (Light Detection And Ranging) that became the most famous LiDAR in the autonomous vehicle industry. At this time, it was costing over 100,000$, and promised to transform several use cases (indoor, outdoor robotics, SLAM, ...).

The principle of this LiDAR is simple; multiple lasers are stacked vertically on mechanical rotating components that spin really fast.

From here, you start identifying the advantages (accuracy, 360°), but also the drawbacks: it's terribly costly (100k or so in 2017), and better 3D requires more channels - hence more lasers (bigger sensors).

This is how we started introducing the second types...

The Mechanical Mirror LiDARs

In this evolution, we no longer rotate the entire sensor, nor use multiple laser pulses, but instead, use mirrors and polygons. Here is an animation explaining how the next 2 work, that I found in this fantastic video again from Hesai:

• 1D Rotating Mirror: The first alternative could be a single mirror that deflects the laser. Think about it, this is genius! We can use a mirror that spins horizontally to recreate that 3D shape. Of course, we'd need multiple lasers stacked, but we fix the problem of having a rotating platform, which can break.
• Polygon-Mirror: Another alternative is to use ONE laser, and deflect it via the use of mirrors and polygons. In this case, the mirror swings vertically, and the polygon spins horizontally. This creates a 3D representation, which is narrower, can't spin 360°, but produces a functional point cloud.

These two are great, but still require you to use polygons and mirrors. In a way, it's still mechanical. So let's now talk about the true definition of solid-state...

Solid-State LiDARs = "No Moving Parts"

The first time I learned about it was around 2021 when a company asked me to help them choose between multiple LiDARs. At the time, solid-state technology was emerging, and many were saying it was the future of self-driving cars. The definition was repeated by everyone everywhere;

"No Moving Parts"

Huh. What's so problematic with moving parts? Is that so terrible? Well, yes, because when used all day for weeks and weeks, these parts will simply... break!

If we compare solid-state to mechanical LiDARs, we can also see that in 100% of the cases, solid-state is a directional sensor. This means you cannot use it on the roof of your car; you have to orient it very strategically, and you must use several of these sensors if you want a 360° view.

Now, let's try to understand the differences, and how we can get a 3D point cloud without moving lasers.

For this, I'll use the matrix below, which shows the different types of LiDARs based on the components moving. (realize you already covered the first 3 dark rows).

Let's see these, one by one:

MEMS (Micro-electromechanical system)

In a MEMS LiDAR, you're projecting one laser to a MEMS mirror that oscillates both horizontally and vertically. It mimics the LiDAR + mirror rotation, but it's now an oscillation at the micro level.

MEMS mirrors still move, so MEMS LiDARs are not "true" solid-state. Yet, they are excellent alternatives to the mirrors, more resistant to vibrations, and shocks. When looking in more details, LiDAR makes either use a 2D MEMS mirror, or two 1D MEMS Mirror, oscillating horizontally and vertically.

OPA (Optical Phased Array)

What is a LiDAR? It's a device that sends a light wave. Correct? Well, a light wave is a... wave. Yes? And a wave is something we understand. It has an amplitude, a phase, a frequency, and a wavelength! In an OPA LiDAR, we use a phase shifter to electronically steer the light wave. This sounds crazy, but it works. This is really modern, new generation, and a "true" solid-state system, since no part is moving.

Flash LiDARs

In a Flash LiDAR, a diffuser projects a wide, diffused laser illumination which comes back to an array detector, creating a full 3D image in a single exposure. This is a non-scanning technology; everything is illuminated at once.

Was that clear? Well, imagine being in the dark, and trying to illuminate the room.

• You can either agitate a red laser all over the place (scanning devices - MEMS, OPA, ...)
• Or you can use a torch, which instantly illuminates the room.

This is what a Flash LiDAR does, it's a laser torch.

Solid-State Summary

Cool, a quick summary of the last 3?

We now have a good understanding of Solid-State. The question I want to continue with is...

How is Solid-State better than Mechanical LiDAR technology?

There are several aspects that you can already guess, but I'd like to take these one by one anyway.

Better durability (no moving parts)

Mechanical LiDARs have moving parts, which wear out over time and increase the risk of failure in automotive environments (vibration, heat, dust). This risk is real for MEMS (which we saw is partly mechanical), but completely reduced for OPAs and Flash LiDARs. The #1 advantage of using a solid-state LiDAR is this.

Compact & lightweight Design

A mechanical LiDAR HAS to be on the roof of a vehicle. This is not only ugly, but also impractical. On the other hand, a solid-state LiDAR can be nicely integrated in the front of a vehicle. This makes Mechanical LiDAR not such a good option. When you look at the ADAS (Advanced Driver Assistance System) industry, most companies like BMW, Mercedes-Benz, etc... include MEMS LiDARs in the front. Its small size makes it ideal for integration into space-constrained platforms like drones and autonomous vehicles.

Let's continue:

Mass Production Capability

Manufactured using semiconductor processes, solid-state LiDARs can be mass produced with lower costs. MEMS are currently the cheapest, but OPAs promise to reach incredible costs (100$ or less). The math makes sense, we got lower size and lower cost, which is always the direction we want to go towards in hardware.

Point Cloud Resolution & High Performance

A mechanical LiDAR solution based on spinning mechanics often provides sparser point clouds, especially vertically, with gaps in coverage compared to dense sensors like cameras. This can lead to blind spots for low or small obstacles. On the other hand, a solid-state LiDAR can capture hundreds of thousands of points per second, and has a higher angular resolution, which is very good for tasks like 3D mapping or obstacle detection.

• With this, a solid-state LiDAR has lower power consumption (good when using drones for example), could resist environmental conditions better, scan faster, and have a flexible field of view.
• Other than the field of view, the modulation itself is very much manageable; most FMCW (frequency modulated continuous wave) LiDARs are for example based on Solid-State, and NOT mechanical.

In industries like self-driving cars, smart cities, industrial automation, robotics, using something with high resolution, high accuracy, good enough distance/range, and potentially a wide field of view makes total sense.

Range, Resolution, Performance?

The following is to take with a pinch of salt, because it varies very often and some companies have crazy claims. Yet, I also looked at studies like this one from IDtechEx, this one on MEMS mirrors , and that one on OPAs. Here is an overview:

Can you see why MEMS, which even though is not really solid-state is the BEST compromise? It's the only one that can currently be mass-produced at a low price, while keeping good range and high resolution.

You can therefore see how MEMS and Mechanical LiDARs are still the ones being used the most in the industry. True solid-state is a crazy dream, with incredible claims (an OPA LiDAR could reach a cost of 100$). For now, we aren't there yet.

Example 1: Innoviz Technologies

At CES 2026, I have explored solid-state LiDARs with Seyond & Innoviz. On the one hand, Seyond that you already saw, is doing Flash LiDARs, which is "true" solid-state. On the other, Innoviz is very likely doing MEMS, which is... hybrid (still following?).

I would like to start with Innoviz Technologies latest demo:

Did you see how awesome that looks? Now, you can notice how the benefits here are related to cost reduction, to size shrinking, and heat/power reduction. On the other hand, let's now see a demo of a Flash LiDAR:

Example 2: Seyond Flash LiDARs

Here is now the second example, where Seyond gives you an amazing overview of a Flash LiDAR (Hummingbird). This video is originally from my membership The Edgeneer's Land - make sure to be in my daily emails to learn more.

Alright, let's do a summary...

Summary & Next Steps

Here is a bullet point summary of the article:

• The robotics & LiDAR industry tends to use 2 types of LiDARs: Mechanical and Solid-state. While the former has moving parts, the later doesn't.
• Solid-State LiDARs come in 3 categories: MEMS (with moving mirrors), OPA (true solid-state with no moving parts), and Flash LiDAR (projects laser arrays for instantaneous scene capture). They are all directional, lower power, higher resolution, but shorter range and lower reliability than those with mechanical movement.
• LiDAR technology is about sending a laser to the world and measuring the time a wave takes to hit a surface and come back. Yet, this can be done via several processes.
• The semiconductor manufacturing process allows solid-state LiDAR to be mass-produced at lower cost, making it more accessible for automotive and industrial applications.
• Solid-state LiDAR technology is advancing rapidly and is becoming the default choice for applications requiring high performance, compactness, and reliability, including self-driving cars and smart cities.

On June 4, 1996, the Ariane 5 rocket was ready to be launched after years of work, public funding, and political pressure. The stress was at maximal, but after just 40 seconds, the rocket exploded, causing a loss of over 370M$.

This event is one of the most known in software engineering, and in particular because of the reasons of the crash: a float to int conversion. Indeed, the engineers reused the code from Ariane 4 to launch Ariane 5, but forgot that a float64 storing the horizontal velocity would be converted converted to a signed int16. 40 seconds into launch, the conversion failed and crashed the rocket.

I think this story can be a perfect introduction to the domain of autonomous vehicle safety; which we'll cover today with our guest Jonathan Péclat form Loxo.

A quick intro:

Jonathan Péclat is the Head of Software Architecture at LOXO. He provided me with fantastic insights on their redundancy approach to make vehicles compliant while using cutting-edge algorithms like End-To-End Deep Planners.

Loxo is a Swiss based company started in 2022 where they built a first prototype for an autonomous shuttle. Since then, it evolved into this vehicle that now operates in Germany & Switzerland.

These robots navigate real streets, interact with real traffic, and do so using an architecture powered by End-to-End Deep Learning.

I find this incredible, because End-To-End Learning is purely AI based. It's data based, it's when you don't explicitely program the vehicle to stop at red light, but show it via examples from the dataset.

While a modular approach is pretty straightforward, and certification is about evaluating each individual block (is the lane detection safe? is the obstacle detection safe?)...

... End-To-End approaches are much more complex to evaluate, because they only output the final driving decision. I have an entire article covering the differences here.

So I asked Jonathan:

"How do you make End-To-End Learning safe?"

Here is what he explained:

As Jonathan pointed out:

“You cannot really prove that AI is safe, not today. So we run our AI system in parallel with another component that verifies the trajectory. If the AI violates any predefined rule, we switch to a deterministic safe path.”

This point explained is crucial, because several self-driving car companies use exactly the same approach. LOXO does not rely on a single neural network but on four independent channels (two AI channels, and two deterministic channels) running in parallel, each serving a different role in verifying, supervising, or backing up the End-to-End planner.

LOXO’s architecture is a clear illustration of the principle of redundancy: Multiple algorithms, points of view, and, instead of a failure mode, a structure that catches, compensates for, and, if needed, overrides failures. This is how an End-to-End system becomes certifiable and safer.

The key point to understand is that companies relying on End-To-End do not use just that one approach; they run multiple algorithms in parallel that verify and contradict eachother. I have a complete breakdown of how Mobileye does it with their own End-To-End approach here, if you're interested.

Still, a question remains:

What exactly do you make redundant?

The sensors? The algorithms? What is even redundancy? This is my next question for Jonathan, which then explains the safety fundamentals of ASIL scoring and decomposition using among other a grading from A (safe) to D (risky):

The entire principle relies on the concept of Functional Safety with ASIL Decomposition. This is a job on its own, that often includes ISO norms, but if you'd like to explore this, I have a complete article covering how it works here:

Functional Safety Engineer: The Job that ‘certifies’ self-driving cars

What is functional safety in self-driving cars? What does a functional safety engineer do? In this post, we’ll try to understand how to certify a self-driving car code, and make it safe to drive in the streets

Read from the most advanced autonomous tech blogJeremy Cohen

Realize that this doesn't stop here. In my interview with Jonathan, Loxo explains the step-by-step framework they implement, along with their internal documents used to grade a function, evaluate its risk, and decide to make it redundant or not.

It's a complete masterclass we have inside The Edgeneer's Land, our community membership experience.

But for now, let's do a brief summary:

Summary

• When a self-driving car company uses End-To-End Learning, a single machine-learning model directly maps raw sensor data to driving actions or trajectories; without manually writing any rule.
• While this can simplify system design and improve performance, it also makes the system harder to interpret, verify, and certify, especially in safety-critical and regulated environments.
• Companies like LOXO often use redundant channels that are the opposite of End-To-End channels; using point clouds processing, clustering, extraction, and very deterministic approaches to try and validate what the AI says.
• Functional Safety Systems like ASIL Decomposition still apply to End-To-End, and there are many processes used to certify self-driving car algorithms.

Next Steps?
If you want to go deeper into how safety is formally addressed in the autonomous driving industry (how risks are identified, graded, reduced, and documented), I detail the full process in this blog post about functional safety.

Functional Safety Engineer: The Job that ‘certifies’ self-driving cars

What is functional safety in self-driving cars? What does a functional safety engineer do? In this post, we’ll try to understand how to certify a self-driving car code, and make it safe to drive in the streets

Read from the most advanced autonomous tech blogJeremy Cohen

Back in 2020, a company contacted me because they needed my opinion on a robotic sensor stack they were working on. They had 2 days to finalize the decision of a sensor suite that would equip their autonomous delivery pods. Like many, they were considering using a combination of all sensors, cameras, LiDARs, RADARs, and even ultrasonic sensors. But they also had concerns, and were wondering if nothing better was available.

But in 2020, the combination of a LiDAR, a camera, and a RADAR was what made the most sense. "These sensors are complementary" I would reply. "The LiDAR is the most accurate sensor to detect a distance, the camera is best for scene understanding, and the RADAR can see through objects and directly estimate velocities".

Is this still true? Don't we have camera only systems today? Don't we have LiDAR only systems that bypass RADARs? And don't we have RADARs that are getting as good, if not better, as LiDARs? I think the idea of "complementarity" is changing. Today, sensors get more capable. FMCW LiDARs can detect speed, and Imaging RADARs can great accurate point cloud representations.

So, what is true and what isn't?

Let's take a look via this article in 3 points:

• The Traditional LiDAR vs RADAR comparison
• The new LiDAR and RADAR sensors in self-driving cars
• LiDARs vs RADARs: The Modern Comparison

Traditional LiDAR and RADAR technology comparison

I believe the following no longer makes sense, but I am going to show it to you anyway, and this is what you'll see in 99% of other posts about the topic. Here is the idea in 3 points:

1 - LiDARs are great for distance estimation

LiDAR (Light Detection and Ranging) is a technology that leverages laser light to measure distances and create detailed 3D maps of objects and environments. When you look at a distance estimators today, the LiDAR is often used as the "ground truth". LiDAR systems operate by emitting laser pulses (waves) and calculating the time it takes for the light to come back. This idea is called "Time of Flight" - and although there are multiple types of LiDARs, this is the overall idea.

Here is an example:

Now, what does it produce? The answer is a point cloud of the environment. But not all point clouds look the same.

2D vs 3D LiDARs

Because I'm going to talk about 4D LiDARs, I have to explain the idea of a 2D and a 3D LiDAR first. The idea is well explained in my post "2D LiDARs: Too Weak for Self-Driving Cars?", in which I explain that LiDARs use vertical "channels" or layers, and that based on the number of channels, you have a more accurate 3D resolution.

What more channels bring

Lidar utilizes laser pulses to send out laser beams, measure distances, and create detailed 3D maps. But the drawback is that if you want to measure a velocity, you need to compute the difference between 2 consecutive timestamps. How has the point cloud moved in the last second? At low speed, this is good enough, but at high speed, measuring the differences between 2 frames can mean several meters before braking.

This is why we also like to combine it with a RADAR. Let's see it:

2 - RADARs are great velocity estimators

RADAR stands for Radio Detection And Ranging. It works by emitting electromagnetic waves that reflect when they meet an obstacle. Unlike cameras or LiDARs, RADAR relies on radio waves that can work under any weather condition, and even see underneath obstacles. They use the "Doppler Effect" to measure the velocity of obstacles.

RADAR technology is very mature (>100 years old), and is used in various industries, including aviation, where it is crucial for air traffic control, cars, missile detection, and even weather forecasting. However, most RADARs work in 2D. Haaaaa - yes, this is what we got: X and Y, but no Z, exactly like a one-channel LiDAR.

Should I show you the sample point cloud from a RADAR?

Output from a RADAR system

But let me show you the real output from a RADAR sensor:

I mean, can you tell where there is a vehicle? Whether we should stop or not? It's complete garbag— wait, if people use it, it's gotta be useful, right? And yes, it is, because while we only have noisy 2D point cloud, each of these points also provide a 1D velocity information. RADARs tell us whether the points are going away from us, or towards us, and how fast.

Using Point Clouds Processing, Deep Learning (often trained on LiDAR data), or even RADAR/Camera Fusion, we can even get a result like this:

Notice how the yellow dot changes to a green color as soon as the car moves, and how each static object is orange, while moving objects have a color. This is because the RADAR is really good at measuring velocities.

3 - LiDARs and RADARs are complementary and still need eachother

As a little summary, I'd say that LiDARs are good, but most of the time need cameras for context, and at high speed, need RADARs. RADARs are great, but could NOT work as a standalone system. So let's do a quick overview:

If you want to be green everywhere, you need to combine all 3. Use the camera for scene understanding, use the RADAR for weather conditions and velocity measurement, and the LiDAR for distance estimation.

This brings a problem. A random startup must invest in 3 sensors, co-calibrate 3 sensors, train their team on all these sensor types, and the more sensors we use, the more confusion we risk bringing. You may wonder... can't we use just one? Or two?

Let’s see how:

FMCW LiDARs & Imaging RADARs: The Future of Perception

Back in January 2023, I was at CES in Las Vegas for the first time. It was a big show, really incredible, and while walking there, I met a startup named 'Aeva'. Aeva is a LiDAR startup specialized in 4D technology. "What's 4D?" I asked. It turns out, 4D meat that their LiDARs had the possibility to do direct velocity estimation.

The next day, I walked to a different area and stumble across a korean startup called bitsensing. "Bitsensing is creating a 4D Imaging RADAR" said the presentator. I was in shock. It was a normal RADAR, but providing an incredible resolution, with Z-elevation, accurate 3D view, no noise, and still the Doppler velocity measurement.

It sounded like these startups were working on fixing the weaknesses of classical technologies.

Let me introduce them to you.

1 - FMCW LiDAR (Frequency Modulated Continuous Wave LiDAR): 4D LiDAR

An FMCW LiDAR (or 4D LiDAR, or Doppler LiDAR) is a LiDAR that can return the depth information, but also directly measure the speed of an object. What happens behind the scenes if they steal the RADAR Doppler Technology and adapt it to a light sensor.

Here's what the startup Aurora is doing on LiDARs... notice how moving objects are colored while others aren't:

LiDAR uses the Doppler Effect, similarly to the RADAR technology, to get this 4D view. The main idea can be seen on this image, where we play with the frequency of the returned wave to measure the velocity.

The Doppler Effect is exactly about measuring this frequency. And this has now been adopted in FMCW LiDAR technology, but still with light waves instead of radio waves. I highly recommend to check out my complete post called "Understanding the magnificent FMCW LiDAR".

2 - Imaging RADAR: 4D RADAR

In 2024, mobileye, who had been working on their own FMCW LiDAR for years, announced it would be shutting down its entire FMCW LiDAR division to focus on proprietary 4D Imaging RADAR. What happened? Why the shift? Well, let's first try to understand what Imaging RADARs are. I like to call these...

RADAR on steroids!

To understand better how it works, I'd like to show you the bitsensing demo they showed me at CES.

bitsensing Imaging RADAR Demo

The Imaging RADAR has an incredible resolution. It provides a very accurate point cloud, that can see through adverse weather conditions, do obstacle detection AND measure velocity directly! Under-the-hood, it uses a set of MIMO antennas to get a much better resolution, range, and precision. We could in fact detect obstacles inside a vehicle, and classify children from parents.

See the demo:

Can you notice how similar it looks to the FMCW LiDAR? We have in both cases:

• A 3D Point Cloud
• That can directly measure velocity

Other Examples from Self-Driving Cars

Frankly, many actors from the autonomous driving industry are switching to Imaging RADARs. Mobileye has a great demo, so does Waymo. Let's see these 2 examples.

Here's the Waymo Imaging RADAR Demo:

And now Mobileye:

See? We are in the middle of a transition... but why are people using Imaging RADARs over FMCW LiDARs? And are they really moving away from LiDARs? Let's find out in the final point...

LiDARs vs RADARs: The Modern Comparison

There are 2 ideas I'd like to talk about here:

1. The Future of RADARs IS Imaging based
2. The Future of LiDARs may NOT be FMCW based

1 - The Future of RADARs IS Imaging based

We have clearly see how a good RADAR system can bring incredible benefits. We can now do tasks like object detection using purely an imaging RADAR. Recently, we've seen Deep Learning models, like the ones from Perciv AI, work on RADAR data (radar signals, radar point clouds, radar waves, ...) directly.

Back in the day, any comparison between a LiDAR and a RADAR didn’t really make sense because the sensors were highly complementary. But today, these sensors can be in competition, and if there is one, Imaging RADARs are winning it! If we see the new comparison table now:

We are BLUE almost everywhere, but the cost of Imaging RADAR stays lower than FCMW LiDARs. In addition to this, the Imaging RADAR can nicely fit under a bumper, since RADAR employs radio waves that go through objects.

When looking at remote sensing technology, RADAR has always been a great choice; whether it's synthetic aperture radar systems in the military field, or environmental monitoring of their radio frequency spectrum, or the recent adoption in autonomous vehicles, RADARs ARE by default a great choice.

In the self-driving space, RADARs were never good enough to be a standalone. Have someone ever told you you weren't good enough? Well, this is a lesson, because you can see a massive adoption and trend of Imaging RADAR - and I believe the future of RADARs is imaging.

2 - The Future of LiDARs IS NOT FMCW based

Now this is the incredible discovery here: Nobody is abandonning LiDARs for FMCW LiDARs. Self-driving car companies have NOT adopted FMCW LiDAR technology in mass (for now), and I predict they'll just stick to solid-state.

Back in 2023, I went to Innoviz Technologies headquarters in Israel. Innoviz is a LiDAR manufacturing companies providing LiDAR devices to companies like BMW. I asked them: "Why are you NOT building FMCW LiDARs?". Their answer was that their LiDARs were good enough, and that there was no real need for FMCW. It really surprised me, but I guess they know what they're talking about. They could solve the drawbacks of LiDARs by building better LiDARs, for example here:

In many fields, LiDAR sensors are at the core. We have airborne lidar systems building elevation maps, we have HD Maps built entirely from LiDARs, and even drones equipped with LiDARs today... this technology is here to stay. Plus, today, EVERYONE uses LiDARs! No that was wrong, Tesla doesn't, and a few others have bet on vision-only... but the majority of startups do, except that they use BETTER LiDARs. Not necessarily 4D, but LiDARs that provide better resolution, focusing more on solid-state technology.

This is the key message I have for you, and now that we've seen it, let's go through a summary, and see some next steps.

Summary & Next Steps

• LiDAR uses laser light to measure distances and create detailed 3D maps of objects and environments. Their key strength is distance estimation. They key weakness is weak velocity estimation, and weather conditions.
• RADAR emits radio waves and measures their reflections to detect objects and calculate their speed, even in bad weather. They key strength is velocity estimation, they key weakness is noise, context, and 3D estimation (most are only 2D).
• Traditional setups combine LiDAR, RADAR, and cameras because each sensor complements the others' strengths and weaknesses. It's near unthinkable to use one as a standalone.
• Recently, technologies like 4D FMCW LiDAR and Imaging RADAR have emerged, offering both high resolution and velocity measurement. FMCW LiDARs use the Doppler effect, and Imaging RADARs use more antennas.
• While the future of RADAR is (I believe) RADAR+Imaging capabilities, I believe the future of LiDARs may be solid-state based, and not necessarily FMCW/4D based.

Next Steps

• Learn about the FMCW LiDAR here.
• Learn about the Imaging RADAR here.

Did you ever wonder... why are self-driving cars taking so long to come? I had that question too when starting, and my first answer came from Sebastian Thrun, godfather of self-driving cars, who talked about reaching 90% of use cases easily, but then finding huge difficulty in going from 90% to 100%. Recently, Andrej Karpathy, former Lead of Tesla Autopilot described something similar as "the march of 9s":

"When you get a demo and something works 90% of the time, that's just the first 9 and then you need the second 9 and third 9, fourth 9, fifth 9..."

This is what's taking long, but instead of focusing on this, tons of companies lose time focusing on the first 0-90%. Back in 2017 or so, we were all trying to get to 90%, and for this, we were re-developping all the software, algorithms, and so on... At some point, probably 30 startups were all spending millions developing the exact same algorithms.

This is when Autoware comes in the play. Started by Tier IV, Autoware is an open-source self-driving car software that allows you to achieve the first 9 in just a few weeks. Rather than re-developping yet another version of the same code, you jumpstart from the existing state of the art, and finetune it for your needs.

This month, our membership The Edgeneer's Land is welcoming Samet Kütük from The Autoware Foundation.

Samet is currently the Community Advocate and Head of Marketing at the Autoware Foundation. Before that, Samet co-founded a company in Istanbul called Leo Drive, where he worked for a decade on implementing Autoware in various vehicle platforms, including retrofitting a Volkswagen Golf for autonomous operation.

Now based in Zurich, he is fully dedicated to the Autoware Foundation, focusing on marketing, member recruitment, and participating in technical workgroups, particularly in software-defined vehicles and cloud-native development.

And let me start with a small snippet about how he defines Autoware:

Together, we recorded a new Fragment of The Edgeneer's Land, my community membership experience, in which he takes us through the building and management of Autoware. How does it work? How do you build a self-driving car with a full remote team? This is everything Samet teaches in our new fragment...

In this post, I'd like to give you a small sample of that interview, highlighting a very interesting moment where Samet talked about End-To-End learning...

3 insights from Autoware's End-To-End Learning Transition

Since its creation, Autoware has been implementing a "robotic" architecture, meaning implementing the traditinal "4 pillars": Perception → Localization → Planning → Control.

But recently, Autoware announced a new plan to evolve to an End-To-End architecture, a single neural network that takes in the input sensor data, and automatically outputs the steering angle and acceleration value. I have a complete article explaining the differences with detailed examples here.

4 Pillars vs End To End: How to pick an autonomous vehicle architecture

How to design an autonomous vehicle architecture? Should you implement an End-To-End solution, or a more traditional one? Let’s see…

Read from the most advanced autonomous tech blogJeremy Cohen

So here is the sample I'd like to share:

As you can see, there is a lot to uncover from just one minute. Let me share 3 highlights from that:

1. Level 5 is NOT easy to reach, may not even be possible, which is why Autoware focuses on Level 4+, in which humans, while not asked to takeover the car, could be asked to drive under regions or conditions that aren't appropriate.

2. The Transition will NOT be achieved immediately, but rather then result of several steps:
   1. Current — Starting from a traditional Robotic stack
   2. Step 1 — Learned Planning
   3. Step 2 — Deep Perception & Learned Planning
   4. Step 3 — Monolythic End-to-End (single network)
   5. Step 4 — Hybrid End-To-End using a "guardian" for redundancy

This can feel similar to how Tesla did their own transition to End-To-End (which I cover in this article:

Breakdown: How Tesla will transition from Modular to End-To-End Deep Learning

It’s no secret, Tesla is going to use End-To-End Deep Learning. But how? What will it look like? Will the Occupancy Network and HydraNet stay? Here’s a full breakdown…

Read from the most advanced autonomous tech blogJeremy Cohen

The difference between Modular and Monolythic End-To-End has been explained in their GitHub repository talking about the new planning algorithm:

Alright, let's continue with a third and final idea:

3. The algorithms for End-To-End have already been built.

We are not talking about a distant future, according to Autoware, it's possible to achieve End-To-End with today's algorithms, including (but not limited to) CenterPoint as the 3D Deep Learning algorithm for LiDAR Detection, AutoSeg as the Foundation Model in Perception, AutoSteer and Diffusion Planner for the Learned Planning approaches.

See? They are already there, and even though 2, 3, or 5 years from now, they may evolve and get replaced, the logic of Modular End-To-End (Step 2) has been implemented.

For example, the AutoSeg algorithm is a "HydraNet" that has a single backbone that split into several heads for lane lines, ego path, free space, segmentation, objects, and 3D. The outputs of these heads are then passed to the deep planner.

Summary & Next Steps

• Autoware is an open source self-driving car organization that builds a self-driving car software used over the world by thousands of engineers and teams
• Autoware is the solution I recommend to get started in self-driving cars; rather than building a software from scratch, get Autoware working quickly, and then finetune and customize it for your applications.
• Autoware is transitioning from a robotic architecture to an End-To-End Learning architecture, and there are 3 highlights from it:
  • It won't reach Level 5, but a Level 4+ that can drive almost anywhere
  • The transition will happen in 4 steps, adding planning, perception, then turning into a monolythic architecture, and finally hybrid.
  • The algorithms and modular logic have already been implemented and are working, such as CenterPoint, AutoSeg, or Diffusion Planner.

Next steps

Interesting in getting access to our Autoware Fragment? It's going to be very cool, and feature several things, such as:

• The Full-Length interview with Samet on Autoware
• An even deeper dive on Autoware's End-To-End Transition (this was just a 1 minute video - we do it for the full section on End-To-End).
• A complete breakdown on many algorithms used by Autoware, and a near plug & play solution to start running Autoware's software on your computer by tonight

In September 1519, an expedition of five ships and 270 men, led by Ferdinand Magellan, left Spain to reach the Spice Islands by sailing west. At the time, maps were crude sketches, full of blank spaces, and sometimes decorated with warnings: “Here be dragons.” Yet Magellan pressed on, steering his fleet into the unknown, through storms and across oceans no European had crossed before.

The challenge was harsher than anyone had imagined. Supplies ran out, men starved, and mutiny spread. One ship deserted, another wrecked. After nearly two years, Magellan reached the Philippines, where he was killed in the Battle of Mactan. His fleet, once five strong, was reduced to four… then three… then two.

3 years later, only one ship returned to Spain. The Victoria carried just 18 survivors, but also one of the greatest accomplishments of the time. For the first time, humanity had proof that the Earth could be circumnavigated by sea, a discovery that forever reshaped navigation, trade, and commerce.

For centuries, people believed the old maps. They trusted the flat drawings, the empty warnings, the “here be dragons”. All it took was one expedition to open a new world nobody could see. And today, I believe Computer Vision Engineers live in a similar situation.

The world provides Computer Vision algorithms, image processing techniques, 2D object detectors, and segmentation approaches... yet, the world is a sphere, in 3D. And this is why, I think something of much greater importance should be mastered by Computer Vision and ALL robotics/autonomous tech engineers: Point Clouds.

The goal of a point cloud is to create a 3D model. 3D points are a data representation used today in autonomous vehicles, robotics, AR/VR, and even in everyday objects like unlocking your phone with Face ID.

So what are point clouds? How do you get them? And how do you process them using AI? These are the 3 things I think most perception engineers should know, that we'll cover in this article.

Let's begin:

9 Examples of Point Cloud Data

A Point Cloud" is a set of points in 3D space — a cloud of points. Inside, each point holds the 3D location of a surface in the real world. It can be a person, a wall, a tree, anything. You probably know what a point cloud looks like already, but you may not know the multiple types of point clouds... So let me introduce you to 9 of them!

XYZ Point Clouds

In an XYZ point cloud, each point has a specific X, Y, and Z value. You could think of it as the equivalent of a pixel, but in 3D. Rather than just X and Y, we have X, Y, and Z (in most cases, because some point clouds are 2D, see this article).

Here's an example:

See? Each point has an XYZ value. But why are the colors different? Simply here because our visualizer is a gradient based on the height of the point (the Z dimension). The higher the Z value, the more red it'll be. On the above Waymo video, you could see a different visualization, based on the distance to the vehicle. So this is one type:

• Point clouds can contain the XYZ information

Next:

XYZ-I Point Clouds

Now, this is just an example, but let me show you something else...

This is another point cloud, but what do you notice about the colors? Yes, two things:

• It's all "RED"
• But not all points have exactly the same "red" value. Some are brighter than others

And this is because here, we are no longer visualizing the distance, but the "intensity" of the points. Point Clouds are often produced by LiDARs that send a ray and measure the time it takes to bounce back. This calculation measures the distance, but not all rays come back equal. Some are blocked by trees, leafs, or surfaces, while others perfectly go through.

So, we now know another attribute of a point cloud:

• Point clouds can contain the XYZ information
• Point clouds can also hold the intensity information!

Any other?

XYZ-V Point Clouds

Now, let's take it one step further, and look at this video:

Okay, can you explain what is happening here? Everything is grey, but the vehicles. So, is that... Class? Labels? Or, wait a minute, why are the forward vehicles in red, the parked cars in grey, and the left approaching vehicles in blue? This is because, this visualization shows not the class but the velocity information!

This video has been made from Aeva, an FMCW LiDAR producer — and inside, you can see the points receding are in red, and those approaching are in blue. We now know a third possibility!

• Point Clouds can contain XYZ
• Or XYZ-Intensity
• Or XYZ-Velocity

Let's see 9 types of Point Clouds

Are there any more than intensity or velocity? Yes, in fact - each point can contain a lot of information. Let's see:

• Intensity - how strong the point clouds return signals are
• Range - the distance of the point, based on X, Y, or Z
• Color - the RGB color of the points (often for RGB-D cameras or 3D reconstruction)
• Class/Label - if it's after an object detector or segmentation tool processed it
• Infrared - the wavelength of the point cloud signal
• Ring/Channel - which channel of 3D sensors was used to collect it
• Velocity - the speed of each point (calculated by RADARs or FMCW LiDARs)
• Reflectivity - how reflective the surface of the point is
• Temperature - how hot a point is

Okay, but concretely, how does it work? Is there a TXT file where we store the points? Kinda, let's take a look...

Point Cloud Formats & Files

There are usually two types of files: ASCII and Binary. One is easier to read, the other is more suited to real-time/embedded. Take a look at the beginning of both files:

See? On the left, the PLY file contains X, Y, Z as floats, followed by a list of point coordinates. This is the point cloud! On the right, you can see a header describing the point cloud, in format XYZ-Intensity, and then, the points are not readable.

2 Main Ways to create a point cloud

There are basically 2 types of approaches, active and passive. Active techniques actively emit signals like light or sound to measure distances and create point clouds, such as LiDAR and structured light systems. In contrast, passive techniques rely on capturing existing environmental data, like photogrammetry, which reconstructs 3D points from multiple camera images without emitting any signals.

Active Techniques: LiDARs, RGB-D & RADARs

In the first case, point clouds come from sensors built to create them. When a camera takes a picture, it aims to get pixels. Well, when a LiDAR makes a measurement, its aim is to create a point cloud. Let's see 3 ways to do it:

1) How to get point clouds using Structured Light RGB-D systems

Ever played the Microsoft Kinect? I can't say that I have. I was a Wii player all the way when they were competing. Yet, I've always been impressed by how the Kinect produced point clouds using its RGB-D camera, working with the Structured Light Principle.

The Kinect shines a special pattern of light around, then uses an infrared camera to take a picture of how that light bounces back. By seeing how the pattern changes, the camera can figure out how far away things are. It combines this distance information with the colors it sees to create a 3D image called one final point cloud.

In robotics, you probably know the Intel Realsense 435i, or other equivalents. Their goal is to build a Depth Map, then turned into a point cloud.

2) How LiDAR point clouds are produced

The most common and popular technique is to use a LiDAR (Light Detection And Ranging). There are many types of LiDARs around, but let's focus on the simple Time-Of-Flight principle. In this setup, a laser scanner sends a light beam and measure the time it takes to reflect and come back to the receiver. Similar to this image:

LiDAR scanners produce raw data of the world up to 300-400 meters in the automotive industry. Each scan can generate millions of points in the three dimensional space. I highly recommend checking my article on the types of LiDARs to learn more.

3) RADAR Point Clouds

The third technique is to use not a LiDAR but a RADAR to create the point cloud data. This is not very straightforward to do. RADARs usually return signal information based on Doppler (velocity), Range (distance), and Azimuth (direction/angle). Using these, we can do some calculations to retrieve the point cloud data.

Here is an example on a very low quality RADAR:

Today, we can use Imaging RADARs to get 3D point clouds. I invite you to check out my Imaging RADAR article to learn more about it.

Now that we've seen the Active ways, using sensors - I'd like to take a minute to talk about the passive ways.

Passive Point Clouds Generation: Photogrammetry & 3D Reconstruction

The idea of passive is that you do not attempt to create a point cloud from your sensors. The main way to do this is by leveraging 3D Reconstruction. Ideas like Structure From Motion, Multi-View Stereo, NeRFs, Gaussian Splatting, or others are used.

The idea? To convert 2 or more images to a 3D point cloud using triangulation, geometry, and depth maps.

If you're interested in this, I highly recommend reading my 3 article Series on PyImageSearch blog, or my article on Pseudo-LiDARs.

Alright, so you now know all about the point cloud types, and the ways to get them. One thing remains...

How to Process Point Cloud Data?

Do you remember in the point cloud types when I showed the "label/class" of each point? This is not something sensors can measure, it's built by algorithms. There are 3 things that really matter here:

1. Understanding the main libraries/tools to work with
2. Understanding the core algorithms to use on raw point cloud data
3. Being able to use them in the applications

Libraries: Open3D and Point Cloud Library (PCL)

There are many libraries used to process point clouds. These implement the algorithms. For example, the Point Cloud Library is one of the most popular to work with. Open3D is also a very common one, it contains fewer algorithms, but is easier to process thanks to the Python interface. I would recommend to get started with this one.

On a similar topic, you could want to know at least one point cloud dataset. I would recommend you check out this article.

Which algorithms can be used to process point clouds?

In point cloud processing, you can either go with traditional algorithms or 3D Deep Learning. The split is, I would say, dependent on the applications. When companies want to detect objects in 3D to get bounding boxes, they usually use 3D Deep Learning algorithms like PointPillars or VoxelNet. Let's see an example:

Outside of 3D Object Detection and 3D Segmentation, the entire world runs on traditional processing approaches. Since you have points, you can create tons of automated pipelines to process them. For example, you can do plane segmentation, clustering, outlier removal, normal estimation, point data cropping, surface reconstruction, filtering of unwanted data points, and so on, you'll use traditional approaches.

For example, you could calculate the surface normals and filter out the objects that belong or don't below to the street.

Another technique can involve point cloud registration and alignment. When you have multiple point clouds, for example coming from 2 LiDARs, you can align them together into a single object. An example below from one of my LiDAR courses, notice how we start with 2 point clouds, a blue and a red, and we end up aligning them perfectly. This makes something better than the raw data.

In the algorithm category, there are countless applications. Everything related to SLAM or Odometry is also extremely in use today.

Applications: Which jobs can you target with Point Cloud skills?

Regarding the applications, we could write an entire article. Yet, let me give you 3 or 4 core jobs you can target with point clouds processing skills:

• Perception Engineer, Autonomous Vehicles: Process LiDARs and RADARs to find objects in the 3D space. Use Sensor Fusion to mix the output with Computer Vision. Build autonomous vehicles, shuttles, delivery robots, and create the future.
• Nuclear SLAM Engineer, Robotics: Use point clouds processing techniques inside robots that explore caves or regions humans can't go to, such as nuclear sites, and build maps of the world.
• BIM Engineer, Architecture: Create digital models of buildings and structure by processing raw point cloud data captured from laser scanners or photogrammetry. These models help architects and engineers visualize object properties, plan renovations, and ensure precise construction. The role often involves using processing software to convert points into computer aided design (CAD) models, performing manual correction to refine the data, and integrating the results into architectural workflows for improved design and quality inspection.
• Medical Imaging Engineer: Apply point cloud techniques to CT-Scans, IMRs, and other 3D data types to detect diseases and save lives. There is both a commercial and research use.
• Drone Engineer, Agriculture: Process Cameras and RADARs/LiDAR information to navigate and help drones fasten agriculture and solve population needs.
• and many, many more...

Alright, now let's see a summary...

Summary & Next Steps

• A point cloud is a series of 2D or 3D points. A point cloud is to the LiDAR what a pixel is to a camera.
  (Re-read that one.)
• Each point of a cloud usually contains at least the XYZ information, but many sensors or technique allow to also get Intensity, Reflectivity, Velocity, Ring/Channel, Color, Temperature, Infrared, and more...
• A point cloud output format is of 2 types: ASCII or Binary. An ASCII file is more readable for humans, Binary is more readable for robots. Each file is a list of points and their information.
• There are 2 ways to build a point cloud: Active and Passive. Active techniques involve sensors like LiDARs, RADARs, or RGB-D cameras, while passive techniques use photogrammetry and 3D reconstruction to retrieve 3D models.
• Point Clouds Processing typically involves 3 stages: the tools/libraries, the algorithms, and the applications. Tools are libraries like Open3D or PCL, algorithms are either traditional or deep learning, and applications go from self-driving cars to robotics, drones, augmented reality, the architecture industry, agriculture, and beyond.

Where do you think humans come from? Growing up, I wrestled with two conflicting ideas about it. One teacher taught me Darwin’s theory of evolution: a gradual process of adaptation, rooted in science but riddled with gaps and errors. The other taught me the Bible’s Old Testament: the story of divine creation, in which even though nothing was ever proven false, this isn't built on any "proof".

Since then, I realized they were trying to answer the same question, but operated in entirely different realms, each with its own logic and purpose. One belonged to science, the other to faith. Making head-to-head comparison was almost meaningless.

And I think this contradiction also exists in self-driving cars, especially when opposing 2 giants: Tesla and Waymo. Both seem to chase the same prize of "Level 5" autonomy, but when you look closer, their paths are so distinct they’re barely comparable.

In this article, we're going to try and understand who has what I call the best "Map to Level 5", we'll take a side-by-side comparison, and I'll give my opinion on 3 aspects:

• The sensor suite
• The algorithms
• The "map", meaning strategy, vision, and more...

Let's begin with the sensors...

Tesla vs Waymo: Who has the best Sensor Suite?

Back when I was studying driverless cars, it was around 2017, when I heard an interview with Sebastian Thrun, godfather of self-driving cars, talking about who was ahead in the race. I vividly remember his words: "Nissan is doing pretty good, but I think the company who is ahead of everyone is actually Tesla".

It was 2017, and I remember feeling surprised by this comment, because at the time, Tesla only had a light ADAS feature working with mobileye, and companies like Waymo, Mercedes, Nissan, and others seemed to be covered everywhere in the media, have "real" self-driving car abilities, and more potential.

What about today? Who is ahead? Closer to remove human drivers? Who has the better vision? The better algorithms? The better sensors? Is it Waymo, or Tesla... or someone else?

In this first part, I want to answer it from a sensor angle. And to do so, I'm going to start by a screenshot of a very popular X (tweet?) from Elon Musk about RADARs, LiDARs, and cameras from August 2025.

This comment raised an army of furious engineers and fusion experts, mentioning Kalman Filters, and Redundancy, and Sensor Fusion. So before we dive into the exactness of this comment, I would like to describe what each company is doing...

Waymo: 29 Cameras, 6 RADARs, 5 LiDARs

If you look at a Waymo car, you're going to see exactly the opposite of Tesla: tons of sensors all over the place. There are RADAR sensors on the front, side and rear, there's 29 cameras, and 5 LiDARs. The question we can ask is... "Is Waymo trying to kill a fly with a bazooka?".

Just in terms of calibration, it must be an absolute nightmare for engineers. Calibrating a camera with a LiDAR is already a long task, but 29 cameras with 5 LiDARs? Then there is the fusion of all of these sensors together, and this is ONLY the "robotaxi" version, because if you look at their Zeekr shuttles, they also have their own types of sensors, with different generation codes. Their stack is therefore always evolving, and depends on the vehicle they drive on.

What type of LiDAR, Camera, and RADAR is Waymo using?

Let's take a brief look at what each sensor sees:

Waymo uses different types of cameras. They work with high-res long-range cameras (color, telephoto lenses) for object detection far down the road, wide-angle cameras for close-range coverage (pedestrians, cyclists, intersections), and near-infrared cameras for night vision / low-light perception.

Regarding LiDARs, they're using their own sensors called "Laser Bear Honeycomb". There is one forward LiDAR, 2 side LiDARs, and one at the rear. But these are short range, solid-state LiDARs. They are excellent for blind spots and front facing vehicles, but complex to drive on highways because they don't see far. This is why there is the roof LiDAR, which is mechanical, and sees several hundred meters away.

On the animation below, you can see LiDARs both in point clouds format and in range-view — and you can learn about types of LiDARs here.

Regarding RADARs, Waymo uses their own line of Imaging RADARs. Imaging RADARs are what I call 4D RADARs. Unlike normal RADARs who see in 2D and measure the velocity, these ones see in 3D and measure the velocity.

Do you want to see what all of this looks like together? Okay, here it is:

Waymo uses a powerful array of sensors allowing them to see every possible object. We'll come back to the utility of LiDARs and RADARs, but for now, let's look at cameras...

2. Tesla's Sensor Design: 8 Cameras, that's it

Unlike Waymo vehicles, Tesla's approach relies only on cameras. Tesla's autopilot aims to solve autonomous driving using vision-only.

On this illustration, you can see a very simple design that almost never changed. In red, you can count 8 outside cameras, and in orange, you can see 12 ultrasonic sensors used to detect static objects when parking. Among the 8 cameras, there 2 on the windshield, used for stereo vision, one on the front bumper, one on the rear bumper, 2 on the doors, and 2 on the wheels. See the difference? I can't even begin to count Waymo's cameras, but I can easily show you Tesla's sensor stack.

So let's visualize what the cameras see:

Interesting, or not. Now, let's try and understand, who has the best sensor suite?

3. Who has the best sensor suite... Tesla or Waymo?

I am going to show you an image, and I would like you to ONLY look at the left part. Ignore the right for now. Can you tell me what you see? You see a car, don't you?

But did you notice the pedestrian? This is one of the limits of the vision-only approach. When you look at Tesla's miles driven without disengagement reports, Tesla FSD clearly shows a limit on bad weather. They don't drive well on cloudy foggy, rainy, or snowy scenes, and they don't drive at all during storm and sleet (when ice falls from the sky).

Reports show that FSD13 significantly improved driving by night, but there is still a physical limitation to driving with cameras only that is not solvable with better algorithms. In a robotaxi situation in which you'd sit in the passenger seat, you would be stuck. The way humans drive involves more than cameras, we hear street sound, we sense people, and we don't use wide-angle cameras to detect other cars.

If we were to come back to Elon Musk's comment now, do you remember the "If LiDARs/RADARs disagree with cameras, which one wins???". Waymo shows that redundancy is key to safety. If a camera misses something, your RADAR may not. And with Kalman Filters, you can certainly develop a powerful fusion module to account for disagreements.

When we consider the physical ability of a LiDAR to generate point clouds, you can understand how powerful having them is. Other than seeing through night or other situations, they physically build point clouds. Recently, a YouTube video has shown a Tesla vs LiDAR-equipped car driving on a wall resembling a street. The Tesla FSD crashed on the wall, confusing it with the highway; but the LiDAR-equipped car stopped.

You may wonder... Okay, but we'll never see fake walls in real life, so what's the point? The point is that vision only has its limitations. We've seen Tesla confuse the moon with objects, miss a red light, do phantom breaks, and even miss truck trailers. For all of these reasons, I would say that the Waymo approach including Camera+LiDAR+Imaging RADARs is a better choice. They get the #1 point.

One caveat is that the algorithms, processing power, and energy required to operate this vehicle is insane. LiDAR sensors consume a lot of energy and record a lot of data. Tesla is much cleaner in that perspective.

Speaking of algorithms, let's now move to this second point.

Tesla vs Waymo: Who has the best algorithms?

If we now look at the algorithms, who is closer to build self-driving cars? Tesla and Waymo started off with very different architectures, but now seem to converge towards End-To-End Learning. So let's take a look...

1. Tesla FSD Algorithms: HydraNets, Occupancy Networks, and End-To-End Learning

In my article breakdown on Tesla, I'm doing a full deep dive on the Tesla's algorithm, and how they work; so I won't do that here, but I will still show you the overview of how they built their autonomous vehicle architecture. Note that it's according to the Tesla data; which moved to private in 2023.

You can see 3 main blocks:

• Lane & Object HydraNet: The lane and object Hydranet is a multi-task learning network that takes in the 8 cameras, learns features from each using a CNN, fuses them spatially and temporally via a Vision Transformer, and then outputs several heads. Heads are trained to detect objects, lanes, positions, and so on... You can read more details here.
• Occupancy Network: The Occupancy Network is also processing all 8 cameras spatially and temporally, except that it's trained to leverage spatial data. This is a 3D network that aims to build voxels and assign a free/occupied state to each. You can read more details here.
• Planning & Control: The Planning & Control node used to be (in the drawing) done via a Monte-Carlo Tree Search. This is traditional artificial intelligence. In 2024, they replaced this with a Neural Network planner. While we don't have details on how it works, the "End-To-End" comes from this node moving to Deep Learning, making the entire network differentiable.

To push the explanation even further, let me show you the typical visualizers on a Tesla, and see how they both refer to an algorithm:

Now, let's see Waymo...

2. Waymo's Algorithms: 3D Deep Learning, Diffusion Planners, and more...

It's a bit harder to fully track the state of Waymo's algorithms, because they continuously release multiple research papers, and we don't know which ones are in production, and which are just pure research. Still, according to my research, there are 3 core pillars Waymo relies on to drive...

• LiDARs
• Prediction/Tracking
• Imitation/End-To-End

LiDARs

Early on, Waymo pioneered work on LiDARs with 3D Object Detection algorithms like SW-Former. These algorithms process LiDAR point clouds and output bounding boxes in 3D. This architecture has been updated a few times, but now serves as the "core" detection algorithm of Waymo. From there, it's encapsulated into other pipelines, like the Late-To-Early Fusion:

See what's happening? We have a temporal fusion algorithm that processes LiDARs and boxes from t-1, t-2, and so on... and each of these go to SWFormer to output a final head. This is turning SWFormer into a temporal detector, and not just a frame-by-frame detector.

Prediction & Tracking

Going even further, we have Prediction & Tracking. Waymo bets big on tracking, and one of the core algorithms I noticed there is an architecture recently released called Stateful Track Transformer which is doing exactly the job of tracking from SWFormer. Over the years, Waymo released TONS of prediction and tracking architectures.

End-To-End/Imitation

Back in the early 2020s, I remember vividly Waymo mentioning an algorithm called ChauffeurNet, who was behaving exactly like Tesla's HydraNet, but was outputting trajectories. Since then, the approach evolved, and the later published papers and public talks mention the End-To-End architecture named EMMA, as well as Vision Language Models.

As you can see, this turned in to an Encoder/Decoder architecture, where the encoder learns features from each sensors, then fuses spatially and temporally, to learn a compressed representation of the scenes. Then, the decoder is a generative part, built on VLMs to predict a trajectory.

While many claimed Waymo has a "modular" approach, while Tesla has the advanced End-To-End approach; this is simply no longer the case. Although we don't know whether Waymo uses EMMA in production, or still relies on their traditional pipeline, we definitely know they're heading towards it.

3. Who has the better algorithms? Waymo or Tesla?

Let me write 4 or 5 bullet points explaining what I think:

• Both Tesla and Waymo seem to be headed towards End-To-End Learning, because the Modular approach has the car behave in a "robotic" fashion. This is not smooth, feels robotic, and rule based. So both go towards End-To-End...
• But to make End-To-End work well, you need LOTS of data. Tesla has that from their fleet of millions of cars all across the world, but Waymo has a small fleet driving only in very specific regions. Scaling via End-To-End will be extremely painful for them.
• On top of that, Tesla only needs to process camera data, which makes the algorithms likely faster, using less power and less time consuming.
• End-To-End's biggest problem is edge cases. Tesla has experience driving several million miles in construction zones, parking lots, or in a new city, and so on... Waymo, on the other hand, is stuck with HD Maps and can have issues moving towards End-To-End...
• Tesla also has advanced techniques for Edge Case Detection and retraining, such as Trigger Classifiers (see my detailed overview here), as well as Dojo and Self-Supervised Learning. They seem far more prepared for End-To-End Learning in my perspective than Waymo. It's as if Tesla had paved the way for this for a decade, while Waymo pivoted last minute

Given these, and under the assumption that the price here is End-To-End, I would give my points to Tesla.

Waymo vs Tesla: Who has the better Map to Level 5?

In this last point, I would like to get off the sensors and technique, and see these as businesses primarily. Their goal is to sell self-driving cars, or autonomous rides, and thus... who's leading in that sense?

1. Strategy

First of all, something very important to understand:

• Tesla sells self-driving cars
• Waymo rents autonomous transportation services

This is essential, because this means their software, sensor stack, philosophies, business models, and even strategies to reach Level 5 are totally opposed. This is very clear when you see the graph below, which shows Waymo starting with a very capable vehicle, but only in ONE geo-fenced area, with ONE car, while Tesla starts with millions of car, but none of them are autonomous. Both don't scale the same thing:

Tesla's goal is to make millions of cars at a 25,000$ price point. Imagine Tesla in 2016, being told to integrate LiDAR technology, which cost over 50k USD/unit. Wouldn't they be better off starting with a light camera + RADAR Level 2, and gradually improving it? Of course they would, if they believe it's possible.

On the other hand, Waymo had a fleet of maybe 20 vehicles at the time. With 29 cameras, 5 RADARs, and 6 LiDARs, a Waymo car costs significantly more than a Tesla car. In fact, each car was estimated to cost around 250,000$ a few years ago. Since then, the LiDAR price dropped, Waymo grew its fleet to over 2,000 cars, and each is now estimated around 150,000$. Can you see how they cost is less and less of a problem to them over time?

Waymo's LiDARs certainly cost a lot, but this cost get absorbed as they do more paid rides, until it 'supposedly' becomes profitable. Supposedly, because there is maintenance, replacement, and growth of the fleet, which can make the cost a never ending fight. Even there, Waymo has more leverage to afford to LiDARs, by raising the ride cost of millions of people from 7$ to maybe 13$ (made up numbers, terribly off — just making a point).

There are great statistics on this blog post from Ben Buchanan that show how Waymo's car get more and more affordable over time. A LiDAR costs 500-1,000$ today, this completely challenges the vision-only philosophy. Unlike Tesla, time in on Waymo's side.

You can therefore understand how each one wins:

• Tesla starts with TONS of disengagements and tries to decreases that number to 0. The more capable the algorithm, the more cars the will sell.
• Waymo starts with very few disengagements and tries to scale the number of rides and regions without increasing this number. The more regions they will cover, the more rides they will sell.

2. HD Maps

Waymo is betting on a serious "HD Map" strategy that Tesla refused to adopt. According to Tesla, the car should be able to drive anywhere in the US, so there's only using normal Google Maps or OpenStreetMap. It does not mean they don't use HD Maps; they do (see screenshot below), but they don't require them to drive. If the car ends on a parking lot with no map, it should still be able to drive.

On the other hand, Waymo maps every squared inch of every place they drive in. This means every traffic sign, every bumper, every crossroad, every traffic light, every roadwork, lane lines, speed limit... everything is continuously mapped and updated. You can see on the image the HD Maps of Waymo and a screenshot of Tesla's HD Maps on a car in debug.

From what I read, Tesla performs poorly on regions where they don't know the maps. While this doesn't block them, this certainly causes disengagements. They are therefore equal in this sense.

3. Miles Driven & Disengagements

We can't conclude this article without first looking at who slams the brakes the most. Yet, I feel you already know the answer, from the first two points. It's obviously Tesla, because they have many more cars, miles driven (5B for Tesla vs 100M for Waymo), and different situations. So how can we really vote who has the better map to Level 5?

Let's first look at disengagements. What is a disengagement? Is overtaking a stuck vehicle a disengagement? Is accelerating? What is the definition, and do Tesla and Waymo use the same? Well, Waymo uses safety drivers, who obey specific instructions. Tesla drivers disengage for virtually any reason, even if they simply feel like it. This is why Tesla FSD Tracker shows, as of September 25, 24% of FSD drives have a disengagement, and 3% have critical disengagements (US).

Continuing with more stats: Tesla drives on average 213 miles before a disengagement on a highway, and most disengagements are caused by Lane Issues. Notice the jump in miles driven without a disengagement with FSD 12.6.

For Waymo, it's a different story. Waymo has a driver permit in California, which had them recently release the 2024 disengagement report to California DMV. It clearly showed a number of 9,793 miles driven before disengagement. You may ask... How is Tesla at 200 but Waymo at 9,700 miles driven without disengagement? It's because, as we said, the definition, reason for disengagement, and cities, are extremely different.

If a company wants to drive on a straight line for 500 million miles and show 0 disengagement, they technically can. This is why I think, we can't really trust any of these numbers — only their relative evolution on the same conditions as before 🤷🏻‍♂️

So who has the better Map to Level 5? I will tell you, but only after we have made a quick summary of what we've seen...

Summary: Who is ahead, Waymo or Tesla?

• Tesla and Waymo both have different sensor stacks. While Waymo relies on a stack of 29 cameras, 6 radars, and 5 LiDARs, Tesla takes a different route and relies on 8 cameras only.
• A sensor setup with LiDARs and RADAR redundancy is safer and gives more reliability than a vision-only setup. LiDARs can detect objects and brake the car even if no object is identified by the camera or an algorithm. Waymo also drives in more weather conditions than Tesla, who is physically limited by the lack of other sensors.
• Waymo uses an architecture based on LiDARs, with algorithms like SW-Former, Prediction/Tracking, and EMMA as an End-To-End system.
• Tesla uses an End-To-End approach involving HydraNets, Occupancy Networks, and Deep Planning. This approach is reinforced by powerful Trigger classifiers, Self-Supervised Learning Dojo, a powerful data fleet — making them win on the algorithm aspect.
• Waymo depends heavily on detailed HD Maps and can only work on these mapped, geo-fenced areas. Tesla’s system is designed to drive anywhere without necessarily relying on HD maps. In practice, they drive much better when they have maps.
• Waymo boasts a much better disengagement rate, with safety drivers rarely needing to take control compared to Tesla’s more frequent interventions; but the conditions they drive in and disengage are 100% under their control.
• The two companies have very different business models: Waymo sells autonomous ride services, Tesla sells cars with self-driving features. As a result, their map to level 5 is not the same.

And now, you're all caught up. So, the Map to Level 5?

The Map to Level 5

Here is what I think:

To reach Level 5, Tesla will need to reduce the number of disengagements. To me, they will have to include LiDARs or RADARs at some point. At today's cost, that would probably be feasible. In fact, I'm pretty sure Tesla is so competent FSD would be solved right now if they didn't chose to play the game on hard more. But now, after all this time, can they really afford to do this? There's ego, brand image, and the "FSD capable" computers they already sold. How can they? Tesla is in the camera game for good.

To reach Level 5, Waymo has to increase the number of areas they drive without increasing the disengagement rate. This means adapting the "HD Map" strategy, which still relies on them, and making algorithms capable to adapt to a new region faster. To make autonomous driving a reality for the entire world, Waymo will need to go faster. Their autonomous vehicles surely are capable, but their current scaling strategy takes too long.

Next Steps

Have you ever heard the story of the iPod? It started in January 2001, right after Apple announced a loss of $195 million, and had missed the shift to digital music. The company was lost, and had one last chance of survival: building an MP3 player to catchup with the competitors.

If you are old enough to remember the MP3 players back then, the were confusing to use, overloaded with buttons and menus, and made the experience painful for customers. Apple was looking for a solution for months, but had no clue how to make it better.

Until one day, when Apple's Head of Marketing Phil Schiller suggested using a scroll wheel. Wheels already existed in mouses and dial phones, but had never been never used in music players. With this, he suggested that the menus should scroll faster the longer the wheel is turned, a stroke of genius that would distinguish the iPod from the agony of using competing player.

The rest is history: Apple developed the iPod in the greatest secrecy, launched it, and changed the world with "1000 songs in your pocket".

What made it so successful? It's not that it looked good, or had buttons, or could store more songs. No, the genius was in the smarter experience scroll wheel.

If you're in the autonomous vehicles market, we probably witnessed a similar pattern: companies have been collecting more and more data endlessly, building data centers, simulators, hiring people to analyze the data generated, and so on... Until some companies came up with smarter ways, not involving just "collecting more data", but rethinking the experience to focus on events instead.

In this article, I would like to tell you about the way automotive data processing works nowadays, and how the AI revolution is going to reshape it.

We are going to learn about 3 ideas:

1. The first part is going to focus on the Manual Era (where we collect and process it all) and the Cloud Era (where we use DataLakes)
2. The second part will be a case-study provided by an autonomous tech startup, revealing the 10 biggest problems of the Cloud Era.
3. The last part will show you the Edge Intelligence Era & the Autonomous Era, which, as you'll see, if an incredibly more intelligent way to do

Let's begin with point #1.

1. Data Management: How self-driving car companies collect and process data in the Cloud Era

One of the things we heard the most this past decade was that Data is king. And for a long time, collecting as much data as you can in order to train heavy machine learning models has been the only way to do. Let's talk about data collection, and then processing.

How do autonomous vehicles collect data?

We know that when a self-driving car drives, all the data (sensors, images, messages, hardware status, algorithm decision, ...) is being recorded.

Should we give an intro line explaining how?

The process is simple, and looks like this:

1. You plug your sensors to your system (for example, Robotic OS/ROS)
2. You press record

I'm sure somebody out there worked hard to find a more complex process, but if you're using a tool like ROS, recording data is as simple as using one command line. In the video below, you can see me recording LiDAR point clouds, camera images, GPS positions, algorithms outputs, and mostly all the messages passed through the self-driving car while we drive...

When I'm done recording, the output is a file in the .bag extension (for ROS 1) that can vary from a few Gb to Terrabytes of data. Let me show you an example below from the TIERS dataset. Notice the duration and sizes of the recordings below — the last one is just 8 minutes long, and yet weights 200Gb. This is 2.4Gb/minute!

The bag is the first element. Then comes what we do with it.

The Manual Era: How do we process and analyze data?

The first "era" I'd like to tell you about is the 1.0 era. Back when I worked on autonomous shuttles, each of our fully autonomous vehicles was driving and collecting data to SSD drives. When the day was over, we had hundreds of Gb to process. So we started coming up with file naming conventions, involving the date, event, and so on...

Then, when back to the office, we could replay our algorithms on it, train our models on the data, and so on... Below is an example of a drivable area segmentation algorithm I've been training on the data collected:

This was fine for a small startup of 8 people, and it's probably still okay for small companies that don't need extensive processing, but most autonomous vehicle companies have turned to the cloud...

The Cloud Era: How Advanced Driver Assistance Systems (ADAS) & most of the Automotive Industry is using Data Lakes

If you record data every day, and each recording is hours long, you're never going to find the events you need. This is why I'm showing you a more sophisticated, let's say '1.5' version, which makes data collection part of a pipeline.

It looks like this:

1. You record the data
2. You upload it to AWS/Azure
3. The R&D team then processes it weeks later, replaying all the events, searching for 10% possibly interesting scenarios, or events, and so on...

If you'd like to see real-world concepts, you can see AWS and Azure Data Lakes:

A lot of companies in the self-driving car market use these "data lakes". Let's look at the Azure Data Lake in a simplified view:

ADAS & fully autonomous cars use it. After recording, the 4 key blocks are:

1. DataOps: Where we analyze data, clean it, label it, augment it, tag it, and so on... Notice the interaction with external labellers; that idea is called "human-in-the-loop".
2. MLOps: The machine learning algorithms, training, testing, and so on...
3. ValidationOps: The validation part, involving visualization, scenario, and simulation.
4. MetaData: After the DataOps tagged the data, we can search for it.

You can see how it's placing data at an element in the chain.

So what are the problems of this? Before telling you about the 2.0 Autonomous Era, let's try to see a case study with real ADAS or artificial intelligence companies using it...

2. [Case Study] ADAS Actors reveals their 10 biggest problems with Data-Driven Approaches

In this section, before talking about the '2.0' approach, I would like to tell you about the core problems companies who process large volumes of data reported.

Before writing this article, I got the opportunity to talk to Heex Technologies, a french startup specialized in Event Based Data Management... and I asked them "Which problem do you solve?". To answer, they shared a 20 page PDF listing all the problems their biggest Advanced Driver Assistance Systems (ADAS), autonomous driving, or robotic clients from the automotive industry.

In the PDF, I spotted a lot of interesting problems. Let me share the main ones with you:

If I were to list down the 10 main problems, you'd see: Slow access to critical events, Manual data processing, Exploding Cloud costs, Fragmented data, Delayed visualization (no real-time), Manual Extraction of scenarios, Useless streaming data, Physical SSD Extraction, Blind Debugging, and Inefficient ROS Bag Processing.

Notice all these terms I underlined? These are the problems of today's data management systems.

Let's take some examples...

• If you collect the data on Day 1, and process it on Day 3, you have slow/delayed access to critical events; like a missed pedestrian. So you're driving, notice something wrong, but you have to wait until 2 days later to even look for the data, and start searching for that event you noticed...
• Similarly, can you see how the 'fragmented' data processing is a problem? Especially when you are with a team. Engineer A grabs bag A, and makes decisions based on it... Engineer B grabs bag B and makes a different decision based on it... The entire decision cycle happens in silos.
• The Physical SSD extraction is a problem too. In May 2025, I was at the Stuttgart ADAS & AV Expo, and I met a company who invented a "swap" disk system... All of this is great, but that's still the same problem of storing, copy/pasting data, etc... to a system.

In each of these problems, I noticed a time and money waste.

For example, the client reporting: "Engineers waited several days to weeks to access specific events due to the time-intensive process of uploading, filtering, and classifying raw data in the cloud." is clearly facing a time problem, reviewing large amount of data... The other client who mentioned: "The full data pipeline we built —from data capture to processing and storage—incurred high cloud costs and consumed engineering resources" faces a money problem...

In this same report shared by Heex, all the companies reported improvement in their pipeline. Whether it was better decision making, more time freed, or money saved. This is why the next part is so important, so let's now focus on it: Event Driven Data Processing for autonomous cars.

3. Event Driven Data Management for autonomous cars

Back when I started learning autonomous driving algorithms, I listened to an interview from Sebastian Thrun, acknowledged as te godfather of self-driving cars, who at some point, said something that marked me: [paraphrased]: "With a team of 2/3, you can build a self-driving car that drives 90% of scenarios in a weekend. Then to get to 95%, it takes a few weeks, and to complete these last 5%, it takes years."

This idea is called the "long-tail" problem.

When looking at traffic accidents involving autonomous vehicles, you often see rare events or edge cases at the root cause. The person wearing a stop sign t-shirt, the truck with a donkey on the trailer, the traffic sign burned by parisian riots, all of these unusual scenes totally different from empty highways cars are used to.

Some companies solve it with data generation, others with simulation, or with End-To-End Learning. Yet, the root of all evils here is data, and thus, this is what we have to change.

A decade ago, the term "data" became king, and everybody became a Data Scientist, Data Engineer, Data Ops, Data Something. It was the case until recently when the data revolution passed, and breakthrough innovations happened not thanks to more data, but thanks to smarter training systems (like self-supervised learning), or more powerful architectures (like transformers). "More data" was ultimately not the solution, and thus, we have to switch our thinking...

The Edge Intelligence Era: From Data Management to Event Management

After companies have recorded a few laps of the neighborhood they drive in, recording more of this same scene doesn't make sense. Companies record more and more, just to spot the 1% of long-tail events. What if we worked on these events only, from the beginning?

It can be done, by setting up a "triggers" in your system, that will act as a filter and only capture the scene when interesting events happen, such as:

• Objects Missed: If one camera misses an object that another sensor sees
• Near Pedestrian Collision: If pedestrians are within 2 meters of our car, and we drive over 30km/h
• Human Intervention: If a human driver manually took over
• Shakes: If the camera physically moved due to a bumper or small shock
• Ego Collision: If a collision with the ego vehicle happened
• and so on...

All of these are valid events we'd like to record. The rest? When it's all smooth? Well, we already have millions of it.

An example with Heex Technologies, and their platform allowing to set triggers:

If I were to show you the 2.0 process, it'd look like this:

• You have the same car with LiDARs generating the same 10Gb/h data
• Rather than recording all data available, you define triggers.
• You intelligently record the events, like the near pedestrian collision, and not all the data
• You get instant notifications, labels, and can do real-time decision making

Seems smarter, isn't it?

Now that you have this in mind, I'd like to show you the last era...

The Autonomous Era: AI does it for you

The next step is to create algorithms to do it automatically for us. For example, Tesla patented a concept called trigger classifiers. The idea is to train their HydraNet backbone to classify whether the general scene it's learning from contains unusual events or not. If it does, let's say above a certain confidence score, then the machine learning models will trigger a warning.

Whether you're working on spreadsheets or building autonomous vehicle technology, automating manual items like labelling or searching for data makes sense. In this case, just like the Edge Intelligence Era, you can see the events being captured live while driving, and not after.

The 2.0 Vision

This is going with the "2.0" vision of self-driving cars that companies now define. A vision driven by Deep Learning first, where data matters, but where more data isn't the solution. In the 2.0 vision, quality is better then quantity; contextual intelligence is needed, learning should be real-time, and training should be done on relevant data.

If the 1.0 vision involved heavy test vehicles, modular architectures; the 2.0 vision is about AI & efficiency.

Now, let's see an example of a company specialized in this...

Example: How Heex Technologies turns Data into Event Management

One of the companies that captured this vision the best is Heex Technologies. They built a SaaS platform that implements exactly these ideas of "triggers" — and their motto is that rater than focusing on the data, they focus on events. As I already showed you the triggers, we could see it in action:

Let's look at their pipeline, which you'll notice also works backwards — once the bag is generated:

From a heavy bag, we get a smart bag. The data is definitely smarter when automatically annotated, categorized, and when relevant events are flagged. We can then re-inject this data into the training pipeline, without having to worry about the rest of the dataset.

As an entrepreneur myself, I can only admire the focus on one specific and painful problem like this one. When you can anticipate customer needs, and enable automakers and automotive engineers move away from a complex process to focus on their core job (self driving technology)... you win!

Alright, let's do a summary and see what to do next:

Summary & Next Steps

• Collecting data is essential to train AI models and develop self-driving vehicles. Yet, "more data" is not the solution to create breakthrough, and solve the "long tail" problem, which cause a significant challenge.
• The first Era of data processing is the Manual Era, in which we record and process everything manually. (1.0)
• The second era (1.5) is the cloud version, in which you work with data lakes and build a real "chain" that contains DataOps, MLOps, ValidationOps, and so on...
• The third era moves to the 2.0. It's where we stop obsessing on the data, and focus on events. We can use triggers and platforms like Heex Technologies to do it.
• The fourth era is the AI Era. (2+) This is where we have AI automatically find events, and train itself continuously on these.

Which solution is right for you? In reality, they can all work. A small startup can work manually, until they find their hard problem to solve, and they have a budget to invest in data lakes... Companies can work with data lakes, but for bigger fleets, it'd make much more sense to think in terms of events instead.

Next Steps

 If you realise you have these problems of recording everything, having your data staying a bit « dumb », and would like to know exactly how to stop recording everything by this afternoon (without losing the important information)...

... Then I’d recommend to check out Heex free discovery quiz, which will reveal tell you exactly what you’re doing wrong today, and (based on your answers) show you what to do this afternoon to save hours of recording, data processing, etc...

It’s free, and you can get access below:

Heex Free Discovery Quiz

Is your data strategy a silent obstacle?

Take the Quiz

You can also take a look at Heex' product here: https://www.heex.io/en-gb/smarter-data-faster-decisions

How much impact do you believe your job has? Is your job saving someone time? Or money? Or... his life? Well, what if you worked on projects that saved people's lives? Such companies exist, in self-driving cars, in healthcare, and in the case of this article... in Autonomous Defense!

This summer, I have interviewed Vibhav Ganesh from Shield AI, a U.S.-based defense technology company that develops autonomous systems for military and government use.

I would write a big paragraph here, but let me instead show you a quick sample from the interview...

Vibhav Ganesh is the Director of Engineering, past Chief of Staff to the CTO, and Employee #20 of Shield AI.

Vibhav has played a pivotal role in the company's growth and innovation. With a background in visual inertial odometry and SLAM, he has been at the forefront of developing autonomous systems like the Nova 2 quadcopter.

Let's read his intro to Shield AI and to their core products: the V-BAT and the ViDAR.

There are a lot of things to note about their products: the V-BAT can last 10 hours, which is a technological achievement itself, thanks to V-TOL (vertical takeoff and landing)... ViDAR is also a very interesting product, which stands for Visual Detection And Ranging... and HiveMind (not shown here) is their AI, or as they call it, "The World's Best AI Pilot".

Let me take you to the v-BAT first, as it's the core product, by showing you this LinkedIn post we did together, where Vibhav Ganesh introduces us to Shield AI.

Together, we recorded an exclusive Fragment of The Edgeneer's Land, my community membership experience, in which he takes us through Shield AI. What is autonomous defense? What are the main technologies involved? What is the range of products?

In this post, I'd like to give you a small sample of that interview, highlighting a very interesting moment where Vibhav talked about infiltration drones.

Inside Shield AI's Tactical Infiltration Drones

Impressive, isn't it? What I really love about it is that it's down to earth. I could see myself assembling a drone kit, adding a camera, a 2D LiDAR, and starting experimenting with Visual SLAM projects to map a room. This is basically what Shield AI did, when they got started. Except that their drone was (1) targetted to a specific client and (2) better than all competition.

There are 2/3 insights I'd like to share with you, from Vibhav:

1) Self-Driving Car > Autonomous Transfer doesn't work as we'd think

Now, here is something important to note:

A lot of what you learn in autonomous robots CANNOT simply be transferred to drones.

I did think that it was a matter of copy and paste. But I understood I got it wrong when making this fragment, especially with Vibhav Ganesh who told me that their drones don't have LiDARs, fly over seas, or over deserts, no man's land zones, between mountains, campaigns, with 3D constraints, and no map!

I thought about it for a minute, and I realized... "Wait, it's absolutely NOT like autonomous cars!"

And in fact, when you start looking into autonomous drone architecture, they absolutely don't look like self-driving car architectures! For example with Shield AI, they have a Control, a Station, RTOS, a ViDAR, but also a Flight Controller powered with frameworks like PX4 and Maven. This is an entire set of libraries to learn.

From there, here is a second insight:

2) Visual SLAM is mostly used

Coming back to the idea that we are NOT like self-driving cars... the other main difference is that we use no map. So without a map, and with just a camera, they have no choice but to use...Visual SLAM!

And Vibhav explains really well what kind of SLAM they're using, how they implement the mapping, even though there is 0 starting point, and so on. Here is a sample of a vSLAM project I've tested with drones:

This is the nitty gritty of Shield AI's work. Once you build a SLAM MAP, you can then feed that map to the Motion Planner, which sends a flight order to the drone. If you'd like more insights on this technology, I highly recommend my vSLAM article.

Visual SLAM Algorithms: The 6 Components Explained

How does Visual SLAM work? How is it different from normal SLAM? What are the 6 main steps of a Visual SLAM system? Let’s find out!

ADVANCED ARTICLES FOR CUTTING-EDGE ROBOTICS & AV ENGINEERSJeremy Cohen

Okay, would you like to see some samples?

Shield AI in Action

Let's take a look at 3 samples here:

• On the left, you can see drones being launched. These drones cannot do the "vertical takeoff and landing". They are projected to the air by launchers and then fly like a plane.
• In the middle, you can see the tactical quadcopters we discussed. Notice how they use vSLAM at the end of the shot.
• On the right, you can see a mission of the v-BAT searching for a vessel in a sea canal.

This is really cutting-edge, and totally applied what we are building in the autonomous tech space. Alright, time to wrap up!

Summary & Next Steps

• The defense industry is extremely active. Hundreds of companies work on the autonomous generations of drones, anti-missile detectors, infiltration equipment, RADARs, and more...
• Shield AI is an active actor of the defence space, with a range of multiple products, such as the v-BAT, the ViDAR, and HiveMind.
• Shield AI started with NOVA 1, a quadcopter that could infiltrate into buildings, builds maps, surveys, without the need to send humans into the region. This allowed to prevent human losses due to buildings collapsing or being trapped.
• Besides from being safer, infiltration drones are also more efficient. Shield AI tested their drone against 6 NAVY SEALs clearing a building, and it finished in a third of the time.
• The transfer of autonomous car/robot technology to autonomous drone isn't as simple as we'd think. Architectures are different, products are different, regions/environments are differents, and even the inside technologies and algorithms change.
• On the other hand, some technologies really apply well to autonomous drones, such as Visual SLAM.