Have you ever doubted your loving partner? In 1945, a Canadian spy named Max Vatan just received an intel that changed everything. His wife Marianne was suspected of being a German spy. His orders were now to find out if it was true, within 72 hours. If she was guilty, he then had to kill her. This is NOT a true story, but the story of the movie Allied, with Brad Pitt and Marion Cotillard.

So how do you find out if your wife is a german spy? Max had easy options and complex ones. He could have her followed. He could interrogate her friends, search the house, pull her files... but that was complex. So instead, he made a simpler choice; let her overhear a phone conversation sharing false intelligence, and see if the enemy acted on it. Unfortunately for him, they did.

This technique is called the canary trap. It's used to identify leaks by feeding slightly different versions of the same false information to different suspects. Whoever's version shows up in the enemy's hands is the traitor. There is something I love about it: it's extremely simple. While many techniques are based on heavy profiling, and probabilities... this one is just relying one one piece of information...

That simplicity, I think Occupancy Grid Mapping in robotics shares it. You could build a world with a complex 3D representation, bounding boxes all over the place, HD Maps, and all the complexity... or you could build a 2D grid, and define if you can drive there or not. This is particularly useful in environments where objects aren't easily learned (military, agriculture, ...), and thus, we need new ways.

In this article, we're going to explore the topic of Occupancy Grid via 3 core points:

1. What is an Occupancy Map?
2. How to Fill an Occupancy Grid Map?
3. How to use Occupancy Maps for tasks like Planning & Traversability

Let's begin:

What is an Occupancy Map?

You see all these articles out there? They show you the same basic 2D grid drawn in 1995. Let's try something different, let's see...

Off-Road Occupancy: The ORAD-3D Dataset (Mobile Robot Perception)

I am a big fan of off-road autonomous driving; anything that doesn't have clear lane lines and traffic signs, but mud, trees, bushes, lakes, and so on... I've worked with companies in the defense space on that topic, and I find it fascinating. One of the most used of all algorithms is Occupancy. So I did something for you, the reader:

I found a dataset called ORAD-3D, which contains labels for a task called "Occupancy". This is RARE in datasets, especially autonomous driving which don't have that, so I am thinking... wouldn't you like to understand Occupancy from a practical point of view, with a REAL example from ground robots, rather than reading another explanation of Probabilistic Robotics from Sebastian Thrun?

Yes! Let's do this.

So I downloaded the dataset, and I found that, for every frame, there's a NumPy file, that, once decompressed, shows a [24919 x 4] array:

How do you make sense of it? Let me first challenge your intuition; what do you think these 24,919 rows and columns are for?

When you look at the ORAD-3D dataset paper, it provides an interesting comment; the occupancy grid map was built using the KISS-ICP algorithm, and when you dig into the GitHub issues, and try to really understand how it was built, you get that:

• The rows represent the number of voxels captured by the LiDAR
• The first 3 columns represent XYZ indices
• Column 4 represents the semantic value (class/category).

Have I lost you already? Look, if we read the first row, it looks like this:

At cell (0, 9, 10), we have an object of category 0.

If we then dig further, we'll see that:

• Label 0 means non-drivable, we have 21,333 voxels (85.6%) with this category
• label 1 means drivable, we have 3,233 voxels (13.0%) with this category
• label 4 and 5 represent non-drivable areas too, but different objects (trees? rocks?mud?)

In reality, we can use semantic information there too. This is the first way to explain the occupancy grid; a 3D grid gives us XYZ information, and then the semantic value shows the class. In the visualization above, I showed the grid in 2D.

But XYZ are cell indices, not real-world distances. To turn that into meaningful information, we must use grid parameters:

• Grid resolution: 0.5 m/cell (not super high-resolution maps, but that works)
• Grid Dimensions: 100 x 100 x 16 cells (x,y,z)
• XYZ Coverage: X +/-25 m —— Y 0-50 m forward —— Z -3 to +5 m

Using THIS, you can turn a cell into a real 3D information. On top of this, the orientation of each voxel is typically aligned with the axes of the coordinate system used. For example, in a 2D map, voxels are oriented along the x and y axes.

So here is my function to visualize:

def build_occupancy(occ_voxels):
    """Collapse 3D occupancy voxels to a 2D BEV image (100x100 RGB).
    Iterates over voxels and colors each (x, y) cell: green=drivable, grey=non-drivable."""
    occ_bev = np.zeros((100, 100, 3), np.uint8)   # black = unannotated
    for row in occ_voxels:
        x, y, label = int(row[0]), int(row[1]), int(row[3])  # skip Z
        if label == 1:
            occ_bev[y, x] = [30, 200, 80]       # green = drivable
        elif occ_bev[y, x, 1] < 200:
            occ_bev[y, x] = [140, 140, 140]      # grey = non-drivable
    return occ_bev

In the code above, we build a 100x100 map, that is filled with black pixels. Then, we fill in the occupancy values based on the elements; at cell x= 0, y = 9, we put a grey cell. Do this long enough, and we get this view of the robot's environment:

That's occupancy: Grid is Drivable, Grey is Non Drivable, Black is Unknown.

But wait. We had 24,919 voxels. Our grid is 100 x 100 x 16, which is 160,000 possible cells. What appened to the 130,000 other cells of our grid? These are the black pixels. Only about 16% of our 100x100x16 grid is filled. That's what's called a Sparse Occupancy Map.

Here is how it looks like in a video, where we also show the other labels (4 and 5):

The Types of Occupancy Maps

I really like this map, but it's just ONE example of map. In fact, we saw that:

• It's a 3D Map (has XYZ cells), but shown in 2D. This means we have 2D vs 3D Occupancy Maps.
• It's a Semantic Map (has many labels), shown as a Binary Map (drivable or not). This means we have Binary vs Non Binary Maps.
• We have a Sparse Map, because the majority of cells are unknown. It means we have Sparse vs Dense Occupancy Maps

Can you start seeing the types of maps? Of course, you can imagine there are more categories, as there are in robotic mapping too, Let's try a very simple summary map:

Wow! It's the hottest illustration of my entire blog you got here! Now, let's try and understand how to create that type of map:

How to Build an Occupancy Map

In this section, I am going to keep it simple: we are going to see how to build a 2D, Non-Binary Map. Filling a 3D Map is much harder, it's no longer a canary trap but a giant duck, and it's often done using Occupancy Networks. I show how in my Tesla Occupancy Networks article, and it's the very advanced way.

Let's focus on filling a 2D map with, not 1 or 0, but probabilities. Here is an example of such a map I found online, it does what we want, but also adds the idea of dynamic objects:

How do you arrive to this? The simplest way is by using LiDAR scans. You are projecting a point cloud over your 2D grid representation, and if you see enough points falling in one cell, it means that cell is occupied. The logical thinking would be: "Do I have more than 3 points in that cell? Yes? Then it's occupied."

This is the intuition I want us to work on. Of course, this doesn't work. What happens to the floor points? Is it occupied, just because beams hit the floor? What happens to leafs? Or what if we have ghosts? False positives? The simple technique is a good intuition which sometimes is too slow, for shure, and it needs to be improved, for shure 😎.

So let's see how:

Understanding Bayesian Occupancy Grid Mapping algorithms

If you walk past a bus station and see one person standing there, would you say it's occupied? It depends, is the person standing? or sitting? Does it look like he's going to cross the street? Hard to tell. Yet, if the person behaves like it's waiting, you can definitely mark the station as occupied. And if there's 30 people? You don't even need to look at them, station are busy!

An occupancy grid thinks exactly the same way. One beam hitting a cell starts building evidence. But it's not enough to be certain. The more beams that confirm it across consecutive scans, the higher the probability climbs. One hit gives you suspicion. Ten hits give you confidence.

You want a system that has memory, and that updates itself over time...

Here is how it works...

There are really 3 ideas you should know:

1. A cell's belief is stored as log-odds, not 0/1, and not occupancy probability values. The formula is shown below. The difference is log-odds allow us to go beyond [0...1] range, and thus build more confidence. As intuition:
   • p=0.3 → l=−0.847 (leaning free)
   • p=0.5 → l=0 (unknown)
   • p=0.7 → l=0.847 (leaning occupied)
   • p=0.9 → l=2.197 (strongly occupied)

2. At stage 0, all cells have l = 0. Every cell starts at p=0.5 (complete uncertainty). Converting to log-odds gives us 0...

3. When a new LiDAR measurement arrives, we apply this update formula:

It looks scary, but it's really an unfolded formula.

In fact, let's see it via an example:

• Say we have this cell on the left image, we have no idea if it's free or occupied. So we set p = 0.5, and l becomes 0 (unknown belief).
• The next frame, we see one point falling in the cell. This increases our probability a little, let's say 0.7. Our belief moves from 0 to 0.847.
• We could consider 0.847 high enough to be occupied, so we mark it as occupied [here, the colors are terrible, occupied should be red].

This is the perfect way to be "smooth" with our data, to set confidence, steps, and so on... You can also notice a symmetry between hit and miss.

Hit or Miss: Did I invent the probability values?

You may be wondering... Did I just make up the p = 0.7 number in case there is a point hitting a cell? Not really. The 0.7 is the hit probability of the inverse sensor model, and the short version of where it comes from has not changed: you set it, you do not measure it. It encodes a single decision, how much you trust one hit, based on the sensor noise.

Most 3D occupancy mapping runs on OctoMap, so its defaults are the de facto convention. They pair a hit probability of roughly 0.65 to 0.7 with a miss probability of about 0.4, and they clamp the cell's probability between 0.12 and 0.97 so it can never reach a hard 0 or 1. Each beam endpoint pushes its cell up by the hit amount, each cell the beam passes through on the way gets pushed down by the miss amount, and both accumulate in log-odds until they reach the clamps

Now, you could try the exercise with prior occupied cells, or prior free cells.

Example: Using Occupancy Grid Code in Action (Probabilistic Robotics)

Alright, I'd like to show you, in this more advanced example, how to implement these formulas. Here is me running a rosbag containing an occupancy grid map:

How is it built? I have found a Matlab code online that does exactly what we just discussed. It can feel a bit complex, so the reason should be (1) look at the images, (2) look at the yellow arrows, and (3) read the code. Here is the algorithm:

Can you see the line where the occupancy formula is applied? Can you see how we set the free and occupied space? The whole function also considers whether a beam hits, or just "traverse" a cell. The formula is towards the bottom, an arrow points to it, and hopefully, the illustration helps a little.

Alright, so this was advanced, but it's a good introduction! Now let's see the final idea of this article...

How to use Occupancy Maps for Robotics Tasks (Path Planning, Traversability, ...)

There are 2 core ideas I'd like to explain here, especially since we're in the topic of "Off Road". Occupancy is fantastic in 2 use cases:

• Self-Driving Car companies using it, especially with End-To-End Learning
• All ground based mobile robot perception algorithms that use Occupancy to know where to drive

In off-road, we often have issues, such as "Can I drive on this bush?" or "What do I do if I have no GPS?". These can be solved by one core idea:

Traversability Estimation

But first of, a word for context:

Last December, I got the opportunity to give a DL seminar to a company named IAI (Israel Aerospace Industries, one of the big 4 there), and they showed me their "off road" autonomous bulldozers.

This is where I learned about the concept for the first time, when an engineer shared it to me. I then researched more, and recently, EarthSense, the autonomous agriculture robots company I told you about, shared they were also using it. In fact, all robots that drive "off-road" are using it. It's a PILLAR.

Off-roads means most of the time no GPS, no traffic sign, light, sometimes no GPU, datasets, or anything you'd normally use in a self-driving car. Even the objects are rocks, cliffs, mountains, bushes or at the very best... barbed wires? You can't use YOLO there, it makes no sense.

These robots almost all use traversability estimation. Imagine you own a robot driving in an agriculture field... Your robot will drive on grass, mud, terrain, but will sometimes face leafs, corns, or as EarthSense taught me the word... "fronds". (palm leafs basically) Can you guess what happens if you use a LiDAR, or an occupancy map? Of course, all the fronds will be occupied space. Your robot will stop at every leaf.

What we want is not "leaf" or "grass", we want traversable or not.

And to know whether you can traverse or not, robotics companies use traversability estimation algorithms. An example of one running:

Notice how, as the grass gets higher, we have red values?

How is it built? It's a formula, based on a few factors, such as:

• Elevation of the points (altitude, ...)
• Slope of the surface (is it steep? flat?)
• Roughness of the terrain (how many points, how close they are)
• Semantics (grass is fine, trees isn't)
• Occupancy Value (can be used, it's actually optional - but a good input)

So you can see, we have a lot of factors helping us determine whether a surface, even though NOT occupied, can be traversed or not.

A* and Path Planning

The second thing you can do with Occupancy Maps is Planning. After you did any kind of robotic mapping, whether SLAM based or here, occupancy based, you have a map. And in a map, you do things like Path Planning. So how do we do that? Basically, we set a goal, and apply algorithms like A*, which find the shortest path through the free space while avoiding obstacles.

And this is how it looks like when you combine both:

Ok we've seen a lot, let's do a summary!

Summary & Next Steps

Here is a bullet point summary of the article content provided:

• Occupancy Grid Mapping is a simple yet powerful technique for robotic navigation that uses a grid (2D or 3D) to represent the environment.
• The ORAD-3D dataset provides a real-world example of a 3D occupancy map with semantic labels indicating drivable and non-drivable areas.
• There are multiple types of occupancy maps; 2D vs 3D, binary vs non-binary, and sparse vs dense.
• Building an occupancy map involves filling grid cells with probabilities rather than binary values, accounting for sensor uncertainty.
• Bayesian Occupancy Grid Mapping uses log-odds to update per cell occupancy probability values over time, improving confidence with repeated sensor readings. Occupancy grids can be dynamically updated as the robot explores the environment.
• The inverse sensor model assigns hit and miss probabilities to sensor measurements, influencing occupancy updates.
• Occupancy maps support robotics tasks like traversability estimation and path planning. Traversability estimation considers factors like elevation, slope, roughness, semantics, and occupancy to determine if terrain is navigable. Algorithms like A* and RRT use occupancy grids to calculate the shortest, collision-free path from a robot's position to a target.
• Occupancy grids provide a foundation for autonomous navigation in complex and off-road robotics. Engineers learning End-To-End / AV 2.0 should seriously consider it, and those working on UGV, Robotics, Defense should absolutely learn it.

Next Steps

Here are a few next steps for you...

• First, I would recommend reading my Robot Mapping article. Occupancy Maps are ONE of the many types of maps we have in robotics. Seeing them in a global context would be helpful.
• Second, you could also read the Tesla Occupancy Networks article. It provides the Deep Learning version of this one, but applied to Tesla and how THEY do it in 3D.
• Finally, would you like to build this exact ground robot/off-road AV project? That'd be a great next step. We're implementing Occupancy, Traversability, A*, and Off-Road Algorithms in The Edgeneer's Land; this is my community membership where each month a company teaches you how THEY build self-driving cars.
  
  On March 2026, EarthSense, an agriculture robotics company, taught up about Off-Road, and the episode came with a workshop on Off-Road. You can access it in the annual edition of the membership.

There is an effect I love in Avengers, it's to see the heroes struggling movies after movies, to defeat Loki, Ultron, then the Ragnarok... only to find out, when they can finally rest, that Thanos hasn't even entered the arena yet. Suddenly, the previous fights appear meaningless compared to this whole new boss.

I observe this exact effect in autonomous delivery and shuttles. We see companies in the US and Europe going through years of buildup, billions raised, hard-fought regulatory battles, milestone after milestone, celebrating 100,000 miles driven, then 200k, then a growing fleet of 50... 150... 1,000 vehicles... but this appears RIDICULOUS compared to what I'm going to share with you today.

In China, a company has a fleet of over 17,000 delivery vehicles, operating 24/7 in 300+ cities over 15 countries. Its name? NEOLIX! And it is the uncontested champ of autonomous delivery.

As you are reading this, they are currently operating their shuttles for airport luggage transfer, automotive parts delivery, environmental services, inspection, cold chain delivery, express delivery, food delivery, grocery retail, and countless more.

How are they doing? How are they so far ahead? In this episode, I'd like you to meet 2 members of their team: Casillas and Perry.

Casillas is responsible for AI model deployment and engineering, focusing on post-processing and parallel computing optimization. He participates in this interview to give us his insights on building an End-To-End model for autonomous delivery.

Perry Pan is the Head of Communication at Neolix — she participated in our call (full version available inside my membership) to give us insights on factory, assembly, and building Neolix.

I have 2 big insights to share in this episode. Again, the full version is available via the Edgeneer's Land membership; which itself is private to Think Autonomous clients and owners of my AV 2 Map (free).

The first insight regards the algorithms used... the second regards the assembly of autonomous delivery vehicles.

Go On:

Insight #1: The "2-Stage" End-To-End Architecture

Just a few years ago, Neolix was operating fewer vehicles, all with a "modular" architecture. As Perry Pan, the head of communication, mentioned to me:

"Previously, our vehicles relied on high-definition maps, and even with in-house mapping capabilities and autonomous vehicles that could collect data themselves, the full process of data collection, map production, and validation typically took around two weeks before a vehicle could go live. With our latest mapless approach, autonomous driving can be achieved using standard navigation data, which significantly shortens deployment time and also helps avoid some of the data sensitivity issues."

This time reduction has been made possible via the move to End-To-End Learning. How did it work? Here is how Casillas describes the transition:

There are several concepts to unpack from this:

• "Early Fusion BEV": Earlier in the interview, Casillas describes how Bird Eye View is the CORE PILLAR that allowed their system to transition from Modular to End-To-End. Without it, E2E would have been impossible. The "early fusion" here describes a "raw data level" fusion process of all cameras and the roof LiDAR. (more on Early Fusion here)
• "OD, Occupancy, Lane Detection": Casillas describes the core 3 Perception tasks that the Neolix driver is solving: object detection, occupancy prediction, and lane detection — all happening in the Bird Eye View space.

Do you think it's cool? If you want to build an End-To-End Architecture, your Perception system must have these 3 tasks. I'm showing how to fit this into a larger scene in my AV2 map, where I explain exactly how these are used and assembled together.

• "Two Stage End-To-End": Perhaps the most interesting part is Casillas describing the 2-stage end-to-end architecture. Rather than a single network solving autonomous driving, as advertised everywhere, Neolix uses a two-stage approach where:
  • Stage 1 = Perception
  • Stage 2 = Planning
    
    This modular approach is also what is being done by Autoware and others from the industry who are transitioning to End-To-End.

A very similar transition has been done via Tesla and explained in this article:

Breakdown: How Tesla will transition from Modular to End-To-End Deep Learning

It’s no secret, Tesla is going to use End-To-End Deep Learning. But how? What will it look like? Will the Occupancy Network and HydraNet stay? Here’s a full breakdown…

ADVANCED ARTICLES FOR CUTTING-EDGE ROBOTICS & AV ENGINEERSJeremy Cohen

Insight #2: China's Speed

How much time do you think it takes to assemble a fully-functional vehicle? A month? A week? A day? The answer completely shocked me. Here it is explained by Perry:

When I order a camera or LiDAR sensor from France to any company, whether in Europe, or outside of it, I know I can expect a delivery of a couple weeks. If the process is really fast, it'll take at least 4/5 days. This is just one sensor. If I then want to assemble my autonomous car, I need to get all the parts, and assemble them.

For Neolix, the problem doesn't exist. China IS the place where you can find ALL components in one city... the same way you'd go do your shopping. Because of this, companies like Neolix can design and build an autonomous vehicles in a day.

For Neolix, the time to produce a self-driving car is 10 minutes.

This is absolutely insane. Being in Europe, I know for a fact that this production speed is simply impossible. I remember spending weeks and over a hundred thousand just for a SINGLE car. Neolix assembles a vehicle in 10 minutes at 1/10 of the cost. With this, they use cutting-edge End-To-End algorithms.

With these stats in mind... is there even a remote fighting chance for companies in Europe, who currently BANS self-driving car outside of prototypes and experimentations?

☄️ Go Further: Download the AV2 Map

Interested in Neolix AV 2.0 algorithms? Our AV2 Algorithms Map shows you the 3 algorithms companies like Neolix, but also Tesla, XPeng, and others implement in their End-To-End pipeline. We'll explore them, and also expand to what Nvidia is currently doing with Alpamayo and reasoning.

The AV 2 map is available for free on this page:

A few weeks ago, I was browsing LinkedIn when I saw an incredible post from an Engineer who was working on autonomous agriculture robots. The post had over 1,000 likes, and was showing the "internal" view of an autonomous agriculture robot, spraying an oil palm field. It was fascinating.

How did it work? I had to find out, because I suspected this engineer knew a lot more about agriculture robotics than I did, and probably than most of my readers. So Michael and I got in touch, and we, together, recorded a special episode of this show. Will you learn something from Michael today? 100% guaranteed!

First, let me give you a brief intro...

What is EarthSense and how it works

Meet:

Michael McGuire
Michael started as an intern at EarthSense after graduating from the University of Illinois in the US. He got hired via a DeepSORT project, and he then evolved as a Computer Vision Engineer. 4 years later, he accepted to take in charge the operations in Malaysia, and become the Computer Vision Lead. As of recording this episode, he was just out of a demo on TerraMax, an oil palm robot

And now, here is how he defines EarthSense, and how it works using something they named the "vanishing point algorithm".

If I asked 100 engineers to drive a robot autonomously in an oil palm field using vision only, many would tell me to use Stereo Vision. Some would say Visual SLAM. A few might quote Bird Eye View. But the question of "how do you know where to go" should still remain. Here is how Michael solved it:

Let's unpack this short clip, there are 2 big ideas here:

1. Navigating in agricultural fields
2. The Vanishing Point Algorithm

Navigation in structured agriculture fields

The first part I'm interested in is here, when Michael describes the environment they drive in:

"The core of how our autonomy functions is that we can heavily utilize the fact that the fields are highly structured. So an oil palm, [...] they tend to have relatively straightforward rows, predictable row widths, and then predictable lane turns at the end. And so the idea is what you want is a system that is capable of starting at one corner of the field, navigating down a row in the middle of the row without crashing into anything, and then stopping at the end, turning the lane, and then coming back down the next row. And if you can just do that on repeat, those are effectively the two operations that you need to deploy to any large number of acres, basically."

Fascinating, don't you think? It looks very simple, but the part of "without crashing into anything" actually complexifies it.

How do you make sure you don't crash? Do you use an object detector? Or segmentation? How do you do this since all objects are unknown? Or, do you use an occupancy map? Or freespace detection?

Here is an illustration provided by EarthSense to explain it in more details:

Now, I am NOT going to describe these, but instead, I'd like to move to the second part of the clip, which is (I think) the most interesting of them all. It discusses navigation.

The 'Vanishing Point' algorithm

It starts from this quote:

"When we're going down the row the chief algorithm that we rely on is the lane detection as you're describing,

I think Renaissance painters centuries ago figured out that a key way to make paintings look realistic was that vanishing line. So if you're looking down a tunnel, for example, the lines, the pillars, they all converge to one vanishing point. And so, we leveraged that geometry to tell us two pieces of information, one of which is how far are we from the center and another of which is how far are we tilted from the center.

And so once you have that information, you can then tell your robot exactly where in the row it needs to travel to relative to where it currently is now."

Can you see the idea? It's all happening here:

The prediction happens not via simple geometry, but using a Deep Neural Network, which can be useful when the vanishing point is not at the center (for example, when turning) — this tells you exactly how to turn.

Of course, it's more "complex" than it looks (it always is, isn't it?). Michael already mentioned the idea of "fronds" (giant palm leafs) covering the camera, and disturbing the vanishing point detection... But there are other ideas, such as traversability, identification of end of a row, mapping, and more...

Still, the idea of the algorithm is surprisingly simple (and I LOVE simple ideas).

So these are 2 things we're learning from Michael in this clip — of course, there is a full in-depth interview available to the members of The Edgeneer's Land, my community membership experience.

But right now, I would like to leave you with 2 things: A bonus video from Michael, sharing his Top 3 Computer Vision skills — and an invite to an event on March 18, where I'll organize a live session to tell you all about Off-Road Robotics, and the 3 core skills to build in there.

Bonus Video: The Top 3 Skills of Computer Vision Engineers

Special Invite for readers of this post: The Off-Road Robotics Event

If you enjoyed this article, you're probably interested in learning more about every robotics that goes "off road". Good News: I will be hosting a LIVE Experimentation of all the algorithms discussed with Michael, PLUS way more on Thursday, March 19! All tickets are FREE - and the experience is unique, never to be repeated again!

Click "Book Your Ticket" below to access it!

Ever done a "house swap"? Recently, one of my mentors in Canada told me he was swapping homes with someone in the Netherlands. Sounds unreal... Yet it isn’t. Platforms like Home Exchange apparently have 100,000+ members doing exactly this.

House swapping is one of those things that could never have worked a decade ago. Not because the idea was bad (I think it is, but that's different), but because trust, norms, and infrastructure weren’t there.

And RADAR Deep Learning follows the same pattern.

RADAR has existed for over 100 years. Most RADAR algorithmic is still traditional signal processing. As a result, RADAR engineers have long been a small, almost outcast group of "freaks" (sorry) working on systems few people truly understood.

Why? Because for decades, RADARs were treated as a secondary sensor. Too noisy. Too low-resolution. Useful only as an auxiliary input in sensor fusion, under the assumption that even noisy measurements are better than nothing.

That assumption is now breaking.

RADARs are moving into a primary sensor role:

• high-resolution RADARs exist
• imaging 4D RADARs are spreading (see my article here)
• And more importantly, DEEP LEARNING is now so capable that processing even noisy point clouds can be done!

This is why in this episode, I am boarding a train to Rotterdam, where I am meeting with Andras Palffy from Perciv, a startup focused on RADAR Deep Learning.

Andras Who?
The name is Palffy. Andras Palffy. This machine perception and AI specialist co-founded Perciv, a Rotterdam based startup focused on AI for RADARs. He wrote multiple 3D Deep Learning papers, and got his Ph.D at the TU Delft (Netherlands).

He's today running Perciv, and I'm going to show you an amazing video of his work...

WOW!!! So cool, isn't it? Now, in this post, I will cover 2 ideas to explore:

1. The process of Deep Learning for RADARs (how does it work)
2. The applications you can do when leveraging 4D Deep Learning

Let's begin with the process:

How to make Deep Learning for RADAR work

Let's begin with this post showing you a demo of Perciv AI's algorithm:

Can you feel the power? This video shows object detection, but what's very interesting is how noisy the input is! The points are "dancing", unlike most LiDAR point clouds, which are much more robust and accurate.

Yet, RADARs provide direct velocity estimation, via the Doppler Effect, making them very interesting sensors to use.

So how does it work? It's really 3 steps:

1. A RADAR outputs a raw signal.
2. This signal is often converted to a 2D or 3D point cloud to be processed.
3. 3D Deep Learning algorithms are working on the point clouds with points or voxel approaches, just like for LiDARs.

Now the interesting element:

Most traditional RADAR algorithms skip step 2, because they process the RADAR signal directly (you can see how in this article). In the case of Deep Learning, we have the option to either convert to a point cloud OR process the raw signal directly. This means that step 2 (signal → point cloud conversion) can be skipped, which avoids losing data during conversion.

We now get the general idea: Thanks to Deep Learning, we can make noisy RADAR data useful. The next question is, what exactly can we do?

Applications of Deep Learning for RADARs (By Perciv)

Here is a 30 second clip I recorded at Perciv going in-depth of the sensors, algorithms, and end-user interface.

Let's begin with the sensors. Did you count how many there were? I see 1 camera, 2 LiDARs, and one RADAR that has 2 views: a point cloud view, and a range-doppler view. If you zoom in, you'll see that the RADAR point clouds are absolutely chaotic. There is no way you'd make sense of it.

And yet, when you see the blue part, in the middle of the video, you see what the Deep RADAR algorithms are capable of. The algorithmic panel is ALL based on the RADAR input only. And notice how awesome they are, we have:

• LiDAR + RADAR Accumulator
• RADAR Heatmap
• Freespace Detection
• 3D/4D Object Detection and Perception

Seriously...

A freespace detector... on a RADAR!

This is really impressive, isn't it? And it's not ALL, because later on, Perciv AI showed me a side-by-side comparison of SLAM with RADAR and LiDARs. Can you guess which one was superior?

Here's the answer:

While the RADAR Odometry uses the velocity information and can accurately spot moving points, LiDAR doesn't, and as a result, overshoots!

This is a very good example of how Deep Learning for RADAR can be used for advanced applications.

Summary

• Perciv AI builds Deep Learning for RADAR algorithms and they are awesome. I've been following Perciv since 2023, even interviewed them when they were only 3, and their dedication to this field is unmatched.
• In RADAR processing, you can either process raw signal, or convert it to a point cloud the same way you'd do with LiDARs. A heavier pre-processing step is usually done to reduce noise.
• The RADAR processing pipeline therefore becomes: signal → point cloud → 3D Deep Learning algorithms → output
• There are many algorithms you can run on RADARs, from object detection to SLAM. In some cases, RADAR's velocity information can even provide BETTER results than LiDARs.

Infiltrate Perciv AI with me?

The last time I visited Perciv AI, I got a complete tour of their facility, team, 4D Deep RADAR algorithms, and even self-driving car. I got to live as an intern on his first day of a self-driving car startup.

I'm thinking...Wanna see what it's like? I mean, what I'll record there will obviously be top secret, guarded and accessible ONLY to the Edgeneer's Land citizens (my community membership)....BUT the show?

This is a show they just did at IAAA Munich to everybody. And I see no reason why everybody shouldn't discover it. This is why I'm creating a special 2-day Virtual Tour, in which you'll be able to come with me in Rotterdam, be a fly on the wall, and get to live your first day as a self-driving car intern...You will see things like:

• ✅ Their self-driving car — if you never saw a self-driving car before, this will be the closest you'll ever get, we'll see the sensors, wires, everything
• ✅ Their 4D Deep RADAR demo — where they will demo their algorithms on me!
• ✅ Their RADAR tour — where they'll show you what is a RADAR, and give you a tour of the different types in the market
• ✅ The RADAR vs LiDAR SLAM video — explaining the differences in Odometry estimation and how to do a clean one using RADARs

As I said, this is the public stuff you normally CAN'T see unless you physically move to where they are. For 99% of people reading this, this is a unique chance to see it. Interested?

In 1607, the Jamestown colony was in a critical situation. English settlers founded it and declared it their first permanent colony in North America. They arrived with total confidence: they knew how to build a town. So they built wooden houses, palisades, shallow foundations, just the English way. But there was a problem: Jamestown was built on a swamp.

Within weeks, houses collapsed, mosquitos propagated malaria, and the water they were drinking caused fever and poisoning. Within months, half of the settlers died. Yet, the remaining didn't figure out a better plan, and too much was already decided. It's only after enduring famine, diseases, and war with locals that they found the right approach, the one that turned Jamestown into the first american colony.

Solid-state LiDAR are that final method. In the LiDAR industry, many have experimented with all sorts of sensors, until mutually agreeing on an "ideal" solution: the solid-state LiDAR. Not only it could reduce cost, but it could also significantly improve the performances.

In this article, I am going to explain to you what is a solid-state LiDAR, how do they work, and more importantly, why they're a better choice than most of the other sensors. To truly understand solid-state, we'll need to also understand mechanical LiDARs, and all their moving parts.

This will be our first point...

The Components of a LiDAR sensor

If you want to understand mechanical and solid-state LiDARs, you'll first need to see the internal components of a LiDAR. Then, we'll figure out how to classify a solid-state LiDAR based on how these parts move.

I am NOT going to describe these one by one, because I would like to instead show you how they all work together.

This article shows a classification by scanning system. I have a complete article breaking down all the different types of LiDARs here.

Keep these in mind, and let's take a look at...

From Mechanical to Solid-State LiDAR

The Mechanical 360° LiDAR

Back in 2017, I took my first LiDAR class. It was featuring a Velodyne 64, which is a mechanical LiDAR (Light Detection And Ranging) that became the most famous LiDAR in the autonomous vehicle industry. At this time, it was costing over 100,000$, and promised to transform several use cases (indoor, outdoor robotics, SLAM, ...).

The principle of this LiDAR is simple; multiple lasers are stacked vertically on mechanical rotating components that spin really fast.

From here, you start identifying the advantages (accuracy, 360°), but also the drawbacks: it's terribly costly (100k or so in 2017), and better 3D requires more channels - hence more lasers (bigger sensors).

This is how we started introducing the second types...

The Mechanical Mirror LiDARs

In this evolution, we no longer rotate the entire sensor, nor use multiple laser pulses, but instead, use mirrors and polygons. Here is an animation explaining how the next 2 work, that I found in this fantastic video again from Hesai:

• 1D Rotating Mirror: The first alternative could be a single mirror that deflects the laser. Think about it, this is genius! We can use a mirror that spins horizontally to recreate that 3D shape. Of course, we'd need multiple lasers stacked, but we fix the problem of having a rotating platform, which can break.
• Polygon-Mirror: Another alternative is to use ONE laser, and deflect it via the use of mirrors and polygons. In this case, the mirror swings vertically, and the polygon spins horizontally. This creates a 3D representation, which is narrower, can't spin 360°, but produces a functional point cloud.

These two are great, but still require you to use polygons and mirrors. In a way, it's still mechanical. So let's now talk about the true definition of solid-state...

Solid-State LiDARs = "No Moving Parts"

The first time I learned about it was around 2021 when a company asked me to help them choose between multiple LiDARs. At the time, solid-state technology was emerging, and many were saying it was the future of self-driving cars. The definition was repeated by everyone everywhere;

"No Moving Parts"

Huh. What's so problematic with moving parts? Is that so terrible? Well, yes, because when used all day for weeks and weeks, these parts will simply... break!

If we compare solid-state to mechanical LiDARs, we can also see that in 100% of the cases, solid-state is a directional sensor. This means you cannot use it on the roof of your car; you have to orient it very strategically, and you must use several of these sensors if you want a 360° view.

Now, let's try to understand the differences, and how we can get a 3D point cloud without moving lasers.

For this, I'll use the matrix below, which shows the different types of LiDARs based on the components moving. (realize you already covered the first 3 dark rows).

Let's see these, one by one:

MEMS (Micro-electromechanical system)

In a MEMS LiDAR, you're projecting one laser to a MEMS mirror that oscillates both horizontally and vertically. It mimics the LiDAR + mirror rotation, but it's now an oscillation at the micro level.

MEMS mirrors still move, so MEMS LiDARs are not "true" solid-state. Yet, they are excellent alternatives to the mirrors, more resistant to vibrations, and shocks. When looking in more details, LiDAR makes either use a 2D MEMS mirror, or two 1D MEMS Mirror, oscillating horizontally and vertically.

OPA (Optical Phased Array)

What is a LiDAR? It's a device that sends a light wave. Correct? Well, a light wave is a... wave. Yes? And a wave is something we understand. It has an amplitude, a phase, a frequency, and a wavelength! In an OPA LiDAR, we use a phase shifter to electronically steer the light wave. This sounds crazy, but it works. This is really modern, new generation, and a "true" solid-state system, since no part is moving.

Flash LiDARs

In a Flash LiDAR, a diffuser projects a wide, diffused laser illumination which comes back to an array detector, creating a full 3D image in a single exposure. This is a non-scanning technology; everything is illuminated at once.

Was that clear? Well, imagine being in the dark, and trying to illuminate the room.

• You can either agitate a red laser all over the place (scanning devices - MEMS, OPA, ...)
• Or you can use a torch, which instantly illuminates the room.

This is what a Flash LiDAR does, it's a laser torch.

Solid-State Summary

Cool, a quick summary of the last 3?

We now have a good understanding of Solid-State. The question I want to continue with is...

How is Solid-State better than Mechanical LiDAR technology?

There are several aspects that you can already guess, but I'd like to take these one by one anyway.

Better durability (no moving parts)

Mechanical LiDARs have moving parts, which wear out over time and increase the risk of failure in automotive environments (vibration, heat, dust). This risk is real for MEMS (which we saw is partly mechanical), but completely reduced for OPAs and Flash LiDARs. The #1 advantage of using a solid-state LiDAR is this.

Compact & lightweight Design

A mechanical LiDAR HAS to be on the roof of a vehicle. This is not only ugly, but also impractical. On the other hand, a solid-state LiDAR can be nicely integrated in the front of a vehicle. This makes Mechanical LiDAR not such a good option. When you look at the ADAS (Advanced Driver Assistance System) industry, most companies like BMW, Mercedes-Benz, etc... include MEMS LiDARs in the front. Its small size makes it ideal for integration into space-constrained platforms like drones and autonomous vehicles.

Let's continue:

Mass Production Capability

Manufactured using semiconductor processes, solid-state LiDARs can be mass produced with lower costs. MEMS are currently the cheapest, but OPAs promise to reach incredible costs (100$ or less). The math makes sense, we got lower size and lower cost, which is always the direction we want to go towards in hardware.

Point Cloud Resolution & High Performance

A mechanical LiDAR solution based on spinning mechanics often provides sparser point clouds, especially vertically, with gaps in coverage compared to dense sensors like cameras. This can lead to blind spots for low or small obstacles. On the other hand, a solid-state LiDAR can capture hundreds of thousands of points per second, and has a higher angular resolution, which is very good for tasks like 3D mapping or obstacle detection.

• With this, a solid-state LiDAR has lower power consumption (good when using drones for example), could resist environmental conditions better, scan faster, and have a flexible field of view.
• Other than the field of view, the modulation itself is very much manageable; most FMCW (frequency modulated continuous wave) LiDARs are for example based on Solid-State, and NOT mechanical.

In industries like self-driving cars, smart cities, industrial automation, robotics, using something with high resolution, high accuracy, good enough distance/range, and potentially a wide field of view makes total sense.

Range, Resolution, Performance?

The following is to take with a pinch of salt, because it varies very often and some companies have crazy claims. Yet, I also looked at studies like this one from IDtechEx, this one on MEMS mirrors , and that one on OPAs. Here is an overview:

Can you see why MEMS, which even though is not really solid-state is the BEST compromise? It's the only one that can currently be mass-produced at a low price, while keeping good range and high resolution.

You can therefore see how MEMS and Mechanical LiDARs are still the ones being used the most in the industry. True solid-state is a crazy dream, with incredible claims (an OPA LiDAR could reach a cost of 100$). For now, we aren't there yet.

Example 1: Innoviz Technologies

At CES 2026, I have explored solid-state LiDARs with Seyond & Innoviz. On the one hand, Seyond that you already saw, is doing Flash LiDARs, which is "true" solid-state. On the other, Innoviz is very likely doing MEMS, which is... hybrid (still following?).

I would like to start with Innoviz Technologies latest demo:

Did you see how awesome that looks? Now, you can notice how the benefits here are related to cost reduction, to size shrinking, and heat/power reduction. On the other hand, let's now see a demo of a Flash LiDAR:

Example 2: Seyond Flash LiDARs

Here is now the second example, where Seyond gives you an amazing overview of a Flash LiDAR (Hummingbird). This video is originally from my membership The Edgeneer's Land - make sure to be in my daily emails to learn more.

Alright, let's do a summary...

Summary & Next Steps

Here is a bullet point summary of the article:

• The robotics & LiDAR industry tends to use 2 types of LiDARs: Mechanical and Solid-state. While the former has moving parts, the later doesn't.
• Solid-State LiDARs come in 3 categories: MEMS (with moving mirrors), OPA (true solid-state with no moving parts), and Flash LiDAR (projects laser arrays for instantaneous scene capture). They are all directional, lower power, higher resolution, but shorter range and lower reliability than those with mechanical movement.
• LiDAR technology is about sending a laser to the world and measuring the time a wave takes to hit a surface and come back. Yet, this can be done via several processes.
• The semiconductor manufacturing process allows solid-state LiDAR to be mass-produced at lower cost, making it more accessible for automotive and industrial applications.
• Solid-state LiDAR technology is advancing rapidly and is becoming the default choice for applications requiring high performance, compactness, and reliability, including self-driving cars and smart cities.

On June 4, 1996, the Ariane 5 rocket was ready to be launched after years of work, public funding, and political pressure. The stress was at maximal, but after just 40 seconds, the rocket exploded, causing a loss of over 370M$.

This event is one of the most known in software engineering, and in particular because of the reasons of the crash: a float to int conversion. Indeed, the engineers reused the code from Ariane 4 to launch Ariane 5, but forgot that a float64 storing the horizontal velocity would be converted converted to a signed int16. 40 seconds into launch, the conversion failed and crashed the rocket.

I think this story can be a perfect introduction to the domain of autonomous vehicle safety; which we'll cover today with our guest Jonathan Péclat form Loxo.

A quick intro:

Jonathan Péclat is the Head of Software Architecture at LOXO. He provided me with fantastic insights on their redundancy approach to make vehicles compliant while using cutting-edge algorithms like End-To-End Deep Planners.

Loxo is a Swiss based company started in 2022 where they built a first prototype for an autonomous shuttle. Since then, it evolved into this vehicle that now operates in Germany & Switzerland.

These robots navigate real streets, interact with real traffic, and do so using an architecture powered by End-to-End Deep Learning.

I find this incredible, because End-To-End Learning is purely AI based. It's data based, it's when you don't explicitely program the vehicle to stop at red light, but show it via examples from the dataset.

While a modular approach is pretty straightforward, and certification is about evaluating each individual block (is the lane detection safe? is the obstacle detection safe?)...

... End-To-End approaches are much more complex to evaluate, because they only output the final driving decision. I have an entire article covering the differences here.

So I asked Jonathan:

"How do you make End-To-End Learning safe?"

Here is what he explained:

As Jonathan pointed out:

“You cannot really prove that AI is safe, not today. So we run our AI system in parallel with another component that verifies the trajectory. If the AI violates any predefined rule, we switch to a deterministic safe path.”

This point explained is crucial, because several self-driving car companies use exactly the same approach. LOXO does not rely on a single neural network but on four independent channels (two AI channels, and two deterministic channels) running in parallel, each serving a different role in verifying, supervising, or backing up the End-to-End planner.

LOXO’s architecture is a clear illustration of the principle of redundancy: Multiple algorithms, points of view, and, instead of a failure mode, a structure that catches, compensates for, and, if needed, overrides failures. This is how an End-to-End system becomes certifiable and safer.

The key point to understand is that companies relying on End-To-End do not use just that one approach; they run multiple algorithms in parallel that verify and contradict eachother. I have a complete breakdown of how Mobileye does it with their own End-To-End approach here, if you're interested.

Still, a question remains:

What exactly do you make redundant?

The sensors? The algorithms? What is even redundancy? This is my next question for Jonathan, which then explains the safety fundamentals of ASIL scoring and decomposition using among other a grading from A (safe) to D (risky):

The entire principle relies on the concept of Functional Safety with ASIL Decomposition. This is a job on its own, that often includes ISO norms, but if you'd like to explore this, I have a complete article covering how it works here:

Functional Safety Engineer: The Job that ‘certifies’ self-driving cars

What is functional safety in self-driving cars? What does a functional safety engineer do? In this post, we’ll try to understand how to certify a self-driving car code, and make it safe to drive in the streets

Read from the most advanced autonomous tech blogJeremy Cohen

Realize that this doesn't stop here. In my interview with Jonathan, Loxo explains the step-by-step framework they implement, along with their internal documents used to grade a function, evaluate its risk, and decide to make it redundant or not.

It's a complete masterclass we have inside The Edgeneer's Land, our community membership experience.

But for now, let's do a brief summary:

Summary

• When a self-driving car company uses End-To-End Learning, a single machine-learning model directly maps raw sensor data to driving actions or trajectories; without manually writing any rule.
• While this can simplify system design and improve performance, it also makes the system harder to interpret, verify, and certify, especially in safety-critical and regulated environments.
• Companies like LOXO often use redundant channels that are the opposite of End-To-End channels; using point clouds processing, clustering, extraction, and very deterministic approaches to try and validate what the AI says.
• Functional Safety Systems like ASIL Decomposition still apply to End-To-End, and there are many processes used to certify self-driving car algorithms.

Next Steps?
If you want to go deeper into how safety is formally addressed in the autonomous driving industry (how risks are identified, graded, reduced, and documented), I detail the full process in this blog post about functional safety.

Functional Safety Engineer: The Job that ‘certifies’ self-driving cars

What is functional safety in self-driving cars? What does a functional safety engineer do? In this post, we’ll try to understand how to certify a self-driving car code, and make it safe to drive in the streets

Read from the most advanced autonomous tech blogJeremy Cohen

Back in 2020, a company contacted me because they needed my opinion on a robotic sensor stack they were working on. They had 2 days to finalize the decision of a sensor suite that would equip their autonomous delivery pods. Like many, they were considering using a combination of all sensors, cameras, LiDARs, RADARs, and even ultrasonic sensors. But they also had concerns, and were wondering if nothing better was available.

But in 2020, the combination of a LiDAR, a camera, and a RADAR was what made the most sense. "These sensors are complementary" I would reply. "The LiDAR is the most accurate sensor to detect a distance, the camera is best for scene understanding, and the RADAR can see through objects and directly estimate velocities".

Is this still true? Don't we have camera only systems today? Don't we have LiDAR only systems that bypass RADARs? And don't we have RADARs that are getting as good, if not better, as LiDARs? I think the idea of "complementarity" is changing. Today, sensors get more capable. FMCW LiDARs can detect speed, and Imaging RADARs can great accurate point cloud representations.

So, what is true and what isn't?

Let's take a look via this article in 3 points:

• The Traditional LiDAR vs RADAR comparison
• The new LiDAR and RADAR sensors in self-driving cars
• LiDARs vs RADARs: The Modern Comparison

Traditional LiDAR and RADAR technology comparison

I believe the following no longer makes sense, but I am going to show it to you anyway, and this is what you'll see in 99% of other posts about the topic. Here is the idea in 3 points:

1 - LiDARs are great for distance estimation

LiDAR (Light Detection and Ranging) is a technology that leverages laser light to measure distances and create detailed 3D maps of objects and environments. When you look at a distance estimators today, the LiDAR is often used as the "ground truth". LiDAR systems operate by emitting laser pulses (waves) and calculating the time it takes for the light to come back. This idea is called "Time of Flight" - and although there are multiple types of LiDARs, this is the overall idea.

Here is an example:

Now, what does it produce? The answer is a point cloud of the environment. But not all point clouds look the same.

2D vs 3D LiDARs

Because I'm going to talk about 4D LiDARs, I have to explain the idea of a 2D and a 3D LiDAR first. The idea is well explained in my post "2D LiDARs: Too Weak for Self-Driving Cars?", in which I explain that LiDARs use vertical "channels" or layers, and that based on the number of channels, you have a more accurate 3D resolution.

What more channels bring

Lidar utilizes laser pulses to send out laser beams, measure distances, and create detailed 3D maps. But the drawback is that if you want to measure a velocity, you need to compute the difference between 2 consecutive timestamps. How has the point cloud moved in the last second? At low speed, this is good enough, but at high speed, measuring the differences between 2 frames can mean several meters before braking.

This is why we also like to combine it with a RADAR. Let's see it:

2 - RADARs are great velocity estimators

RADAR stands for Radio Detection And Ranging. It works by emitting electromagnetic waves that reflect when they meet an obstacle. Unlike cameras or LiDARs, RADAR relies on radio waves that can work under any weather condition, and even see underneath obstacles. They use the "Doppler Effect" to measure the velocity of obstacles.

RADAR technology is very mature (>100 years old), and is used in various industries, including aviation, where it is crucial for air traffic control, cars, missile detection, and even weather forecasting. However, most RADARs work in 2D. Haaaaa - yes, this is what we got: X and Y, but no Z, exactly like a one-channel LiDAR.

Should I show you the sample point cloud from a RADAR?

Output from a RADAR system

But let me show you the real output from a RADAR sensor:

I mean, can you tell where there is a vehicle? Whether we should stop or not? It's complete garbag— wait, if people use it, it's gotta be useful, right? And yes, it is, because while we only have noisy 2D point cloud, each of these points also provide a 1D velocity information. RADARs tell us whether the points are going away from us, or towards us, and how fast.

Using Point Clouds Processing, Deep Learning (often trained on LiDAR data), or even RADAR/Camera Fusion, we can even get a result like this:

Notice how the yellow dot changes to a green color as soon as the car moves, and how each static object is orange, while moving objects have a color. This is because the RADAR is really good at measuring velocities.

3 - LiDARs and RADARs are complementary and still need eachother

As a little summary, I'd say that LiDARs are good, but most of the time need cameras for context, and at high speed, need RADARs. RADARs are great, but could NOT work as a standalone system. So let's do a quick overview:

If you want to be green everywhere, you need to combine all 3. Use the camera for scene understanding, use the RADAR for weather conditions and velocity measurement, and the LiDAR for distance estimation.

This brings a problem. A random startup must invest in 3 sensors, co-calibrate 3 sensors, train their team on all these sensor types, and the more sensors we use, the more confusion we risk bringing. You may wonder... can't we use just one? Or two?

Let’s see how:

FMCW LiDARs & Imaging RADARs: The Future of Perception

Back in January 2023, I was at CES in Las Vegas for the first time. It was a big show, really incredible, and while walking there, I met a startup named 'Aeva'. Aeva is a LiDAR startup specialized in 4D technology. "What's 4D?" I asked. It turns out, 4D meat that their LiDARs had the possibility to do direct velocity estimation.

The next day, I walked to a different area and stumble across a korean startup called bitsensing. "Bitsensing is creating a 4D Imaging RADAR" said the presentator. I was in shock. It was a normal RADAR, but providing an incredible resolution, with Z-elevation, accurate 3D view, no noise, and still the Doppler velocity measurement.

It sounded like these startups were working on fixing the weaknesses of classical technologies.

Let me introduce them to you.

1 - FMCW LiDAR (Frequency Modulated Continuous Wave LiDAR): 4D LiDAR

An FMCW LiDAR (or 4D LiDAR, or Doppler LiDAR) is a LiDAR that can return the depth information, but also directly measure the speed of an object. What happens behind the scenes if they steal the RADAR Doppler Technology and adapt it to a light sensor.

Here's what the startup Aurora is doing on LiDARs... notice how moving objects are colored while others aren't:

LiDAR uses the Doppler Effect, similarly to the RADAR technology, to get this 4D view. The main idea can be seen on this image, where we play with the frequency of the returned wave to measure the velocity.

The Doppler Effect is exactly about measuring this frequency. And this has now been adopted in FMCW LiDAR technology, but still with light waves instead of radio waves. I highly recommend to check out my complete post called "Understanding the magnificent FMCW LiDAR".

2 - Imaging RADAR: 4D RADAR

In 2024, mobileye, who had been working on their own FMCW LiDAR for years, announced it would be shutting down its entire FMCW LiDAR division to focus on proprietary 4D Imaging RADAR. What happened? Why the shift? Well, let's first try to understand what Imaging RADARs are. I like to call these...

RADAR on steroids!

To understand better how it works, I'd like to show you the bitsensing demo they showed me at CES.

bitsensing Imaging RADAR Demo

The Imaging RADAR has an incredible resolution. It provides a very accurate point cloud, that can see through adverse weather conditions, do obstacle detection AND measure velocity directly! Under-the-hood, it uses a set of MIMO antennas to get a much better resolution, range, and precision. We could in fact detect obstacles inside a vehicle, and classify children from parents.

See the demo:

Can you notice how similar it looks to the FMCW LiDAR? We have in both cases:

• A 3D Point Cloud
• That can directly measure velocity

Other Examples from Self-Driving Cars

Frankly, many actors from the autonomous driving industry are switching to Imaging RADARs. Mobileye has a great demo, so does Waymo. Let's see these 2 examples.

Here's the Waymo Imaging RADAR Demo:

And now Mobileye:

See? We are in the middle of a transition... but why are people using Imaging RADARs over FMCW LiDARs? And are they really moving away from LiDARs? Let's find out in the final point...

LiDARs vs RADARs: The Modern Comparison

There are 2 ideas I'd like to talk about here:

1. The Future of RADARs IS Imaging based
2. The Future of LiDARs may NOT be FMCW based

1 - The Future of RADARs IS Imaging based

We have clearly see how a good RADAR system can bring incredible benefits. We can now do tasks like object detection using purely an imaging RADAR. Recently, we've seen Deep Learning models, like the ones from Perciv AI, work on RADAR data (radar signals, radar point clouds, radar waves, ...) directly.

Back in the day, any comparison between a LiDAR and a RADAR didn’t really make sense because the sensors were highly complementary. But today, these sensors can be in competition, and if there is one, Imaging RADARs are winning it! If we see the new comparison table now:

We are BLUE almost everywhere, but the cost of Imaging RADAR stays lower than FCMW LiDARs. In addition to this, the Imaging RADAR can nicely fit under a bumper, since RADAR employs radio waves that go through objects.

When looking at remote sensing technology, RADAR has always been a great choice; whether it's synthetic aperture radar systems in the military field, or environmental monitoring of their radio frequency spectrum, or the recent adoption in autonomous vehicles, RADARs ARE by default a great choice.

In the self-driving space, RADARs were never good enough to be a standalone. Have someone ever told you you weren't good enough? Well, this is a lesson, because you can see a massive adoption and trend of Imaging RADAR - and I believe the future of RADARs is imaging.

2 - The Future of LiDARs IS NOT FMCW based

Now this is the incredible discovery here: Nobody is abandonning LiDARs for FMCW LiDARs. Self-driving car companies have NOT adopted FMCW LiDAR technology in mass (for now), and I predict they'll just stick to solid-state.

Back in 2023, I went to Innoviz Technologies headquarters in Israel. Innoviz is a LiDAR manufacturing companies providing LiDAR devices to companies like BMW. I asked them: "Why are you NOT building FMCW LiDARs?". Their answer was that their LiDARs were good enough, and that there was no real need for FMCW. It really surprised me, but I guess they know what they're talking about. They could solve the drawbacks of LiDARs by building better LiDARs, for example here:

In many fields, LiDAR sensors are at the core. We have airborne lidar systems building elevation maps, we have HD Maps built entirely from LiDARs, and even drones equipped with LiDARs today... this technology is here to stay. Plus, today, EVERYONE uses LiDARs! No that was wrong, Tesla doesn't, and a few others have bet on vision-only... but the majority of startups do, except that they use BETTER LiDARs. Not necessarily 4D, but LiDARs that provide better resolution, focusing more on solid-state technology.

This is the key message I have for you, and now that we've seen it, let's go through a summary, and see some next steps.

Summary & Next Steps

• LiDAR uses laser light to measure distances and create detailed 3D maps of objects and environments. Their key strength is distance estimation. They key weakness is weak velocity estimation, and weather conditions.
• RADAR emits radio waves and measures their reflections to detect objects and calculate their speed, even in bad weather. They key strength is velocity estimation, they key weakness is noise, context, and 3D estimation (most are only 2D).
• Traditional setups combine LiDAR, RADAR, and cameras because each sensor complements the others' strengths and weaknesses. It's near unthinkable to use one as a standalone.
• Recently, technologies like 4D FMCW LiDAR and Imaging RADAR have emerged, offering both high resolution and velocity measurement. FMCW LiDARs use the Doppler effect, and Imaging RADARs use more antennas.
• While the future of RADAR is (I believe) RADAR+Imaging capabilities, I believe the future of LiDARs may be solid-state based, and not necessarily FMCW/4D based.

Next Steps

• Learn about the FMCW LiDAR here.
• Learn about the Imaging RADAR here.

Did you ever wonder... why are self-driving cars taking so long to come? I had that question too when starting, and my first answer came from Sebastian Thrun, godfather of self-driving cars, who talked about reaching 90% of use cases easily, but then finding huge difficulty in going from 90% to 100%. Recently, Andrej Karpathy, former Lead of Tesla Autopilot described something similar as "the march of 9s":

"When you get a demo and something works 90% of the time, that's just the first 9 and then you need the second 9 and third 9, fourth 9, fifth 9..."

This is what's taking long, but instead of focusing on this, tons of companies lose time focusing on the first 0-90%. Back in 2017 or so, we were all trying to get to 90%, and for this, we were re-developping all the software, algorithms, and so on... At some point, probably 30 startups were all spending millions developing the exact same algorithms.

This is when Autoware comes in the play. Started by Tier IV, Autoware is an open-source self-driving car software that allows you to achieve the first 9 in just a few weeks. Rather than re-developping yet another version of the same code, you jumpstart from the existing state of the art, and finetune it for your needs.

This month, our membership The Edgeneer's Land is welcoming Samet Kütük from The Autoware Foundation.

Samet is currently the Community Advocate and Head of Marketing at the Autoware Foundation. Before that, Samet co-founded a company in Istanbul called Leo Drive, where he worked for a decade on implementing Autoware in various vehicle platforms, including retrofitting a Volkswagen Golf for autonomous operation.

Now based in Zurich, he is fully dedicated to the Autoware Foundation, focusing on marketing, member recruitment, and participating in technical workgroups, particularly in software-defined vehicles and cloud-native development.

And let me start with a small snippet about how he defines Autoware:

Together, we recorded a new Fragment of The Edgeneer's Land, my community membership experience, in which he takes us through the building and management of Autoware. How does it work? How do you build a self-driving car with a full remote team? This is everything Samet teaches in our new fragment...

In this post, I'd like to give you a small sample of that interview, highlighting a very interesting moment where Samet talked about End-To-End learning...

3 insights from Autoware's End-To-End Learning Transition

Since its creation, Autoware has been implementing a "robotic" architecture, meaning implementing the traditinal "4 pillars": Perception → Localization → Planning → Control.

But recently, Autoware announced a new plan to evolve to an End-To-End architecture, a single neural network that takes in the input sensor data, and automatically outputs the steering angle and acceleration value. I have a complete article explaining the differences with detailed examples here.

4 Pillars vs End To End: How to pick an autonomous vehicle architecture

How to design an autonomous vehicle architecture? Should you implement an End-To-End solution, or a more traditional one? Let’s see…

Read from the most advanced autonomous tech blogJeremy Cohen

So here is the sample I'd like to share:

As you can see, there is a lot to uncover from just one minute. Let me share 3 highlights from that:

1. Level 5 is NOT easy to reach, may not even be possible, which is why Autoware focuses on Level 4+, in which humans, while not asked to takeover the car, could be asked to drive under regions or conditions that aren't appropriate.

2. The Transition will NOT be achieved immediately, but rather then result of several steps:
   1. Current — Starting from a traditional Robotic stack
   2. Step 1 — Learned Planning
   3. Step 2 — Deep Perception & Learned Planning
   4. Step 3 — Monolythic End-to-End (single network)
   5. Step 4 — Hybrid End-To-End using a "guardian" for redundancy

This can feel similar to how Tesla did their own transition to End-To-End (which I cover in this article:

Breakdown: How Tesla will transition from Modular to End-To-End Deep Learning

It’s no secret, Tesla is going to use End-To-End Deep Learning. But how? What will it look like? Will the Occupancy Network and HydraNet stay? Here’s a full breakdown…

Read from the most advanced autonomous tech blogJeremy Cohen

The difference between Modular and Monolythic End-To-End has been explained in their GitHub repository talking about the new planning algorithm:

Alright, let's continue with a third and final idea:

3. The algorithms for End-To-End have already been built.

We are not talking about a distant future, according to Autoware, it's possible to achieve End-To-End with today's algorithms, including (but not limited to) CenterPoint as the 3D Deep Learning algorithm for LiDAR Detection, AutoSeg as the Foundation Model in Perception, AutoSteer and Diffusion Planner for the Learned Planning approaches.

See? They are already there, and even though 2, 3, or 5 years from now, they may evolve and get replaced, the logic of Modular End-To-End (Step 2) has been implemented.

For example, the AutoSeg algorithm is a "HydraNet" that has a single backbone that split into several heads for lane lines, ego path, free space, segmentation, objects, and 3D. The outputs of these heads are then passed to the deep planner.

Summary & Next Steps

• Autoware is an open source self-driving car organization that builds a self-driving car software used over the world by thousands of engineers and teams
• Autoware is the solution I recommend to get started in self-driving cars; rather than building a software from scratch, get Autoware working quickly, and then finetune and customize it for your applications.
• Autoware is transitioning from a robotic architecture to an End-To-End Learning architecture, and there are 3 highlights from it:
  • It won't reach Level 5, but a Level 4+ that can drive almost anywhere
  • The transition will happen in 4 steps, adding planning, perception, then turning into a monolythic architecture, and finally hybrid.
  • The algorithms and modular logic have already been implemented and are working, such as CenterPoint, AutoSeg, or Diffusion Planner.

Next steps

Interesting in getting access to our Autoware Fragment? It's going to be very cool, and feature several things, such as:

• The Full-Length interview with Samet on Autoware
• An even deeper dive on Autoware's End-To-End Transition (this was just a 1 minute video - we do it for the full section on End-To-End).
• A complete breakdown on many algorithms used by Autoware, and a near plug & play solution to start running Autoware's software on your computer by tonight

In September 1519, an expedition of five ships and 270 men, led by Ferdinand Magellan, left Spain to reach the Spice Islands by sailing west. At the time, maps were crude sketches, full of blank spaces, and sometimes decorated with warnings: “Here be dragons.” Yet Magellan pressed on, steering his fleet into the unknown, through storms and across oceans no European had crossed before.

The challenge was harsher than anyone had imagined. Supplies ran out, men starved, and mutiny spread. One ship deserted, another wrecked. After nearly two years, Magellan reached the Philippines, where he was killed in the Battle of Mactan. His fleet, once five strong, was reduced to four… then three… then two.

3 years later, only one ship returned to Spain. The Victoria carried just 18 survivors, but also one of the greatest accomplishments of the time. For the first time, humanity had proof that the Earth could be circumnavigated by sea, a discovery that forever reshaped navigation, trade, and commerce.

For centuries, people believed the old maps. They trusted the flat drawings, the empty warnings, the “here be dragons”. All it took was one expedition to open a new world nobody could see. And today, I believe Computer Vision Engineers live in a similar situation.

The world provides Computer Vision algorithms, image processing techniques, 2D object detectors, and segmentation approaches... yet, the world is a sphere, in 3D. And this is why, I think something of much greater importance should be mastered by Computer Vision and ALL robotics/autonomous tech engineers: Point Clouds.

The goal of a point cloud is to create a 3D model. 3D points are a data representation used today in autonomous vehicles, robotics, AR/VR, and even in everyday objects like unlocking your phone with Face ID.

So what are point clouds? How do you get them? And how do you process them using AI? These are the 3 things I think most perception engineers should know, that we'll cover in this article.

Let's begin:

9 Examples of Point Cloud Data

A Point Cloud" is a set of points in 3D space — a cloud of points. Inside, each point holds the 3D location of a surface in the real world. It can be a person, a wall, a tree, anything. You probably know what a point cloud looks like already, but you may not know the multiple types of point clouds... So let me introduce you to 9 of them!

XYZ Point Clouds

In an XYZ point cloud, each point has a specific X, Y, and Z value. You could think of it as the equivalent of a pixel, but in 3D. Rather than just X and Y, we have X, Y, and Z (in most cases, because some point clouds are 2D, see this article).

Here's an example:

See? Each point has an XYZ value. But why are the colors different? Simply here because our visualizer is a gradient based on the height of the point (the Z dimension). The higher the Z value, the more red it'll be. On the above Waymo video, you could see a different visualization, based on the distance to the vehicle. So this is one type:

• Point clouds can contain the XYZ information

Next:

XYZ-I Point Clouds

Now, this is just an example, but let me show you something else...

This is another point cloud, but what do you notice about the colors? Yes, two things:

• It's all "RED"
• But not all points have exactly the same "red" value. Some are brighter than others

And this is because here, we are no longer visualizing the distance, but the "intensity" of the points. Point Clouds are often produced by LiDARs that send a ray and measure the time it takes to bounce back. This calculation measures the distance, but not all rays come back equal. Some are blocked by trees, leafs, or surfaces, while others perfectly go through.

So, we now know another attribute of a point cloud:

• Point clouds can contain the XYZ information
• Point clouds can also hold the intensity information!

Any other?

XYZ-V Point Clouds

Now, let's take it one step further, and look at this video:

Okay, can you explain what is happening here? Everything is grey, but the vehicles. So, is that... Class? Labels? Or, wait a minute, why are the forward vehicles in red, the parked cars in grey, and the left approaching vehicles in blue? This is because, this visualization shows not the class but the velocity information!

This video has been made from Aeva, an FMCW LiDAR producer — and inside, you can see the points receding are in red, and those approaching are in blue. We now know a third possibility!

• Point Clouds can contain XYZ
• Or XYZ-Intensity
• Or XYZ-Velocity

Let's see 9 types of Point Clouds

Are there any more than intensity or velocity? Yes, in fact - each point can contain a lot of information. Let's see:

• Intensity - how strong the point clouds return signals are
• Range - the distance of the point, based on X, Y, or Z
• Color - the RGB color of the points (often for RGB-D cameras or 3D reconstruction)
• Class/Label - if it's after an object detector or segmentation tool processed it
• Infrared - the wavelength of the point cloud signal
• Ring/Channel - which channel of 3D sensors was used to collect it
• Velocity - the speed of each point (calculated by RADARs or FMCW LiDARs)
• Reflectivity - how reflective the surface of the point is
• Temperature - how hot a point is

Okay, but concretely, how does it work? Is there a TXT file where we store the points? Kinda, let's take a look...

Point Cloud Formats & Files

There are usually two types of files: ASCII and Binary. One is easier to read, the other is more suited to real-time/embedded. Take a look at the beginning of both files:

See? On the left, the PLY file contains X, Y, Z as floats, followed by a list of point coordinates. This is the point cloud! On the right, you can see a header describing the point cloud, in format XYZ-Intensity, and then, the points are not readable.

2 Main Ways to create a point cloud

There are basically 2 types of approaches, active and passive. Active techniques actively emit signals like light or sound to measure distances and create point clouds, such as LiDAR and structured light systems. In contrast, passive techniques rely on capturing existing environmental data, like photogrammetry, which reconstructs 3D points from multiple camera images without emitting any signals.

Active Techniques: LiDARs, RGB-D & RADARs

In the first case, point clouds come from sensors built to create them. When a camera takes a picture, it aims to get pixels. Well, when a LiDAR makes a measurement, its aim is to create a point cloud. Let's see 3 ways to do it:

1) How to get point clouds using Structured Light RGB-D systems

Ever played the Microsoft Kinect? I can't say that I have. I was a Wii player all the way when they were competing. Yet, I've always been impressed by how the Kinect produced point clouds using its RGB-D camera, working with the Structured Light Principle.

The Kinect shines a special pattern of light around, then uses an infrared camera to take a picture of how that light bounces back. By seeing how the pattern changes, the camera can figure out how far away things are. It combines this distance information with the colors it sees to create a 3D image called one final point cloud.

In robotics, you probably know the Intel Realsense 435i, or other equivalents. Their goal is to build a Depth Map, then turned into a point cloud.

2) How LiDAR point clouds are produced

The most common and popular technique is to use a LiDAR (Light Detection And Ranging). There are many types of LiDARs around, but let's focus on the simple Time-Of-Flight principle. In this setup, a laser scanner sends a light beam and measure the time it takes to reflect and come back to the receiver. Similar to this image:

LiDAR scanners produce raw data of the world up to 300-400 meters in the automotive industry. Each scan can generate millions of points in the three dimensional space. I highly recommend checking my article on the types of LiDARs to learn more.

3) RADAR Point Clouds

The third technique is to use not a LiDAR but a RADAR to create the point cloud data. This is not very straightforward to do. RADARs usually return signal information based on Doppler (velocity), Range (distance), and Azimuth (direction/angle). Using these, we can do some calculations to retrieve the point cloud data.

Here is an example on a very low quality RADAR:

Today, we can use Imaging RADARs to get 3D point clouds. I invite you to check out my Imaging RADAR article to learn more about it.

Now that we've seen the Active ways, using sensors - I'd like to take a minute to talk about the passive ways.

Passive Point Clouds Generation: Photogrammetry & 3D Reconstruction

The idea of passive is that you do not attempt to create a point cloud from your sensors. The main way to do this is by leveraging 3D Reconstruction. Ideas like Structure From Motion, Multi-View Stereo, NeRFs, Gaussian Splatting, or others are used.

The idea? To convert 2 or more images to a 3D point cloud using triangulation, geometry, and depth maps.

If you're interested in this, I highly recommend reading my 3 article Series on PyImageSearch blog, or my article on Pseudo-LiDARs.

Alright, so you now know all about the point cloud types, and the ways to get them. One thing remains...

How to Process Point Cloud Data?

Do you remember in the point cloud types when I showed the "label/class" of each point? This is not something sensors can measure, it's built by algorithms. There are 3 things that really matter here:

1. Understanding the main libraries/tools to work with
2. Understanding the core algorithms to use on raw point cloud data
3. Being able to use them in the applications

Libraries: Open3D and Point Cloud Library (PCL)

There are many libraries used to process point clouds. These implement the algorithms. For example, the Point Cloud Library is one of the most popular to work with. Open3D is also a very common one, it contains fewer algorithms, but is easier to process thanks to the Python interface. I would recommend to get started with this one.

On a similar topic, you could want to know at least one point cloud dataset. I would recommend you check out this article.

Which algorithms can be used to process point clouds?

In point cloud processing, you can either go with traditional algorithms or 3D Deep Learning. The split is, I would say, dependent on the applications. When companies want to detect objects in 3D to get bounding boxes, they usually use 3D Deep Learning algorithms like PointPillars or VoxelNet. Let's see an example:

Outside of 3D Object Detection and 3D Segmentation, the entire world runs on traditional processing approaches. Since you have points, you can create tons of automated pipelines to process them. For example, you can do plane segmentation, clustering, outlier removal, normal estimation, point data cropping, surface reconstruction, filtering of unwanted data points, and so on, you'll use traditional approaches.

For example, you could calculate the surface normals and filter out the objects that belong or don't below to the street.

Another technique can involve point cloud registration and alignment. When you have multiple point clouds, for example coming from 2 LiDARs, you can align them together into a single object. An example below from one of my LiDAR courses, notice how we start with 2 point clouds, a blue and a red, and we end up aligning them perfectly. This makes something better than the raw data.

In the algorithm category, there are countless applications. Everything related to SLAM or Odometry is also extremely in use today.

Applications: Which jobs can you target with Point Cloud skills?

Regarding the applications, we could write an entire article. Yet, let me give you 3 or 4 core jobs you can target with point clouds processing skills:

• Perception Engineer, Autonomous Vehicles: Process LiDARs and RADARs to find objects in the 3D space. Use Sensor Fusion to mix the output with Computer Vision. Build autonomous vehicles, shuttles, delivery robots, and create the future.
• Nuclear SLAM Engineer, Robotics: Use point clouds processing techniques inside robots that explore caves or regions humans can't go to, such as nuclear sites, and build maps of the world.
• BIM Engineer, Architecture: Create digital models of buildings and structure by processing raw point cloud data captured from laser scanners or photogrammetry. These models help architects and engineers visualize object properties, plan renovations, and ensure precise construction. The role often involves using processing software to convert points into computer aided design (CAD) models, performing manual correction to refine the data, and integrating the results into architectural workflows for improved design and quality inspection.
• Medical Imaging Engineer: Apply point cloud techniques to CT-Scans, IMRs, and other 3D data types to detect diseases and save lives. There is both a commercial and research use.
• Drone Engineer, Agriculture: Process Cameras and RADARs/LiDAR information to navigate and help drones fasten agriculture and solve population needs.
• and many, many more...

Alright, now let's see a summary...

Summary & Next Steps

• A point cloud is a series of 2D or 3D points. A point cloud is to the LiDAR what a pixel is to a camera.
  (Re-read that one.)
• Each point of a cloud usually contains at least the XYZ information, but many sensors or technique allow to also get Intensity, Reflectivity, Velocity, Ring/Channel, Color, Temperature, Infrared, and more...
• A point cloud output format is of 2 types: ASCII or Binary. An ASCII file is more readable for humans, Binary is more readable for robots. Each file is a list of points and their information.
• There are 2 ways to build a point cloud: Active and Passive. Active techniques involve sensors like LiDARs, RADARs, or RGB-D cameras, while passive techniques use photogrammetry and 3D reconstruction to retrieve 3D models.
• Point Clouds Processing typically involves 3 stages: the tools/libraries, the algorithms, and the applications. Tools are libraries like Open3D or PCL, algorithms are either traditional or deep learning, and applications go from self-driving cars to robotics, drones, augmented reality, the architecture industry, agriculture, and beyond.

Where do you think humans come from? Growing up, I wrestled with two conflicting ideas about it. One teacher taught me Darwin’s theory of evolution: a gradual process of adaptation, rooted in science but riddled with gaps and errors. The other taught me the Bible’s Old Testament: the story of divine creation, in which even though nothing was ever proven false, this isn't built on any "proof".

Since then, I realized they were trying to answer the same question, but operated in entirely different realms, each with its own logic and purpose. One belonged to science, the other to faith. Making head-to-head comparison was almost meaningless.

And I think this contradiction also exists in self-driving cars, especially when opposing 2 giants: Tesla and Waymo. Both seem to chase the same prize of "Level 5" autonomy, but when you look closer, their paths are so distinct they’re barely comparable.

In this article, we're going to try and understand who has what I call the best "Map to Level 5", we'll take a side-by-side comparison, and I'll give my opinion on 3 aspects:

• The sensor suite
• The algorithms
• The "map", meaning strategy, vision, and more...

Let's begin with the sensors...

Tesla vs Waymo: Who has the best Sensor Suite?

Back when I was studying driverless cars, it was around 2017, when I heard an interview with Sebastian Thrun, godfather of self-driving cars, talking about who was ahead in the race. I vividly remember his words: "Nissan is doing pretty good, but I think the company who is ahead of everyone is actually Tesla".

It was 2017, and I remember feeling surprised by this comment, because at the time, Tesla only had a light ADAS feature working with mobileye, and companies like Waymo, Mercedes, Nissan, and others seemed to be covered everywhere in the media, have "real" self-driving car abilities, and more potential.

What about today? Who is ahead? Closer to remove human drivers? Who has the better vision? The better algorithms? The better sensors? Is it Waymo, or Tesla... or someone else?

In this first part, I want to answer it from a sensor angle. And to do so, I'm going to start by a screenshot of a very popular X (tweet?) from Elon Musk about RADARs, LiDARs, and cameras from August 2025.

This comment raised an army of furious engineers and fusion experts, mentioning Kalman Filters, and Redundancy, and Sensor Fusion. So before we dive into the exactness of this comment, I would like to describe what each company is doing...

Waymo: 29 Cameras, 6 RADARs, 5 LiDARs

If you look at a Waymo car, you're going to see exactly the opposite of Tesla: tons of sensors all over the place. There are RADAR sensors on the front, side and rear, there's 29 cameras, and 5 LiDARs. The question we can ask is... "Is Waymo trying to kill a fly with a bazooka?".

Just in terms of calibration, it must be an absolute nightmare for engineers. Calibrating a camera with a LiDAR is already a long task, but 29 cameras with 5 LiDARs? Then there is the fusion of all of these sensors together, and this is ONLY the "robotaxi" version, because if you look at their Zeekr shuttles, they also have their own types of sensors, with different generation codes. Their stack is therefore always evolving, and depends on the vehicle they drive on.

What type of LiDAR, Camera, and RADAR is Waymo using?

Let's take a brief look at what each sensor sees:

Waymo uses different types of cameras. They work with high-res long-range cameras (color, telephoto lenses) for object detection far down the road, wide-angle cameras for close-range coverage (pedestrians, cyclists, intersections), and near-infrared cameras for night vision / low-light perception.

Regarding LiDARs, they're using their own sensors called "Laser Bear Honeycomb". There is one forward LiDAR, 2 side LiDARs, and one at the rear. But these are short range, solid-state LiDARs. They are excellent for blind spots and front facing vehicles, but complex to drive on highways because they don't see far. This is why there is the roof LiDAR, which is mechanical, and sees several hundred meters away.

On the animation below, you can see LiDARs both in point clouds format and in range-view — and you can learn about types of LiDARs here.

Regarding RADARs, Waymo uses their own line of Imaging RADARs. Imaging RADARs are what I call 4D RADARs. Unlike normal RADARs who see in 2D and measure the velocity, these ones see in 3D and measure the velocity.

Do you want to see what all of this looks like together? Okay, here it is:

Waymo uses a powerful array of sensors allowing them to see every possible object. We'll come back to the utility of LiDARs and RADARs, but for now, let's look at cameras...

2. Tesla's Sensor Design: 8 Cameras, that's it

Unlike Waymo vehicles, Tesla's approach relies only on cameras. Tesla's autopilot aims to solve autonomous driving using vision-only.

On this illustration, you can see a very simple design that almost never changed. In red, you can count 8 outside cameras, and in orange, you can see 12 ultrasonic sensors used to detect static objects when parking. Among the 8 cameras, there 2 on the windshield, used for stereo vision, one on the front bumper, one on the rear bumper, 2 on the doors, and 2 on the wheels. See the difference? I can't even begin to count Waymo's cameras, but I can easily show you Tesla's sensor stack.

So let's visualize what the cameras see:

Interesting, or not. Now, let's try and understand, who has the best sensor suite?

3. Who has the best sensor suite... Tesla or Waymo?

I am going to show you an image, and I would like you to ONLY look at the left part. Ignore the right for now. Can you tell me what you see? You see a car, don't you?

But did you notice the pedestrian? This is one of the limits of the vision-only approach. When you look at Tesla's miles driven without disengagement reports, Tesla FSD clearly shows a limit on bad weather. They don't drive well on cloudy foggy, rainy, or snowy scenes, and they don't drive at all during storm and sleet (when ice falls from the sky).

Reports show that FSD13 significantly improved driving by night, but there is still a physical limitation to driving with cameras only that is not solvable with better algorithms. In a robotaxi situation in which you'd sit in the passenger seat, you would be stuck. The way humans drive involves more than cameras, we hear street sound, we sense people, and we don't use wide-angle cameras to detect other cars.

If we were to come back to Elon Musk's comment now, do you remember the "If LiDARs/RADARs disagree with cameras, which one wins???". Waymo shows that redundancy is key to safety. If a camera misses something, your RADAR may not. And with Kalman Filters, you can certainly develop a powerful fusion module to account for disagreements.

When we consider the physical ability of a LiDAR to generate point clouds, you can understand how powerful having them is. Other than seeing through night or other situations, they physically build point clouds. Recently, a YouTube video has shown a Tesla vs LiDAR-equipped car driving on a wall resembling a street. The Tesla FSD crashed on the wall, confusing it with the highway; but the LiDAR-equipped car stopped.

You may wonder... Okay, but we'll never see fake walls in real life, so what's the point? The point is that vision only has its limitations. We've seen Tesla confuse the moon with objects, miss a red light, do phantom breaks, and even miss truck trailers. For all of these reasons, I would say that the Waymo approach including Camera+LiDAR+Imaging RADARs is a better choice. They get the #1 point.

One caveat is that the algorithms, processing power, and energy required to operate this vehicle is insane. LiDAR sensors consume a lot of energy and record a lot of data. Tesla is much cleaner in that perspective.

Speaking of algorithms, let's now move to this second point.

Tesla vs Waymo: Who has the best algorithms?

If we now look at the algorithms, who is closer to build self-driving cars? Tesla and Waymo started off with very different architectures, but now seem to converge towards End-To-End Learning. So let's take a look...

1. Tesla FSD Algorithms: HydraNets, Occupancy Networks, and End-To-End Learning

In my article breakdown on Tesla, I'm doing a full deep dive on the Tesla's algorithm, and how they work; so I won't do that here, but I will still show you the overview of how they built their autonomous vehicle architecture. Note that it's according to the Tesla data; which moved to private in 2023.

You can see 3 main blocks:

• Lane & Object HydraNet: The lane and object Hydranet is a multi-task learning network that takes in the 8 cameras, learns features from each using a CNN, fuses them spatially and temporally via a Vision Transformer, and then outputs several heads. Heads are trained to detect objects, lanes, positions, and so on... You can read more details here.
• Occupancy Network: The Occupancy Network is also processing all 8 cameras spatially and temporally, except that it's trained to leverage spatial data. This is a 3D network that aims to build voxels and assign a free/occupied state to each. You can read more details here.
• Planning & Control: The Planning & Control node used to be (in the drawing) done via a Monte-Carlo Tree Search. This is traditional artificial intelligence. In 2024, they replaced this with a Neural Network planner. While we don't have details on how it works, the "End-To-End" comes from this node moving to Deep Learning, making the entire network differentiable.

To push the explanation even further, let me show you the typical visualizers on a Tesla, and see how they both refer to an algorithm:

Now, let's see Waymo...

2. Waymo's Algorithms: 3D Deep Learning, Diffusion Planners, and more...

It's a bit harder to fully track the state of Waymo's algorithms, because they continuously release multiple research papers, and we don't know which ones are in production, and which are just pure research. Still, according to my research, there are 3 core pillars Waymo relies on to drive...

• LiDARs
• Prediction/Tracking
• Imitation/End-To-End

LiDARs

Early on, Waymo pioneered work on LiDARs with 3D Object Detection algorithms like SW-Former. These algorithms process LiDAR point clouds and output bounding boxes in 3D. This architecture has been updated a few times, but now serves as the "core" detection algorithm of Waymo. From there, it's encapsulated into other pipelines, like the Late-To-Early Fusion:

See what's happening? We have a temporal fusion algorithm that processes LiDARs and boxes from t-1, t-2, and so on... and each of these go to SWFormer to output a final head. This is turning SWFormer into a temporal detector, and not just a frame-by-frame detector.

Prediction & Tracking

Going even further, we have Prediction & Tracking. Waymo bets big on tracking, and one of the core algorithms I noticed there is an architecture recently released called Stateful Track Transformer which is doing exactly the job of tracking from SWFormer. Over the years, Waymo released TONS of prediction and tracking architectures.

End-To-End/Imitation

Back in the early 2020s, I remember vividly Waymo mentioning an algorithm called ChauffeurNet, who was behaving exactly like Tesla's HydraNet, but was outputting trajectories. Since then, the approach evolved, and the later published papers and public talks mention the End-To-End architecture named EMMA, as well as Vision Language Models.

As you can see, this turned in to an Encoder/Decoder architecture, where the encoder learns features from each sensors, then fuses spatially and temporally, to learn a compressed representation of the scenes. Then, the decoder is a generative part, built on VLMs to predict a trajectory.

While many claimed Waymo has a "modular" approach, while Tesla has the advanced End-To-End approach; this is simply no longer the case. Although we don't know whether Waymo uses EMMA in production, or still relies on their traditional pipeline, we definitely know they're heading towards it.

3. Who has the better algorithms? Waymo or Tesla?

Let me write 4 or 5 bullet points explaining what I think:

• Both Tesla and Waymo seem to be headed towards End-To-End Learning, because the Modular approach has the car behave in a "robotic" fashion. This is not smooth, feels robotic, and rule based. So both go towards End-To-End...
• But to make End-To-End work well, you need LOTS of data. Tesla has that from their fleet of millions of cars all across the world, but Waymo has a small fleet driving only in very specific regions. Scaling via End-To-End will be extremely painful for them.
• On top of that, Tesla only needs to process camera data, which makes the algorithms likely faster, using less power and less time consuming.
• End-To-End's biggest problem is edge cases. Tesla has experience driving several million miles in construction zones, parking lots, or in a new city, and so on... Waymo, on the other hand, is stuck with HD Maps and can have issues moving towards End-To-End...
• Tesla also has advanced techniques for Edge Case Detection and retraining, such as Trigger Classifiers (see my detailed overview here), as well as Dojo and Self-Supervised Learning. They seem far more prepared for End-To-End Learning in my perspective than Waymo. It's as if Tesla had paved the way for this for a decade, while Waymo pivoted last minute

Given these, and under the assumption that the price here is End-To-End, I would give my points to Tesla.

Waymo vs Tesla: Who has the better Map to Level 5?

In this last point, I would like to get off the sensors and technique, and see these as businesses primarily. Their goal is to sell self-driving cars, or autonomous rides, and thus... who's leading in that sense?

1. Strategy

First of all, something very important to understand:

• Tesla sells self-driving cars
• Waymo rents autonomous transportation services

This is essential, because this means their software, sensor stack, philosophies, business models, and even strategies to reach Level 5 are totally opposed. This is very clear when you see the graph below, which shows Waymo starting with a very capable vehicle, but only in ONE geo-fenced area, with ONE car, while Tesla starts with millions of car, but none of them are autonomous. Both don't scale the same thing:

Tesla's goal is to make millions of cars at a 25,000$ price point. Imagine Tesla in 2016, being told to integrate LiDAR technology, which cost over 50k USD/unit. Wouldn't they be better off starting with a light camera + RADAR Level 2, and gradually improving it? Of course they would, if they believe it's possible.

On the other hand, Waymo had a fleet of maybe 20 vehicles at the time. With 29 cameras, 5 RADARs, and 6 LiDARs, a Waymo car costs significantly more than a Tesla car. In fact, each car was estimated to cost around 250,000$ a few years ago. Since then, the LiDAR price dropped, Waymo grew its fleet to over 2,000 cars, and each is now estimated around 150,000$. Can you see how they cost is less and less of a problem to them over time?

Waymo's LiDARs certainly cost a lot, but this cost get absorbed as they do more paid rides, until it 'supposedly' becomes profitable. Supposedly, because there is maintenance, replacement, and growth of the fleet, which can make the cost a never ending fight. Even there, Waymo has more leverage to afford to LiDARs, by raising the ride cost of millions of people from 7$ to maybe 13$ (made up numbers, terribly off — just making a point).

There are great statistics on this blog post from Ben Buchanan that show how Waymo's car get more and more affordable over time. A LiDAR costs 500-1,000$ today, this completely challenges the vision-only philosophy. Unlike Tesla, time in on Waymo's side.

You can therefore understand how each one wins:

• Tesla starts with TONS of disengagements and tries to decreases that number to 0. The more capable the algorithm, the more cars the will sell.
• Waymo starts with very few disengagements and tries to scale the number of rides and regions without increasing this number. The more regions they will cover, the more rides they will sell.

2. HD Maps

Waymo is betting on a serious "HD Map" strategy that Tesla refused to adopt. According to Tesla, the car should be able to drive anywhere in the US, so there's only using normal Google Maps or OpenStreetMap. It does not mean they don't use HD Maps; they do (see screenshot below), but they don't require them to drive. If the car ends on a parking lot with no map, it should still be able to drive.

On the other hand, Waymo maps every squared inch of every place they drive in. This means every traffic sign, every bumper, every crossroad, every traffic light, every roadwork, lane lines, speed limit... everything is continuously mapped and updated. You can see on the image the HD Maps of Waymo and a screenshot of Tesla's HD Maps on a car in debug.

From what I read, Tesla performs poorly on regions where they don't know the maps. While this doesn't block them, this certainly causes disengagements. They are therefore equal in this sense.

3. Miles Driven & Disengagements

We can't conclude this article without first looking at who slams the brakes the most. Yet, I feel you already know the answer, from the first two points. It's obviously Tesla, because they have many more cars, miles driven (5B for Tesla vs 100M for Waymo), and different situations. So how can we really vote who has the better map to Level 5?

Let's first look at disengagements. What is a disengagement? Is overtaking a stuck vehicle a disengagement? Is accelerating? What is the definition, and do Tesla and Waymo use the same? Well, Waymo uses safety drivers, who obey specific instructions. Tesla drivers disengage for virtually any reason, even if they simply feel like it. This is why Tesla FSD Tracker shows, as of September 25, 24% of FSD drives have a disengagement, and 3% have critical disengagements (US).

Continuing with more stats: Tesla drives on average 213 miles before a disengagement on a highway, and most disengagements are caused by Lane Issues. Notice the jump in miles driven without a disengagement with FSD 12.6.

For Waymo, it's a different story. Waymo has a driver permit in California, which had them recently release the 2024 disengagement report to California DMV. It clearly showed a number of 9,793 miles driven before disengagement. You may ask... How is Tesla at 200 but Waymo at 9,700 miles driven without disengagement? It's because, as we said, the definition, reason for disengagement, and cities, are extremely different.

If a company wants to drive on a straight line for 500 million miles and show 0 disengagement, they technically can. This is why I think, we can't really trust any of these numbers — only their relative evolution on the same conditions as before 🤷🏻‍♂️

So who has the better Map to Level 5? I will tell you, but only after we have made a quick summary of what we've seen...

Summary: Who is ahead, Waymo or Tesla?

• Tesla and Waymo both have different sensor stacks. While Waymo relies on a stack of 29 cameras, 6 radars, and 5 LiDARs, Tesla takes a different route and relies on 8 cameras only.
• A sensor setup with LiDARs and RADAR redundancy is safer and gives more reliability than a vision-only setup. LiDARs can detect objects and brake the car even if no object is identified by the camera or an algorithm. Waymo also drives in more weather conditions than Tesla, who is physically limited by the lack of other sensors.
• Waymo uses an architecture based on LiDARs, with algorithms like SW-Former, Prediction/Tracking, and EMMA as an End-To-End system.
• Tesla uses an End-To-End approach involving HydraNets, Occupancy Networks, and Deep Planning. This approach is reinforced by powerful Trigger classifiers, Self-Supervised Learning Dojo, a powerful data fleet — making them win on the algorithm aspect.
• Waymo depends heavily on detailed HD Maps and can only work on these mapped, geo-fenced areas. Tesla’s system is designed to drive anywhere without necessarily relying on HD maps. In practice, they drive much better when they have maps.
• Waymo boasts a much better disengagement rate, with safety drivers rarely needing to take control compared to Tesla’s more frequent interventions; but the conditions they drive in and disengage are 100% under their control.
• The two companies have very different business models: Waymo sells autonomous ride services, Tesla sells cars with self-driving features. As a result, their map to level 5 is not the same.

And now, you're all caught up. So, the Map to Level 5?

The Map to Level 5

Here is what I think:

To reach Level 5, Tesla will need to reduce the number of disengagements. To me, they will have to include LiDARs or RADARs at some point. At today's cost, that would probably be feasible. In fact, I'm pretty sure Tesla is so competent FSD would be solved right now if they didn't chose to play the game on hard more. But now, after all this time, can they really afford to do this? There's ego, brand image, and the "FSD capable" computers they already sold. How can they? Tesla is in the camera game for good.

To reach Level 5, Waymo has to increase the number of areas they drive without increasing the disengagement rate. This means adapting the "HD Map" strategy, which still relies on them, and making algorithms capable to adapt to a new region faster. To make autonomous driving a reality for the entire world, Waymo will need to go faster. Their autonomous vehicles surely are capable, but their current scaling strategy takes too long.

Next Steps

Have you ever heard the story of the iPod? It started in January 2001, right after Apple announced a loss of $195 million, and had missed the shift to digital music. The company was lost, and had one last chance of survival: building an MP3 player to catchup with the competitors.

If you are old enough to remember the MP3 players back then, the were confusing to use, overloaded with buttons and menus, and made the experience painful for customers. Apple was looking for a solution for months, but had no clue how to make it better.

Until one day, when Apple's Head of Marketing Phil Schiller suggested using a scroll wheel. Wheels already existed in mouses and dial phones, but had never been never used in music players. With this, he suggested that the menus should scroll faster the longer the wheel is turned, a stroke of genius that would distinguish the iPod from the agony of using competing player.

The rest is history: Apple developed the iPod in the greatest secrecy, launched it, and changed the world with "1000 songs in your pocket".

What made it so successful? It's not that it looked good, or had buttons, or could store more songs. No, the genius was in the smarter experience scroll wheel.

If you're in the autonomous vehicles market, we probably witnessed a similar pattern: companies have been collecting more and more data endlessly, building data centers, simulators, hiring people to analyze the data generated, and so on... Until some companies came up with smarter ways, not involving just "collecting more data", but rethinking the experience to focus on events instead.

In this article, I would like to tell you about the way automotive data processing works nowadays, and how the AI revolution is going to reshape it.

We are going to learn about 3 ideas:

1. The first part is going to focus on the Manual Era (where we collect and process it all) and the Cloud Era (where we use DataLakes)
2. The second part will be a case-study provided by an autonomous tech startup, revealing the 10 biggest problems of the Cloud Era.
3. The last part will show you the Edge Intelligence Era & the Autonomous Era, which, as you'll see, if an incredibly more intelligent way to do

Let's begin with point #1.

1. Data Management: How self-driving car companies collect and process data in the Cloud Era

One of the things we heard the most this past decade was that Data is king. And for a long time, collecting as much data as you can in order to train heavy machine learning models has been the only way to do. Let's talk about data collection, and then processing.

How do autonomous vehicles collect data?

We know that when a self-driving car drives, all the data (sensors, images, messages, hardware status, algorithm decision, ...) is being recorded.

Should we give an intro line explaining how?

The process is simple, and looks like this:

1. You plug your sensors to your system (for example, Robotic OS/ROS)
2. You press record

I'm sure somebody out there worked hard to find a more complex process, but if you're using a tool like ROS, recording data is as simple as using one command line. In the video below, you can see me recording LiDAR point clouds, camera images, GPS positions, algorithms outputs, and mostly all the messages passed through the self-driving car while we drive...

When I'm done recording, the output is a file in the .bag extension (for ROS 1) that can vary from a few Gb to Terrabytes of data. Let me show you an example below from the TIERS dataset. Notice the duration and sizes of the recordings below — the last one is just 8 minutes long, and yet weights 200Gb. This is 2.4Gb/minute!

The bag is the first element. Then comes what we do with it.

The Manual Era: How do we process and analyze data?

The first "era" I'd like to tell you about is the 1.0 era. Back when I worked on autonomous shuttles, each of our fully autonomous vehicles was driving and collecting data to SSD drives. When the day was over, we had hundreds of Gb to process. So we started coming up with file naming conventions, involving the date, event, and so on...

Then, when back to the office, we could replay our algorithms on it, train our models on the data, and so on... Below is an example of a drivable area segmentation algorithm I've been training on the data collected:

This was fine for a small startup of 8 people, and it's probably still okay for small companies that don't need extensive processing, but most autonomous vehicle companies have turned to the cloud...

The Cloud Era: How Advanced Driver Assistance Systems (ADAS) & most of the Automotive Industry is using Data Lakes

If you record data every day, and each recording is hours long, you're never going to find the events you need. This is why I'm showing you a more sophisticated, let's say '1.5' version, which makes data collection part of a pipeline.

It looks like this:

1. You record the data
2. You upload it to AWS/Azure
3. The R&D team then processes it weeks later, replaying all the events, searching for 10% possibly interesting scenarios, or events, and so on...

If you'd like to see real-world concepts, you can see AWS and Azure Data Lakes:

A lot of companies in the self-driving car market use these "data lakes". Let's look at the Azure Data Lake in a simplified view:

ADAS & fully autonomous cars use it. After recording, the 4 key blocks are:

1. DataOps: Where we analyze data, clean it, label it, augment it, tag it, and so on... Notice the interaction with external labellers; that idea is called "human-in-the-loop".
2. MLOps: The machine learning algorithms, training, testing, and so on...
3. ValidationOps: The validation part, involving visualization, scenario, and simulation.
4. MetaData: After the DataOps tagged the data, we can search for it.

You can see how it's placing data at an element in the chain.

So what are the problems of this? Before telling you about the 2.0 Autonomous Era, let's try to see a case study with real ADAS or artificial intelligence companies using it...

2. [Case Study] ADAS Actors reveals their 10 biggest problems with Data-Driven Approaches

In this section, before talking about the '2.0' approach, I would like to tell you about the core problems companies who process large volumes of data reported.

Before writing this article, I got the opportunity to talk to Heex Technologies, a french startup specialized in Event Based Data Management... and I asked them "Which problem do you solve?". To answer, they shared a 20 page PDF listing all the problems their biggest Advanced Driver Assistance Systems (ADAS), autonomous driving, or robotic clients from the automotive industry.

In the PDF, I spotted a lot of interesting problems. Let me share the main ones with you:

If I were to list down the 10 main problems, you'd see: Slow access to critical events, Manual data processing, Exploding Cloud costs, Fragmented data, Delayed visualization (no real-time), Manual Extraction of scenarios, Useless streaming data, Physical SSD Extraction, Blind Debugging, and Inefficient ROS Bag Processing.

Notice all these terms I underlined? These are the problems of today's data management systems.

Let's take some examples...

• If you collect the data on Day 1, and process it on Day 3, you have slow/delayed access to critical events; like a missed pedestrian. So you're driving, notice something wrong, but you have to wait until 2 days later to even look for the data, and start searching for that event you noticed...
• Similarly, can you see how the 'fragmented' data processing is a problem? Especially when you are with a team. Engineer A grabs bag A, and makes decisions based on it... Engineer B grabs bag B and makes a different decision based on it... The entire decision cycle happens in silos.
• The Physical SSD extraction is a problem too. In May 2025, I was at the Stuttgart ADAS & AV Expo, and I met a company who invented a "swap" disk system... All of this is great, but that's still the same problem of storing, copy/pasting data, etc... to a system.

In each of these problems, I noticed a time and money waste.

For example, the client reporting: "Engineers waited several days to weeks to access specific events due to the time-intensive process of uploading, filtering, and classifying raw data in the cloud." is clearly facing a time problem, reviewing large amount of data... The other client who mentioned: "The full data pipeline we built —from data capture to processing and storage—incurred high cloud costs and consumed engineering resources" faces a money problem...

In this same report shared by Heex, all the companies reported improvement in their pipeline. Whether it was better decision making, more time freed, or money saved. This is why the next part is so important, so let's now focus on it: Event Driven Data Processing for autonomous cars.

3. Event Driven Data Management for autonomous cars

Back when I started learning autonomous driving algorithms, I listened to an interview from Sebastian Thrun, acknowledged as te godfather of self-driving cars, who at some point, said something that marked me: [paraphrased]: "With a team of 2/3, you can build a self-driving car that drives 90% of scenarios in a weekend. Then to get to 95%, it takes a few weeks, and to complete these last 5%, it takes years."

This idea is called the "long-tail" problem.

When looking at traffic accidents involving autonomous vehicles, you often see rare events or edge cases at the root cause. The person wearing a stop sign t-shirt, the truck with a donkey on the trailer, the traffic sign burned by parisian riots, all of these unusual scenes totally different from empty highways cars are used to.

Some companies solve it with data generation, others with simulation, or with End-To-End Learning. Yet, the root of all evils here is data, and thus, this is what we have to change.

A decade ago, the term "data" became king, and everybody became a Data Scientist, Data Engineer, Data Ops, Data Something. It was the case until recently when the data revolution passed, and breakthrough innovations happened not thanks to more data, but thanks to smarter training systems (like self-supervised learning), or more powerful architectures (like transformers). "More data" was ultimately not the solution, and thus, we have to switch our thinking...

The Edge Intelligence Era: From Data Management to Event Management

After companies have recorded a few laps of the neighborhood they drive in, recording more of this same scene doesn't make sense. Companies record more and more, just to spot the 1% of long-tail events. What if we worked on these events only, from the beginning?

It can be done, by setting up a "triggers" in your system, that will act as a filter and only capture the scene when interesting events happen, such as:

• Objects Missed: If one camera misses an object that another sensor sees
• Near Pedestrian Collision: If pedestrians are within 2 meters of our car, and we drive over 30km/h
• Human Intervention: If a human driver manually took over
• Shakes: If the camera physically moved due to a bumper or small shock
• Ego Collision: If a collision with the ego vehicle happened
• and so on...

All of these are valid events we'd like to record. The rest? When it's all smooth? Well, we already have millions of it.

An example with Heex Technologies, and their platform allowing to set triggers:

If I were to show you the 2.0 process, it'd look like this:

• You have the same car with LiDARs generating the same 10Gb/h data
• Rather than recording all data available, you define triggers.
• You intelligently record the events, like the near pedestrian collision, and not all the data
• You get instant notifications, labels, and can do real-time decision making

Seems smarter, isn't it?

Now that you have this in mind, I'd like to show you the last era...

The Autonomous Era: AI does it for you

The next step is to create algorithms to do it automatically for us. For example, Tesla patented a concept called trigger classifiers. The idea is to train their HydraNet backbone to classify whether the general scene it's learning from contains unusual events or not. If it does, let's say above a certain confidence score, then the machine learning models will trigger a warning.

Whether you're working on spreadsheets or building autonomous vehicle technology, automating manual items like labelling or searching for data makes sense. In this case, just like the Edge Intelligence Era, you can see the events being captured live while driving, and not after.

The 2.0 Vision

This is going with the "2.0" vision of self-driving cars that companies now define. A vision driven by Deep Learning first, where data matters, but where more data isn't the solution. In the 2.0 vision, quality is better then quantity; contextual intelligence is needed, learning should be real-time, and training should be done on relevant data.

If the 1.0 vision involved heavy test vehicles, modular architectures; the 2.0 vision is about AI & efficiency.

Now, let's see an example of a company specialized in this...

Example: How Heex Technologies turns Data into Event Management

One of the companies that captured this vision the best is Heex Technologies. They built a SaaS platform that implements exactly these ideas of "triggers" — and their motto is that rater than focusing on the data, they focus on events. As I already showed you the triggers, we could see it in action:

Let's look at their pipeline, which you'll notice also works backwards — once the bag is generated:

From a heavy bag, we get a smart bag. The data is definitely smarter when automatically annotated, categorized, and when relevant events are flagged. We can then re-inject this data into the training pipeline, without having to worry about the rest of the dataset.

As an entrepreneur myself, I can only admire the focus on one specific and painful problem like this one. When you can anticipate customer needs, and enable automakers and automotive engineers move away from a complex process to focus on their core job (self driving technology)... you win!

Alright, let's do a summary and see what to do next:

Summary & Next Steps

• Collecting data is essential to train AI models and develop self-driving vehicles. Yet, "more data" is not the solution to create breakthrough, and solve the "long tail" problem, which cause a significant challenge.
• The first Era of data processing is the Manual Era, in which we record and process everything manually. (1.0)
• The second era (1.5) is the cloud version, in which you work with data lakes and build a real "chain" that contains DataOps, MLOps, ValidationOps, and so on...
• The third era moves to the 2.0. It's where we stop obsessing on the data, and focus on events. We can use triggers and platforms like Heex Technologies to do it.
• The fourth era is the AI Era. (2+) This is where we have AI automatically find events, and train itself continuously on these.

Which solution is right for you? In reality, they can all work. A small startup can work manually, until they find their hard problem to solve, and they have a budget to invest in data lakes... Companies can work with data lakes, but for bigger fleets, it'd make much more sense to think in terms of events instead.

Next Steps

 If you realise you have these problems of recording everything, having your data staying a bit « dumb », and would like to know exactly how to stop recording everything by this afternoon (without losing the important information)...

... Then I’d recommend to check out Heex free discovery quiz, which will reveal tell you exactly what you’re doing wrong today, and (based on your answers) show you what to do this afternoon to save hours of recording, data processing, etc...

It’s free, and you can get access below:

Heex Free Discovery Quiz

Is your data strategy a silent obstacle?

Take the Quiz

You can also take a look at Heex' product here: https://www.heex.io/en-gb/smarter-data-faster-decisions

How much impact do you believe your job has? Is your job saving someone time? Or money? Or... his life? Well, what if you worked on projects that saved people's lives? Such companies exist, in self-driving cars, in healthcare, and in the case of this article... in Autonomous Defense!

This summer, I have interviewed Vibhav Ganesh from Shield AI, a U.S.-based defense technology company that develops autonomous systems for military and government use.

I would write a big paragraph here, but let me instead show you a quick sample from the interview...

Vibhav Ganesh is the Director of Engineering, past Chief of Staff to the CTO, and Employee #20 of Shield AI.

Vibhav has played a pivotal role in the company's growth and innovation. With a background in visual inertial odometry and SLAM, he has been at the forefront of developing autonomous systems like the Nova 2 quadcopter.

Let's read his intro to Shield AI and to their core products: the V-BAT and the ViDAR.

There are a lot of things to note about their products: the V-BAT can last 10 hours, which is a technological achievement itself, thanks to V-TOL (vertical takeoff and landing)... ViDAR is also a very interesting product, which stands for Visual Detection And Ranging... and HiveMind (not shown here) is their AI, or as they call it, "The World's Best AI Pilot".

Let me take you to the v-BAT first, as it's the core product, by showing you this LinkedIn post we did together, where Vibhav Ganesh introduces us to Shield AI.

Together, we recorded an exclusive Fragment of The Edgeneer's Land, my community membership experience, in which he takes us through Shield AI. What is autonomous defense? What are the main technologies involved? What is the range of products?

In this post, I'd like to give you a small sample of that interview, highlighting a very interesting moment where Vibhav talked about infiltration drones.

Inside Shield AI's Tactical Infiltration Drones

Impressive, isn't it? What I really love about it is that it's down to earth. I could see myself assembling a drone kit, adding a camera, a 2D LiDAR, and starting experimenting with Visual SLAM projects to map a room. This is basically what Shield AI did, when they got started. Except that their drone was (1) targetted to a specific client and (2) better than all competition.

There are 2/3 insights I'd like to share with you, from Vibhav:

1) Self-Driving Car > Autonomous Transfer doesn't work as we'd think

Now, here is something important to note:

A lot of what you learn in autonomous robots CANNOT simply be transferred to drones.

I did think that it was a matter of copy and paste. But I understood I got it wrong when making this fragment, especially with Vibhav Ganesh who told me that their drones don't have LiDARs, fly over seas, or over deserts, no man's land zones, between mountains, campaigns, with 3D constraints, and no map!

I thought about it for a minute, and I realized... "Wait, it's absolutely NOT like autonomous cars!"

And in fact, when you start looking into autonomous drone architecture, they absolutely don't look like self-driving car architectures! For example with Shield AI, they have a Control, a Station, RTOS, a ViDAR, but also a Flight Controller powered with frameworks like PX4 and Maven. This is an entire set of libraries to learn.

From there, here is a second insight:

2) Visual SLAM is mostly used

Coming back to the idea that we are NOT like self-driving cars... the other main difference is that we use no map. So without a map, and with just a camera, they have no choice but to use...Visual SLAM!

And Vibhav explains really well what kind of SLAM they're using, how they implement the mapping, even though there is 0 starting point, and so on. Here is a sample of a vSLAM project I've tested with drones:

This is the nitty gritty of Shield AI's work. Once you build a SLAM MAP, you can then feed that map to the Motion Planner, which sends a flight order to the drone. If you'd like more insights on this technology, I highly recommend my vSLAM article.

The 6 Components of a Visual SLAM Algorithm

How does Visual SLAM work? How is it different from normal SLAM? What are the 6 main steps of a Visual SLAM system? Let’s find out!

Read from the most advanced autonomous tech blogJeremy Cohen

Okay, would you like to see some samples?

Shield AI in Action

Let's take a look at 3 samples here:

• On the left, you can see drones being launched. These drones cannot do the "vertical takeoff and landing". They are projected to the air by launchers and then fly like a plane.
• In the middle, you can see the tactical quadcopters we discussed. Notice how they use vSLAM at the end of the shot.
• On the right, you can see a mission of the v-BAT searching for a vessel in a sea canal.

This is really cutting-edge, and totally applied what we are building in the autonomous tech space. Alright, time to wrap up!

Summary & Next Steps

• The defense industry is extremely active. Hundreds of companies work on the autonomous generations of drones, anti-missile detectors, infiltration equipment, RADARs, and more...
• Shield AI is an active actor of the defence space, with a range of multiple products, such as the v-BAT, the ViDAR, and HiveMind.
• Shield AI started with NOVA 1, a quadcopter that could infiltrate into buildings, builds maps, surveys, without the need to send humans into the region. This allowed to prevent human losses due to buildings collapsing or being trapped.
• Besides from being safer, infiltration drones are also more efficient. Shield AI tested their drone against 6 NAVY SEALs clearing a building, and it finished in a third of the time.
• The transfer of autonomous car/robot technology to autonomous drone isn't as simple as we'd think. Architectures are different, products are different, regions/environments are differents, and even the inside technologies and algorithms change.
• On the other hand, some technologies really apply well to autonomous drones, such as Visual SLAM.

On September 23, 1999, NASA’s Mars Climate Orbiter—a $125 million spacecraft—was set to enter Mars' orbit to study its climate and atmosphere. But just as it approached the planet, something went terribly wrong. Instead of entering a stable orbit, the spacecraft plunged into Mars’ atmosphere and was destroyed.

After analysis, NASA found a unit mismatch: their Jet Propulser used metric units (newtons), while the spacecraft they got from Lockheed Martin used imperial units (pound-force). This caused navigation errors, making the spacecraft descend far too low into the Martian atmosphere; and causing a 125m$ loss.

Human errors happen every day in all sorts of domains. In 2016, an alarming report from Johns Hopkins estimated that medical errors (including misdiagnoses) cause over 250,000 deaths annually in the U.S., making them the third leading cause of death. Many are due to errors in analysis of medical images, such as MRIs, X-Rays, CT Stans, and more.

In this article, I would like to show you how Medical Image Segmentation can be used to counter this problem, and I'll do it in 3 points:

1. 2D Medical Image Segmentation
2. 3D Medical Image Segmentation
3. Examples/Demo

Let's get started...

Intro to 2D Medical Image Segmentation

In 2019, I hosted the biggest AI Healthcare hackathon ever held, happening simultaneously over 20 cities! The goal at the time was to mix companies, healthcare groups, and engineers to build healthcare solutions using Deep Learning. After the 48 hours of coding, the winning team would win 10,000 USD, the second 4,000 USD, and then team 3, 4, 5, and 6 would win 2,500 USD each!

Great computer vision projects happened, and in fact, Paris (my city) finished the competition #2 via Spot Implant, a Shazam for Tooth Implants project that then became a startup. At the time, everybody was working on 2D Images. We had projects like Skin Melanoma detection, X-Ray segmentation, Brain Segmentation, and more...

Let me show you a few tasks in Medical Image Segmentation, and then we'll look at algorithms.

2D Medical Image Segmentation Tasks

X-Ray (the most common)

First, we have X-Rays. X-Rays are the 2D representation of a body. We often see bones and organs there, and it's the most common image you'll find in Deep Learning x Healthcare. Using medical image segmentation, we can assist doctors in finding bone fractures, lung diseases, and other abnormalities. It can also help in screening large volumes of X-rays for tuberculosis, which is particularly useful in low-income countries with limited access to radiologists.

This really is the most known among Deep Learning Engineers. I would like to show you other applications of segmentation...

Dermoscopy Segmentation (skin lesion segmentation)

Dermoscopy segmentation was the health hackathon's top pick. It's all about using medical image segmentation to spot and separate skin lesions in dermoscopic images. By applying deep learning on medical images, we can quickly and accurately detect skin conditions like melanoma. This helps dermatologists diagnose and treat patients faster and manage large amounts of data more efficiently.

Let's see one or two more...

Mammography Segmentation

Mammograms are specialized X-ray images designed to reveal the inner structure of breast tissue. These images typically come in a flat, 2D format, capturing the breast from multiple angles to ensure a comprehensive view. The details in mammograms can show everything from dense tissue patterns to potential abnormalities like lumps or calcifications.

Look at the image below: see how the role of a doctor/radiologist is to find these highlighted areas. The role of image segmentation (in this case instance segmentation) is to assist the doctor, so he's not alone doing that high stake task of spotting problems (of course, it goes without saying that doctors also do much more than spotting, from understanding how bad a calcification can be, to finding the treatment, and so on...).

Other Types: Ultrasound 👶🏽, Endoscopy 🤢, and more...

We just saw 3 types: X-Rays, Dermoscopy, and Mammography. There are other types, such as ultrasound images (baby for examples), which can be 2D or 3D; or endoscopy, and more... The image below shows many 2D segmentation applications:

So how do you build the segmentation results? What do you use? Let's take a look...

2D Medical Image Segmentation Models

Ever heard of UNet? You know, that 2015 model subtitled "Convolutional Networks for Biomedical Image Segmentation". Well, it may be from 2015, but it's a great way to start! In fact, there have been lots of improvements of UNet, to UNet++, Trans-UNet, Swin-UNet, all keeping that "U" shape, but using different pattern recognition techniques like Swin Transformers, CNNs, etc...

This is one family of semantic image segmentation algorithms, and here is what the results look like:

To get true numbers, Dice-Similarity coefficient (DSC) and average Hausdorff Distance (HD) are used as evaluation metric to evaluate these algorithms.

These are great, but what happens when you don't have millions of labeled data? In healthcare, getting access to labeled, free-to-use data isn't easy; especially for certain types of diseases that are specific to certain hospitals, and so on... In these cases, you can use more "foundational" semantic segmentation models such as SAM (Segment Anything) or SAM2. These have been trained using Self-Supervised Learning on "the entire internet", and thus are supposed to find problems better.

For example, MEDSAM is a Medical SAM (Segment Anything) is what I used for the images above. It's the regular SAM, but tweaked for medical image segmentation, to boost the segmentation performances. The model performance is quite high, and we get a top notch Computer Vision project using image segmentation... It can can even take your prompt as a region of interest bounding box to return the segmented masks:

So this is for the first part on 2D images... Now what are 3D images?

3D Medical Image Segmentation: CT Scans & MRIs

Now come 3D images! For this part, I'll talk about the two use cases (CT Scans & MRIs) and discuss the algorithms together.

CT Scans: Use Cases & Algorithms

Use Cases for CT Scans & 3D Representation

In the FLARE 2022 dataset (Fast and Low-resource semi-supervised Abdominal oRgan sEgmentation), we get access to a few hundred labeled and unlabeled cases with liver, kidney, spleen, or pancreas diseases as well as examples of uterine corpus endometrial, urothelial bladder, stomach, sarcomas, or ovarian diseases.

Hey, relax. I'm just scaring you. I didn't get a clue of what that meant either. Except that:

These are CT SCANS (Computed Tomography Scans). A CT scan uses X-rays to create detailed, cross-sectional (slice-by-slice) images of the inside of the body. They're more detailed than traditional X-rays because they produce 3D images by taking multiple X-ray images from different angles and combining them using a computer.

So what's the "3D" output like? Voxels? Point Clouds? Not exactly. As I said, these are images taken with multiple "layers" (or dimensions). So your input image dimension isn't (512, 512, 3) but (512, 512, 129) or something like this. You have a multi-dimensional image on which you can apply image segmentation to each of the 2D slices:

In this example, I used MedSAM to process individual 2D images. If you do it on the entire 3D CT Scan, you get something like this:

If you get it, you understand that from these images, we can put that into a software that is going to reconstruct the scan to 3D:

From there, people go absolutely nuts and even try to make it into a point cloud (I'm not sure why, but this is cool, shoutout to Beau Seymour's video).

MRI Scans: Advanced Medical Image Computing

Magnetic Resonance Imaging (MRI) Scans are another powerful tool in medical imaging. Unlike CT scans, MRIs use powerful magnets and radio waves to create detailed images of organs and tissues within the body. This technique is particularly great for soft tissue contrast, making it ideal for brain, spinal cord, and joint imaging. By leveraging medical image segmentation, MRI scans can aid in the precise identification of tumors, neurological disorders, and musculoskeletal issues.

Here is an example of MRI scan and its segmentation task:

So now, let's see how to process that...

Algorithms in the 3D Medical Image Segmentation Domain

We already discussed SAM (Segment Anything) and how it can work on individual slices. The reality is, medical image segmentation involves a lot of complex "job" knowledge; and it would probably be better to use a specialized artificial intelligence model for optimal model performance. Today, in AI, we have two types of models:

• Foundation Models, that are very general and know everything
• Specific & Labeled Models, that can only process the images it's been trained on

I would like to show you two models doing both: TotalSegmentor & Vista-3D.

Total Segmentator: A specific model for 2D and 3D Segmentation

Perhaps one of the most used and well known "framework" for image segmentation of both 2D and 3D data is TotalSegmentator. Rather than being a simple machine learning model, it's a complete framework that does the automatic labelling.

The number of classes for CT and MRI data it can segment is gigantic:

And the model is based on the nn-UNet architecture, which is similar to UNet, but can also take in different medical imaging modalities.

VISTA-3D: Foundation Model for 3D Medical Image Segmentation

VISTA-3D is a 2024 "Foundation model" from Nvidia that works on the 3D patch directly. While being named "foundation" model, it's incredibly specific to the medical image segmentation tasks. Here, we are PURELY in 3D Deep Learning.

So we've seen a lot:

• 2D Segmentation can be done with models like UNet, UNet++, etc... (specific), or SAM (foundation)
• 3D Segmentation can be done with models like nnUNet/TotalSegmentator (specific), or Vista-3D & SAM (foundation)

Let's see examples now...

Example 1: CT Scan Segmentation with Vista-3D

In this platform from Nvidia, I am able to select a CT Scan and call Vista-3D to process it.

Notice how we can select an Abdomen, and then pick all the organs we want to segment. Finally, we can get the view from 3 different "angles" and process that too!

Example 2: CT Scan Segmentation with TotalSegmentator

On totalsegmentator.com, we can upload images, and ask for complete segmentation. Here, I am going to upload a scan from the FLARE2022 dataset I mentioned above. The platform return hundreds of organs all in a weird format 'nii.gz' format:

I can visualize some of these, and see what the output is like:

Alright! So this is our second example, and both have playable demos! Let's now do a summary...

Summary & Next Steps

• Medical image segmentation helps reduce human errors by processing 2D and 3D medical images like MRIs and X-rays.
• 2D medical image segmentation tasks include X-Rays, dermoscopy (skin lesion analysis), endoscopy, mammography segmentation (breast), and more...
• UNet and its variants are popular models for 2D medical image analysis, utilizing CNNs or Transformer approaches. Foundation models like SAM (Segment Anything Model) can also be fine-tuned on medical images, like with MedSam.
• 3D medical image segmentation involves CT(computed tomography) and MRI (magnetic resonance imaging) scans. They're called 3D images because they're multiple slices of the same image under different views.
• MedSAM can process 2D slices of 3D scans, allowing individual segmentation of each slice. We can then fit that into a software that will do a reconstruction into a complete 3D image.
• For 3D processing, TotalSegmentator and Vista-3D are solid solutions, being either specific or foundation based.

In 1897, French police faced a difficult problem: a serial killer named Joseph Vacher was stealing and murdering sheperds, and remained impossible to catch. Every time he was arrested, he gave a different name, changed his appearance, used fake mustaches, wigs, and different clothing styles... and got to disappear without the police realizing they just controlled France's most wanted man.

At the time, France had no national ID system, and no way to prove that the man they caught today was the same man they arrested months ago. That was until an officer named Alphonse Bertillon introduced a revolutionary method: anthropometry. It's a system that labeled criminals based on of 12 unchangeable physical measurements like ear shapes, skull sizes, and limb lengths, that could not be faked.

One day, Vacher was caught for attacking a woman, and this time, the police used Bertillon's system to compare his measurements to what they had in their records: they discovered they just caught France's most wanted criminal. This time, he could not escape with a warning, and got sent to... yeah — the guillotine 🤷🏻‍♂️🇫🇷

What got Vacher executed wasn’t just this one-time capture, but the ability to analyze a series of events and not just a one-time event. And this is exactly what this article is about: the shift from frame-by-frame to sequence processing, here in Computer Vision with videos. And this is done via something called video segmentation.

So let's get started:

What is Video Segmentation?

Most Computer Vision Engineers spend time learning about image processing, but never consider what happens when you use a video. Yet, tons of architectures today, whether in surveillance, retail, sports analysis, healthcare, or even robotics and self-driving cars — now process videos instead of images. The sequence brings something individual images don't, just like the Vacher story, where he was able to get judged through all the murders he committed.

So let's take a less deadly scene — shoplifting detection in retail. There is a startup I once interviewed for named Veesion — that has this amazing video on their homepage:

Can you see everything happening here?

• We have the object tracking (the second man is moving from aisle 1 to aisle 2)
• The event detection (at 00:03, a man puts an item in a pocket)
• The action classification (of putting something in a pocket)
• The video decomposition (shoplifting from 00:02 to 00:03 — standing from 00:03 to 00:06)
• The people counting (2 people in the video, one is obstructing the other)
• And more...

Among these, there is the idea of "segmenting" the scene to track the shoplifters through the video. You can see the hands being in red, consistently from frame to frame. So this is the idea of Video Processing, and Video Segmentation is a sub-branch of it focus on the task of segmenting a scene.

There are two types of Video Segmentation tasks:

• Video Object Segmentation (VOS)
• Video Semantic Segmentation (VSS)

Video Object Segmentation

In Video Object Segmentation, we are doing exactly what I did in this video. I define an object to track, send the video to the model, which tracks the object consistently across frames. It's purely "object" based, and is NOT used in a supervised way. For example, you can use semi-supervised video object segmentation, where you define an object on Frame 1, and let the model track it across the next frames... Or you can use totally unsupervised video object segmentation, where you won't even mention the objects to track.

Let me show you an example where I am shoplifting (muahahah):

See? We are able to track my head & hands in blue, and the phone in yellow! That is the idea we're interested in... And even more when we can do this:

Now, to be fair, the floating head experiment may NOT be the most useful thing in this example, but the stolen phone is. Now think of everything we can keep track of cells in health related videos, we can keep track of a player when analysing a football match, and a lot more...

Video Semantic Segmentation

In Video Semantic Segmentation, we'll really go at the pixel level, and rather than focusing on segmenting objects, we focus on the scene. The output is going to look extremely similar to a normal image segmentation task.

Just like image segmentation, you can also use video instance segmentation, video panoptic segmentation, video semantic segmentation, and so on... And of course, there is the benefit of doing background extraction, to then process uniquely what's been segmented, for example in a case like lane detection in self-driving cars:

But from now, you may have a question:

How is Video Segmentation different than Image Segmentation?

I mean, is it really different? It kinda looks similar to. image segmentation, right? And yes, while it may be the case for some examples, like the one I just gave with video semantic segmentation, most of the tasks will be different and give different outputs.

To put it simply: Video Segmentation is about processing videos. You don't process image per image, you process video frames immediately. And this has several advantages:

• The model can track multiple objects even though they're occluded (similar to what object tracking would do, but using video sequences)
• The model can segment specific scenes you're looking for (a blood cell changing sizes, a car entering a scene, a man stealing something)
• It ensures temporal consistency, meaning an object that appears in one frame keeps the same identity/color across the entire video, enabling tracking at the same time.
• It understands object motion, meaning it can predict where an object will be in the next frame instead of treating every frame as an isolated image (thanks mainly to video instance segmentation)
• For some models, it can be more efficient, since instead of running image segmentation on each frame separately, the model processes a video sequence, leveraging temporal information to process frames together, reducing redundant computations.

So, how does that work? What type of model does this? I do NOT have a specific "do this do that" template to share with you, but by studying examples, we could probably understand what's required to make a Video Segmentation algorithm work...

Example 1: VisTR (Video Instance Segmentation Transformer)

The first paper looks terribly simple. Let's try to understand the different blocks:

• Input: First, we process raw video data, it's purely a sequence of images sent to the CNN
• Backbone: Then a normal 2D CNN processes each frame independently before concatenating the feature maps
• Video Processing: This is fed to a Transformer, known to process sequences quite well. However, we modify this transformer a bit to not just receive a positional encoding, but also a temporal encoding.
• Output: Finally, the output of the decoder predicts instances for each pixel, with a sequence matching strategy

The training is done after obtaining labeled data from the YoutubeVIS dataset, and the backbone is initialized with the weights of DETR.

The detailed version looks like this:

As you can see, we have a video processing pipeline, where the transformer is actually aware of the frames. The segmentation process ends by matching pixels with instances. This is done using Bipartite Matching (the Hungarian Algorithm). More subtle blocks exist, and I invite you to read the paper for more...

Example 2: SAM 2 (Segment Anything 2)

If you didn't live in a cave around 2023, you probably heard of Segment Anything — the segmentation model that could find any object in an image. Recently, it got an upgraded version called SAM2, which is designed to process videos. Let's take a look:

As you can see, SAM 2 differs from SAM by the addition of a memory block, made of a memory attention module, a memory encoder, and a memory bank that stores the past frames, and helps with temporal consistency.

If you played with the online demo, you will find that the model starts by asking you to click on an object, so it can keep tracking it. So at frame 0, you click the object you want to track, and then the model tracks it on the entire sequence...

This is called "Promptable" Visual Segmentation.

This is no different than the original SAM model, and in fact, it's using the same "prompt encoder". So let's the see the details of the model:

• Prompt Encoder: As expected, we begin by clicking objects, which generates a "prompt", and send this to the same encoder as Segment Anything to track object across each image
• Image Encoder: We then send the entire vide to the image encoder which is a masked autoencoder
• Memory Attention: Uses vanilla attention to condition the current frame features on the past frames features and predictions as well as on any new prompts
• Memory Bank: It retains information about past predictions for the target object in the video by maintaining a FIFO (first in first out) queue of memories of up to N recent frames.
• Mask Decoder (prediction): Similar to SAM, but accounting for previous memory information

So, you saw a second way to build a video segmentation algorithm. The first way was fully transformer based; and this second way has the somewhat robotic "memory bank"; and this because this model is a "hybrid" between 100% video processing, and frame-by-frame processing.

Image vs Video Segmentation: Worth the trouble?

I would say yes, especially considering all the use cases that can benefit video segmentation. For example, surveillance with massive occlusions (in a crowd, with walls, trees, ...) where standard object tracking would be limited, video editing, where for example, we want to remove an object not from one frame, but from an entire scene, sports analytics, entirely based on motion, cell tracking (for example, division of cells, which can only be seen via videos), shoplifting detection (which can't really be seen in an image), fire spreading, and more...

You can see this article for the normal image segmentation use cases, and I highly recommend you augment it in your mind with these video examples I provided. So as a rule:

• For most cases, don't replace all your image segmentation pipelines with video pipelines
• But for the cases where segmentation fails because you need to understand video, do it!

Alright, we've seen a lot, let's do a summary...

Summary & Next Steps

Congratulations on getting so far! Let's summarize what we learned:

• In many cases, analyzing one event fails. When video is essential, you have to use Video Computer Vision models.
• Video segmentation is segmentation applied to video processing, it's used in in various fields like surveillance, retail, sports analysis, shoplifting detection (or detecting suspicious behavior of any kind) and healthcare.
• Video Segmentation splits into two categories: Video Object Segmentation and Video Semantic Segmentation.
• Video Object Segmentation (VOS) focuses on tracking defined objects across video frames. Many applications like SAM2 are semi-supervised, because you give the model a prompt and an initial object to track.
• Video Semantic Segmentation focuses on pixel-level scene segmentation, it can also be instance or panoptic based, and the output may resemble the one of standard image segmentation.
• Some models like VisTR can be 100% video processing based. This model uses transformers for video instance segmentation.
• Other models can process frames one by one, but rely on a memory bank. In the case of SAM2, frames are processes both as a video and one by one (to keep track of a same object)

Next Steps

A few articles you can read:

• Introduction to Video Processing — an old post (you can see my writing style is much different) but good overview on Video Processing.
• A complete overview of Object Tracking Algorithms in Computer Vision & Self-Driving Cars - very related to video object processing, but without the segmentation part (bounding boxes).

In 2019, I was an Autonomous Shuttle Engineer, working for a company that got a thrilling opportunity: to equip Paris' transportation system with our autonomous shuttles. This was a golden opportunity many don't have, but the client was known to be ruthless selectors. Many others perished while trying to be "approved".

With high hopes, our team prepared for the demo day for months. We meticulously reviewed the client's 100+-point checklist, ensuring our shuttle met all requirements from real-time operations to autonomy measures. One day, a team of 5 was called to begin process in a secret underground site. It was going to begin.

The experimentation lasted days, in which each of the items were reviewed. Came the final test: Cyber-Security. The client made a phone call, and within 30 seconds, an engineer with a thinkpad came and entered the shuttle. "Oh great! We can charge our phones!" He said amused. "What a mistake!". My colleagues were sweating, horrified at the vision of what this young men could do.... and they were right: In just five minutes, using only a USB stick, he had taken control of the vehicle, got it to drive all across the room. The room went silent, as everyone realized our chance had slipped away.

Checkmate.

Many engineers join the self-driving car world for the same reasons I did: it's exciting, it's interesting, it's a passionating, it's impacting, it's just... wow. Yet, nearly all the engineers who are on the "learning" group and have never joined a real self-driving car company yet have absolute zero vision on what it takes to certify a vehicle. We could talk cyber security, but even at the automotive level, the software level, and more...

So in this post, I will try to sensibilize you to the concept of safety, from an autonomous tech engineer point of view. This means — this post won't be for expert functional safety engineers, but for those who want an introduction.

Let's begin with the fundamentals:

What is Functional Safety?

Functional Safety is about making sure machines and systems stay safe, even if something goes wrong. For example, in self-driving cars, it means making sure the car can still drive safely if a part stops working. It can mean verifying that an algorithm works under all conditions, but also that it's never going to crash, and that if it does, the system has a backup.

To make it work, we use functional safety standards that determine what is safe to include in a self-driving car by evaluating the potential risks associated with each function and scenario. You can therefore understand the entire point of functional safety:

To reduce risk to an acceptable level.

Okay, but this shouldn't be your job, right? It's someone else's problem! So you may wonder...

Why should I bother learning about Functional Safety?

Let's say you decide to build an autonomous tech startup and run your algorithms. Some are open source, some are designed by you. You decide that these are good algorithms, the accuracy is near perfect, and you're a brutal C++ coder. There is no way you missed anything. Let's even pretend you really ARE a super-hero and really, the system is perfect...

You convinced me... but can you convince recruiters? Or your management? Or the suits giving your startup a self-driving permit? Hey — you can't test without the permit. No matter how good your system looks, you will need to convince the state to deliver you a permit. It can be the State of California, or the Ministry of Transport, or whoever delivers authorizations.

The problem? They are NOT experts in safety or self-driving cars. So they will ask you to go via independent organizations, who run functional safety certification programs. Organisms like TÜV Rheinland and TÜV SUD (Germany) are the ones 'certifying' you. They're verifying your safety functions, even the safety critical functions (emergency braking), and doing all kinds of silly tests before issuing you the certification.

Their job is to verify you are compliant with the industry norms.

But which norms are we talking about?

What are the different functional safety norms used?

When we say we want to "reduce risk to an acceptable level"... What is an acceptable level? Are you the one defining it? If an object detector works at 95%... is this okay? No? Yes? Who defines it? If your blinkers fail once every 300,000 miles... is this fine? Or is it every 3 millions miles?

You can't be the deciding entity, this is what norms and industry standards are for. For example, ISO 26262 is a norm. It's focusing on electronics (buttons, A/C, windows, sensors, computers, ...), and defines a complete process to develop & test your cars. It also tells you how to test scenarios, how to grade the risk of any event, and how to reduce that risk.

Let me share some norms we use in the industry:

• ✅ ISO-26262 is the norm that focuses on failures in electronic and software systems. It's going to deal with the question "What happens if the object detector crashes mid-drive? Is there any backup?"​​​​​​ Based on how your system is implemented, you will comply more or less with the norm.
• ✅ ISO-21448 verifies the Safety of the Intended Function (SOTIF). It ensures perception systems like LiDAR, cameras, and object detection perform safely in all conditions. "Is your object detector working on all pedestrians? Really? Even in the dark?"
• ✅ ISO-21434 is the norm focused on cyber-security of the system. It solves my USB-stick story. And it tells you everything you need to do to ensure your model is free from cyber attacks.
• ✅ A-SPICE is focused on how your project is coded, tested, and maintained. This means the requirements, the modular and maintainable code, the coding standards & reviews, the software testing, software versions and revisions, bug fixing, lifecycle of the product, etc...
• ✅ UNECE WP.29 Regulations is the compliance with EU autonomous driving laws. You need at least this one to be allowed to drive autonomously.
• and more... depending on what you want to certify.

While these are not mandatory, the more of these norms you check, the safer you'll look.

So comes a question:

How to know if your robot complies with Functional Safety norms?

There are TONS of ways to do this, and it's really a profession, but let me share with you 2 important functional safety concepts:

1. The V-Model
2. The Functional Safety Process to "certify" a function

The V-Model

The V-Model is a widely used framework in functional safety management and software development. You will find it when trying to comply with ISO26262, but also A-SPICE for example. It is structured like a "V," where the left side represents the concepts/requirements/design phase, the bottom part is the coding phase, and the right side corresponds to the validation/integration/testing phase.

You can see it as a continuous process, where you continuously verify that your system behaves as intended in the concept phase. If not, you rework it. It's evolving, it's alive, promoting a systematic approach to achieving functional safety in safety related systems.

In most companies that seriously want to comply with the ISO norms and get the functional safety accreditation, using the V-Model is the best starting point.

Next:

The Functional Safety Process to "certify" a function

As we said, we have ISO26262 focusing on electronics, SOTIF focusing on algorithms, and A-SPICE focusing on code/software. Each of these is using the V-Model. Then, to comply with these norms, you'll need a "process". This means defining clearly what each of these phases are.

Here is a 7-Step process:

The job of a functional safety engineer is to implement this. This is the "bridge" between systems and production I was telling you about earlier.

Let me briefly define each element: (credit to a client of Think Autonomous named Mayur Wagchoure for helping me write this one)

1. Define the System

First, we want to define the system we're testing. For example, lane detection. We want to define the purpose, the scope, the dependencies, and even the normal and edge cases.

2. HARA: Hazard Analysis and Risk Assessment

The second point is HARA, in which we want to do:

• HA — Hazard Analysis (what could go wrong?)
• RA  — Risk Assessment (how bad would that be if it went wrong?)

Hazard Analysis

If you want to comply with functional safety standards, the first thing you'll need to do is account for the different scenarios. I see them into 4 main sections: Car Status, Scenario, Environment, Driving Status.

Your car could be turned on, driving in a country road, with rainy conditions, and driving at low speed. Or you could drive at high speed, and accelerate. Or suddenly brake. Or drive in dry roads. Or wet roads. Putting categories into each of these is a way to avoid the summer/winter rookie mistake.

Risk Assessment

To "grade" each function, you then use the formula defined by ISO26262: Risk = Severity * Exposure * Controllability.

For example:

• I am testing the emergency braking function, and the risk that it doesn't activate (Severity = S3)
• I'm driving in urban environment, at 30-60 km/h, which happens all the time (Exposure = E4)
• Urban areas have many pedestrians, it's very hard to control (Controllability = C3)

Then what?

The ISO26262 provides what's called the ASIL (Automotive Safety Integrity Level) Table:

I am NOT going to describe how we do in this article, but the "RA" phase is about assigning, for every single function and every single scenario, what's called an ASIL level. These can be A (safe), B (safe), C (risky), or D (risky). We're trying to see, for each function, is it risky or safe?

For example:

If you're testing an emergency braking system, in a highway scenario, with wet road, snow, and fog... you can imagine it's an ASIL-D score. Now if you're on the same scenario, but testing the radio, it's probably A or B.

3. Set Safety Goals

From every potential hazard and risk we have, we want to turn this into a safety goal. Basically, turn the failure into an opportunity to design a better system. If I have just one LiDAR, and it's working bad under snow, could I have a better LiDAR and a camera instead?

Here, we will create a list of requirements for the new system. It's still the "concept" phase, where we identify the breaking points, and turn this into a better solution.This is the work where you try to think about reducing risk to an acceptable level.

4. Functional Safety Analysis

Then, we implement things like FMEA (Failure Mode and Effects Analysis) to assess potential failure causes, effects, and mitigation strategies. We can also run FTA (Fault Tree Analysis) to explore how faults propagate and lead to hazards. We want to identify all causes of errors.

5. Design Safety Mechanisms

Then, we're introducing mechanisms to detect, isolate, or prevent failures (e.g., redundancy, diagnostics, fail-safe systems). This can be watchdog timers, dual-channel systems, degraded operational modes, ...

For example, one of the Functional Safety Methods is to implement redundancy. If you have an ASIL-D component (unsafe); you could turn it into 2 ASIL-B ones (somewhat safe). This way, your overall ASIL score is better, and you become compliant.

In this example, we could imagine that the second LiDAR is different, or that the algorithms behind it are more "deterministic", don't use AI, and therefore are safer. The goal of functional safety is to try and reduce as many components to ASIL-A and ASIL-B as possible. >>> This is the acceptable level.

6. Validation and Verification

How do we test? This can be field tests, but also simulations, hardware-in-the-loop (HIL), and fault injection testing. You can also here test the Safety of Intended Functionality (SOtIF) — how performant is your algorithm? Is it really THAT good?

Finally:

7. Iterate, Validate, and Document

You want to iterate, improve, and document your safety analysis results. In the end, it's a very technical job, but that has a lot of paperwork, documentation, diagrams, schematics, grading, because these are the papers giving you authorizations.

We have now seen:

• What is functional safety?
• What are the different norms we should comply with?
• How do we comply with these norms (overview)

Let's see an example:

Example: Mobileye's Primary Guardian Fallback / "True Redundancy" System

Mobileye, Intel's self-driving car company, is has a very strong functional safety focus. Their algorithm has 3 distinct channels that are completely different:

The lane detection is the main channel used to fine lane lines. This can work for example with modular deep lane detection. It is verified with HD Map Extraction & Localization. If they agree, then we're good, but if they don't, they'll extract the lanes from a parallel end-to-end deep learning algorithm that will act as the "judge" or guardian.

Do you realize how many algorithms are running in parallel? They implemented these automatic protection functions in case of failure. They also implemented these safety requirements across the entire system, meaning the electronic systems, the software components, and so on...

When doing something like this, it's very important that each function is run using a separate method, possibly with a separate computer, separate sensors, etc... so that there cannot be a single point of failure (for example, if everything uses the same camera, and this one fails, it's not functionally safe).

Wait... Does everybody really do all of this?

No.

In fact, many startups don't have a functional safety team, or even have a safety system in place. In this case, they try to do it in the safety critical systems, while waiting for the certification process. Some are also in a more favorable state/country that gives permits more easily (to enhance innovation and startups work on the technology).

It's important to understand that complying with ISO norms is NOT mandatory. In the European Union, you need to comply with the UNECE WP.29 Regulations (traffic laws) but I don't think the ISO norms are mandatory.

In fact, Tesla doesn't comply with the norms, and they are approved to drive in the streets. They sell cars, and they even sell autonomous cars all across the world. But you'll note that some of their functions, like FSD (Full Self-Driving) are currently (early 2025) NOT authorized everywhere, like in Europe, because they don't comply with all the norms.

Okay, okay, I think we have ENOUGH! Let's do a summary...

Summary & Next Steps

• Functional safety makes sure robots and algorithms operate safely, even when something goes wrong, by reducing risks to an acceptable level.
• Every engineer working in the field should be introduced to safety. This defines how you code, but also whether a startup gets authorzations to drive or not.
• Key functional safety norms include ISO 26262 for electronics, ISO 21448 for algorithms, ISO 21434 for cybersecurity, and UNECE WP.29 for EU compliance.
• The V-Model is a structured approach in functional safety management, covering concept, coding, and validation phases to achieve compliance. It has a V shape doing Conception - Coding - Testing.
• Functional Safety is a 7-step process includes defining systems, hazard analysis, setting safety goals, and implementing safety mechanisms to ensure compliance.
• The ISO26262 norm defines risks as Exposure * Severity * Controllability. An ASIL table then defines for each function, which grade it has.
• When something is risky (ASIL-C, ASIL-D), we introduce redundancy, diagnostics, and fail-safe systems to detect, isolate, or prevent failures, enhancing the overall safety integrity level.
• We want to test through simulations, field tests, and fault injection to ensure safety functions perform under all conditions, meeting the required safety standards.

Alright, I think we are good!