Ever done a "house swap"? Recently, one of my mentors in Canada told me he was swapping homes with someone in the Netherlands. Sounds unreal... Yet it isn’t. Platforms like Home Exchange apparently have 100,000+ members doing exactly this.

House swapping is one of those things that could never have worked a decade ago. Not because the idea was bad (I think it is, but that's different), but because trust, norms, and infrastructure weren’t there.

And RADAR Deep Learning follows the same pattern.

RADAR has existed for over 100 years. Most RADAR algorithmic is still traditional signal processing. As a result, RADAR engineers have long been a small, almost outcast group of "freaks" (sorry) working on systems few people truly understood.

Why? Because for decades, RADARs were treated as a secondary sensor. Too noisy. Too low-resolution. Useful only as an auxiliary input in sensor fusion, under the assumption that even noisy measurements are better than nothing.

That assumption is now breaking.

RADARs are moving into a primary sensor role:

• high-resolution RADARs exist
• imaging 4D RADARs are spreading (see my article here)
• And more importantly, DEEP LEARNING is now so capable that processing even noisy point clouds can be done!

This is why in this episode, I am boarding a train to Rotterdam, where I am meeting with Andras Palffy from Perciv, a startup focused on RADAR Deep Learning.

He's today running Perciv, and I'm going to show you an amazing video of his work...

WOW!!! So cool, isn't it? Now, in this post, I will cover 2 ideas to explore:

1. The process of Deep Learning for RADARs (how does it work)
2. The applications you can do when leveraging 4D Deep Learning

Let's begin with the process:

How to make Deep Learning for RADAR work

Let's begin with this post showing you a demo of Perciv AI's algorithm:

Can you feel the power? This video shows object detection, but what's very interesting is how noisy the input is! The points are "dancing", unlike most LiDAR point clouds, which are much more robust and accurate.

Yet, RADARs provide direct velocity estimation, via the Doppler Effect, making them very interesting sensors to use.

So how does it work? It's really 3 steps:

1. A RADAR outputs a raw signal.
2. This signal is often converted to a 2D or 3D point cloud to be processed.
3. 3D Deep Learning algorithms are working on the point clouds with points or voxel approaches, just like for LiDARs.

Now the interesting element:

Most traditional RADAR algorithms skip step 2, because they process the RADAR signal directly (you can see how in this article). In the case of Deep Learning, we have the option to either convert to a point cloud OR process the raw signal directly. This means that step 2 (signal → point cloud conversion) can be skipped, which avoids losing data during conversion.

We now get the general idea: Thanks to Deep Learning, we can make noisy RADAR data useful. The next question is, what exactly can we do?

Applications of Deep Learning for RADARs (By Perciv)

Here is a 30 second clip I recorded at Perciv going in-depth of the sensors, algorithms, and end-user interface.

Let's begin with the sensors. Did you count how many there were? I see 1 camera, 2 LiDARs, and one RADAR that has 2 views: a point cloud view, and a range-doppler view. If you zoom in, you'll see that the RADAR point clouds are absolutely chaotic. There is no way you'd make sense of it.

And yet, when you see the blue part, in the middle of the video, you see what the Deep RADAR algorithms are capable of. The algorithmic panel is ALL based on the RADAR input only. And notice how awesome they are, we have:

• LiDAR + RADAR Accumulator
• RADAR Heatmap
• Freespace Detection
• 3D/4D Object Detection and Perception

Seriously...

A freespace detector... on a RADAR!

This is really impressive, isn't it? And it's not ALL, because later on, Perciv AI showed me a side-by-side comparison of SLAM with RADAR and LiDARs. Can you guess which one was superior?

Here's the answer:

While the RADAR Odometry uses the velocity information and can accurately spot moving points, LiDAR doesn't, and as a result, overshoots!

This is a very good example of how Deep Learning for RADAR can be used for advanced applications.

Summary

• Perciv AI builds Deep Learning for RADAR algorithms and they are awesome. I've been following Perciv since 2023, even interviewed them when they were only 3, and their dedication to this field is unmatched.
• In RADAR processing, you can either process raw signal, or convert it to a point cloud the same way you'd do with LiDARs. A heavier pre-processing step is usually done to reduce noise.
• The RADAR processing pipeline therefore becomes: signal → point cloud → 3D Deep Learning algorithms → output
• There are many algorithms you can run on RADARs, from object detection to SLAM. In some cases, RADAR's velocity information can even provide BETTER results than LiDARs.

Infiltrate Perciv AI with me?

The last time I visited Perciv AI, I got a complete tour of their facility, team, 4D Deep RADAR algorithms, and even self-driving car. I got to live as an intern on his first day of a self-driving car startup.

I'm thinking...Wanna see what it's like? I mean, what I'll record there will obviously be top secret, guarded and accessible ONLY to the Edgeneer's Land citizens (my community membership)....BUT the show?

This is a show they just did at IAAA Munich to everybody. And I see no reason why everybody shouldn't discover it. This is why I'm creating a special 2-day Virtual Tour, in which you'll be able to come with me in Rotterdam, be a fly on the wall, and get to live your first day as a self-driving car intern...You will see things like:

• ✅ Their self-driving car — if you never saw a self-driving car before, this will be the closest you'll ever get, we'll see the sensors, wires, everything
• ✅ Their 4D Deep RADAR demo — where they will demo their algorithms on me!
• ✅ Their RADAR tour — where they'll show you what is a RADAR, and give you a tour of the different types in the market
• ✅ The RADAR vs LiDAR SLAM video — explaining the differences in Odometry estimation and how to do a clean one using RADARs

As I said, this is the public stuff you normally CAN'T see unless you physically move to where they are. For 99% of people reading this, this is a unique chance to see it. Interested?

In 1607, the Jamestown colony was in a critical situation. English settlers founded it and declared it their first permanent colony in North America. They arrived with total confidence: they knew how to build a town. So they built wooden houses, palisades, shallow foundations, just the English way. But there was a problem: Jamestown was built on a swamp.

Within weeks, houses collapsed, mosquitos propagated malaria, and the water they were drinking caused fever and poisoning. Within months, half of the settlers died. Yet, the remaining didn't figure out a better plan, and too much was already decided. It's only after enduring famine, diseases, and war with locals that they found the right approach, the one that turned Jamestown into the first american colony.

Solid-state LiDAR are that final method. In the LiDAR industry, many have experimented with all sorts of sensors, until mutually agreeing on an "ideal" solution: the solid-state LiDAR. Not only it could reduce cost, but it could also significantly improve the performances.

In this article, I am going to explain to you what is a solid-state LiDAR, how do they work, and more importantly, why they're a better choice than most of the other sensors. To truly understand solid-state, we'll need to also understand mechanical LiDARs, and all their moving parts.

This will be our first point...

The Components of a LiDAR sensor

If you want to understand mechanical and solid-state LiDARs, you'll first need to see the internal components of a LiDAR. Then, we'll figure out how to classify a solid-state LiDAR based on how these parts move.

I am NOT going to describe these one by one, because I would like to instead show you how they all work together.

Keep these in mind, and let's take a look at...

From Mechanical to Solid-State LiDAR

The Mechanical 360° LiDAR

Back in 2017, I took my first LiDAR class. It was featuring a Velodyne 64, which is a mechanical LiDAR (Light Detection And Ranging) that became the most famous LiDAR in the autonomous vehicle industry. At this time, it was costing over 100,000$, and promised to transform several use cases (indoor, outdoor robotics, SLAM, ...).

The principle of this LiDAR is simple; multiple lasers are stacked vertically on mechanical rotating components that spin really fast.

From here, you start identifying the advantages (accuracy, 360°), but also the drawbacks: it's terribly costly (100k or so in 2017), and better 3D requires more channels - hence more lasers (bigger sensors).

This is how we started introducing the second types...

The Mechanical Mirror LiDARs

In this evolution, we no longer rotate the entire sensor, nor use multiple laser pulses, but instead, use mirrors and polygons. Here is an animation explaining how the next 2 work, that I found in this fantastic video again from Hesai:

• 1D Rotating Mirror: The first alternative could be a single mirror that deflects the laser. Think about it, this is genius! We can use a mirror that spins horizontally to recreate that 3D shape. Of course, we'd need multiple lasers stacked, but we fix the problem of having a rotating platform, which can break.
• Polygon-Mirror: Another alternative is to use ONE laser, and deflect it via the use of mirrors and polygons. In this case, the mirror swings vertically, and the polygon spins horizontally. This creates a 3D representation, which is narrower, can't spin 360°, but produces a functional point cloud.

These two are great, but still require you to use polygons and mirrors. In a way, it's still mechanical. So let's now talk about the true definition of solid-state...

Solid-State LiDARs = "No Moving Parts"

The first time I learned about it was around 2021 when a company asked me to help them choose between multiple LiDARs. At the time, solid-state technology was emerging, and many were saying it was the future of self-driving cars. The definition was repeated by everyone everywhere;

"No Moving Parts"

Huh. What's so problematic with moving parts? Is that so terrible? Well, yes, because when used all day for weeks and weeks, these parts will simply... break!

If we compare solid-state to mechanical LiDARs, we can also see that in 100% of the cases, solid-state is a directional sensor. This means you cannot use it on the roof of your car; you have to orient it very strategically, and you must use several of these sensors if you want a 360° view.

Now, let's try to understand the differences, and how we can get a 3D point cloud without moving lasers.

For this, I'll use the matrix below, which shows the different types of LiDARs based on the components moving. (realize you already covered the first 3 dark rows).

Let's see these, one by one:

Deep Point Clouds Waitlist

Deep Point Clouds Waitlist

MEMS (Micro-electromechanical system)

In a MEMS LiDAR, you're projecting one laser to a MEMS mirror that oscillates both horizontally and vertically. It mimics the LiDAR + mirror rotation, but it's now an oscillation at the micro level.

MEMS mirrors still move, so MEMS LiDARs are not "true" solid-state. Yet, they are excellent alternatives to the mirrors, more resistant to vibrations, and shocks. When looking in more details, LiDAR makes either use a 2D MEMS mirror, or two 1D MEMS Mirror, oscillating horizontally and vertically.

OPA (Optical Phased Array)

What is a LiDAR? It's a device that sends a light wave. Correct? Well, a light wave is a... wave. Yes? And a wave is something we understand. It has an amplitude, a phase, a frequency, and a wavelength! In an OPA LiDAR, we use a phase shifter to electronically steer the light wave. This sounds crazy, but it works. This is really modern, new generation, and a "true" solid-state system, since no part is moving.

Flash LiDARs

In a Flash LiDAR, a diffuser projects a wide, diffused laser illumination which comes back to an array detector, creating a full 3D image in a single exposure. This is a non-scanning technology; everything is illuminated at once.

Was that clear? Well, imagine being in the dark, and trying to illuminate the room.

• You can either agitate a red laser all over the place (scanning devices - MEMS, OPA, ...)
• Or you can use a torch, which instantly illuminates the room.

This is what a Flash LiDAR does, it's a laser torch.

Solid-State Summary

Cool, a quick summary of the last 3?

We now have a good understanding of Solid-State. The question I want to continue with is...

How is Solid-State better than Mechanical LiDAR technology?

There are several aspects that you can already guess, but I'd like to take these one by one anyway.

Better durability (no moving parts)

Mechanical LiDARs have moving parts, which wear out over time and increase the risk of failure in automotive environments (vibration, heat, dust). This risk is real for MEMS (which we saw is partly mechanical), but completely reduced for OPAs and Flash LiDARs. The #1 advantage of using a solid-state LiDAR is this.

Compact & lightweight Design

A mechanical LiDAR HAS to be on the roof of a vehicle. This is not only ugly, but also impractical. On the other hand, a solid-state LiDAR can be nicely integrated in the front of a vehicle. This makes Mechanical LiDAR not such a good option. When you look at the ADAS (Advanced Driver Assistance System) industry, most companies like BMW, Mercedes-Benz, etc... include MEMS LiDARs in the front. Its small size makes it ideal for integration into space-constrained platforms like drones and autonomous vehicles.

Let's continue:

Mass Production Capability

Manufactured using semiconductor processes, solid-state LiDARs can be mass produced with lower costs. MEMS are currently the cheapest, but OPAs promise to reach incredible costs (100$ or less). The math makes sense, we got lower size and lower cost, which is always the direction we want to go towards in hardware.

Point Cloud Resolution & High Performance

A mechanical LiDAR solution based on spinning mechanics often provides sparser point clouds, especially vertically, with gaps in coverage compared to dense sensors like cameras. This can lead to blind spots for low or small obstacles. On the other hand, a solid-state LiDAR can capture hundreds of thousands of points per second, and has a higher angular resolution, which is very good for tasks like 3D mapping or obstacle detection.

• With this, a solid-state LiDAR has lower power consumption (good when using drones for example), could resist environmental conditions better, scan faster, and have a flexible field of view.
• Other than the field of view, the modulation itself is very much manageable; most FMCW (frequency modulated continuous wave) LiDARs are for example based on Solid-State, and NOT mechanical.

In industries like self-driving cars, smart cities, industrial automation, robotics, using something with high resolution, high accuracy, good enough distance/range, and potentially a wide field of view makes total sense.

Range, Resolution, Performance?

The following is to take with a pinch of salt, because it varies very often and some companies have crazy claims. Yet, I also looked at studies like this one from IDtechEx, this one on MEMS mirrors , and that one on OPAs. Here is an overview:

Can you see why MEMS, which even though is not really solid-state is the BEST compromise? It's the only one that can currently be mass-produced at a low price, while keeping good range and high resolution.

You can therefore see how MEMS and Mechanical LiDARs are still the ones being used the most in the industry. True solid-state is a crazy dream, with incredible claims (an OPA LiDAR could reach a cost of 100$). For now, we aren't there yet.

Example 1: Innoviz Technologies

At CES 2026, I have explored solid-state LiDARs with Seyond & Innoviz. On the one hand, Seyond that you already saw, is doing Flash LiDARs, which is "true" solid-state. On the other, Innoviz is very likely doing MEMS, which is... hybrid (still following?).

I would like to start with Innoviz Technologies latest demo:

Did you see how awesome that looks? Now, you can notice how the benefits here are related to cost reduction, to size shrinking, and heat/power reduction. On the other hand, let's now see a demo of a Flash LiDAR:

Example 2: Seyond Flash LiDARs

Here is now the second example, where Seyond gives you an amazing overview of a Flash LiDAR (Hummingbird). This video is originally from my membership The Edgeneer's Land - make sure to be in my daily emails to learn more.

Alright, let's do a summary...

Summary & Next Steps

Here is a bullet point summary of the article:

• The robotics & LiDAR industry tends to use 2 types of LiDARs: Mechanical and Solid-state. While the former has moving parts, the later doesn't.
• Solid-State LiDARs come in 3 categories: MEMS (with moving mirrors), OPA (true solid-state with no moving parts), and Flash LiDAR (projects laser arrays for instantaneous scene capture). They are all directional, lower power, higher resolution, but shorter range and lower reliability than those with mechanical movement.
• LiDAR technology is about sending a laser to the world and measuring the time a wave takes to hit a surface and come back. Yet, this can be done via several processes.
• The semiconductor manufacturing process allows solid-state LiDAR to be mass-produced at lower cost, making it more accessible for automotive and industrial applications.
• Solid-state LiDAR technology is advancing rapidly and is becoming the default choice for applications requiring high performance, compactness, and reliability, including self-driving cars and smart cities.

Deep Point Clouds Waitlist

Deep Point Clouds Waitlist

On June 4, 1996, the Ariane 5 rocket was ready to be launched after years of work, public funding, and political pressure. The stress was at maximal, but after just 40 seconds, the rocket exploded, causing a loss of over 370M$.

This event is one of the most known in software engineering, and in particular because of the reasons of the crash: a float to int conversion. Indeed, the engineers reused the code from Ariane 4 to launch Ariane 5, but forgot that a float64 storing the horizontal velocity would be converted converted to a signed int16. 40 seconds into launch, the conversion failed and crashed the rocket.

I think this story can be a perfect introduction to the domain of autonomous vehicle safety; which we'll cover today with our guest Jonathan Péclat form Loxo.

A quick intro:

Loxo is a Swiss based company started in 2022 where they built a first prototype for an autonomous shuttle. Since then, it evolved into this vehicle that now operates in Germany & Switzerland.

These robots navigate real streets, interact with real traffic, and do so using an architecture powered by End-to-End Deep Learning.

I find this incredible, because End-To-End Learning is purely AI based. It's data based, it's when you don't explicitely program the vehicle to stop at red light, but show it via examples from the dataset.

While a modular approach is pretty straightforward, and certification is about evaluating each individual block (is the lane detection safe? is the obstacle detection safe?)...

... End-To-End approaches are much more complex to evaluate, because they only output the final driving decision. I have an entire article covering the differences here.

So I asked Jonathan:

"How do you make End-To-End Learning safe?"

Here is what he explained:

As Jonathan pointed out:

“You cannot really prove that AI is safe, not today. So we run our AI system in parallel with another component that verifies the trajectory. If the AI violates any predefined rule, we switch to a deterministic safe path.”

This point explained is crucial, because several self-driving car companies use exactly the same approach. LOXO does not rely on a single neural network but on four independent channels (two AI channels, and two deterministic channels) running in parallel, each serving a different role in verifying, supervising, or backing up the End-to-End planner.

LOXO’s architecture is a clear illustration of the principle of redundancy: Multiple algorithms, points of view, and, instead of a failure mode, a structure that catches, compensates for, and, if needed, overrides failures. This is how an End-to-End system becomes certifiable and safer.

The key point to understand is that companies relying on End-To-End do not use just that one approach; they run multiple algorithms in parallel that verify and contradict eachother. I have a complete breakdown of how Mobileye does it with their own End-To-End approach here, if you're interested.

Still, a question remains:

What exactly do you make redundant?

The sensors? The algorithms? What is even redundancy? This is my next question for Jonathan, which then explains the safety fundamentals of ASIL scoring and decomposition using among other a grading from A (safe) to D (risky):

The entire principle relies on the concept of Functional Safety with ASIL Decomposition. This is a job on its own, that often includes ISO norms, but if you'd like to explore this, I have a complete article covering how it works here:

Functional Safety Engineer: The Job that ‘certifies’ self-driving cars

What is functional safety in self-driving cars? What does a functional safety engineer do? In this post, we’ll try to understand how to certify a self-driving car code, and make it safe to drive in the streets

Read from the most advanced autonomous tech blogJeremy Cohen

Realize that this doesn't stop here. In my interview with Jonathan, Loxo explains the step-by-step framework they implement, along with their internal documents used to grade a function, evaluate its risk, and decide to make it redundant or not.

It's a complete masterclass we have inside The Edgeneer's Land, our community membership experience.

But for now, let's do a brief summary:

Summary

• When a self-driving car company uses End-To-End Learning, a single machine-learning model directly maps raw sensor data to driving actions or trajectories; without manually writing any rule.
• While this can simplify system design and improve performance, it also makes the system harder to interpret, verify, and certify, especially in safety-critical and regulated environments.
• Companies like LOXO often use redundant channels that are the opposite of End-To-End channels; using point clouds processing, clustering, extraction, and very deterministic approaches to try and validate what the AI says.
• Functional Safety Systems like ASIL Decomposition still apply to End-To-End, and there are many processes used to certify self-driving car algorithms.

Next Steps?
If you want to go deeper into how safety is formally addressed in the autonomous driving industry (how risks are identified, graded, reduced, and documented), I detail the full process in this blog post about functional safety.

Functional Safety Engineer: The Job that ‘certifies’ self-driving cars

What is functional safety in self-driving cars? What does a functional safety engineer do? In this post, we’ll try to understand how to certify a self-driving car code, and make it safe to drive in the streets

Read from the most advanced autonomous tech blogJeremy Cohen

If you'd like to get the complete masterclass from Jonathan, I recommend checking out The Edgeneer's Land.

Back in 2020, a company contacted me because they needed my opinion on a robotic sensor stack they were working on. They had 2 days to finalize the decision of a sensor suite that would equip their autonomous delivery pods. Like many, they were considering using a combination of all sensors, cameras, LiDARs, RADARs, and even ultrasonic sensors. But they also had concerns, and were wondering if nothing better was available.

But in 2020, the combination of a LiDAR, a camera, and a RADAR was what made the most sense. "These sensors are complementary" I would reply. "The LiDAR is the most accurate sensor to detect a distance, the camera is best for scene understanding, and the RADAR can see through objects and directly estimate velocities".

Is this still true? Don't we have camera only systems today? Don't we have LiDAR only systems that bypass RADARs? And don't we have RADARs that are getting as good, if not better, as LiDARs? I think the idea of "complementarity" is changing. Today, sensors get more capable. FMCW LiDARs can detect speed, and Imaging RADARs can great accurate point cloud representations.

So, what is true and what isn't?

Let's take a look via this article in 3 points:

• The Traditional LiDAR vs RADAR comparison
• The new LiDAR and RADAR sensors in self-driving cars
• LiDARs vs RADARs: The Modern Comparison

Traditional LiDAR and RADAR technology comparison

I believe the following no longer makes sense, but I am going to show it to you anyway, and this is what you'll see in 99% of other posts about the topic. Here is the idea in 3 points:

1 - LiDARs are great for distance estimation

LiDAR (Light Detection and Ranging) is a technology that leverages laser light to measure distances and create detailed 3D maps of objects and environments. When you look at a distance estimators today, the LiDAR is often used as the "ground truth". LiDAR systems operate by emitting laser pulses (waves) and calculating the time it takes for the light to come back. This idea is called "Time of Flight" - and although there are multiple types of LiDARs, this is the overall idea.

Here is an example:

Now, what does it produce? The answer is a point cloud of the environment. But not all point clouds look the same.

2D vs 3D LiDARs

Because I'm going to talk about 4D LiDARs, I have to explain the idea of a 2D and a 3D LiDAR first. The idea is well explained in my post "2D LiDARs: Too Weak for Self-Driving Cars?", in which I explain that LiDARs use vertical "channels" or layers, and that based on the number of channels, you have a more accurate 3D resolution.

What more channels bring

Lidar utilizes laser pulses to send out laser beams, measure distances, and create detailed 3D maps. But the drawback is that if you want to measure a velocity, you need to compute the difference between 2 consecutive timestamps. How has the point cloud moved in the last second? At low speed, this is good enough, but at high speed, measuring the differences between 2 frames can mean several meters before braking.

This is why we also like to combine it with a RADAR. Let's see it:

2 - RADARs are great velocity estimators

RADAR stands for Radio Detection And Ranging. It works by emitting electromagnetic waves that reflect when they meet an obstacle. Unlike cameras or LiDARs, RADAR relies on radio waves that can work under any weather condition, and even see underneath obstacles. They use the "Doppler Effect" to measure the velocity of obstacles.

RADAR technology is very mature (>100 years old), and is used in various industries, including aviation, where it is crucial for air traffic control, cars, missile detection, and even weather forecasting. However, most RADARs work in 2D. Haaaaa - yes, this is what we got: X and Y, but no Z, exactly like a one-channel LiDAR.

Should I show you the sample point cloud from a RADAR?

Output from a RADAR system

But let me show you the real output from a RADAR sensor:

I mean, can you tell where there is a vehicle? Whether we should stop or not? It's complete garbag— wait, if people use it, it's gotta be useful, right? And yes, it is, because while we only have noisy 2D point cloud, each of these points also provide a 1D velocity information. RADARs tell us whether the points are going away from us, or towards us, and how fast.

Using Point Clouds Processing, Deep Learning (often trained on LiDAR data), or even RADAR/Camera Fusion, we can even get a result like this:

Notice how the yellow dot changes to a green color as soon as the car moves, and how each static object is orange, while moving objects have a color. This is because the RADAR is really good at measuring velocities.

3 - LiDARs and RADARs are complementary and still need eachother

As a little summary, I'd say that LiDARs are good, but most of the time need cameras for context, and at high speed, need RADARs. RADARs are great, but could NOT work as a standalone system. So let's do a quick overview:

If you want to be green everywhere, you need to combine all 3. Use the camera for scene understanding, use the RADAR for weather conditions and velocity measurement, and the LiDAR for distance estimation.

This brings a problem. A random startup must invest in 3 sensors, co-calibrate 3 sensors, train their team on all these sensor types, and the more sensors we use, the more confusion we risk bringing. You may wonder... can't we use just one? Or two?

Let’s see how:

FMCW LiDARs & Imaging RADARs: The Future of Perception

Back in January 2023, I was at CES in Las Vegas for the first time. It was a big show, really incredible, and while walking there, I met a startup named 'Aeva'. Aeva is a LiDAR startup specialized in 4D technology. "What's 4D?" I asked. It turns out, 4D meat that their LiDARs had the possibility to do direct velocity estimation.

The next day, I walked to a different area and stumble across a korean startup called bitsensing. "Bitsensing is creating a 4D Imaging RADAR" said the presentator. I was in shock. It was a normal RADAR, but providing an incredible resolution, with Z-elevation, accurate 3D view, no noise, and still the Doppler velocity measurement.

It sounded like these startups were working on fixing the weaknesses of classical technologies.

Let me introduce them to you.

1 - FMCW LiDAR (Frequency Modulated Continuous Wave LiDAR): 4D LiDAR

An FMCW LiDAR (or 4D LiDAR, or Doppler LiDAR) is a LiDAR that can return the depth information, but also directly measure the speed of an object. What happens behind the scenes if they steal the RADAR Doppler Technology and adapt it to a light sensor.

Here's what the startup Aurora is doing on LiDARs... notice how moving objects are colored while others aren't:

LiDAR uses the Doppler Effect, similarly to the RADAR technology, to get this 4D view. The main idea can be seen on this image, where we play with the frequency of the returned wave to measure the velocity.

The Doppler Effect is exactly about measuring this frequency. And this has now been adopted in FMCW LiDAR technology, but still with light waves instead of radio waves. I highly recommend to check out my complete post called "Understanding the magnificent FMCW LiDAR".

2 - Imaging RADAR: 4D RADAR

In 2024, mobileye, who had been working on their own FMCW LiDAR for years, announced it would be shutting down its entire FMCW LiDAR division to focus on proprietary 4D Imaging RADAR. What happened? Why the shift? Well, let's first try to understand what Imaging RADARs are. I like to call these...

RADAR on steroids!

To understand better how it works, I'd like to show you the bitsensing demo they showed me at CES.

bitsensing Imaging RADAR Demo

The Imaging RADAR has an incredible resolution. It provides a very accurate point cloud, that can see through adverse weather conditions, do obstacle detection AND measure velocity directly! Under-the-hood, it uses a set of MIMO antennas to get a much better resolution, range, and precision. We could in fact detect obstacles inside a vehicle, and classify children from parents.

See the demo:

Can you notice how similar it looks to the FMCW LiDAR? We have in both cases:

• A 3D Point Cloud
• That can directly measure velocity

Other Examples from Self-Driving Cars

Frankly, many actors from the autonomous driving industry are switching to Imaging RADARs. Mobileye has a great demo, so does Waymo. Let's see these 2 examples.

Here's the Waymo Imaging RADAR Demo:

And now Mobileye:

See? We are in the middle of a transition... but why are people using Imaging RADARs over FMCW LiDARs? And are they really moving away from LiDARs? Let's find out in the final point...

LiDARs vs RADARs: The Modern Comparison

There are 2 ideas I'd like to talk about here:

1. The Future of RADARs IS Imaging based
2. The Future of LiDARs may NOT be FMCW based

1 - The Future of RADARs IS Imaging based

We have clearly see how a good RADAR system can bring incredible benefits. We can now do tasks like object detection using purely an imaging RADAR. Recently, we've seen Deep Learning models, like the ones from Perciv AI, work on RADAR data (radar signals, radar point clouds, radar waves, ...) directly.

Back in the day, any comparison between a LiDAR and a RADAR didn’t really make sense because the sensors were highly complementary. But today, these sensors can be in competition, and if there is one, Imaging RADARs are winning it! If we see the new comparison table now:

We are BLUE almost everywhere, but the cost of Imaging RADAR stays lower than FCMW LiDARs. In addition to this, the Imaging RADAR can nicely fit under a bumper, since RADAR employs radio waves that go through objects.

When looking at remote sensing technology, RADAR has always been a great choice; whether it's synthetic aperture radar systems in the military field, or environmental monitoring of their radio frequency spectrum, or the recent adoption in autonomous vehicles, RADARs ARE by default a great choice.

In the self-driving space, RADARs were never good enough to be a standalone. Have someone ever told you you weren't good enough? Well, this is a lesson, because you can see a massive adoption and trend of Imaging RADAR - and I believe the future of RADARs is imaging.

2 - The Future of LiDARs IS NOT FMCW based

Now this is the incredible discovery here: Nobody is abandonning LiDARs for FMCW LiDARs. Self-driving car companies have NOT adopted FMCW LiDAR technology in mass (for now), and I predict they'll just stick to solid-state.

Back in 2023, I went to Innoviz Technologies headquarters in Israel. Innoviz is a LiDAR manufacturing companies providing LiDAR devices to companies like BMW. I asked them: "Why are you NOT building FMCW LiDARs?". Their answer was that their LiDARs were good enough, and that there was no real need for FMCW. It really surprised me, but I guess they know what they're talking about. They could solve the drawbacks of LiDARs by building better LiDARs, for example here:

In many fields, LiDAR sensors are at the core. We have airborne lidar systems building elevation maps, we have HD Maps built entirely from LiDARs, and even drones equipped with LiDARs today... this technology is here to stay. Plus, today, EVERYONE uses LiDARs! No that was wrong, Tesla doesn't, and a few others have bet on vision-only... but the majority of startups do, except that they use BETTER LiDARs. Not necessarily 4D, but LiDARs that provide better resolution, focusing more on solid-state technology.

This is the key message I have for you, and now that we've seen it, let's go through a summary, and see some next steps.

Summary & Next Steps

• LiDAR uses laser light to measure distances and create detailed 3D maps of objects and environments. Their key strength is distance estimation. They key weakness is weak velocity estimation, and weather conditions.
• RADAR emits radio waves and measures their reflections to detect objects and calculate their speed, even in bad weather. They key strength is velocity estimation, they key weakness is noise, context, and 3D estimation (most are only 2D).
• Traditional setups combine LiDAR, RADAR, and cameras because each sensor complements the others' strengths and weaknesses. It's near unthinkable to use one as a standalone.
• Recently, technologies like 4D FMCW LiDAR and Imaging RADAR have emerged, offering both high resolution and velocity measurement. FMCW LiDARs use the Doppler effect, and Imaging RADARs use more antennas.
• While the future of RADAR is (I believe) RADAR+Imaging capabilities, I believe the future of LiDARs may be solid-state based, and not necessarily FMCW/4D based.

Next Steps

• Learn about the FMCW LiDAR here.
• Learn about the Imaging RADAR here.

Did you ever wonder... why are self-driving cars taking so long to come? I had that question too when starting, and my first answer came from Sebastian Thrun, godfather of self-driving cars, who talked about reaching 90% of use cases easily, but then finding huge difficulty in going from 90% to 100%. Recently, Andrej Karpathy, former Lead of Tesla Autopilot described something similar as "the march of 9s":

"When you get a demo and something works 90% of the time, that's just the first 9 and then you need the second 9 and third 9, fourth 9, fifth 9..."

This is what's taking long, but instead of focusing on this, tons of companies lose time focusing on the first 0-90%. Back in 2017 or so, we were all trying to get to 90%, and for this, we were re-developping all the software, algorithms, and so on... At some point, probably 30 startups were all spending millions developing the exact same algorithms.

This is when Autoware comes in the play. Started by Tier IV, Autoware is an open-source self-driving car software that allows you to achieve the first 9 in just a few weeks. Rather than re-developping yet another version of the same code, you jumpstart from the existing state of the art, and finetune it for your needs.

This month, our membership The Edgeneer's Land is welcoming Samet Kütük from The Autoware Foundation.

And let me start with a small snippet about how he defines Autoware:

Together, we recorded a new Fragment of The Edgeneer's Land, my community membership experience, in which he takes us through the building and management of Autoware. How does it work? How do you build a self-driving car with a full remote team? This is everything Samet teaches in our new fragment...

In this post, I'd like to give you a small sample of that interview, highlighting a very interesting moment where Samet talked about End-To-End learning...

3 insights from Autoware's End-To-End Learning Transition

Since its creation, Autoware has been implementing a "robotic" architecture, meaning implementing the traditinal "4 pillars": Perception → Localization → Planning → Control.

But recently, Autoware announced a new plan to evolve to an End-To-End architecture, a single neural network that takes in the input sensor data, and automatically outputs the steering angle and acceleration value. I have a complete article explaining the differences with detailed examples here.

4 Pillars vs End To End: How to pick an autonomous vehicle architecture

How to design an autonomous vehicle architecture? Should you implement an End-To-End solution, or a more traditional one? Let’s see…

Read from the most advanced autonomous tech blogJeremy Cohen

So here is the sample I'd like to share:

As you can see, there is a lot to uncover from just one minute. Let me share 3 highlights from that:

1. Level 5 is NOT easy to reach, may not even be possible, which is why Autoware focuses on Level 4+, in which humans, while not asked to takeover the car, could be asked to drive under regions or conditions that aren't appropriate.

2. The Transition will NOT be achieved immediately, but rather then result of several steps:
   1. Current — Starting from a traditional Robotic stack
   2. Step 1 — Learned Planning
   3. Step 2 — Deep Perception & Learned Planning
   4. Step 3 — Monolythic End-to-End (single network)
   5. Step 4 — Hybrid End-To-End using a "guardian" for redundancy

This can feel similar to how Tesla did their own transition to End-To-End (which I cover in this article:

Breakdown: How Tesla will transition from Modular to End-To-End Deep Learning

It’s no secret, Tesla is going to use End-To-End Deep Learning. But how? What will it look like? Will the Occupancy Network and HydraNet stay? Here’s a full breakdown…

Read from the most advanced autonomous tech blogJeremy Cohen

The difference between Modular and Monolythic End-To-End has been explained in their GitHub repository talking about the new planning algorithm:

Alright, let's continue with a third and final idea:

3. The algorithms for End-To-End have already been built.

We are not talking about a distant future, according to Autoware, it's possible to achieve End-To-End with today's algorithms, including (but not limited to) CenterPoint as the 3D Deep Learning algorithm for LiDAR Detection, AutoSeg as the Foundation Model in Perception, AutoSteer and Diffusion Planner for the Learned Planning approaches.

See? They are already there, and even though 2, 3, or 5 years from now, they may evolve and get replaced, the logic of Modular End-To-End (Step 2) has been implemented.

For example, the AutoSeg algorithm is a "HydraNet" that has a single backbone that split into several heads for lane lines, ego path, free space, segmentation, objects, and 3D. The outputs of these heads are then passed to the deep planner.

Summary & Next Steps

• Autoware is an open source self-driving car organization that builds a self-driving car software used over the world by thousands of engineers and teams
• Autoware is the solution I recommend to get started in self-driving cars; rather than building a software from scratch, get Autoware working quickly, and then finetune and customize it for your applications.
• Autoware is transitioning from a robotic architecture to an End-To-End Learning architecture, and there are 3 highlights from it:
  • It won't reach Level 5, but a Level 4+ that can drive almost anywhere
  • The transition will happen in 4 steps, adding planning, perception, then turning into a monolythic architecture, and finally hybrid.
  • The algorithms and modular logic have already been implemented and are working, such as CenterPoint, AutoSeg, or Diffusion Planner.

Next steps

Interesting in getting access to our Autoware Fragment? It's going to be very cool, and feature several things, such as:

• The Full-Length interview with Samet on Autoware
• An even deeper dive on Autoware's End-To-End Transition (this was just a 1 minute video - we do it for the full section on End-To-End).
• A complete breakdown on many algorithms used by Autoware, and a near plug & play solution to start running Autoware's software on your computer by tonight

Intersted? Apply to the Edgeneer's Land at the button below before October 25, 2025 (use the direct link if you're already a pre-approved client).

Fragment #13: Open Source AV Secrets

Exclusive Interview with Autoware - Autoware E2E Transition Full Breakdown - Autoware Platform Run & Algorithm Dive

Learn More about Fragment 13

In September 1519, an expedition of five ships and 270 men, led by Ferdinand Magellan, left Spain to reach the Spice Islands by sailing west. At the time, maps were crude sketches, full of blank spaces, and sometimes decorated with warnings: “Here be dragons.” Yet Magellan pressed on, steering his fleet into the unknown, through storms and across oceans no European had crossed before.

The challenge was harsher than anyone had imagined. Supplies ran out, men starved, and mutiny spread. One ship deserted, another wrecked. After nearly two years, Magellan reached the Philippines, where he was killed in the Battle of Mactan. His fleet, once five strong, was reduced to four… then three… then two.

3 years later, only one ship returned to Spain. The Victoria carried just 18 survivors, but also one of the greatest accomplishments of the time. For the first time, humanity had proof that the Earth could be circumnavigated by sea, a discovery that forever reshaped navigation, trade, and commerce.

For centuries, people believed the old maps. They trusted the flat drawings, the empty warnings, the “here be dragons”. All it took was one expedition to open a new world nobody could see. And today, I believe Computer Vision Engineers live in a similar situation.

The world provides Computer Vision algorithms, image processing techniques, 2D object detectors, and segmentation approaches... yet, the world is a sphere, in 3D. And this is why, I think something of much greater importance should be mastered by Computer Vision and ALL robotics/autonomous tech engineers: Point Clouds.

The goal of a point cloud is to create a 3D model. 3D points are a data representation used today in autonomous vehicles, robotics, AR/VR, and even in everyday objects like unlocking your phone with Face ID.

So what are point clouds? How do you get them? And how do you process them using AI? These are the 3 things I think most perception engineers should know, that we'll cover in this article.

Let's begin:

9 Examples of Point Cloud Data

A Point Cloud" is a set of points in 3D space — a cloud of points. Inside, each point holds the 3D location of a surface in the real world. It can be a person, a wall, a tree, anything. You probably know what a point cloud looks like already, but you may not know the multiple types of point clouds... So let me introduce you to 9 of them!

XYZ Point Clouds

In an XYZ point cloud, each point has a specific X, Y, and Z value. You could think of it as the equivalent of a pixel, but in 3D. Rather than just X and Y, we have X, Y, and Z (in most cases, because some point clouds are 2D, see this article).

Here's an example:

See? Each point has an XYZ value. But why are the colors different? Simply here because our visualizer is a gradient based on the height of the point (the Z dimension). The higher the Z value, the more red it'll be. On the above Waymo video, you could see a different visualization, based on the distance to the vehicle. So this is one type:

• Point clouds can contain the XYZ information

Next:

XYZ-I Point Clouds

Now, this is just an example, but let me show you something else...

This is another point cloud, but what do you notice about the colors? Yes, two things:

• It's all "RED"
• But not all points have exactly the same "red" value. Some are brighter than others

And this is because here, we are no longer visualizing the distance, but the "intensity" of the points. Point Clouds are often produced by LiDARs that send a ray and measure the time it takes to bounce back. This calculation measures the distance, but not all rays come back equal. Some are blocked by trees, leafs, or surfaces, while others perfectly go through.

So, we now know another attribute of a point cloud:

• Point clouds can contain the XYZ information
• Point clouds can also hold the intensity information!

Any other?

XYZ-V Point Clouds

Now, let's take it one step further, and look at this video:

Okay, can you explain what is happening here? Everything is grey, but the vehicles. So, is that... Class? Labels? Or, wait a minute, why are the forward vehicles in red, the parked cars in grey, and the left approaching vehicles in blue? This is because, this visualization shows not the class but the velocity information!

This video has been made from Aeva, an FMCW LiDAR producer — and inside, you can see the points receding are in red, and those approaching are in blue. We now know a third possibility!

• Point Clouds can contain XYZ
• Or XYZ-Intensity
• Or XYZ-Velocity

Let's see 9 types of Point Clouds

Are there any more than intensity or velocity? Yes, in fact - each point can contain a lot of information. Let's see:

• Intensity - how strong the point clouds return signals are
• Range - the distance of the point, based on X, Y, or Z
• Color - the RGB color of the points (often for RGB-D cameras or 3D reconstruction)
• Class/Label - if it's after an object detector or segmentation tool processed it
• Infrared - the wavelength of the point cloud signal
• Ring/Channel - which channel of 3D sensors was used to collect it
• Velocity - the speed of each point (calculated by RADARs or FMCW LiDARs)
• Reflectivity - how reflective the surface of the point is
• Temperature - how hot a point is

Okay, but concretely, how does it work? Is there a TXT file where we store the points? Kinda, let's take a look...

Point Cloud Formats & Files

There are usually two types of files: ASCII and Binary. One is easier to read, the other is more suited to real-time/embedded. Take a look at the beginning of both files:

See? On the left, the PLY file contains X, Y, Z as floats, followed by a list of point coordinates. This is the point cloud! On the right, you can see a header describing the point cloud, in format XYZ-Intensity, and then, the points are not readable.

2 Main Ways to create a point cloud

There are basically 2 types of approaches, active and passive. Active techniques actively emit signals like light or sound to measure distances and create point clouds, such as LiDAR and structured light systems. In contrast, passive techniques rely on capturing existing environmental data, like photogrammetry, which reconstructs 3D points from multiple camera images without emitting any signals.

Active Techniques: LiDARs, RGB-D & RADARs

In the first case, point clouds come from sensors built to create them. When a camera takes a picture, it aims to get pixels. Well, when a LiDAR makes a measurement, its aim is to create a point cloud. Let's see 3 ways to do it:

1) How to get point clouds using Structured Light RGB-D systems

Ever played the Microsoft Kinect? I can't say that I have. I was a Wii player all the way when they were competing. Yet, I've always been impressed by how the Kinect produced point clouds using its RGB-D camera, working with the Structured Light Principle.

The Kinect shines a special pattern of light around, then uses an infrared camera to take a picture of how that light bounces back. By seeing how the pattern changes, the camera can figure out how far away things are. It combines this distance information with the colors it sees to create a 3D image called one final point cloud.

In robotics, you probably know the Intel Realsense 435i, or other equivalents. Their goal is to build a Depth Map, then turned into a point cloud.

2) How LiDAR point clouds are produced

The most common and popular technique is to use a LiDAR (Light Detection And Ranging). There are many types of LiDARs around, but let's focus on the simple Time-Of-Flight principle. In this setup, a laser scanner sends a light beam and measure the time it takes to reflect and come back to the receiver. Similar to this image:

LiDAR scanners produce raw data of the world up to 300-400 meters in the automotive industry. Each scan can generate millions of points in the three dimensional space. I highly recommend checking my article on the types of LiDARs to learn more.

3) RADAR Point Clouds

The third technique is to use not a LiDAR but a RADAR to create the point cloud data. This is not very straightforward to do. RADARs usually return signal information based on Doppler (velocity), Range (distance), and Azimuth (direction/angle). Using these, we can do some calculations to retrieve the point cloud data.

Here is an example on a very low quality RADAR:

Today, we can use Imaging RADARs to get 3D point clouds. I invite you to check out my Imaging RADAR article to learn more about it.

Now that we've seen the Active ways, using sensors - I'd like to take a minute to talk about the passive ways.

Passive Point Clouds Generation: Photogrammetry & 3D Reconstruction

The idea of passive is that you do not attempt to create a point cloud from your sensors. The main way to do this is by leveraging 3D Reconstruction. Ideas like Structure From Motion, Multi-View Stereo, NeRFs, Gaussian Splatting, or others are used.

The idea? To convert 2 or more images to a 3D point cloud using triangulation, geometry, and depth maps.

If you're interested in this, I highly recommend reading my 3 article Series on PyImageSearch blog, or my article on Pseudo-LiDARs.

Alright, so you now know all about the point cloud types, and the ways to get them. One thing remains...

How to Process Point Cloud Data?

Do you remember in the point cloud types when I showed the "label/class" of each point? This is not something sensors can measure, it's built by algorithms. There are 3 things that really matter here:

1. Understanding the main libraries/tools to work with
2. Understanding the core algorithms to use on raw point cloud data
3. Being able to use them in the applications

Libraries: Open3D and Point Cloud Library (PCL)

There are many libraries used to process point clouds. These implement the algorithms. For example, the Point Cloud Library is one of the most popular to work with. Open3D is also a very common one, it contains fewer algorithms, but is easier to process thanks to the Python interface. I would recommend to get started with this one.

On a similar topic, you could want to know at least one point cloud dataset. I would recommend you check out this article.

Which algorithms can be used to process point clouds?

In point cloud processing, you can either go with traditional algorithms or 3D Deep Learning. The split is, I would say, dependent on the applications. When companies want to detect objects in 3D to get bounding boxes, they usually use 3D Deep Learning algorithms like PointPillars or VoxelNet. Let's see an example:

Outside of 3D Object Detection and 3D Segmentation, the entire world runs on traditional processing approaches. Since you have points, you can create tons of automated pipelines to process them. For example, you can do plane segmentation, clustering, outlier removal, normal estimation, point data cropping, surface reconstruction, filtering of unwanted data points, and so on, you'll use traditional approaches.

For example, you could calculate the surface normals and filter out the objects that belong or don't below to the street.

Another technique can involve point cloud registration and alignment. When you have multiple point clouds, for example coming from 2 LiDARs, you can align them together into a single object. An example below from one of my LiDAR courses, notice how we start with 2 point clouds, a blue and a red, and we end up aligning them perfectly. This makes something better than the raw data.

In the algorithm category, there are countless applications. Everything related to SLAM or Odometry is also extremely in use today.

Applications: Which jobs can you target with Point Cloud skills?

Regarding the applications, we could write an entire article. Yet, let me give you 3 or 4 core jobs you can target with point clouds processing skills:

• Perception Engineer, Autonomous Vehicles: Process LiDARs and RADARs to find objects in the 3D space. Use Sensor Fusion to mix the output with Computer Vision. Build autonomous vehicles, shuttles, delivery robots, and create the future.
• Nuclear SLAM Engineer, Robotics: Use point clouds processing techniques inside robots that explore caves or regions humans can't go to, such as nuclear sites, and build maps of the world.
• BIM Engineer, Architecture: Create digital models of buildings and structure by processing raw point cloud data captured from laser scanners or photogrammetry. These models help architects and engineers visualize object properties, plan renovations, and ensure precise construction. The role often involves using processing software to convert points into computer aided design (CAD) models, performing manual correction to refine the data, and integrating the results into architectural workflows for improved design and quality inspection.
• Medical Imaging Engineer: Apply point cloud techniques to CT-Scans, IMRs, and other 3D data types to detect diseases and save lives. There is both a commercial and research use.
• Drone Engineer, Agriculture: Process Cameras and RADARs/LiDAR information to navigate and help drones fasten agriculture and solve population needs.
• and many, many more...

Alright, now let's see a summary...

Summary & Next Steps

• A point cloud is a series of 2D or 3D points. A point cloud is to the LiDAR what a pixel is to a camera.
  (Re-read that one.)
• Each point of a cloud usually contains at least the XYZ information, but many sensors or technique allow to also get Intensity, Reflectivity, Velocity, Ring/Channel, Color, Temperature, Infrared, and more...
• A point cloud output format is of 2 types: ASCII or Binary. An ASCII file is more readable for humans, Binary is more readable for robots. Each file is a list of points and their information.
• There are 2 ways to build a point cloud: Active and Passive. Active techniques involve sensors like LiDARs, RADARs, or RGB-D cameras, while passive techniques use photogrammetry and 3D reconstruction to retrieve 3D models.
• Point Clouds Processing typically involves 3 stages: the tools/libraries, the algorithms, and the applications. Tools are libraries like Open3D or PCL, algorithms are either traditional or deep learning, and applications go from self-driving cars to robotics, drones, augmented reality, the architecture industry, agriculture, and beyond.

Where do you think humans come from? Growing up, I wrestled with two conflicting ideas about it. One teacher taught me Darwin’s theory of evolution: a gradual process of adaptation, rooted in science but riddled with gaps and errors. The other taught me the Bible’s Old Testament: the story of divine creation, in which even though nothing was ever proven false, this isn't built on any "proof".

Since then, I realized they were trying to answer the same question, but operated in entirely different realms, each with its own logic and purpose. One belonged to science, the other to faith. Making head-to-head comparison was almost meaningless.

And I think this contradiction also exists in self-driving cars, especially when opposing 2 giants: Tesla and Waymo. Both seem to chase the same prize of "Level 5" autonomy, but when you look closer, their paths are so distinct they’re barely comparable.

In this article, we're going to try and understand who has what I call the best "Map to Level 5", we'll take a side-by-side comparison, and I'll give my opinion on 3 aspects:

• The sensor suite
• The algorithms
• The "map", meaning strategy, vision, and more...

Let's begin with the sensors...

Tesla vs Waymo: Who has the best Sensor Suite?

Back when I was studying driverless cars, it was around 2017, when I heard an interview with Sebastian Thrun, godfather of self-driving cars, talking about who was ahead in the race. I vividly remember his words: "Nissan is doing pretty good, but I think the company who is ahead of everyone is actually Tesla".

It was 2017, and I remember feeling surprised by this comment, because at the time, Tesla only had a light ADAS feature working with mobileye, and companies like Waymo, Mercedes, Nissan, and others seemed to be covered everywhere in the media, have "real" self-driving car abilities, and more potential.

What about today? Who is ahead? Closer to remove human drivers? Who has the better vision? The better algorithms? The better sensors? Is it Waymo, or Tesla... or someone else?

In this first part, I want to answer it from a sensor angle. And to do so, I'm going to start by a screenshot of a very popular X (tweet?) from Elon Musk about RADARs, LiDARs, and cameras from August 2025.

This comment raised an army of furious engineers and fusion experts, mentioning Kalman Filters, and Redundancy, and Sensor Fusion. So before we dive into the exactness of this comment, I would like to describe what each company is doing...

Waymo: 29 Cameras, 6 RADARs, 5 LiDARs

If you look at a Waymo car, you're going to see exactly the opposite of Tesla: tons of sensors all over the place. There are RADAR sensors on the front, side and rear, there's 29 cameras, and 5 LiDARs. The question we can ask is... "Is Waymo trying to kill a fly with a bazooka?".

Just in terms of calibration, it must be an absolute nightmare for engineers. Calibrating a camera with a LiDAR is already a long task, but 29 cameras with 5 LiDARs? Then there is the fusion of all of these sensors together, and this is ONLY the "robotaxi" version, because if you look at their Zeekr shuttles, they also have their own types of sensors, with different generation codes. Their stack is therefore always evolving, and depends on the vehicle they drive on.

What type of LiDAR, Camera, and RADAR is Waymo using?

Let's take a brief look at what each sensor sees:

Waymo uses different types of cameras. They work with high-res long-range cameras (color, telephoto lenses) for object detection far down the road, wide-angle cameras for close-range coverage (pedestrians, cyclists, intersections), and near-infrared cameras for night vision / low-light perception.

Regarding LiDARs, they're using their own sensors called "Laser Bear Honeycomb". There is one forward LiDAR, 2 side LiDARs, and one at the rear. But these are short range, solid-state LiDARs. They are excellent for blind spots and front facing vehicles, but complex to drive on highways because they don't see far. This is why there is the roof LiDAR, which is mechanical, and sees several hundred meters away.

On the animation below, you can see LiDARs both in point clouds format and in range-view — and you can learn about types of LiDARs here.

Regarding RADARs, Waymo uses their own line of Imaging RADARs. Imaging RADARs are what I call 4D RADARs. Unlike normal RADARs who see in 2D and measure the velocity, these ones see in 3D and measure the velocity.

Do you want to see what all of this looks like together? Okay, here it is:

Waymo uses a powerful array of sensors allowing them to see every possible object. We'll come back to the utility of LiDARs and RADARs, but for now, let's look at cameras...

2. Tesla's Sensor Design: 8 Cameras, that's it

Unlike Waymo vehicles, Tesla's approach relies only on cameras. Tesla's autopilot aims to solve autonomous driving using vision-only.

On this illustration, you can see a very simple design that almost never changed. In red, you can count 8 outside cameras, and in orange, you can see 12 ultrasonic sensors used to detect static objects when parking. Among the 8 cameras, there 2 on the windshield, used for stereo vision, one on the front bumper, one on the rear bumper, 2 on the doors, and 2 on the wheels. See the difference? I can't even begin to count Waymo's cameras, but I can easily show you Tesla's sensor stack.

So let's visualize what the cameras see:

Interesting, or not. Now, let's try and understand, who has the best sensor suite?

3. Who has the best sensor suite... Tesla or Waymo?

I am going to show you an image, and I would like you to ONLY look at the left part. Ignore the right for now. Can you tell me what you see? You see a car, don't you?

But did you notice the pedestrian? This is one of the limits of the vision-only approach. When you look at Tesla's miles driven without disengagement reports, Tesla FSD clearly shows a limit on bad weather. They don't drive well on cloudy foggy, rainy, or snowy scenes, and they don't drive at all during storm and sleet (when ice falls from the sky).

Reports show that FSD13 significantly improved driving by night, but there is still a physical limitation to driving with cameras only that is not solvable with better algorithms. In a robotaxi situation in which you'd sit in the passenger seat, you would be stuck. The way humans drive involves more than cameras, we hear street sound, we sense people, and we don't use wide-angle cameras to detect other cars.

If we were to come back to Elon Musk's comment now, do you remember the "If LiDARs/RADARs disagree with cameras, which one wins???". Waymo shows that redundancy is key to safety. If a camera misses something, your RADAR may not. And with Kalman Filters, you can certainly develop a powerful fusion module to account for disagreements.

When we consider the physical ability of a LiDAR to generate point clouds, you can understand how powerful having them is. Other than seeing through night or other situations, they physically build point clouds. Recently, a YouTube video has shown a Tesla vs LiDAR-equipped car driving on a wall resembling a street. The Tesla FSD crashed on the wall, confusing it with the highway; but the LiDAR-equipped car stopped.

You may wonder... Okay, but we'll never see fake walls in real life, so what's the point? The point is that vision only has its limitations. We've seen Tesla confuse the moon with objects, miss a red light, do phantom breaks, and even miss truck trailers. For all of these reasons, I would say that the Waymo approach including Camera+LiDAR+Imaging RADARs is a better choice. They get the #1 point.

One caveat is that the algorithms, processing power, and energy required to operate this vehicle is insane. LiDAR sensors consume a lot of energy and record a lot of data. Tesla is much cleaner in that perspective.

Speaking of algorithms, let's now move to this second point.

Tesla vs Waymo: Who has the best algorithms?

If we now look at the algorithms, who is closer to build self-driving cars? Tesla and Waymo started off with very different architectures, but now seem to converge towards End-To-End Learning. So let's take a look...

1. Tesla FSD Algorithms: HydraNets, Occupancy Networks, and End-To-End Learning

In my article breakdown on Tesla, I'm doing a full deep dive on the Tesla's algorithm, and how they work; so I won't do that here, but I will still show you the overview of how they built their autonomous vehicle architecture. Note that it's according to the Tesla data; which moved to private in 2023.

You can see 3 main blocks:

• Lane & Object HydraNet: The lane and object Hydranet is a multi-task learning network that takes in the 8 cameras, learns features from each using a CNN, fuses them spatially and temporally via a Vision Transformer, and then outputs several heads. Heads are trained to detect objects, lanes, positions, and so on... You can read more details here.
• Occupancy Network: The Occupancy Network is also processing all 8 cameras spatially and temporally, except that it's trained to leverage spatial data. This is a 3D network that aims to build voxels and assign a free/occupied state to each. You can read more details here.
• Planning & Control: The Planning & Control node used to be (in the drawing) done via a Monte-Carlo Tree Search. This is traditional artificial intelligence. In 2024, they replaced this with a Neural Network planner. While we don't have details on how it works, the "End-To-End" comes from this node moving to Deep Learning, making the entire network differentiable.

To push the explanation even further, let me show you the typical visualizers on a Tesla, and see how they both refer to an algorithm:

Now, let's see Waymo...

2. Waymo's Algorithms: 3D Deep Learning, Diffusion Planners, and more...

It's a bit harder to fully track the state of Waymo's algorithms, because they continuously release multiple research papers, and we don't know which ones are in production, and which are just pure research. Still, according to my research, there are 3 core pillars Waymo relies on to drive...

• LiDARs
• Prediction/Tracking
• Imitation/End-To-End

LiDARs

Early on, Waymo pioneered work on LiDARs with 3D Object Detection algorithms like SW-Former. These algorithms process LiDAR point clouds and output bounding boxes in 3D. This architecture has been updated a few times, but now serves as the "core" detection algorithm of Waymo. From there, it's encapsulated into other pipelines, like the Late-To-Early Fusion:

See what's happening? We have a temporal fusion algorithm that processes LiDARs and boxes from t-1, t-2, and so on... and each of these go to SWFormer to output a final head. This is turning SWFormer into a temporal detector, and not just a frame-by-frame detector.

Prediction & Tracking

Going even further, we have Prediction & Tracking. Waymo bets big on tracking, and one of the core algorithms I noticed there is an architecture recently released called Stateful Track Transformer which is doing exactly the job of tracking from SWFormer. Over the years, Waymo released TONS of prediction and tracking architectures.

End-To-End/Imitation

Back in the early 2020s, I remember vividly Waymo mentioning an algorithm called ChauffeurNet, who was behaving exactly like Tesla's HydraNet, but was outputting trajectories. Since then, the approach evolved, and the later published papers and public talks mention the End-To-End architecture named EMMA, as well as Vision Language Models.

As you can see, this turned in to an Encoder/Decoder architecture, where the encoder learns features from each sensors, then fuses spatially and temporally, to learn a compressed representation of the scenes. Then, the decoder is a generative part, built on VLMs to predict a trajectory.

While many claimed Waymo has a "modular" approach, while Tesla has the advanced End-To-End approach; this is simply no longer the case. Although we don't know whether Waymo uses EMMA in production, or still relies on their traditional pipeline, we definitely know they're heading towards it.

3. Who has the better algorithms? Waymo or Tesla?

Let me write 4 or 5 bullet points explaining what I think:

• Both Tesla and Waymo seem to be headed towards End-To-End Learning, because the Modular approach has the car behave in a "robotic" fashion. This is not smooth, feels robotic, and rule based. So both go towards End-To-End...
• But to make End-To-End work well, you need LOTS of data. Tesla has that from their fleet of millions of cars all across the world, but Waymo has a small fleet driving only in very specific regions. Scaling via End-To-End will be extremely painful for them.
• On top of that, Tesla only needs to process camera data, which makes the algorithms likely faster, using less power and less time consuming.
• End-To-End's biggest problem is edge cases. Tesla has experience driving several million miles in construction zones, parking lots, or in a new city, and so on... Waymo, on the other hand, is stuck with HD Maps and can have issues moving towards End-To-End...
• Tesla also has advanced techniques for Edge Case Detection and retraining, such as Trigger Classifiers (see my detailed overview here), as well as Dojo and Self-Supervised Learning. They seem far more prepared for End-To-End Learning in my perspective than Waymo. It's as if Tesla had paved the way for this for a decade, while Waymo pivoted last minute

Given these, and under the assumption that the price here is End-To-End, I would give my points to Tesla.

Waymo vs Tesla: Who has the better Map to Level 5?

In this last point, I would like to get off the sensors and technique, and see these as businesses primarily. Their goal is to sell self-driving cars, or autonomous rides, and thus... who's leading in that sense?

1. Strategy

First of all, something very important to understand:

• Tesla sells self-driving cars
• Waymo rents autonomous transportation services

This is essential, because this means their software, sensor stack, philosophies, business models, and even strategies to reach Level 5 are totally opposed. This is very clear when you see the graph below, which shows Waymo starting with a very capable vehicle, but only in ONE geo-fenced area, with ONE car, while Tesla starts with millions of car, but none of them are autonomous. Both don't scale the same thing:

Tesla's goal is to make millions of cars at a 25,000$ price point. Imagine Tesla in 2016, being told to integrate LiDAR technology, which cost over 50k USD/unit. Wouldn't they be better off starting with a light camera + RADAR Level 2, and gradually improving it? Of course they would, if they believe it's possible.

On the other hand, Waymo had a fleet of maybe 20 vehicles at the time. With 29 cameras, 5 RADARs, and 6 LiDARs, a Waymo car costs significantly more than a Tesla car. In fact, each car was estimated to cost around 250,000$ a few years ago. Since then, the LiDAR price dropped, Waymo grew its fleet to over 2,000 cars, and each is now estimated around 150,000$. Can you see how they cost is less and less of a problem to them over time?

Waymo's LiDARs certainly cost a lot, but this cost get absorbed as they do more paid rides, until it 'supposedly' becomes profitable. Supposedly, because there is maintenance, replacement, and growth of the fleet, which can make the cost a never ending fight. Even there, Waymo has more leverage to afford to LiDARs, by raising the ride cost of millions of people from 7$ to maybe 13$ (made up numbers, terribly off — just making a point).

There are great statistics on this blog post from Ben Buchanan that show how Waymo's car get more and more affordable over time. A LiDAR costs 500-1,000$ today, this completely challenges the vision-only philosophy. Unlike Tesla, time in on Waymo's side.

You can therefore understand how each one wins:

• Tesla starts with TONS of disengagements and tries to decreases that number to 0. The more capable the algorithm, the more cars the will sell.
• Waymo starts with very few disengagements and tries to scale the number of rides and regions without increasing this number. The more regions they will cover, the more rides they will sell.

2. HD Maps

Waymo is betting on a serious "HD Map" strategy that Tesla refused to adopt. According to Tesla, the car should be able to drive anywhere in the US, so there's only using normal Google Maps or OpenStreetMap. It does not mean they don't use HD Maps; they do (see screenshot below), but they don't require them to drive. If the car ends on a parking lot with no map, it should still be able to drive.

On the other hand, Waymo maps every squared inch of every place they drive in. This means every traffic sign, every bumper, every crossroad, every traffic light, every roadwork, lane lines, speed limit... everything is continuously mapped and updated. You can see on the image the HD Maps of Waymo and a screenshot of Tesla's HD Maps on a car in debug.

From what I read, Tesla performs poorly on regions where they don't know the maps. While this doesn't block them, this certainly causes disengagements. They are therefore equal in this sense.

3. Miles Driven & Disengagements

We can't conclude this article without first looking at who slams the brakes the most. Yet, I feel you already know the answer, from the first two points. It's obviously Tesla, because they have many more cars, miles driven (5B for Tesla vs 100M for Waymo), and different situations. So how can we really vote who has the better map to Level 5?

Let's first look at disengagements. What is a disengagement? Is overtaking a stuck vehicle a disengagement? Is accelerating? What is the definition, and do Tesla and Waymo use the same? Well, Waymo uses safety drivers, who obey specific instructions. Tesla drivers disengage for virtually any reason, even if they simply feel like it. This is why Tesla FSD Tracker shows, as of September 25, 24% of FSD drives have a disengagement, and 3% have critical disengagements (US).

Continuing with more stats: Tesla drives on average 213 miles before a disengagement on a highway, and most disengagements are caused by Lane Issues. Notice the jump in miles driven without a disengagement with FSD 12.6.

For Waymo, it's a different story. Waymo has a driver permit in California, which had them recently release the 2024 disengagement report to California DMV. It clearly showed a number of 9,793 miles driven before disengagement. You may ask... How is Tesla at 200 but Waymo at 9,700 miles driven without disengagement? It's because, as we said, the definition, reason for disengagement, and cities, are extremely different.

If a company wants to drive on a straight line for 500 million miles and show 0 disengagement, they technically can. This is why I think, we can't really trust any of these numbers — only their relative evolution on the same conditions as before 🤷🏻‍♂️

So who has the better Map to Level 5? I will tell you, but only after we have made a quick summary of what we've seen...

Summary: Who is ahead, Waymo or Tesla?

• Tesla and Waymo both have different sensor stacks. While Waymo relies on a stack of 29 cameras, 6 radars, and 5 LiDARs, Tesla takes a different route and relies on 8 cameras only.
• A sensor setup with LiDARs and RADAR redundancy is safer and gives more reliability than a vision-only setup. LiDARs can detect objects and brake the car even if no object is identified by the camera or an algorithm. Waymo also drives in more weather conditions than Tesla, who is physically limited by the lack of other sensors.
• Waymo uses an architecture based on LiDARs, with algorithms like SW-Former, Prediction/Tracking, and EMMA as an End-To-End system.
• Tesla uses an End-To-End approach involving HydraNets, Occupancy Networks, and Deep Planning. This approach is reinforced by powerful Trigger classifiers, Self-Supervised Learning Dojo, a powerful data fleet — making them win on the algorithm aspect.
• Waymo depends heavily on detailed HD Maps and can only work on these mapped, geo-fenced areas. Tesla’s system is designed to drive anywhere without necessarily relying on HD maps. In practice, they drive much better when they have maps.
• Waymo boasts a much better disengagement rate, with safety drivers rarely needing to take control compared to Tesla’s more frequent interventions; but the conditions they drive in and disengage are 100% under their control.
• The two companies have very different business models: Waymo sells autonomous ride services, Tesla sells cars with self-driving features. As a result, their map to level 5 is not the same.

And now, you're all caught up. So, the Map to Level 5?

The Map to Level 5

Here is what I think:

To reach Level 5, Tesla will need to reduce the number of disengagements. To me, they will have to include LiDARs or RADARs at some point. At today's cost, that would probably be feasible. In fact, I'm pretty sure Tesla is so competent FSD would be solved right now if they didn't chose to play the game on hard more. But now, after all this time, can they really afford to do this? There's ego, brand image, and the "FSD capable" computers they already sold. How can they? Tesla is in the camera game for good.

To reach Level 5, Waymo has to increase the number of areas they drive without increasing the disengagement rate. This means adapting the "HD Map" strategy, which still relies on them, and making algorithms capable to adapt to a new region faster. To make autonomous driving a reality for the entire world, Waymo will need to go faster. Their autonomous vehicles surely are capable, but their current scaling strategy takes too long.

Next Steps

Have you ever heard the story of the iPod? It started in January 2001, right after Apple announced a loss of $195 million, and had missed the shift to digital music. The company was lost, and had one last chance of survival: building an MP3 player to catchup with the competitors.

If you are old enough to remember the MP3 players back then, the were confusing to use, overloaded with buttons and menus, and made the experience painful for customers. Apple was looking for a solution for months, but had no clue how to make it better.

Until one day, when Apple's Head of Marketing Phil Schiller suggested using a scroll wheel. Wheels already existed in mouses and dial phones, but had never been never used in music players. With this, he suggested that the menus should scroll faster the longer the wheel is turned, a stroke of genius that would distinguish the iPod from the agony of using competing player.

The rest is history: Apple developed the iPod in the greatest secrecy, launched it, and changed the world with "1000 songs in your pocket".

What made it so successful? It's not that it looked good, or had buttons, or could store more songs. No, the genius was in the smarter experience scroll wheel.

If you're in the autonomous vehicles market, we probably witnessed a similar pattern: companies have been collecting more and more data endlessly, building data centers, simulators, hiring people to analyze the data generated, and so on... Until some companies came up with smarter ways, not involving just "collecting more data", but rethinking the experience to focus on events instead.

In this article, I would like to tell you about the way automotive data processing works nowadays, and how the AI revolution is going to reshape it.

We are going to learn about 3 ideas:

1. The first part is going to focus on the Manual Era (where we collect and process it all) and the Cloud Era (where we use DataLakes)
2. The second part will be a case-study provided by an autonomous tech startup, revealing the 10 biggest problems of the Cloud Era.
3. The last part will show you the Edge Intelligence Era & the Autonomous Era, which, as you'll see, if an incredibly more intelligent way to do

Let's begin with point #1.

1. Data Management: How self-driving car companies collect and process data in the Cloud Era

One of the things we heard the most this past decade was that Data is king. And for a long time, collecting as much data as you can in order to train heavy machine learning models has been the only way to do. Let's talk about data collection, and then processing.

How do autonomous vehicles collect data?

We know that when a self-driving car drives, all the data (sensors, images, messages, hardware status, algorithm decision, ...) is being recorded.

Should we give an intro line explaining how?

The process is simple, and looks like this:

1. You plug your sensors to your system (for example, Robotic OS/ROS)
2. You press record

I'm sure somebody out there worked hard to find a more complex process, but if you're using a tool like ROS, recording data is as simple as using one command line. In the video below, you can see me recording LiDAR point clouds, camera images, GPS positions, algorithms outputs, and mostly all the messages passed through the self-driving car while we drive...

When I'm done recording, the output is a file in the .bag extension (for ROS 1) that can vary from a few Gb to Terrabytes of data. Let me show you an example below from the TIERS dataset. Notice the duration and sizes of the recordings below — the last one is just 8 minutes long, and yet weights 200Gb. This is 2.4Gb/minute!

The bag is the first element. Then comes what we do with it.

The Manual Era: How do we process and analyze data?

The first "era" I'd like to tell you about is the 1.0 era. Back when I worked on autonomous shuttles, each of our fully autonomous vehicles was driving and collecting data to SSD drives. When the day was over, we had hundreds of Gb to process. So we started coming up with file naming conventions, involving the date, event, and so on...

Then, when back to the office, we could replay our algorithms on it, train our models on the data, and so on... Below is an example of a drivable area segmentation algorithm I've been training on the data collected:

This was fine for a small startup of 8 people, and it's probably still okay for small companies that don't need extensive processing, but most autonomous vehicle companies have turned to the cloud...

The Cloud Era: How Advanced Driver Assistance Systems (ADAS) & most of the Automotive Industry is using Data Lakes

If you record data every day, and each recording is hours long, you're never going to find the events you need. This is why I'm showing you a more sophisticated, let's say '1.5' version, which makes data collection part of a pipeline.

It looks like this:

1. You record the data
2. You upload it to AWS/Azure
3. The R&D team then processes it weeks later, replaying all the events, searching for 10% possibly interesting scenarios, or events, and so on...

If you'd like to see real-world concepts, you can see AWS and Azure Data Lakes:

A lot of companies in the self-driving car market use these "data lakes". Let's look at the Azure Data Lake in a simplified view:

ADAS & fully autonomous cars use it. After recording, the 4 key blocks are:

1. DataOps: Where we analyze data, clean it, label it, augment it, tag it, and so on... Notice the interaction with external labellers; that idea is called "human-in-the-loop".
2. MLOps: The machine learning algorithms, training, testing, and so on...
3. ValidationOps: The validation part, involving visualization, scenario, and simulation.
4. MetaData: After the DataOps tagged the data, we can search for it.

You can see how it's placing data at an element in the chain.

So what are the problems of this? Before telling you about the 2.0 Autonomous Era, let's try to see a case study with real ADAS or artificial intelligence companies using it...

2. [Case Study] ADAS Actors reveals their 10 biggest problems with Data-Driven Approaches

In this section, before talking about the '2.0' approach, I would like to tell you about the core problems companies who process large volumes of data reported.

Before writing this article, I got the opportunity to talk to Heex Technologies, a french startup specialized in Event Based Data Management... and I asked them "Which problem do you solve?". To answer, they shared a 20 page PDF listing all the problems their biggest Advanced Driver Assistance Systems (ADAS), autonomous driving, or robotic clients from the automotive industry.

In the PDF, I spotted a lot of interesting problems. Let me share the main ones with you:

If I were to list down the 10 main problems, you'd see: Slow access to critical events, Manual data processing, Exploding Cloud costs, Fragmented data, Delayed visualization (no real-time), Manual Extraction of scenarios, Useless streaming data, Physical SSD Extraction, Blind Debugging, and Inefficient ROS Bag Processing.

Notice all these terms I underlined? These are the problems of today's data management systems.

Let's take some examples...

• If you collect the data on Day 1, and process it on Day 3, you have slow/delayed access to critical events; like a missed pedestrian. So you're driving, notice something wrong, but you have to wait until 2 days later to even look for the data, and start searching for that event you noticed...
• Similarly, can you see how the 'fragmented' data processing is a problem? Especially when you are with a team. Engineer A grabs bag A, and makes decisions based on it... Engineer B grabs bag B and makes a different decision based on it... The entire decision cycle happens in silos.
• The Physical SSD extraction is a problem too. In May 2025, I was at the Stuttgart ADAS & AV Expo, and I met a company who invented a "swap" disk system... All of this is great, but that's still the same problem of storing, copy/pasting data, etc... to a system.

In each of these problems, I noticed a time and money waste.

For example, the client reporting: "Engineers waited several days to weeks to access specific events due to the time-intensive process of uploading, filtering, and classifying raw data in the cloud." is clearly facing a time problem, reviewing large amount of data... The other client who mentioned: "The full data pipeline we built —from data capture to processing and storage—incurred high cloud costs and consumed engineering resources" faces a money problem...

In this same report shared by Heex, all the companies reported improvement in their pipeline. Whether it was better decision making, more time freed, or money saved. This is why the next part is so important, so let's now focus on it: Event Driven Data Processing for autonomous cars.

3. Event Driven Data Management for autonomous cars

Back when I started learning autonomous driving algorithms, I listened to an interview from Sebastian Thrun, acknowledged as te godfather of self-driving cars, who at some point, said something that marked me: [paraphrased]: "With a team of 2/3, you can build a self-driving car that drives 90% of scenarios in a weekend. Then to get to 95%, it takes a few weeks, and to complete these last 5%, it takes years."

This idea is called the "long-tail" problem.

When looking at traffic accidents involving autonomous vehicles, you often see rare events or edge cases at the root cause. The person wearing a stop sign t-shirt, the truck with a donkey on the trailer, the traffic sign burned by parisian riots, all of these unusual scenes totally different from empty highways cars are used to.

Some companies solve it with data generation, others with simulation, or with End-To-End Learning. Yet, the root of all evils here is data, and thus, this is what we have to change.

A decade ago, the term "data" became king, and everybody became a Data Scientist, Data Engineer, Data Ops, Data Something. It was the case until recently when the data revolution passed, and breakthrough innovations happened not thanks to more data, but thanks to smarter training systems (like self-supervised learning), or more powerful architectures (like transformers). "More data" was ultimately not the solution, and thus, we have to switch our thinking...

The Edge Intelligence Era: From Data Management to Event Management

After companies have recorded a few laps of the neighborhood they drive in, recording more of this same scene doesn't make sense. Companies record more and more, just to spot the 1% of long-tail events. What if we worked on these events only, from the beginning?

It can be done, by setting up a "triggers" in your system, that will act as a filter and only capture the scene when interesting events happen, such as:

• Objects Missed: If one camera misses an object that another sensor sees
• Near Pedestrian Collision: If pedestrians are within 2 meters of our car, and we drive over 30km/h
• Human Intervention: If a human driver manually took over
• Shakes: If the camera physically moved due to a bumper or small shock
• Ego Collision: If a collision with the ego vehicle happened
• and so on...

All of these are valid events we'd like to record. The rest? When it's all smooth? Well, we already have millions of it.

An example with Heex Technologies, and their platform allowing to set triggers:

If I were to show you the 2.0 process, it'd look like this:

• You have the same car with LiDARs generating the same 10Gb/h data
• Rather than recording all data available, you define triggers.
• You intelligently record the events, like the near pedestrian collision, and not all the data
• You get instant notifications, labels, and can do real-time decision making

Seems smarter, isn't it?

Now that you have this in mind, I'd like to show you the last era...

The Autonomous Era: AI does it for you

The next step is to create algorithms to do it automatically for us. For example, Tesla patented a concept called trigger classifiers. The idea is to train their HydraNet backbone to classify whether the general scene it's learning from contains unusual events or not. If it does, let's say above a certain confidence score, then the machine learning models will trigger a warning.

Whether you're working on spreadsheets or building autonomous vehicle technology, automating manual items like labelling or searching for data makes sense. In this case, just like the Edge Intelligence Era, you can see the events being captured live while driving, and not after.

The 2.0 Vision

This is going with the "2.0" vision of self-driving cars that companies now define. A vision driven by Deep Learning first, where data matters, but where more data isn't the solution. In the 2.0 vision, quality is better then quantity; contextual intelligence is needed, learning should be real-time, and training should be done on relevant data.

If the 1.0 vision involved heavy test vehicles, modular architectures; the 2.0 vision is about AI & efficiency.

Now, let's see an example of a company specialized in this...

Example: How Heex Technologies turns Data into Event Management

One of the companies that captured this vision the best is Heex Technologies. They built a SaaS platform that implements exactly these ideas of "triggers" — and their motto is that rater than focusing on the data, they focus on events. As I already showed you the triggers, we could see it in action:

Let's look at their pipeline, which you'll notice also works backwards — once the bag is generated:

From a heavy bag, we get a smart bag. The data is definitely smarter when automatically annotated, categorized, and when relevant events are flagged. We can then re-inject this data into the training pipeline, without having to worry about the rest of the dataset.

As an entrepreneur myself, I can only admire the focus on one specific and painful problem like this one. When you can anticipate customer needs, and enable automakers and automotive engineers move away from a complex process to focus on their core job (self driving technology)... you win!

Alright, let's do a summary and see what to do next:

Summary & Next Steps

• Collecting data is essential to train AI models and develop self-driving vehicles. Yet, "more data" is not the solution to create breakthrough, and solve the "long tail" problem, which cause a significant challenge.
• The first Era of data processing is the Manual Era, in which we record and process everything manually. (1.0)
• The second era (1.5) is the cloud version, in which you work with data lakes and build a real "chain" that contains DataOps, MLOps, ValidationOps, and so on...
• The third era moves to the 2.0. It's where we stop obsessing on the data, and focus on events. We can use triggers and platforms like Heex Technologies to do it.
• The fourth era is the AI Era. (2+) This is where we have AI automatically find events, and train itself continuously on these.

Which solution is right for you? In reality, they can all work. A small startup can work manually, until they find their hard problem to solve, and they have a budget to invest in data lakes... Companies can work with data lakes, but for bigger fleets, it'd make much more sense to think in terms of events instead.

Next Steps

 If you realise you have these problems of recording everything, having your data staying a bit « dumb », and would like to know exactly how to stop recording everything by this afternoon (without losing the important information)...

... Then I’d recommend to check out Heex free discovery quiz, which will reveal tell you exactly what you’re doing wrong today, and (based on your answers) show you what to do this afternoon to save hours of recording, data processing, etc...

It’s free, and you can get access below:

Heex Free Discovery Quiz

Is your data strategy a silent obstacle?

Take the Quiz

You can also take a look at Heex' product here: https://www.heex.io/en-gb/smarter-data-faster-decisions

How much impact do you believe your job has? Is your job saving someone time? Or money? Or... his life? Well, what if you worked on projects that saved people's lives? Such companies exist, in self-driving cars, in healthcare, and in the case of this article... in Autonomous Defense!

This summer, I have interviewed Vibhav Ganesh from Shield AI, a U.S.-based defense technology company that develops autonomous systems for military and government use.

I would write a big paragraph here, but let me instead show you a quick sample from the interview...

Let's read his intro to Shield AI and to their core products: the V-BAT and the ViDAR.

There are a lot of things to note about their products: the V-BAT can last 10 hours, which is a technological achievement itself, thanks to V-TOL (vertical takeoff and landing)... ViDAR is also a very interesting product, which stands for Visual Detection And Ranging... and HiveMind (not shown here) is their AI, or as they call it, "The World's Best AI Pilot".

Let me take you to the v-BAT first, as it's the core product, by showing you this LinkedIn post we did together, where Vibhav Ganesh introduces us to Shield AI.

Together, we recorded an exclusive Fragment of The Edgeneer's Land, my community membership experience, in which he takes us through Shield AI. What is autonomous defense? What are the main technologies involved? What is the range of products?

In this post, I'd like to give you a small sample of that interview, highlighting a very interesting moment where Vibhav talked about infiltration drones.

Inside Shield AI's Tactical Infiltration Drones

Impressive, isn't it? What I really love about it is that it's down to earth. I could see myself assembling a drone kit, adding a camera, a 2D LiDAR, and starting experimenting with Visual SLAM projects to map a room. This is basically what Shield AI did, when they got started. Except that their drone was (1) targetted to a specific client and (2) better than all competition.

There are 2/3 insights I'd like to share with you, from Vibhav:

1) Self-Driving Car > Autonomous Transfer doesn't work as we'd think

Now, here is something important to note:

A lot of what you learn in autonomous robots CANNOT simply be transferred to drones.

I did think that it was a matter of copy and paste. But I understood I got it wrong when making this fragment, especially with Vibhav Ganesh who told me that their drones don't have LiDARs, fly over seas, or over deserts, no man's land zones, between mountains, campaigns, with 3D constraints, and no map!

I thought about it for a minute, and I realized... "Wait, it's absolutely NOT like autonomous cars!"

And in fact, when you start looking into autonomous drone architecture, they absolutely don't look like self-driving car architectures! For example with Shield AI, they have a Control, a Station, RTOS, a ViDAR, but also a Flight Controller powered with frameworks like PX4 and Maven. This is an entire set of libraries to learn.

From there, here is a second insight:

2) Visual SLAM is mostly used

Coming back to the idea that we are NOT like self-driving cars... the other main difference is that we use no map. So without a map, and with just a camera, they have no choice but to use...Visual SLAM!

And Vibhav explains really well what kind of SLAM they're using, how they implement the mapping, even though there is 0 starting point, and so on. Here is a sample of a vSLAM project I've tested with drones:

This is the nitty gritty of Shield AI's work. Once you build a SLAM MAP, you can then feed that map to the Motion Planner, which sends a flight order to the drone. If you'd like more insights on this technology, I highly recommend my vSLAM article.

The 6 Components of a Visual SLAM Algorithm

How does Visual SLAM work? How is it different from normal SLAM? What are the 6 main steps of a Visual SLAM system? Let’s find out!

Read from the most advanced autonomous tech blogJeremy Cohen

Okay, would you like to see some samples?

Shield AI in Action

Let's take a look at 3 samples here:

• On the left, you can see drones being launched. These drones cannot do the "vertical takeoff and landing". They are projected to the air by launchers and then fly like a plane.
• In the middle, you can see the tactical quadcopters we discussed. Notice how they use vSLAM at the end of the shot.
• On the right, you can see a mission of the v-BAT searching for a vessel in a sea canal.

This is really cutting-edge, and totally applied what we are building in the autonomous tech space. Alright, time to wrap up!

Summary & Next Steps

• The defense industry is extremely active. Hundreds of companies work on the autonomous generations of drones, anti-missile detectors, infiltration equipment, RADARs, and more...
• Shield AI is an active actor of the defence space, with a range of multiple products, such as the v-BAT, the ViDAR, and HiveMind.
• Shield AI started with NOVA 1, a quadcopter that could infiltrate into buildings, builds maps, surveys, without the need to send humans into the region. This allowed to prevent human losses due to buildings collapsing or being trapped.
• Besides from being safer, infiltration drones are also more efficient. Shield AI tested their drone against 6 NAVY SEALs clearing a building, and it finished in a third of the time.
• The transfer of autonomous car/robot technology to autonomous drone isn't as simple as we'd think. Architectures are different, products are different, regions/environments are differents, and even the inside technologies and algorithms change.
• On the other hand, some technologies really apply well to autonomous drones, such as Visual SLAM.

On September 23, 1999, NASA’s Mars Climate Orbiter—a $125 million spacecraft—was set to enter Mars' orbit to study its climate and atmosphere. But just as it approached the planet, something went terribly wrong. Instead of entering a stable orbit, the spacecraft plunged into Mars’ atmosphere and was destroyed.

After analysis, NASA found a unit mismatch: their Jet Propulser used metric units (newtons), while the spacecraft they got from Lockheed Martin used imperial units (pound-force). This caused navigation errors, making the spacecraft descend far too low into the Martian atmosphere; and causing a 125m$ loss.

Human errors happen every day in all sorts of domains. In 2016, an alarming report from Johns Hopkins estimated that medical errors (including misdiagnoses) cause over 250,000 deaths annually in the U.S., making them the third leading cause of death. Many are due to errors in analysis of medical images, such as MRIs, X-Rays, CT Stans, and more.

In this article, I would like to show you how Medical Image Segmentation can be used to counter this problem, and I'll do it in 3 points:

1. 2D Medical Image Segmentation
2. 3D Medical Image Segmentation
3. Examples/Demo

Let's get started...

Intro to 2D Medical Image Segmentation

In 2019, I hosted the biggest AI Healthcare hackathon ever held, happening simultaneously over 20 cities! The goal at the time was to mix companies, healthcare groups, and engineers to build healthcare solutions using Deep Learning. After the 48 hours of coding, the winning team would win 10,000 USD, the second 4,000 USD, and then team 3, 4, 5, and 6 would win 2,500 USD each!

Great computer vision projects happened, and in fact, Paris (my city) finished the competition #2 via Spot Implant, a Shazam for Tooth Implants project that then became a startup. At the time, everybody was working on 2D Images. We had projects like Skin Melanoma detection, X-Ray segmentation, Brain Segmentation, and more...

Let me show you a few tasks in Medical Image Segmentation, and then we'll look at algorithms.

2D Medical Image Segmentation Tasks

X-Ray (the most common)

First, we have X-Rays. X-Rays are the 2D representation of a body. We often see bones and organs there, and it's the most common image you'll find in Deep Learning x Healthcare. Using medical image segmentation, we can assist doctors in finding bone fractures, lung diseases, and other abnormalities. It can also help in screening large volumes of X-rays for tuberculosis, which is particularly useful in low-income countries with limited access to radiologists.

This really is the most known among Deep Learning Engineers. I would like to show you other applications of segmentation...

Dermoscopy Segmentation (skin lesion segmentation)

Dermoscopy segmentation was the health hackathon's top pick. It's all about using medical image segmentation to spot and separate skin lesions in dermoscopic images. By applying deep learning on medical images, we can quickly and accurately detect skin conditions like melanoma. This helps dermatologists diagnose and treat patients faster and manage large amounts of data more efficiently.

Let's see one or two more...

Mammography Segmentation

Mammograms are specialized X-ray images designed to reveal the inner structure of breast tissue. These images typically come in a flat, 2D format, capturing the breast from multiple angles to ensure a comprehensive view. The details in mammograms can show everything from dense tissue patterns to potential abnormalities like lumps or calcifications.

Look at the image below: see how the role of a doctor/radiologist is to find these highlighted areas. The role of image segmentation is to assist the doctor, so he's not alone doing that high stake task of spotting problems (of course, it goes without saying that doctors also do much more than spotting, from understanding how bad a calcification can be, to finding the treatment, and so on...).

Other Types: Ultrasound 👶🏽, Endoscopy 🤢, and more...

We just saw 3 types: X-Rays, Dermoscopy, and Mammography. There are other types, such as ultrasound images (baby for examples), which can be 2D or 3D; or endoscopy, and more... The image below shows many 2D segmentation applications:

So how do you build the segmentation results? What do you use? Let's take a look...

2D Medical Image Segmentation Models

Ever heard of UNet? You know, that 2015 model subtitled "Convolutional Networks for Biomedical Image Segmentation". Well, it may be from 2015, but it's a great way to start! In fact, there have been lots of improvements of UNet, to UNet++, Trans-UNet, Swin-UNet, all keeping that "U" shape, but using different pattern recognition techniques like Swin Transformers, CNNs, etc...

This is one family of semantic image segmentation algorithms, and here is what the results look like:

To get true numbers, Dice-Similarity coefficient (DSC) and average Hausdorff Distance (HD) are used as evaluation metric to evaluate these algorithms.

These are great, but what happens when you don't have millions of labeled data? In healthcare, getting access to labeled, free-to-use data isn't easy; especially for certain types of diseases that are specific to certain hospitals, and so on... In these cases, you can use more "foundational" semantic segmentation models such as SAM (Segment Anything) or SAM2. These have been trained using Self-Supervised Learning on "the entire internet", and thus are supposed to find problems better.

For example, MEDSAM is a Medical SAM (Segment Anything) is what I used for the images above. It's the regular SAM, but tweaked for medical image segmentation, to boost the segmentation performances. The model performance is quite high, and we get a top notch Computer Vision project using image segmentation... It can can even take your prompt as a region of interest bounding box to return the segmented masks:

So this is for the first part on 2D images... Now what are 3D images?

3D Medical Image Segmentation: CT Scans & MRIs

Now come 3D images! For this part, I'll talk about the two use cases (CT Scans & MRIs) and discuss the algorithms together.

CT Scans: Use Cases & Algorithms

Use Cases for CT Scans & 3D Representation

In the FLARE 2022 dataset (Fast and Low-resource semi-supervised Abdominal oRgan sEgmentation), we get access to a few hundred labeled and unlabeled cases with liver, kidney, spleen, or pancreas diseases as well as examples of uterine corpus endometrial, urothelial bladder, stomach, sarcomas, or ovarian diseases.

Hey, relax. I'm just scaring you. I didn't get a clue of what that meant either. Except that:

These are CT SCANS (Computed Tomography Scans). A CT scan uses X-rays to create detailed, cross-sectional (slice-by-slice) images of the inside of the body. They're more detailed than traditional X-rays because they produce 3D images by taking multiple X-ray images from different angles and combining them using a computer.

So what's the "3D" output like? Voxels? Point Clouds? Not exactly. As I said, these are images taken with multiple "layers" (or dimensions). So your input image dimension isn't (512, 512, 3) but (512, 512, 129) or something like this. You have a multi-dimensional image on which you can apply image segmentation to each of the 2D slices:

In this example, I used MedSAM to process individual 2D images. If you do it on the entire 3D CT Scan, you get something like this:

If you get it, you understand that from these images, we can put that into a software that is going to reconstruct the scan to 3D:

From there, people go absolutely nuts and even try to make it into a point cloud (I'm not sure why, but this is cool, shoutout to Beau Seymour's video).

MRI Scans: Advanced Medical Image Computing

Magnetic Resonance Imaging (MRI) Scans are another powerful tool in medical imaging. Unlike CT scans, MRIs use powerful magnets and radio waves to create detailed images of organs and tissues within the body. This technique is particularly great for soft tissue contrast, making it ideal for brain, spinal cord, and joint imaging. By leveraging medical image segmentation, MRI scans can aid in the precise identification of tumors, neurological disorders, and musculoskeletal issues.

Here is an example of MRI scan and its segmentation task:

So now, let's see how to process that...

Algorithms in the 3D Medical Image Segmentation Domain

We already discussed SAM (Segment Anything) and how it can work on individual slices. The reality is, medical image segmentation involves a lot of complex "job" knowledge; and it would probably be better to use a specialized artificial intelligence model for optimal model performance. Today, in AI, we have two types of models:

• Foundation Models, that are very general and know everything
• Specific & Labeled Models, that can only process the images it's been trained on

I would like to show you two models doing both: TotalSegmentor & Vista-3D.

Total Segmentator: A specific model for 2D and 3D Segmentation

Perhaps one of the most used and well known "framework" for image segmentation of both 2D and 3D data is TotalSegmentator. Rather than being a simple machine learning model, it's a complete framework that does the automatic labelling.

The number of classes for CT and MRI data it can segment is gigantic:

And the model is based on the nn-UNet architecture, which is similar to UNet, but can also take in different medical imaging modalities.

VISTA-3D: Foundation Model for 3D Medical Image Segmentation

VISTA-3D is a 2024 "Foundation model" from Nvidia that works on the 3D patch directly. While being named "foundation" model, it's incredibly specific to the medical image segmentation tasks. Here, we are PURELY in 3D Deep Learning.

So we've seen a lot:

• 2D Segmentation can be done with models like UNet, UNet++, etc... (specific), or SAM (foundation)
• 3D Segmentation can be done with models like nnUNet/TotalSegmentator (specific), or Vista-3D & SAM (foundation)

Let's see examples now...

Example 1: CT Scan Segmentation with Vista-3D

In this platform from Nvidia, I am able to select a CT Scan and call Vista-3D to process it.

Notice how we can select an Abdomen, and then pick all the organs we want to segment. Finally, we can get the view from 3 different "angles" and process that too!

Example 2: CT Scan Segmentation with TotalSegmentator

On totalsegmentator.com, we can upload images, and ask for complete segmentation. Here, I am going to upload a scan from the FLARE2022 dataset I mentioned above. The platform return hundreds of organs all in a weird format 'nii.gz' format:

I can visualize some of these, and see what the output is like:

Alright! So this is our second example, and both have playable demos! Let's now do a summary...

Summary & Next Steps

• Medical image segmentation helps reduce human errors by processing 2D and 3D medical images like MRIs and X-rays.
• 2D medical image segmentation tasks include X-Rays, dermoscopy (skin lesion analysis), endoscopy, mammography segmentation (breast), and more...
• UNet and its variants are popular models for 2D medical image analysis, utilizing CNNs or Transformer approaches. Foundation models like SAM (Segment Anything Model) can also be fine-tuned on medical images, like with MedSam.
• 3D medical image segmentation involves CT(computed tomography) and MRI (magnetic resonance imaging) scans. They're called 3D images because they're multiple slices of the same image under different views.
• MedSAM can process 2D slices of 3D scans, allowing individual segmentation of each slice. We can then fit that into a software that will do a reconstruction into a complete 3D image.
• For 3D processing, TotalSegmentator and Vista-3D are solid solutions, being either specific or foundation based.

In 1897, French police faced a difficult problem: a serial killer named Joseph Vacher was stealing and murdering sheperds, and remained impossible to catch. Every time he was arrested, he gave a different name, changed his appearance, used fake mustaches, wigs, and different clothing styles... and got to disappear without the police realizing they just controlled France's most wanted man.

At the time, France had no national ID system, and no way to prove that the man they caught today was the same man they arrested months ago. That was until an officer named Alphonse Bertillon introduced a revolutionary method: anthropometry. It's a system that labeled criminals based on of 12 unchangeable physical measurements like ear shapes, skull sizes, and limb lengths, that could not be faked.

One day, Vacher was caught for attacking a woman, and this time, the police used Bertillon's system to compare his measurements to what they had in their records: they discovered they just caught France's most wanted criminal. This time, he could not escape with a warning, and got sent to... yeah — the guillotine 🤷🏻‍♂️🇫🇷

What got Vacher executed wasn’t just this one-time capture, but the ability to analyze a series of events and not just a one-time event. And this is exactly what this article is about: the shift from frame-by-frame to sequence processing, here in Computer Vision with videos. And this is done via something called video segmentation.

So let's get started:

What is Video Segmentation?

Most Computer Vision Engineers spend time learning about image processing, but never consider what happens when you use a video. Yet, tons of architectures today, whether in surveillance, retail, sports analysis, healthcare, or even robotics and self-driving cars — now process videos instead of images. The sequence brings something individual images don't, just like the Vacher story, where he was able to get judged through all the murders he committed.

So let's take a less deadly scene — shoplifting detection in retail. There is a startup I once interviewed for named Veesion — that has this amazing video on their homepage:

Can you see everything happening here?

• We have the object tracking (the second man is moving from aisle 1 to aisle 2)
• The event detection (at 00:03, a man puts an item in a pocket)
• The action classification (of putting something in a pocket)
• The video decomposition (shoplifting from 00:02 to 00:03 — standing from 00:03 to 00:06)
• The people counting (2 people in the video, one is obstructing the other)
• And more...

Among these, there is the idea of "segmenting" the scene to track the shoplifters through the video. You can see the hands being in red, consistently from frame to frame. So this is the idea of Video Processing, and Video Segmentation is a sub-branch of it focus on the task of segmenting a scene.

There are two types of Video Segmentation tasks:

• Video Object Segmentation (VOS)
• Video Semantic Segmentation (VSS)

Video Object Segmentation

In Video Object Segmentation, we are doing exactly what I did in this video. I define an object to track, send the video to the model, which tracks the object consistently across frames. It's purely "object" based, and is NOT used in a supervised way. For example, you can use semi-supervised video object segmentation, where you define an object on Frame 1, and let the model track it across the next frames... Or you can use totally unsupervised video object segmentation, where you won't even mention the objects to track.

Let me show you an example where I am shoplifting (muahahah):

See? We are able to track my head & hands in blue, and the phone in yellow! That is the idea we're interested in... And even more when we can do this:

Now, to be fair, the floating head experiment may NOT be the most useful thing in this example, but the stolen phone is. Now think of everything we can keep track of cells in health related videos, we can keep track of a player when analysing a football match, and a lot more...

Video Semantic Segmentation

In Video Semantic Segmentation, we'll really go at the pixel level, and rather than focusing on segmenting objects, we focus on the scene. The output is going to look extremely similar to a normal image segmentation task.

Just like image segmentation, you can also use video instance segmentation, video panoptic segmentation, video semantic segmentation, and so on... And of course, there is the benefit of doing background extraction, to then process uniquely what's been segmented, for example in a case like lane detection in self-driving cars:

But from now, you may have a question:

How is Video Segmentation different than Image Segmentation?

I mean, is it really different? It kinda looks similar to. image segmentation, right? And yes, while it may be the case for some examples, like the one I just gave with video semantic segmentation, most of the tasks will be different and give different outputs.

To put it simply: Video Segmentation is about processing videos. You don't process image per image, you process video frames immediately. And this has several advantages:

• The model can track multiple objects even though they're occluded (similar to what object tracking would do, but using video sequences)
• The model can segment specific scenes you're looking for (a blood cell changing sizes, a car entering a scene, a man stealing something)
• It ensures temporal consistency, meaning an object that appears in one frame keeps the same identity/color across the entire video, enabling tracking at the same time.
• It understands object motion, meaning it can predict where an object will be in the next frame instead of treating every frame as an isolated image (thanks mainly to video instance segmentation)
• For some models, it can be more efficient, since instead of running image segmentation on each frame separately, the model processes a video sequence, leveraging temporal information to process frames together, reducing redundant computations.

So, how does that work? What type of model does this? I do NOT have a specific "do this do that" template to share with you, but by studying examples, we could probably understand what's required to make a Video Segmentation algorithm work...

Example 1: VisTR (Video Instance Segmentation Transformer)

The first paper looks terribly simple. Let's try to understand the different blocks:

• Input: First, we process raw video data, it's purely a sequence of images sent to the CNN
• Backbone: Then a normal 2D CNN processes each frame independently before concatenating the feature maps
• Video Processing: This is fed to a Transformer, known to process sequences quite well. However, we modify this transformer a bit to not just receive a positional encoding, but also a temporal encoding.
• Output: Finally, the output of the decoder predicts instances for each pixel, with a sequence matching strategy

The training is done after obtaining labeled data from the YoutubeVIS dataset, and the backbone is initialized with the weights of DETR.

The detailed version looks like this:

As you can see, we have a video processing pipeline, where the transformer is actually aware of the frames. The segmentation process ends by matching pixels with instances. This is done using Bipartite Matching (the Hungarian Algorithm). More subtle blocks exist, and I invite you to read the paper for more...

Example 2: SAM 2 (Segment Anything 2)

If you didn't live in a cave around 2023, you probably heard of Segment Anything — the segmentation model that could find any object in an image. Recently, it got an upgraded version called SAM2, which is designed to process videos. Let's take a look:

As you can see, SAM 2 differs from SAM by the addition of a memory block, made of a memory attention module, a memory encoder, and a memory bank that stores the past frames, and helps with temporal consistency.

If you played with the online demo, you will find that the model starts by asking you to click on an object, so it can keep tracking it. So at frame 0, you click the object you want to track, and then the model tracks it on the entire sequence...

This is called "Promptable" Visual Segmentation.

This is no different than the original SAM model, and in fact, it's using the same "prompt encoder". So let's the see the details of the model:

• Prompt Encoder: As expected, we begin by clicking objects, which generates a "prompt", and send this to the same encoder as Segment Anything to track object across each image
• Image Encoder: We then send the entire vide to the image encoder which is a masked autoencoder
• Memory Attention: Uses vanilla attention to condition the current frame features on the past frames features and predictions as well as on any new prompts
• Memory Bank: It retains information about past predictions for the target object in the video by maintaining a FIFO (first in first out) queue of memories of up to N recent frames.
• Mask Decoder (prediction): Similar to SAM, but accounting for previous memory information

So, you saw a second way to build a video segmentation algorithm. The first way was fully transformer based; and this second way has the somewhat robotic "memory bank"; and this because this model is a "hybrid" between 100% video processing, and frame-by-frame processing.

Image vs Video Segmentation: Worth the trouble?

I would say yes, especially considering all the use cases that can benefit video segmentation. For example, surveillance with massive occlusions (in a crowd, with walls, trees, ...) where standard object tracking would be limited, video editing, where for example, we want to remove an object not from one frame, but from an entire scene, sports analytics, entirely based on motion, cell tracking (for example, division of cells, which can only be seen via videos), shoplifting detection (which can't really be seen in an image), fire spreading, and more...

You can see this article for the normal image segmentation use cases, and I highly recommend you augment it in your mind with these video examples I provided. So as a rule:

• For most cases, don't replace all your image segmentation pipelines with video pipelines
• But for the cases where segmentation fails because you need to understand video, do it!

Alright, we've seen a lot, let's do a summary...

Summary & Next Steps

Congratulations on getting so far! Let's summarize what we learned:

• In many cases, analyzing one event fails. When video is essential, you have to use Video Computer Vision models.
• Video segmentation is segmentation applied to video processing, it's used in in various fields like surveillance, retail, sports analysis, shoplifting detection (or detecting suspicious behavior of any kind) and healthcare.
• Video Segmentation splits into two categories: Video Object Segmentation and Video Semantic Segmentation.
• Video Object Segmentation (VOS) focuses on tracking defined objects across video frames. Many applications like SAM2 are semi-supervised, because you give the model a prompt and an initial object to track.
• Video Semantic Segmentation focuses on pixel-level scene segmentation, it can also be instance or panoptic based, and the output may resemble the one of standard image segmentation.
• Some models like VisTR can be 100% video processing based. This model uses transformers for video instance segmentation.
• Other models can process frames one by one, but rely on a memory bank. In the case of SAM2, frames are processes both as a video and one by one (to keep track of a same object)

Next Steps

A few articles you can read:

• Introduction to Video Processing — an old post (you can see my writing style is much different) but good overview on Video Processing.
• A complete overview of Object Tracking Algorithms in Computer Vision & Self-Driving Cars - very related to video object processing, but without the segmentation part (bounding boxes).

In 2019, I was an Autonomous Shuttle Engineer, working for a company that got a thrilling opportunity: to equip Paris' transportation system with our autonomous shuttles. This was a golden opportunity many don't have, but the client was known to be ruthless selectors. Many others perished while trying to be "approved".

With high hopes, our team prepared for the demo day for months. We meticulously reviewed the client's 100+-point checklist, ensuring our shuttle met all requirements from real-time operations to autonomy measures. One day, a team of 5 was called to begin process in a secret underground site. It was going to begin.

The experimentation lasted days, in which each of the items were reviewed. Came the final test: Cyber-Security. The client made a phone call, and within 30 seconds, an engineer with a thinkpad came and entered the shuttle. "Oh great! We can charge our phones!" He said amused. "What a mistake!". My colleagues were sweating, horrified at the vision of what this young men could do.... and they were right: In just five minutes, using only a USB stick, he had taken control of the vehicle, got it to drive all across the room. The room went silent, as everyone realized our chance had slipped away.

Checkmate.

Many engineers join the self-driving car world for the same reasons I did: it's exciting, it's interesting, it's a passionating, it's impacting, it's just... wow. Yet, nearly all the engineers who are on the "learning" group and have never joined a real self-driving car company yet have absolute zero vision on what it takes to certify a vehicle. We could talk cyber security, but even at the automotive level, the software level, and more...

So in this post, I will try to sensibilize you to the concept of safety, from an autonomous tech engineer point of view. This means — this post won't be for expert functional safety engineers, but for those who want an introduction.

Let's begin with the fundamentals:

What is Functional Safety?

Functional Safety is about making sure machines and systems stay safe, even if something goes wrong. For example, in self-driving cars, it means making sure the car can still drive safely if a part stops working. It can mean verifying that an algorithm works under all conditions, but also that it's never going to crash, and that if it does, the system has a backup.

To make it work, we use functional safety standards that determine what is safe to include in a self-driving car by evaluating the potential risks associated with each function and scenario. You can therefore understand the entire point of functional safety:

To reduce risk to an acceptable level.

Okay, but this shouldn't be your job, right? It's someone else's problem! So you may wonder...

Why should I bother learning about Functional Safety?

Let's say you decide to build an autonomous tech startup and run your algorithms. Some are open source, some are designed by you. You decide that these are good algorithms, the accuracy is near perfect, and you're a brutal C++ coder. There is no way you missed anything. Let's even pretend you really ARE a super-hero and really, the system is perfect...

You convinced me... but can you convince recruiters? Or your management? Or the suits giving your startup a self-driving permit? Hey — you can't test without the permit. No matter how good your system looks, you will need to convince the state to deliver you a permit. It can be the State of California, or the Ministry of Transport, or whoever delivers authorizations.

The problem? They are NOT experts in safety or self-driving cars. So they will ask you to go via independent organizations, who run functional safety certification programs. Organisms like TÜV Rheinland and TÜV SUD (Germany) are the ones 'certifying' you. They're verifying your safety functions, even the safety critical functions (emergency braking), and doing all kinds of silly tests before issuing you the certification.

Their job is to verify you are compliant with the industry norms.

But which norms are we talking about?

What are the different functional safety norms used?

When we say we want to "reduce risk to an acceptable level"... What is an acceptable level? Are you the one defining it? If an object detector works at 95%... is this okay? No? Yes? Who defines it? If your blinkers fail once every 300,000 miles... is this fine? Or is it every 3 millions miles?

You can't be the deciding entity, this is what norms and industry standards are for. For example, ISO 26262 is a norm. It's focusing on electronics (buttons, A/C, windows, sensors, computers, ...), and defines a complete process to develop & test your cars. It also tells you how to test scenarios, how to grade the risk of any event, and how to reduce that risk.

Let me share some norms we use in the industry:

• ✅ ISO-26262 is the norm that focuses on failures in electronic and software systems. It's going to deal with the question "What happens if the object detector crashes mid-drive? Is there any backup?"​​​​​​ Based on how your system is implemented, you will comply more or less with the norm.
• ✅ ISO-21448 verifies the Safety of the Intended Function (SOTIF). It ensures perception systems like LiDAR, cameras, and object detection perform safely in all conditions. "Is your object detector working on all pedestrians? Really? Even in the dark?"
• ✅ ISO-21434 is the norm focused on cyber-security of the system. It solves my USB-stick story. And it tells you everything you need to do to ensure your model is free from cyber attacks.
• ✅ A-SPICE is focused on how your project is coded, tested, and maintained. This means the requirements, the modular and maintainable code, the coding standards & reviews, the software testing, software versions and revisions, bug fixing, lifecycle of the product, etc...
• ✅ UNECE WP.29 Regulations is the compliance with EU autonomous driving laws. You need at least this one to be allowed to drive autonomously.
• and more... depending on what you want to certify.

While these are not mandatory, the more of these norms you check, the safer you'll look.

So comes a question:

How to know if your robot complies with Functional Safety norms?

There are TONS of ways to do this, and it's really a profession, but let me share with you 2 important functional safety concepts:

1. The V-Model
2. The Functional Safety Process to "certify" a function

The V-Model

The V-Model is a widely used framework in functional safety management and software development. You will find it when trying to comply with ISO26262, but also A-SPICE for example. It is structured like a "V," where the left side represents the concepts/requirements/design phase, the bottom part is the coding phase, and the right side corresponds to the validation/integration/testing phase.

You can see it as a continuous process, where you continuously verify that your system behaves as intended in the concept phase. If not, you rework it. It's evolving, it's alive, promoting a systematic approach to achieving functional safety in safety related systems.

In most companies that seriously want to comply with the ISO norms and get the functional safety accreditation, using the V-Model is the best starting point.

Next:

The Functional Safety Process to "certify" a function

As we said, we have ISO26262 focusing on electronics, SOTIF focusing on algorithms, and A-SPICE focusing on code/software. Each of these is using the V-Model. Then, to comply with these norms, you'll need a "process". This means defining clearly what each of these phases are.

Here is a 7-Step process:

The job of a functional safety engineer is to implement this. This is the "bridge" between systems and production I was telling you about earlier.

Let me briefly define each element: (credit to a client of Think Autonomous named Mayur Wagchoure for helping me write this one)

1. Define the System

First, we want to define the system we're testing. For example, lane detection. We want to define the purpose, the scope, the dependencies, and even the normal and edge cases.

2. HARA: Hazard Analysis and Risk Assessment

The second point is HARA, in which we want to do:

• HA — Hazard Analysis (what could go wrong?)
• RA  — Risk Assessment (how bad would that be if it went wrong?)

Hazard Analysis

If you want to comply with functional safety standards, the first thing you'll need to do is account for the different scenarios. I see them into 4 main sections: Car Status, Scenario, Environment, Driving Status.

Your car could be turned on, driving in a country road, with rainy conditions, and driving at low speed. Or you could drive at high speed, and accelerate. Or suddenly brake. Or drive in dry roads. Or wet roads. Putting categories into each of these is a way to avoid the summer/winter rookie mistake.

Risk Assessment

To "grade" each function, you then use the formula defined by ISO26262: Risk = Severity * Exposure * Controllability.

For example:

• I am testing the emergency braking function, and the risk that it doesn't activate (Severity = S3)
• I'm driving in urban environment, at 30-60 km/h, which happens all the time (Exposure = E4)
• Urban areas have many pedestrians, it's very hard to control (Controllability = C3)

Then what?

The ISO26262 provides what's called the ASIL (Automotive Safety Integrity Level) Table:

I am NOT going to describe how we do in this article, but the "RA" phase is about assigning, for every single function and every single scenario, what's called an ASIL level. These can be A (safe), B (safe), C (risky), or D (risky). We're trying to see, for each function, is it risky or safe?

For example:

If you're testing an emergency braking system, in a highway scenario, with wet road, snow, and fog... you can imagine it's an ASIL-D score. Now if you're on the same scenario, but testing the radio, it's probably A or B.

3. Set Safety Goals

From every potential hazard and risk we have, we want to turn this into a safety goal. Basically, turn the failure into an opportunity to design a better system. If I have just one LiDAR, and it's working bad under snow, could I have a better LiDAR and a camera instead?

Here, we will create a list of requirements for the new system. It's still the "concept" phase, where we identify the breaking points, and turn this into a better solution.This is the work where you try to think about reducing risk to an acceptable level.

4. Functional Safety Analysis

Then, we implement things like FMEA (Failure Mode and Effects Analysis) to assess potential failure causes, effects, and mitigation strategies. We can also run FTA (Fault Tree Analysis) to explore how faults propagate and lead to hazards. We want to identify all causes of errors.

5. Design Safety Mechanisms

Then, we're introducing mechanisms to detect, isolate, or prevent failures (e.g., redundancy, diagnostics, fail-safe systems). This can be watchdog timers, dual-channel systems, degraded operational modes, ...

For example, one of the Functional Safety Methods is to implement redundancy. If you have an ASIL-D component (unsafe); you could turn it into 2 ASIL-B ones (somewhat safe). This way, your overall ASIL score is better, and you become compliant.

In this example, we could imagine that the second LiDAR is different, or that the algorithms behind it are more "deterministic", don't use AI, and therefore are safer. The goal of functional safety is to try and reduce as many components to ASIL-A and ASIL-B as possible. >>> This is the acceptable level.

6. Validation and Verification

How do we test? This can be field tests, but also simulations, hardware-in-the-loop (HIL), and fault injection testing. You can also here test the Safety of Intended Functionality (SOtIF) — how performant is your algorithm? Is it really THAT good?

Finally:

7. Iterate, Validate, and Document

You want to iterate, improve, and document your safety analysis results. In the end, it's a very technical job, but that has a lot of paperwork, documentation, diagrams, schematics, grading, because these are the papers giving you authorizations.

We have now seen:

• What is functional safety?
• What are the different norms we should comply with?
• How do we comply with these norms (overview)

Let's see an example:

Example: Mobileye's Primary Guardian Fallback / "True Redundancy" System

Mobileye, Intel's self-driving car company, is has a very strong functional safety focus. Their algorithm has 3 distinct channels that are completely different:

The lane detection is the main channel used to fine lane lines. This can work for example with modular deep lane detection. It is verified with HD Map Extraction & Localization. If they agree, then we're good, but if they don't, they'll extract the lanes from a parallel end-to-end deep learning algorithm that will act as the "judge" or guardian.

Do you realize how many algorithms are running in parallel? They implemented these automatic protection functions in case of failure. They also implemented these safety requirements across the entire system, meaning the electronic systems, the software components, and so on...

When doing something like this, it's very important that each function is run using a separate method, possibly with a separate computer, separate sensors, etc... so that there cannot be a single point of failure (for example, if everything uses the same camera, and this one fails, it's not functionally safe).

Wait... Does everybody really do all of this?

No.

In fact, many startups don't have a functional safety team, or even have a safety system in place. In this case, they try to do it in the safety critical systems, while waiting for the certification process. Some are also in a more favorable state/country that gives permits more easily (to enhance innovation and startups work on the technology).

It's important to understand that complying with ISO norms is NOT mandatory. In the European Union, you need to comply with the UNECE WP.29 Regulations (traffic laws) but I don't think the ISO norms are mandatory.

In fact, Tesla doesn't comply with the norms, and they are approved to drive in the streets. They sell cars, and they even sell autonomous cars all across the world. But you'll note that some of their functions, like FSD (Full Self-Driving) are currently (early 2025) NOT authorized everywhere, like in Europe, because they don't comply with all the norms.

Okay, okay, I think we have ENOUGH! Let's do a summary...

Summary & Next Steps

• Functional safety makes sure robots and algorithms operate safely, even when something goes wrong, by reducing risks to an acceptable level.
• Every engineer working in the field should be introduced to safety. This defines how you code, but also whether a startup gets authorzations to drive or not.
• Key functional safety norms include ISO 26262 for electronics, ISO 21448 for algorithms, ISO 21434 for cybersecurity, and UNECE WP.29 for EU compliance.
• The V-Model is a structured approach in functional safety management, covering concept, coding, and validation phases to achieve compliance. It has a V shape doing Conception - Coding - Testing.
• Functional Safety is a 7-step process includes defining systems, hazard analysis, setting safety goals, and implementing safety mechanisms to ensure compliance.
• The ISO26262 norm defines risks as Exposure * Severity * Controllability. An ASIL table then defines for each function, which grade it has.
• When something is risky (ASIL-C, ASIL-D), we introduce redundancy, diagnostics, and fail-safe systems to detect, isolate, or prevent failures, enhancing the overall safety integrity level.
• We want to test through simulations, field tests, and fault injection to ensure safety functions perform under all conditions, meeting the required safety standards.

Alright, I think we are good!

A day in 2018, I was buying my first car, an Audi A1, when the sales representative pitched me about the "sportback" version; which was supposedly more powerful. "Why?" I asked. And it turns out, it had more horsepower than the classic version. Horsepower? The idea intrigued me, especially since it's been a century since people replaced horses with cars, and yet, we still use horsepower as the key metric to describe a car.

This metric, while seeming outdated, is still used as the gold standard in automotive... and it reminds me very much of the Faster-RCNN algorithm in object detection.

The Faster RCNN algorithm got introduced to the AI community in 2015, and even though it's been 10+ years now, you still see it listed as the benchmark in most new object detection papers. Somehow, Faster RCNN is still the reference researchers use when they create a new algorithm.

Take, for example, the paper CO-DETR, which is doing Object Detection with Hybrid Transformers, something super-advanced, released late 2023 (almost 10 years after Faster RCNN), and notice the papers it's being compared to: Faster RCNN is part of the list.

Have you noticed? And it's the case for almost every paper!

Why? Back when the algorithm got first released, and somewhere around 2018, I was looking for an object detection model to integrate in my autonomous shuttle, and it seemed that the entire market came to 3 conclusions:

• SSD (Single Shot Detector) is the fastest object detection network
• Faster R-CNN is the best model for accuracy, especially with small objects
• YOLO (You Only Look Once) is the best tradeoff between accuracy and speed

Yet, the original YOLOv3 got replaced several times, and the Faster R CNN model still continued to live. In this article, I'd like to describe the model to you, explain its key components, and help you understand whether you should spend time on it or not.

Before we dive into this algorithm, a quick aside:

When I first learned about object detection, it was through the Udacity Self-Driving Car Nanodegree where we learned a Machine Learning technique called 'HOG+SVM', which worked like this:

The image was sent to an algorithm that ran a sliding window, and for each window, extracted Histogram of Oriented Gradient features that it classified using a Support Vector Machine (SVM). It was old, fully "traditional", but somehow worked. I doesn't win an object detection oscar, but it did work. The idea was what we call a two-stage object detector:

1. We propose regions or bounding boxes (in this case, we defined window dimensions)
2. We classify each region

And here was the output:

The Faster-RCNN algorithm is built exactly on the same "Two Stage" principle, except that it replaces every single one of these techniques by Neural Networks.

Let's see how:

R-CNN: Selective Search

The first idea was to replace Feature Extraction, which was done using Histogram of Oriented Gradients, with a CNN (Convolutional Neural Network). Here it how it worked:

The algorithm had a few steps:

1. Propose 2,000+ Regions using Selective Search Algorithm
2. For each region, extract features with a CNN (Convolutional Neural Network)
3. For each region, classify the features using SVM (Support Vector Machine).

The idea was almost similar to my project, except that the region proposal network was done using Selective Search, an old Computer Vision algorithm, and the extraction was done using CNNs.

The algorithm had several problems: too many useless regions, too much extraction to do, and every region had to be resized/rewarped manually to match the CNN input layer.

This wasn't ideal...

SPP-Net: Adding a Spatial Pyramid Pooling (SPP) block

SPP-Net is an evolution of this paper using a clever technique called Spatial Pyramid Pooling. The idea was as follows:

1. Extract the Features using a CNN first
2. Propose 2,000+ feature map Regions using Selective Search.
3. Use Spatial Pyramid Pooling to avoid cropping/warping regions
4. Send each feature map to FC layers and classify using SVM

And the idea worked! Working on the features instead of the regions helped remove some noise, and introducing Spatial Pyramid Pooling helped look at the image from different aspect ratios using multiple 'Max Pooling' operations.

Let me briefly take you back to the Max Pooling idea: it takes a window (say 2x2) and computes the maximum to reduce the size of the input, so that a 500x500 images becomes 250x250.

Spatial Pyramid Pooling is doing a similar thing, but at multiple different scales, like (1x1), (2x2), (3x3), etc:

The idea: collect information from different scales.

So far, we replaced the Feature Extraction with a CNN, and we added an SPP. What now?

Fast R-CNN: Adding Neural Network Classification & ROI Pooling

Comes the Fast R-CNN detector! And the idea is almost the same, except that it replaces the Spatial Pyramid Pooling with ROI-Pooling; and the final SVM with a Multi-Layer Perception classifier:

Notice the key steps:

1. Extract the Features using a CNN
2. Propose 2,000+ Regions using Selective Search.
3. Use ROI Pooling to avoid cropping/warping regions
4. Send this to FC layers and classify using a neural network

There are two ideas of the Fast R-CNN architecture: ROI Pooling & FC Classification.

1. ROI Pooling: SPP in better

ROI Pooling is a special case of Spatial Pyramid Pooling, with an added idea of focusing on a specific region.

This is particularly useful in two-stage object detection algorithms that first propose regions using algorithms such as Selective Search or as in Faster RCNN, Region Proposal Networks. This last technique is also super fast compared to SPP that computes pooling several times at different scales (it also computes pooling for all regions directly).

One thing that you can notice about this technique is that by working on different scales, it allows the network to be more accurate, especially with different sizes of objects. Many objects detection models use anchor box techniques to find bounding boxes. The problem is, you must manually define the anchor boxes every time. In this process, we find the regions first (rather than some boxes), and then extract information about these regions.

We can see this on the Faster-RCNN paper (which uses this same technique) as well:

2. Fully Connected Classifier: SVM in better

The second addition is to replace the traditional Machine Learning SVM (Support Vector Machine) with a Softmax layer. There isn't much to comment here — we're using softmax on the k region proposals to do object classification for each bounding box. And this is an idea used in Fast R-CNN, but also in Faster-R-CNN.

Speaking of it, there is one last evolution to go from Fast R-CNN to Faster R-CNN:

Faster R-CNN: Replacing Selective Search with a Region Proposal Network (RPN)

When we started the algorithm in 2013, we had almost everything done by traditional techniques:

• We proposed regions using Selective Search (old school computer vision/segmentation)
• We did feature extraction with CNNs (this was Deep Learning)
• We classified the features using SVM (traditional machine learning classification)

And progressively, we replaced SVM with a Fully-Connected Layer, and we improved the CNN extraction with pyramids. What is left to replace with Deep Learning? The Selective Search algorithm! And yes, it was really an old and slow Computer Vision technique:

Faster R-CNN replaces selective search with a Deep Learning based Region proposal Network (RPN). This way, everything is trained end to end using a unified network. It's all a single network!

The full process goes like this:

1. Extract the Features using a CNN
2. Generate Region Proposals using a Region Proposal Network.
3. Use ROI Pooling to avoid cropping/warping regions
4. Send this to FC layers and classify using a neural network

And here we are!

Now what is this RPN doing? It serves as the "attention" of the network. It's designed to generate high quality region proposals and highlight where there might be objects. Taking an image of any size as input, it uses a fully convolutional network to output a set of rectangular bounding boxes, each with an objectness score. When you think about it, the name of the paper Faster-RCNN is "Towards Real-Tme Object detection with Region Proposal Networks".

How is a Region Proposal Network making it any real time? By removing the selective search algorithm and working on the feature maps directly, it kills the need for heavy computations, and does cost free region proposals. The RPN operates on the same convolutional feature maps produced by the backbone CNN (e.g., ResNet or VGG) that are already computed for object detection.

It then uses the concept of 'anchor boxes' to do the region proposal generation. If you're not familiar with the concept of anchor boxes, the idea is to define boxes of multiple aspects and sizes, and try to have objects fit these boxes. For example, a vertical small anchor box is a pedestrian seen by far — a vertical big anchor box is a pedestrian close to the camera.

I highly recommend my article "Finally Understand Anchor Boxes in Object Detection" to grasp the idea.

Using the FCN in Fast R-CNN and this new RPN, the algorithm can simultaneously predicts object bounds and class probabilities.

And we have it! The paper Faster R-CNN: Towards Real-Time Object Detection with Region Proposal Networks is really performant, because it uses all of these ideas. Being a 2-stage algorithm, it's powerful and can find small objects as well as big objects. It's a bit slow, but the enhancements made to the region proposal network are much better

Let's now see an example:

Is Faster-RCNN still used? Where would it fit best?

I don't think people today would use Faster R-CNN as the main choice for their algorithm. There are many powerful, better, and faster object detectors. Yet, if there was one place where I'd use it, it'd be on the task of Traffic Light Detection & Classification.

You see this all the time; traffic lights have a separate network designed just for them. These networks are usually focused on finding smaller objects, detecting the states, and using the long-range camera focused on lights.

Here, you will find Faster-RCNN possibly used. Probably more than anywhere else, because this algorithm is accurate and working well with small objects.

Should I try to recode it on my own?

If you have never implemented object detection, I would recommend to first go to the HOG+SVM traditional techniques. Then, you could try to run Faster R-CNN first, and then maybe implement some bricks. Today, YOLO is more dominant, and understanding this algorithm would make a lot of sense too.

What's to understand is that the real going behind object detection is not necessarily to find objects, but rather to extend it to things like LiDAR/Camera Fusion or object tracking. This is where the real value of object detectors is.

Summary & Next Steps

You've been through the article! Congratulations! Let's do a quick summary of everything we learned. First, you probably now understand this image:

Here is what it was about:

• Faster R-CNN remains a benchmark for object detection, even a decade after its introduction. The algorithm is known for its accuracy, especially with small objects, and is often compared to newer models like YOLO and SSD.
• Faster R-CNN evolved from traditional techniques, replacing each of them with neural networks; from region proposals, to feature extraction and classification.
• HOG feature extraction was replaced with CNNs in Fast-RCN, allowing to build feature maps.
• Spatial Pyramid Pooling, and later ROI Pooling and anchor boxes got added for better extraction, proposal, and understanding.
• Region Proposal Networks (RPN) replaced Selective Search, allowing the model to generate high-quality region proposals in real-time.
• A fully-connected layer replaced SVM for classification.
• Despite being slower than some modern detectors, Faster R-CNN excels in tasks requiring high accuracy, such as traffic light detection.
• The model's two-stage process involves proposing regions and classifying them, making it powerful for detecting objects of various sizes.

Next Steps

Here are a few articles I'd recommend you learn next to continue your journey:

• Finally Understand Anchor Boxes in Object Detection (2D and 3D)
• Computer Vision for Multi-Object Tracking: Live Example
• Instance Segmentation: How adding Masks improves Object Detection

And of course, the most important recommendation of all:

Have you seen the movie 'Limitless'? I really loved this movie when growing up. It's the story of Eddie Morra, a broke writer who suddenly gets access to the NZT drug, unlocking immense cognitive capacities. Using it, he's able to learn quicker, benefit enhanced memory, focus better, build charisma, and instantly has an amazing life.

I love this movie, and in particular the idea of being "limitless", which is similar to the movie Lucy where you start using your brain to 100% capacity, and thus fully exploit it. Tons of Computer Vision students today have the NZT pill in front of them, but refuse to take it. And by this, I mean they have great skills in image processing, and can detect objects and find 2D boxes, but are unable to fully exploit this skill to 100%.

In order to fully exploit your Computer Vision skills, you'd need object tracking, and this is exactly what we'll talk about in this article in 3 points:

• Object Detection vs Object Tracking: Why you shouldn't be an Object Detection Engineer, and why Object Tracking Engineers bring more benefits
• Object Tracking Elements: How to track objects from frame to frame
• Advanced Object Tracking: 3D Tracking, Deep Tracking, and more...

Object Detection vs Object Tracking: Why you shouldn't be an Object Detection Engineer, and why Object Tracking Engineers are better

Back when I started in Computer Vision, I often stumble across courses where the curriculum went like this:

1. CV Fundamentals: Image Processing, Filtering, Resizing, Colorization, OpenCV, computer vision algorithms, etc...
2. Neural Network & CNNs: Neural Nets, MLP Classification, Backpropagation, CNNs Image Classification
3. Advanced Computer Vision: 2D Object Detection, CV Projects (car counting, ...), ...

Somehow, no matter the course I picked — and even at university — object detection was presented as the utmost cutting-edge project you could work on. And somehow, almost 8 years later, it's still the case.

When I worked in the self-driving car field, I realized object detection was merely a handful tool, and NEVER the end goal. No, knowing that a detected object is in a bounding box centered at pixel (200,300) isn't directly useful, and this because we're in the pixel space, but also because we have no information about these objects.

This is something I explain in the presentation page of my obstacle tracking course, but also in general every time I talk about object detection: a 2D bounding box is NOT the end goal. However, if you have information about that bounding box, it could be useful... For example:

• If you know how long each specific object has been in the frame
• Or how fast they're going
• Or where they're heading (even what action they'll take)

Then it becomes useful, and can be the origin of many real-world AI applications in retail, self-driving cars, robots, traffic monitoring, sports analytics, visual object tracking, and more..

Computer Vision might be represented by the YOLO algorithm and object detection, but the reality is, skills like Object Tracking are far more useful to companies, and often directly follow Object Detection.

To put things very clearly:

• The output of an object detector is a list of 2D or 3D Bounding Box coordinates
• The output of an object tracking algorithm is the bounding box PLUS the ID of each, and yes, we can add the speed, class, action, age in the frame, whether it's occluded or not, and so on...

But there are many types of object tracking algorithms, and many different algorithms to consider, so let's immediately jump to the second point:

The Main Elements of Object Tracking: How to track objects from frame to frame

The initial goal of object tracking is, taking two frames, to associate their bounding boxes. To explain the idea very well, let me take the example of this scene:

Say this is the view from our self-driving car, and we want to do vehicle tracking. First, let's remove the background and focus only on the moving objects:

Ohhh this is so cool. What I just did is called Gaussian Decomposition and has been achieved solely to impress you with 3D Gaussian Splatting, but whatever, let's imagine we're the self-driving car stopped and looking at these 3 objects.

How would we assign the IDs? First, we would need to extract video frames from this scene, so let's put our video object tracking lenses and look at it this way:

Now, let's try to do the tracking for Frame 3 and Frame 4.

There are 3 main steps you you need to understand:

1. Object Detection: YOLO — because you only live once
2. Data Association: Bipartite Matching, the Hungarian Algorithm
3. Object Tracking: Kalman Filters

1) Object Detection

The first step is simple, we detect objects — this is where everybody rush and stops. Since we said we'd focus on frame 3 and 4 only, this would be the output of YOLO:

The problem here is, if every car is red, there is no tracking, we need IDs, and we need good tracking. Comes step 2.

2) Data Association: Bipartite Matching & The Hungarian Algorithm

The second step is to attribute IDs to every box, and this for the entire scene. In this article, I'm going to consider that we're tracking multiple objects, and thus doing multi object tracking (MOT). If we keep our same example, it should look like this:

And to be clear, it should NOT look like this:

So how do we do that? You could assign an ID to every single object in frame 3, but these objects have to correspond.

So here is how we are going to make a good matching algorithm:

1. We attribute an ID to each box for the first frame (t-1).
2. We match the box from (t-1) to (t)
3. We assign the ID to each box for the second frame (t)

So:

1. We attribute an ID to each box for the first frame (t-1).

Then:

2. We match the box from (t-1) to (t)

But how?

This is where we run the data association. We are going to, for each box (and this is important), try to find the closest match with the other box. For this, we'll consider matching criteria, such as:

• The euclidean distance of the center of the boxes
• The size & shape change of the box
• The class (don't match a car with a cyclist)
• The IOU (Intersection Over Union) of how these 2 boxes overlap
• The deep association metric each box has with the other
• And more... (in my obstacle tracking course, we implement object tracking using a cost function of tons of these costs and more...)

Let me show you an example for the first car, if we pick the euclidean distance as a metric:

As you can see, if we compute the euclidean distance between the red dot and the left car, we have a very small distance. On the other hand, we have a big distance with the right car. And this is how we can do the matching. You are then going to hold what we call a Bipartite Graph storing all the distances.

Finally:

3. We assign the ID to each box for the second frame frame (t)

The lowest distance is picked and every object is assigned.

And this is how we do it!

Now, I showed you a very simple version, and before I get loaded with "But what if...?" — there is one final idea to talk about:

3) The Kalman Filter

When combined with a Kalman Filter, this object tracking method is called SORT (Simple Online Realtime Tracking). What does a Kalman Filter do? Over time, you're going to have a real motion of objects, and thus, the Kalman Filter is going to predict the next position of the box — so that the distance gets smaller.

If I were to illustrate it, it would be:

The orange box is the predicted box, and the red box is the (t-1) box. When using a Kalman Filter, we are going to use the orange box for matching instead. So rather than matching the box at time (t-1) with the box at time (t), we match it with its forward prediction at time (t). This has a big impact, because if a car suddenly accelerates, or is about to leave the frame, you can predict it and thus improve your tracking results.

How a Kalman Filter works

To clarify a bit, we'll initialize our Kalman Filter with a motion model, say constant velocity, and predict the next position of the box. We may predict that the car is at the same position as it is now. Then, we'll get the "real" data from YOLO at t+1, and thus we know we were a bit "off" — so we calculate that motion, and refine our next prediction... again and again until we're perfectly able to predict the next position.

I have a complete article doing a Live Example of Multi Object Tracking with the actual KF values here.

Awesome, so this is our second point, and before taking questions, I'll just show you some advanced algorithms in our third point...

Questions about the simple tracking methods

A few questions you may have about this object tracking approach:

"Is the euclidean distance standard? What do people use in the field?"

No, the euclidean distance is actually one of the worst metric you can use. Why? Because imaging we have many objects? Or if an object comes in front of another? Suddenly, the shortest distance could be attributed to another object, and thus, we could fail the tracking. People like to use the IOU (Intersection Over Union) instead, or the IOU + the distance between the deep convolutional features (we look at the CNN feature map inside the bounding box). Often, a combined cost is the best solution.

What if 2 object have a minimal distance to the same object?

Haha! That is interesting. Imagine this case:

As you can see, in this example, object 1 is moving, and a third object is entering the scene. It's possible that we don't know exactly how to do the match; maybe the first object moved too much, and thus we have a situation like this:

So what if everybody has to be assigned to the middle object? This can happen extremely often, especially when the two frames don't have the same number of vehicles. This question, and ALL the others like "What if we have occlusions?" or "What if 2 objects are in frame 3 but 5 objects are in frame 4" or "What if an object is mis-detected?"... can be answered with one word: The Hungarian Algorithm.

This is the algorithm behind all matching, even the advanced ones with Deep Learning, and if you'd like to understand how it works, I have a complete article about it here.

You now have a good idea of how to do multiple object tracking, let's move to the advanced algorithms.

Advanced Multi Object Tracking Models

In this section, I will still focus on the Multiple Object Tracking problem. Not the single object tracking (image tracking), which is often a simpler Computer Vision problem answered with other techniques.

In the example I showed you, we were (1) detecting cars and (2) tracking them. We call this tracking-by-detection. In reality, we don't have to do this. Modern approaches today track object immediately: we call this joint detection and tracking. And thus, we have two families:

• Tracking by Detection algorithms
• Joint Detection and Tracking algorithms

Let's briefly take a look at each:

Tracking By Detection

You already saw a simple object tracking approach called SORT, which belongs to the first family; and when you add deep CNN metrics, you have Deep SORT! The idea is, rather than simply looking at the euclidean distances, we also add the Deep CNN feature distances as a metric. For example:

The matching is here done using the IOU/Distance/Shapes, etc... but also adding this new cost, which is going to look "inside" the bounding box.

What are other approaches from this family?

For example, ByteTrack is pretty well-known:

The goal of this algorithm is to match all the detection boxes, even the low confidence ones, which are assigned to unmatched detections (when it makes sense). ByteTrack's "two stage association" approach (of associating both high-confidence and low-confidence detections) makes it highly beneficial in situations where confidence is inconsistent or where external factors make detection challenging.

Now let's see the second family:

Joint Detection & Tracking

Here are a few state of the art algorithms to consider...

FAIR-MOT

FairMOT is a multi-object tracking (MOT) framework that is focused on the re-identification problem. The training loss is optimized for both object detection and re-identification (ReID). It has two branches: one for anchor-free detection and another for ReID feature embedding. This is really effective when you have occlusions, dense crowds, and offers real-time performance without compromising tracking accuracy.

Now is this better then the other one? Which could be better? As you saw, some algorithms are optimized for occlusions, others for using multiple sensors (or from different camera angles), others for sudden velocity changes... etc...

Now let's see some examples:

Example 1: Waymo's Stateful Track Transformers

In 2024, Waymo released an object tracking algorithm called STT: Stateful Track Transformer. Here is the diagram:

As you can see, this algorithm is based on LiDAR (as for almost everything at Waymo), and it begins with the Detection Encoder thad encode all of the 3D detections and extract temporal features for each track. The temporal features are fed into the Track-Detection Interaction module to aggregate information from surrounding detections and produce association scores and predicted states for each track. The Track State Decoder also takes the temporal features to produce track states in the previous frame t − 1.

I have a complete and full explanation of this advanced algorithm in my free app.

Next, let's see another example:

Example 2: 4D Perception with 3D Kalman Filters

In my article on 3D Object Tracking, I talk about object tracking systems that work with 3D Bounding Boxes. In this case, there are 2 different situations:

• We're still using monocular camera, and this time tracking 3D Bounding Boxes
• We're using LiDARs, RADARs, or stereo cameras, and thus track 3D Bounding Boxes, but have access to a different type of data (point clouds, RADAR maps, ...)

In my course MASTER 4D PERCEPTION, I do this with LiDARs, and here is the output:

Now that is stunning!

The key elements of this project are:

• We're using a LiDAR detector like Point-RCNN, PV-RCNN, or Cas-A to find 3D Bounding box coordinates
• We then have several possible association criteria such as the 3D IOU, but also the point cloud shapes, the 3D euclidean distance, and more...
• We're using a second order 3D Kalman Filter tracking XYZ in a constant velocity motion model.

You can learn more about this in my course MASTER 4D PERCEPTION (but warning, it's advanced and I highly recommend you check my introduction course on tracking before going to this one).

You've now been through the end of this article!

Congratulations! Let's do a quick summary of everything you learned:

Summary

First, there is this image that I built, showing the different stages of tracking:

As you can see, most engineers who only learn to use object detectors miss the benefits of association & tracking: ID, Age, Uncertainty, Motion, and even (if you predict) future positions, actions, trajectory, etc...

In a self-driving car, using tracking instead of simple detection is much more useful, and there are many real time object tracking applications that creating fantastic startups.

A few other summaries:

• The main elements of object tracking are (1) detection, (2) association, and (3) prediction via Kalman Filter.
• In association, we build a bipartite graph holding the objects of (t-1) and those of (t) and do a matching based on criteria such as euclidean distance, IOU, deep features, etc... While the majority of CV enthusiast use IOU, I highly recommend a combined cost function instead.
• The Hungarian algorithm is the algorithm responsible for tracking, and handles all the edge cases such as new objects, old objects, mis-detections, occlusions, etc...
• Kalman Filters are responsible for the prediction allowing for smoother tracking. In the case of 3D Tracking, we would use 3D Kalman Filters.
• There are two families of trackers: tracking-by-detection, and joint-detection-and-tracking. The second family promises to do it all in a single network, and can be seen as more advanced; yet, today, both are heavily used (and the first category probably more).
• Many companies from the field use tracking, such as Waymo with their Stateful Track Transformers — and you can learn it all on my free app.

Next Steps

If you enjoyed this article on tracking, there are chances you might enjoy these other ones:

• Computer Vision for Multi-Object Tracking: Live Example
• An Introduction to 3D Object Tracking (Advanced)
• Exactly how the Hungarian Algorithm works

You can also see relevant tracking courses here.

"I made it". These were my first thoughts back in 2017, after I graduated from my diploma in Internet of Things and got my first internship. After years learning about electrical signals, bluetooth, connectivity, networks, prototyping, and time spent building projects using multiple types of sensors... I was going to be an IoT Engineer — and make millions!

Or so I thought... Because my first project ended up being an epic failure. The market wasn't really interested in smart alarms like we all thought it would — there was no interest in smart fridges or other smart objects either... IoT was flopping — and thus, I got sent to another consulting project on AI!

And AI gave me this pair of glasses and told me "See? You learned about all these sensors, but what really matters is the data it collects". That was really an eye-opener for me; to the point where I spent most of my energy learning about AI & Robots... and writing on it.

But sensors matter too — and without them — there wouldn't be any self-driving car in development. This is why I'd like this article to focus on sensors. Not on the AI processing, but really the sensors themselves — explain to you the main types, and how much you should know about them.

This article will split sensors in 2 categories:

1. Exteroceptive sensors — the external sensors
2. Proprioceptive sensors — the internal sensors

Yes, the wording is scary — but you'll see they don't bite. Let's begin.

Exteroceptive Sensors: Looking at the outside world

In robotics or self-driving cars, you need to see the world so you can navigate in it. Every sensor that is external will be part of the exteroceptive sensor category. For example cameras or stereo cameras, LiDARs (light detection and ranging), RADARs (radio detection and ranging), GPS, ultrasonic sensors & proximity sensors, thermal cameras, infrared sensors, and so on...

There are many ways we could categorize this first family, but I like to do it by task, for example:

• Perception Sensors
• Localization Sensors
• Environmental Condition Sensors

Perception Sensors

I assume in this article you have a good idea of what are the Perception sensors. Yet you may not realize how much you need to know about each. Let's see for a few per category, what you should know:

Cameras

So you know what a camera is, right? Well, do you know how to set parameters like ISO? Shutter? Gain? Whether to use grayscale or RGB? Yes? So, do you know how to find the intrinsic and extrinsic parameters of the camera? Or how to use Charuco calibration? What about Stereo Calibration? Do you know how to calibrate using multiple checkerboards?

Knowing cameras isn't just knowing how to load images, but it's mostly knowing very well the camera parameters. When you truly understand cameras, you can do really wonderful applications, like this 3D Reconstruction we do in my course MASTER STEREO VISION:

As you can see, we build a complete 3D environment from just cameras! Cameras may give flat 2D images, but when used in stereo mode, you can leverage their 3D properties.

What's interesting:

LiDARs

Cameras are good with distances, but not as good as LiDARs. LiDARs/lasers are light sensors that can send a light wave and measure the exact distance of an object. It means they work in the dark, and they can construct a very accurate point cloud. The thing is, there are many types of LiDARs (like 2D LiDARs, 3D, or even 4D LiDARs — but also Time of Flight vs others...), and the output may be intuitive, but the inner technology is quite complex.

Below is an example of LiDAR processing we do in my course POINT CLOUDS CONQUEROR:

LiDARs are excellent sensors, but extremely costly (this is why startups like Tesla don't use them), easily affected by weather like fog, rain, snow, or even dust, and they can't measure velocities (when you need to measure the velocity of an object, you have to compute the distance between two frames). This is why most automotive companies use... RADARs!

3D and 4D RADARs (Radio Detection And Ranging)

RADARs are very mature (100 years old or so), used in TONS of industries, but they're super un-intuitive. The first time I saw a RADAR output, it just took 30 minutes to get what was on the screen, and I wasn't even sure. Most Perception Engineers in the AI/Self-Driving Car space avoid them like a disease because of that.

Because they use radio waves, they aren't affected by weather conditions, and because they use the Doppler Effect, they can measure the relative velocity of objects. In the self-driving car world, there are almost 0 courses on RADARs, but I did create a very interesting piece of content for my membership, in which we learn to visualize RADARs, create point clouds, and intuitively understand the output. Here is a look at it:

This output has been done using an Imaging RADAR (4D RADAR) from a startup named bitsensing. I highly recommend you go check out their product, especially because it's 4D technology. Unlike most RADARs that find 2D + Speed, Imaging RADARs find 3D + Speed (they see the Z dimension). Similarly FMCW LiDARs steal the Doppler effect from RADARs to measure speed directly, and become 4D LiDARs as well.

Other Sensors (Infrared, Ultrasonic, Thermal ...)

Let’s continue looking at the types of sensors in the Perception world. When you park your car and hear “BIP BIP”, you work with an ultrasonic sensor. These are great sensors for short range, static objects. On the other hand, RADARs are longer range, and work better with moving objects.

Infrared or Thermal can be a great thing to use with cameras. For example, Luxonis recently announced it was now selling thermal cameras and stereo cameras with Active Stereo. This means using IR sensors with dot projectors to help the camera find depth; which can be very useful for example when an object is in front of a wall (and thus, no detail is visible).

Now, let’s move on to category 2:

Localization Sensors

Even more misunderstood, GPS rely on satellites in space to triangulate your position. They're extremely used in self-driving cars, especially when relying on classic localization and mapping techniques. Other than GPS, RTK (Real-Time Kinematic) GPS are now the standard, because while GPS are accurate at ~1 meter, RTK GPS give centimeter level accuracy?

How?

RTK GPS communicate with a fixed antenna (with a known position) which can measure the error in localization. If the antenna is positioned at position (0,0,0), uses a GPS that says it's at (0.5, 0.5, 0.5), you know you have a 0.5 meter error; and you can send this info back to the cars using RTK GPS.

In my own personal experience, I worked on autonomous shuttles that drove through the Polytechnique Campus in France (the country's most prestigious engineering school), and I remember TONS of problems with GPS, such as:

• Clouds and weather affecting the signal strength
• Tunnels turning our GPS receivers off
• Trees confusing GPS positions
• Or students weird experimentations in some dorm rooms confusing our GPS signals

For some of these cases, we relied on Ultrawide band technology, which consisted in setting up a network of fixed reference nodes throughout the environment. These nodes communicated with mobile tags on the robots, providing precise distance measurements through time-of-flight calculations.

Finally:

Environmental & Contact Sensors

In late 2024, Waymo announced their 6th gen vehicle was using an array of audio sensors to recognize honks, ambient noise, or sirens. When you think about it, it makes a lot of sense for a car to use sound. We rely on sound a lot when driving. But even bigger is outside of the car world. Using this logic, we could think of humidity sensors, gas sensors, radioactivity sensors, ...

Now something related:

If we change the focus from "cars" to things like wheeled robots, humanoid robots, or specialized robots, like surgery robots. They often make something cars don't: physical contact. And thus part of their environment is the other objects they make contact with.

For example, capacitive sensors are the types of sensors that detect tactile feedback, proximity sensing, or materials. You can see them as a "skin" for robots. Some can also sense humidity or a surface, or find moisture, and more...

An example on a robot arm equipped with a torque sensor and a grip sensor:

Alright! We have covered the first category of sensors. If we take a look back, we have something like this:

There are a few ones we didn't cover, like flow sensors in water, or light sensors, photodiodes, etc... but they aren't that used with robots.

What now? Let's look into the second category:

Proprioceptive Sensors

I was able to show off with my cool GIFs in the first part, wasn't I? Well, who's laughing now? Because I have to write about proprioceptive sensors, and truth is, I never worked that much with these types of sensors before.

Why?

Because a proprioceptive sensor is an internal sensor. It's something that measures the INSIDE of your robot or vehicle. For example, an odometer measures the rotations of your wheel to estimate how much you moved. An IMU measures your position, orientation, and measure how YOU are moving. While exteroceptive sensors focus on the others, proprioceptive sensors focus on you (I mean, your robot).

I would list down 3:

• Position sensors
• Motion sensors
• Automotive sensors

Position sensors

Imaging you wake up in the middle of the night, heading for the toilets (oh this is going to be lame). You can't see, but know you should walk 10 steps to get to the throne room. So you walk one step, two steps, three... until you reach 10 steps, and... no toilets? 🚽 You agitate your arm, but nothing seems in the way. Where is the door? So you walk one more step, then two, then thr— BAM! Found the door. What happened here? You probably estimated your number of steps a bit inaccurately. Have you been drinking last night?

A wheel encoder is a bit similar, it measures how many rotations your wheels do, and thus can tell you how many meters your car or robot has travelled. It's extremely useful in localization, SLAM, or similar tasks.

If you look at an Apple Maps vehicle, you will see the wheel encoders plugged at the bottom:

Motion Sensors

There's position, but there's also motion. I told you I wasn't super at ease with proprioceptive sensors, but I did work on odometry, Visual Inertial Odometry, LiDAR-Inertial SLAM, Localization, and all of these words. What do they have in common? They use sensors that calculate their motion. See in my course SLAM UNLEASHED how I present the IMU outputs:

An IMU is an Inertial Measurement Unit that outputs orientation (in quaternion coordinates), an angular velocity, and linear acceleration. It means it's also part of the accelerometer sensors, that tell you how much you moved, how much you are rotated to the left, etc... They also tell the temperature and even measure the magnetic field to account for the Earth's rotation, and thus know exactly how you are moving.

Yes, if you used an IMU to record data in 2011, it's likely different than if you're doing it today; because the earth's magnetic field moved, and thus, you don't travel the same distance. We're getting really precise here, and I do cover this in my robotic architect course already; but I wanted to point it out because it's an important part.

The other types of motion sensors can be accelerometers like you have in your phones to calculate that you make your 10,000 daily steps, or even gyroscopes that measure your velocity.

Finally:

Regular Automotive Sensors

I really didn’t get inspired for this one, but when you think about it; a self-driving car must use all the regular car sensors as well.

For instance, a pressure sensor in the tires monitors the air pressure and converts it into an electrical signal, ensuring optimal performance and safety while driving. I remember one day, I was driving on the highway, and I noticed that I had to apply some steering to the left to keep my car straight. Somehow, it drifted to the side… After a quick check, I realized I had a flat tire. Self-driving cars must do the same.

On another idea, temperature sensors keep tabs on critical components like the engine, brakes, and battery, preventing overheating or failure. Oil level sensors and coolant sensors ensure that the engine runs smoothly, while fuel level sensors provide crucial data for range estimation. They’re kinda chemical sensors. Even the wheel speed sensors, which are integral to anti-lock braking systems (ABS) and traction control, contribute to the decision-making processes of a self-driving car by providing real-time feedback on vehicle dynamics.

Okay, so before seeing some examples, let’s do a brief recap of that part 2:

Example: How to SLAM a Nuclear Facility

A few months ago, I was chatting with a client from my 4D Perception course, who has his own 4D Perception startup — and he told me about one of his past experiences working with SLAM for a client from the nuclear field. "Wait... Nuclear ☢️?" I asked like a Looney Tunes. "Yes! Many SLAM applications are incredible when you have radioactive environments!" he answered before sharing a few details about it.

I got so intrigued by the idea that I started searching online, and of course... I found something! RNENuclear robots has written a paper and made this video about it (if there is no image — the video still works):

When you think about it, there are tons of places we human can't go, and where robots equipped with the right types of sensors can make wonders. In this example, the robot works with gamma dosimeters, these are radioactivity measurement sensors. A common one is the ThermoFisher RadEye unit G10— which can cost several thousands.

Example #2: Sensor Fusion of Multiple types of Sensors

I talked about SLAM here, but most robots don't localize using SLAM; but using localization algorithms. This is often a fusion of GPS and sensors like IMU & Odometers. It's all a fusion of sensors! Often done using an Extended Kalman Filter. In my course ROBOTIC ARCHITECT, I have a complete lesson showing it, but for this example, let me just show you the gist:

The Extended Kalman Filter takes:

• GPS
• IMU
• Odometer

And outputs the localization in the Google Map.

Alright, we covered a lot. To finish this article, let me answer a critical question:

How much should you know about these sensors to become a robotics/self-driving car engineer?

At the bare minimum, you should have a good idea of what these sensors are, how much they're involved, and what they do (what we're seeing in this article). There are other ideas you could expand, such as active sensors — which emit their own waves (LiDARs, RADARs, ...) and passive sensors, which are not really sending anything to the environment.

For beginner positions, it's probably a good idea to learn ONE of these sensors well. For example, the camera — or the LiDAR — or the GPS. When you specialize into one of these, you are also learning the algorithms, and starting to tell the difference between each type, and thus you start building expertise.

For intermediate positions, I recommend to understand 2 or more sensors. If you understand the LiDAR AND the Camera, you can start doing some fusion of these for object detection projects. If you understand the GPS AND the IMU, same idea — you dive into Kalman Filters, and thus have some good value to add.

For advanced positions, the more the better. Experts have a good idea of the differences between sensor types. We discussed active and passive sensors, but there's also digital sensors vs analog sensors (one outputs a discrete or digital data/binary for computers, and the other looks more like "signal"), or you could also dive into one sensor and master it really well.

Summary

• Sensors are categorized into two main types: exteroceptive sensors, which perceive the external environment, and proprioceptive sensors, which measure internal dynamics.
• Exteroceptive sensors are split into 3 categories: Perception, Localization, and Contact/Environmental sensors.
• Perception sensors are cameras, LiDARs, RADARs, thermal cameras, ultrasonics, infrareds, ... They work really well alone, but also fused together to provide an accurate understanding of the objects around.
• Localization sensors are sensors like GPS, RTK-GPS, or even workaround solutions like Ultrawide band.
• Environmental and contact sensors, such as audio sensors, humidity sensors, and gas sensors, help monitoring the surroundings of autonomous vehicles and robots. In robotics, we also have all the contact sensors — that are used for that idea of physical contact.
• Proprioceptive sensors, such as position sensors and motion sensors, monitor how the car moves over time. They can be used together with external sensors like GPS.
• We also use all the automotive sensors, like tire pressure, oil, etc...

Understanding and mastering different types of sensors is important, please refer to the part above to get the gist. There are a few types of sensors we didn't cover; because I wanted this article to focus on robotics & self-driving cars, but these are the main ones.

Next Steps

If you enjoyed this article on sensors — you might love this one on the different types of LiDARs. You may also get interested into this article on Sensor Fusion.