Computer Vision Applications in Self-Driving Cars: The 6-Level Ladder
I remember when I was 14, my father took me to a shop to buy my first book on coding: 'HTML 4: Master the Source Code'. I had been exploring with the idea of learning to code for a few weeks, and felt like I had to take a leap of faith. The book was 500 page long, and after an afternoon, I knew how to build a basic web page with a text on it. Then I learned CSS, and it all became extremely fun.
"DAD! I can build websites for people and make money!" I yelled
He knew nothing about coding, but he told me this: "Just realize all of that is just an introduction... an approach"
"WHAT??? That 500 page book is just an intro???"
It didn't discourage me; but I got that what I considered pretty advanced after a few weeks was actually, to a professional, just the bare minimum.
This same scene is currently happening to tens of thousands of Computer Vision Engineers. The industry has massively changed in the past 2 years. 2 or 3 years ago, we weren't talking about AI and VLMs, and now, it's everywhere.
If you're a computer vision engineer, or aspiring, I want to introduce you to the new Computer Vision world you're getting into.
In this article, we will cover:
- Why YOLO gets you stuck on level 2 computer Vision
- The Computer Vision Ladder: 6 computer vision applications in self-driving cars
- The Roadmap to Computer Vision
Obviously, I intend to make this relevant to today. So YES, we will discuss VLMs, World Models, reasoning, and more... and YES, we will also discuss things as basic as image classification.
Ready?
Okay, so what is...
Why YOLO gets you stuck on "Level 2" Computer Vision
Back in 2018, I felt like I had managed one of the greatest accomplishment of my career: I became a self-driving car engineer. I would be a specialist in Computer Vision. Just a year ago, I was running in circles trying to manage a chatbot project; and now, I was working on the most cutting-edge technologies ever created.
To many people, it was just a job. To me, it was THE job. To get there, I invested HOURS of my life in Computer Vision... From gradients, thresholding, and colour spaces... To image classification, machine learning, deep neural networks, and object detection. I felt proud! All of that work had finally paid off. And the best? I was going to use it in my day job!
Starting with object detection!
I wanted to use it so much, to see the bounding boxes, to estimate the accuracy. I knew I could bring my expertise there. But one day, a project manager came to me. He sat down on the chair next to my desk, and looked at me with a proud smile:
"Look what I've done! 😃"
I glanced at his computer, and saw THIS:
- You've... done that?
- Yes! I just setup a YOLO algorithm this morning and it worked!
- But... You don't know how to code this algorithm right? Have you been through online courses and learned it?
- No need to! It's really not that hard! Anybody who knows how to clone a repo and follow the instructions can do it.
And here I felt it: the threat! I spent months learning how to do it. First with Machine Learning & HOG features. Then I learned that Deep Learning outperforms traditional methods in object detection... that neural networks could output bounding boxes directly at 50 frames per second or more... and this guy came with a git clone command and made it work?
I felt threatened, maybe even a bit defeated... If he could do that, what was my purpose there? YOLO was my superweapon.
If there was no need for my object detection services, was I useful at all?
While meditating about it, I started to think of all the things he could probably NOT do. Add tracking. Maybe do Sensor Fusion. Stereo Vision. Make it work with the camera calibration parameters, to enable 3D... all in real-time. An autonomous vehicle isn't just built with YOLO, far from this.
This is when I understood: I had to EVOLVE; to raise the bar. I could NOT stay the Object Detection guy. To do Computer Vision & Self-Driving Cars, I had to understand BOTH industries.
If you're a beginner in autonomous vehicles, you may have already experienced a similar moment... You may have spent a LOT of time learning CNNs, OpenCV, and other technologies only to see them replicated with one command line by a random stranger.
You gotta face reality: ANYONE can run an object detector today. It was almost the case back in 2018. It certainly is today. You CANNOT Stay on these Level 2 applications.
So what is Level 2?
The way I learned Computer Vision was in 3 layers:
- Layer 1: Image Classification - you classify an image
- Layer 2: Object Detection - you classify portions of an image
- Layer 3: Image Segmentation - you classify every pixel of an image
If you see things that way, I'm sorry to announce that you are STUCK in Level 2. Yes, the most basic thing in Computer Vision is to run an image classifier. We can call this Level 1. But all these other applications of object detection, image segmentation, lane detection, instance segmentation, traffic light classification, and even Object Tracking are ONLY Level 2/6.
This is just how it is. We evolved, and what was once considered advanced is now considered a starting point.
So if this is Level 2, what is Level 3?
The Ladder of Computer Vision: 6 Applications of Computer Vision
Meet the Ladder of Computer Vision:
To give credit where it's due, this comes from Intel CVPR 2026 talk; which I saw live back in June. I find it "incomplete", I will explain after introducing you to it...
- As you can see, you Level 1 is Image Classification.
- Our Level 2 is Object Detection & Segmentation
Level 3: Visual Question Answering
If you look at self-driving cars today, you will see companies like Nvidia, Tesla, Wayve, and others trying to add "reasoning" inside their algorithms. After all, we can't possibly train computer vision models on ALL objects. We need a VLM (Vision Language Model) to tell us what's in the picture. Makes sense? Usually, these are Foundation Models, trained via self-supervised learning on the entire internet; and if you have a cyclist carrying a person on their shoulder, the VLM will "get" what happens.
- Applications: Video Understanding, Video Classification, Edge Case Detection, ...
- Skills Needed: Video Transformers, Foundation Models, Vision Language Models
Level 4: Scene & Depth Understanding
Nothing is MORE important to a computer vision engineer, especially the ones working on the robotics/physical side than to understand 3D. I have an entire webinar built JUST on 3D. Why? Look at ALL the applications we saw so far: Level 1, Level 2, even Level 3 work at the PIXEL level.
At the pixel level, you cannot do much. The information that a pedestrian is on the image at pixel (400, 500) brings us NOTHING. We need its true distance, its dimension, orientation, and others... This is why the next step is 3D.
- Applications: Depth Estimation, 3D Reconstruction, Visual SLAM, Visual Fusion, Bird Eye View Fusion, 3D Occupancy, 3D Segmentation, ...
- Skills Needed: Stereo Vision, Cameras, Calibration, Structure From Motion, Transformers, Spatial Transformer Networks, Geometric Foundation Models, Homographies, Sensor Fusion, ...
See how technical it suddenly got?
Hey, you really should signup for the next session of my 3D Computer Vision Webinar. I will there show you the exact step-by-step roadmap to achieve Level 4.
Level 5: Video & Temporal Reasoning
There are of course multiple "levels" inside each of these. The image above shows something called "Optical Flow", which I teach in my Video Perception course. In Optical Flow, you are predicting in which direction a pixel is moving. This is highly used in applications like Visual SLAM and Odometry. However, it remains the "low-tier" of what can be done today.
We can now also classify videos, and do applications such as Shoplifting Detection. We can predict what will happen inside a video. We can "generate" the next frames of a video, given an action. And even there, we can do it as a video generation tool, or as a "world model". Here are some ideas here:
- Applications: Video Generation, SLAM, Object Prediction, Intent Prediction, Video QA, ...
- Skills Needed: Video Transformers, Vision Language Models, Object Tracking, Feature Detection & Tracking, Optical Flow, Diffusion & Generative Modelling...
Heh, does it feel harder than Level 4 on Depth?
Level 6: Driving Decisions & Spatial Reasoning
The final level is complex to understand. It EATS a bit of Planning. Once you accurately know what happens in a picture, you should then define what you should do.
THIS IS NOT EASY. Let me do a little demo for you. Here is an experiment I ran with students of my membership, where I ask them to prompt the same AI (QWEN) by 3 different ways.
If we ask QWEN to describe the scene, it does identify a bear, something a Level 2 would NOT be able to do. If you then ask it whether it's safe to drive, it tells you YES; but when you ask for a risk level, it says RISKY. Should you drive or not?
I don't even know. I'd likely freeze while my wife takes pictures. The point is, you got a new world here at Level 6.
- Applications: Robotics, Robotic Arms, Humanoid Robotics, Self-Driving Cars, Drone Delivery, ...
- Skills Needed: Imitation Learning, Conditioning, Reinforcement Learning, ...
Alright, so we covered the ladder!
I would now like to move to the next part...
The Roadmap to Computer Vision in Self-Driving Cars
Being an engineer in 2018 was not easy. At the time, MOST people were using traditional approaches, and I made the bet that AI and CNNs were THE thing to learn, and that I had to focus on this. Today, it's even less.
Here is what I would recommend you do:
- Build SOLID foundations of Computer Vision: OpenCV, CNNs, Transformers, and everything discussed in Level 1 and 2. DO NOT skip straight to level 3 and 4 if you don't know these, you are just not there yet. (plus, you can get jobs doing Level 2 computer vision too)
- Master Cameras: You never know enough foundations; especially cameras. In my 3D Computer Vision Webinar, I reveal how most computer vision engineers do NOT know how to correctly calibrate a camera. Many wouldn't even know WHICH camera to buy. Some would have no idea what camera to use to detect traffic lights. What lens. Optic. And what processing to do there. And frankly, the best thing to do here is... buy a camera!
- Specialize within the ladder: Rather than looking to ascend is as quickly as possible, spent time SPECIALIZING in one of these applications, whether it's VLMs, 3D, or video, and make sure you understand not just how to do it ONCE, but also do it on multiple datasets, with simulators, using different algorithms, with and without a neural network, and so on... Master the craft.
- Understand what New-Gen means: Look at any 2026 model, do you see a single image being sent to a deep learning algorithm. Or do you also see text? Do you see videos? Do you see LiDAR point clouds? RADAR signals? Maps? Ego-Motion history? THIS IS NEW GEN. Today, a Computer Vision Engineer is NOT an image processing engineer.
- Build Visual Proof: Portfolio Projects, of course. I have been repeating that for a decade. DO PORTFOLIO Projects. Write articles around them, to solidify your knowledge. Make the projects visually GOOD. Use AI to build great visualization tools. Stop sharing course certificates and start sharing visual proof.
Why the Ladder is incomplete: The case for Sensor Fusion
Here is a question for you: How many cameras does a Tesla use? 8 cameras? Don't they? Otherwise, how can they get 360° coverage? Now what about Waymo use? I think in my Tesla vs Waymo article, I counted 29 cameras, but also 5 RADARs and 6 LiDARs! Isn't it INSANE that the majority of content online pitches self-driving cars as monocular driving.
Expand the range; Nvidia, mobileye, Zoox ALL use way more than a single camera.
The majority of self-driving car systems today are built with LiDARs and RADARs. LiDARs are very good at distance estimation and 3D map creation. They are difficult to scale, because LiDAR driving often goes with pre-mapping environments. RADARs directly measure object velocity. Ignoring these is NOT doing yourself a favor.
Alright friend, I hope this helps.
We'll do a part 2 where we'll tackle the core blocks in more details.
In the meantime, let's do a summary:
Summary & Next Steps
👉🏼 Let's recap:
- Anyone can run an object detector today, which means object detection on its own is no longer a career, it's a starting point.
- Computer Vision in self-driving cars has 6 levels, and most engineers are stuck on Level 2 without realizing there are four more above it.
- Level 3 is Visual Question Answering, where a VLM tells you what's in the scene instead of you training a model for every object that could possibly show up.
- Level 4 is Scene and Depth Understanding, and it's the one that matters most, because everything below it works at the pixel level, and a pixel tells you nothing about distance.
- Level 5 is Video and Temporal Reasoning, from Optical Flow at the low end up to world models that generate the next frames given an action.
- Level 6 is Driving Decisions, and my QWEN experiment showed the same model calling a scene both safe and risky depending on how I asked the question.
- The roadmap is foundations first, then cameras, then specialization, and you don't skip to Level 4, because Level 2 still pays the bills while you climb.
- A 2026 model doesn't take a single image, it takes video, text, point clouds, RADAR signals, maps and ego-motion history. A Computer Vision Engineer is no longer an image processing engineer.
- The Ladder ignores sensor fusion, and real vehicles run 8 to 29 cameras alongside LiDARs and RADARs, so building a career on monocular vision is building on a myth.
If you want to keep going, start with 9 Types of Sensor Fusion Algorithms for the part the Ladder leaves out, then 3D Computer Vision to begin your climb to Level 4.
