Sensor Fusion Algorithms: The 5 Design Decisions Behind Every Fusion
In 2003, LEGO was losing money so fast that the company was on the brink of bankruptcy. For years, its designers had invented a brand-new piece for almost every new box, until the factories had to produce around 12,900 different elements, many of them used in a single set. Every new mold cost money, the warehouses overflowed, and the most famous toy company in the world was drowning in its own pieces.
In 2004, the new CEO, Jørgen Vig Knudstorp, made the move everyone dreaded: he cut the number of elements almost in half, down to about 7,000. And you can imagine the fear inside a toy company making that cut... with fewer pieces, wouldn't kids run out of things to build, and wouldn't LEGO lose the very creativity it was selling?
That same year, a mathematician from the University of Copenhagen, Søren Eilers, gave the answer. Curious about a number he had seen at LEGOLAND, he wrote a program to count every way you can combine six identical 2x4 bricks, let it run for a week, and found 915,103,765 combinations. LEGO published the result in its own company newsletter, and a decade later, the brand that almost died was named the most powerful in the world.
Six bricks were enough to build almost a billion different things.
Now, if I asked you to name the best sensor fusion algorithm, what would you say? Most engineers name the one Waymo uses, the one from the latest paper, or the Kalman Filter they learned first.
In reality, every sensor fusion algorithm is a set of answers to 5 questions: which sensors you fuse, why, when they meet, where, and how.
Change one answer and you get a different algorithm, which is why Tesla, Waymo, Mobileye and Autoware all fuse their sensors in very different ways. In this article, I'll ask you these 5 questions one by one, show you how real companies answered them, test each option on real nuScenes data, and tell you what each choice costs.
What is Sensor Fusion, and How Does it Work?
Sensor fusion is the process of combining data from multiple sensors to get a more accurate, more complete and more reliable view of the world than any single sensor could give. It's mandatory in robotics because it brings reliability, redundancy, and ultimately safety, and you'll also hear it called sensor data fusion, data fusion or multi-sensor fusion.
So how does sensor fusion work in practice? Each sensor measures the world in its own format, the data is brought into a common space, and a fusion algorithm combines it into one result. There are hundreds of these sensor fusion algorithms, and every one of them is built from a few design decisions, so let's go through them.
Decision 1: WHICH Sensors Do You Fuse?
If you had to bet your career on one setup, which one would you pick: cameras only, or cameras with LiDARs and RADARs?
Tesla made its bet in 2021, when it removed the RADAR from its cars and went vision-only. Waymo made the opposite bet and puts cameras, LiDARs and RADARs on every robotaxi, and I compared these two visions in Waymo vs Tesla. Mobileye went even further, with separate systems that double-check each other, as I explained in my breakdown of Mobileye's End-To-End talk.
And even Tesla does sensor fusion, since its cars merge 8 cameras into one view of the world, which I covered in my article on Occupancy Networks.
Most people assume every robot needs all three sensors, and that's simply not true. The camera tells you what an object is, the LiDAR tells you where it is, and the RADAR tells you how fast it moves, even in rain and fog. Which ones you need, and the sensor fusion architecture that goes with them, depends on your robot.
Would you put a LiDAR on a small robot that delivers food on sidewalks? It drives slowly, in places full of people and signs, so cameras might be enough, which is what I saw when I hunted an autonomous delivery robot. Now take a truck that needs to see a stopped car far ahead, at night, on a highway: would you trust cameras alone? Aurora didn't, and its trucks fuse LiDARs, cameras and RADARs, as I explained in how Aurora uses deep learning for sensor fusion. And for a military robot driving through dust and smoke, where cameras and LiDARs go blind, wouldn't the RADAR become your most important sensor?
Every one of these answers leads to a different fusion, and I go deeper into each sensor in my article on the types of sensors.
Decision 2: WHY Are You Fusing Them? (Competitive, Complementary or Coordinated)
Now imagine you've picked your sensors, and they have to work as a team: who leads, and who supports?
In good weather, the camera leads because it understands the scene best, and the LiDAR and RADAR confirm what it sees. At night, the LiDAR takes the lead, and in heavy fog, the RADAR does. The roles change, but every sensor in the team has one of 3 jobs.
Competitive Fusion: Double-Checking
Two sensors watch the same thing, so one can confirm the other, or replace it if it fails. Here, a LiDAR and 3 RADARs on a nuScenes car all see the same vehicles.
You get fewer false detections, and the car doesn't go blind when one sensor breaks. But you pay twice for the same information, and you need a rule for when the sensors disagree, which is often the job of a Kalman Filter.
Complementary Fusion: Seeing More
If your RADAR only looks ahead, you'll miss the car in your blind spot, so you add a second one on the side and fuse them. A nuScenes car has 5 RADARs placed like this, all around the vehicle.
The catch is where their views overlap, because the same car shows up twice, and you have to merge it into one object. It's also where calibration errors show up first.
Coordinated Fusion: Creating New Data
Sometimes, two sensors together can measure something neither of them can measure alone. Two cameras give you depth with stereo vision, and a camera plus a LiDAR give you an RGB-D image, where every pixel has a color and a distance.
It's a cheap way to get dense depth, but both sensors must be perfectly synchronized, since a small error in one ruins the result. My Stereo Vision course covers this from end to end.
Decision 3: WHEN Do the Sensors Meet? (Early, Mid or Late Fusion)
Picture a pedestrian 50 meters away, with only 12 LiDAR points on them and a blurry patch in the camera image, so neither detector finds them alone. Can your fusion still catch them?
It depends on when your sensors meet, and this is where the three classic sensor fusion techniques come from: early, mid and late fusion.
Early Fusion
With early fusion, yes, because you give each LiDAR point the color of its pixel before detecting anything, so the detector gets more information about those 12 points. I cover two ways to do it in my article on early fusion. The risk is calibration: if your sensors are off by a few centimeters or a few milliseconds, the points get the wrong colors, and the detector learns from bad data.
Mid Fusion
With mid fusion, probably, because each sensor goes through its own network, and their features meet before any decision is made. This is how BEVFusion works, along with many of the best methods on the nuScenes leaderboard. The downside is that you need a lot of labeled data, and changing one sensor means training again, as I explain in my BEVFusion article.
Late Fusion
With late fusion, no, because each sensor detects objects on its own, and if both miss the pedestrian, there's nothing left to fuse. Most stacks then add 3D object tracking on top of the fused objects. So why do most production cars still use it? Because each part, like a camera detector or a LiDAR detector, can be built, tested and replaced on its own, and the car keeps working when one sensor fails.
So, would you rather catch more objects, or have a system that's easier to trust?
One warning: projecting LiDAR points onto camera boxes is often called early fusion, but the boxes are already detections, so it's really a mix of early and late fusion. Get this wrong in an interview, and it shows you've only read about sensor fusion.
Decision 4: WHERE Do the Sensors Meet? (Image, 3D or Feature Space)
Say you have a camera, a LiDAR, and a YOLO model that works really well: where do you make them meet?
Fusing in the Image
Most engineers pick the image, so they can reuse YOLO, which is trained on millions of images and works better than any 3D detector at telling a car from a truck. That's what Autoware's ROI Cluster Fusion does. The problem is depth: a point on a car and a point on the wall behind it can land on the same pixel, and when I built the camera + RADAR fusion for this article, my first version placed a car at 24.7 m instead of 17 m for that exact reason.
Fusing in 3D
If the LiDAR is the most expensive sensor on your car, why would you throw away most of its points just to match boxes in an image?
In 3D, you keep the real distances and the full point cloud, with its shapes and clusters, and that's also the space where the car plans its path. The downside is that cameras are bad at 3D, so their boxes are less precise and harder to match.
Fusing in the Feature Space
And what if you let the network decide what to keep from each sensor?
That's fusion in the feature space: the data from each sensor meets inside the network, often in a Bird Eye View grid like in BEVFusion. Nothing gets thrown away by a threshold you picked, but when it fails, you have no box or point to look at, only numbers. I compare these 3 options with real examples in my article on LiDAR camera sensor fusion.
Decision 5: HOW Is the Fusion Done? (Rules, Hybrid or Learned)
Last question, and the most debated one among sensor fusion methods: do you write the fusion yourself, or let a neural network learn it?
Rule-Based Fusion
You write every step yourself: the projection, the matching threshold, and the logic that decides which sensor to trust. Most ADAS sensor fusion in automotive works like this today, because the behavior is predictable and much easier to certify for functional safety standards like ISO 26262. But every new edge case means writing a new rule.
Hybrid Fusion
Many teams go for the middle: a network understands each sensor, and simple geometry does the fusion. PointPainting, which also ships in Autoware, labels the camera image with a network and then copies these labels onto the LiDAR points, so you get the power of deep learning and a fusion you can still debug.
Learned Fusion
BEVFusion and End-To-End models let the network learn the entire fusion. It can find patterns no engineer would think of, but it needs huge datasets recorded with every sensor, and when it fails, explaining why to your safety team takes a very long meeting.
Sensor Fusion Algorithm Examples: 5 Decisions, Decoded
Now take a few famous sensor fusion examples and answer the 5 questions for each one, and they stop looking like separate worlds.
PointPainting and ROI Cluster Fusion, which both ship in Autoware, give the same answers to 3 of the questions, while BEVFusion changes 3 answers at once.
Which combination would you pick for your own robot?
Is End-To-End Making These Decisions Obsolete?
With End-To-End models learning the whole driving task at once, you might think nobody makes these decisions anymore. In fact, the engineers who built the network still chose which sensors go in, when and where their data meets, and how much of the fusion is learned, they just made these choices inside the network, which I break down in my article on autonomous vehicle architecture.
And End-To-End doesn't remove the need for classic fusion around it. Teams still use it to check what the network outputs, to build ground truth, and to keep a safety layer they can certify, which is the problem I discussed in my interview with LOXO on certifying End-To-End. You can also see how Autoware handles it in my article on AutoE2E.
Summary & Next Steps
Here's what to remember about sensor fusion algorithms:
- Every sensor fusion algorithm is a set of answers to 5 questions, which is why Tesla, Waymo, Mobileye and Autoware all fuse differently.
- Which: you don't need every sensor everywhere, and the right setup depends on your robot, from sidewalk delivery to highway trucks.
- Why: competitive to double-check, complementary to see more, coordinated to create new data.
- When: early fusion catches more but is fragile, mid fusion is powerful but needs a lot of data, late fusion is easy to trust but loses information.
- Where and how: in the image to reuse YOLO, in 3D to keep the full point cloud, or in the feature space to let the network decide, with rules, a hybrid, or a learned fusion.
The next time you open a sensor fusion paper, answer the 5 questions before reading the equations. You'll quickly see what the authors chose, what they gave up, and whether their design fits your own sensors.
If you want to keep going, here are 3 articles to read next:
- LiDAR Camera Sensor Fusion: Where Should the Fusion Happen?
- Sensor Fusion using Kalman Filters
- The 6-Step Roadmap to Learn Sensor Fusion