AutoE2E: How Autoware built a VLM-free End-To-End Reasoning Machine with Muhammad Zain Khawaja

How many steps did you walk today? If you’re carrying a phone with you, you probably know the answer. And if you’re like millions of people, you also know the number you’re supposed to hit: 10,000. Every evening, people walk around their living room, around the kitchen table, sometimes around the block at 11 P.M., just to hit that magic health number.

But who decided it had to be 10,000? To find out, you have to go back to Japan in 1964. Tokyo had just hosted the Olympic Games, the whole country was obsessed with fitness, and a company named Yamasa saw an opportunity to release a small pedometer and gave it a name that would travel the world: “Manpo-kei”, which literally means “10,000 steps meter”.

So where did the number come from? A study? A doctor? A clinical trial Apparently not. One popular explanation is that the Japanese character for 10,000 万, looks a bit like a man walking. The number was chosen to sell a device, and for almost 60 years, nobody verified it. Until 2019, a Harvard study revealed the real health number was around 7,500 steps.

I think the self-driving car industry has its own 10,000 steps right now: language. Most of the new models are showcasing how they describe the scene in words before they drive, and billions of parameters go into it.

But is it really needed to drive? This is what we'll find out in this article with the President of the Autoware Foundation himself.

What is AutoE2E?

If you are familiar with Autoware, you know they have 3 core models:

• Autoware Universe (rule-based Level 4)
• Vision Pilot (End-To-End Level 2)
• AutoE2E (End-To-End Level 2++ to 4)

Here is how Muhammad describes them:

For over an hour, Muhammad and I talked about the new AutoE2E architecture, and why it's the best place to start with End-To-End Learning. Here are 2 strong ideas you should get from this:

Why 'You do not need language to drive'

How contrarian. Most of the industry is going big right now into Vision-Language-Action models, World models made of billions of parameters... but Autoware says it's not needed?

Let's take their point of view: if the majority is to start End-To-End from open source, on their current GPUs and architectures, then forcing billions of parameters is the wrong direction.

Here is how Muhammad explains it:

In a recent test, Autoware beat Nvidia Alpamayo with 100x fewer parameters. Yes, AutoE2E has 80M parameters and Nvidia Alpamayo 1 has 8B. What is taking billions? Language. It is not only heavy, but expensive to train, and doesn't run real-time (10Hz or more). It doesn't fit on the automotive chips cars to drive real-time.

So here is Muhammad's take:

"Animals can navigate without speaking or understanding English. Vehicles should also be able to do the same."

If we make the claim that robots don't need anything other than vision to drive (because humans don't), then we can make the claim that robots don't need to articulate phrases about the scene.

So instead, AutoE2E uses a slow-fast design, with 3 branches:

1. The reactive branch (green): It runs real-time (10Hz) and has cameras, a raster map and the car's motion history go in to get a trajectory out. This branch alone is what drives in their demo and works in 90% of use cases.
2. The reasoning branch (yellow) identifies edge cases; the remaining 10% and rather than having a model generate the description (with LLMs), classifies the scene with a transformer.
3. The world action branch (purple) does NOT predict the next image, but the next features. This is what 'JEPA' means in this graph. Muhammad calls it the "fingerprint" level.

That last point is my favorite part of the conversation. A world model like Nvidia Cosmos regenerates every pixel of the next frame, clouds included. If it predicts a car's position perfectly but gets its color wrong, it still gets penalized. For driving, you care about where that car is and how fast it moves. Who cares if it's red or blue? JEPA compares a "fingerprint" of the predicted future with the real one, which is much cheaper to train.

Two more details worth noting:

1. The map is a raster (literally a screenshot of a sat-nav)
2. The planner outputs curvature and acceleration over 6.4 seconds instead of X,Y waypoints, because it's numerically stable and maps straight to the controller.

If you want the background on these design choices, I explain them in my 4 Pillars vs End-To-End article.

Alright, now the part most of you have been asking me about...

How anyone can start with End-To-End Learning

In February 2026, I ran a poll in my community asking where people were in their End-To-End transition:

• 30% replied they were migrating towards End-To-End Learning...
• 50% were exploring exploring learned components, but staying modular
• 20% were modular with no intention to switch.
• 0% replied they already had end-to-end in production.

There is a reason for this, End-To-End is a LEAP. Not just technically, but also conceptually. It's a complete rewriting of everything you do. Who will work on what if you're doing End-To-End? What happens to the LiDAR Engineers? The Fusion Engineers? What about our data? Who handles it... and how? What about explainability? Certification? ISO norms?

So where should a small team start from? Here is Muhammad's take:

From this off-road example, you can also get 2 follow-up ideas:

1. Imitation Learning is where to start. Drive 10 to 30 hours in your own environment, and train the model to predict what the human did. This is great, and you can test this on open-loop examples, but again, it solves only about 90% of use cases.
2. Closed-Loop Reinforcement Learning post-training is where to finish. An imitation model alone will drift off the road, because small errors compound frame after frame. If you let it make mistakes in a photorealistic simulator (Gaussian Splatting, generative world models...), and train it to recover, you can actually solve for a lot of extra use cases.

None of this is new, by the way. NVIDIA did the Imitation Learning part in 2016 by simply warping camera images when the car drifted. Wayve did the Reinforcement Learning part with a real Renault Twizy in Cambridge back in 2017 with their famous video "Learning to drive in one day".

But with today's technology, both these ideas are practically feasible.

So how much does it cost? Muhammad's back-of-the-envelope estimate, starting from AutoE2E: 1 person to reproduce the open-loop results, ~50 hours of raw driving data, and tens of thousands of dollars in compute (not millions).

But Jeremy... this can drive in production? Really?

Muhammad other label isn't President of Autoware but President of Production. He knows way better than me (or a lot of people) what makes a car drive in production.

What did I get from him?

Level 4 still needs a deterministic "safety guardian" (LiDAR, radar, HD maps) running in parallel.

This is close to Mobileye's redundancy philosophy (that I heavily cover in my Autonomous Tech Bunker), and goes with Nvidia's SFF (Safety Force Field) and Halos environment, that even companies like Wayve are built on.

To go deeper on this topic, I highly recommend this article:

Alright, let's see a summary of what you learned in this article:

Summary & Next Steps

• Autoware has 3 core models, all built for different means:
  • Autoware Universe, the classic rule-based Level 4 stack
  • Vision Pilot, monocular Level 2 (an open-source Autopilot)
  • AutoE2E, the End-To-End from Level 2++ up to Level 4 (an open-source FSD)
• AutoE2E is only 80M parameters and doesn't use language at all. It reasons in latent space instead. World Models are built with ideas like JEPA.
• The process to start with End-To-End: Start with imitation learning on your own driving, then go closed-loop in simulation. 50 hours of data and a few tens of thousands of dollars might get you there (an estimate, not a promise!)
• AutoE2E itself is still open-loop for now, but Autoware is working on post-training RL. It's probably the best place to start currently. Models are getting easier and easier to build, so put your energy into data, hardware, and your own first principles.

Want to watch the full interview with Muhammad?

This post only scratches the surface. The full Fragment is inside The Edgeneer's Land, my community membership, and goes much deeper. Inside you get:

• The commented AutoE2E demo, where Muhammad walks through predicted vs human trajectories, including the pedestrian stop and night driving
• How to set up your own closed-loop training, from simple reward design to his "start the car one meter off-center" test
• Exactly what models are inside EACH block of AutoE2E (and novel ideas you will find inside that are seen nowhere else)
• JEV: How to use Foundation Models without LLMs for edge case detection
• Muhammad's advice for engineers who want to start their own robotics company
• What to do at the age of AI and the pressure to push AI generated code
• Behind-The-Scenes: A practical workshop on the "fingerprint" idea of Muhammad — how to work with JEPA World Models for self-driving car scenarios

Get it PLUS TONS of other trainings when joining the membership: