Dear friend, if you are completely FREEZING while hearing the terms "optimization" and "deployment", you have reached the right place.
Let me tell you the story of what happened the first time I deployed a deep learning model on a real self-driving car.
Back in 2018, I was working as a computer vision engineer on autonomous shuttles, and I got handed what sounded like a simple task: Add computer vision models to the shuttle. So I started with YOLO, a done-for-you architecture and repo, and honestly, in about five or ten minutes I had it working with ROS. Everything ran perfectly. The FPS were good, it was going fast, I was really happy about it.
But then I went to level two, and I had to add segmentation models on top.
And this is when the problems started. I couldn't keep both models running fast enough. The GPU locked at a hundred percent, all the cores maxed out, and it got so hot the temperature hit its limit and the thing crashed.
On a car that has to see and react in real time, that's not workable. And that was just two models. So I'm thinking, if two models already put me here, what happens when I add five, ten, maybe twenty models total? All of it with Python scripts, with C++ code, with point cloud processing, radar, GPS, planning...
How to make slow models faster
I did what most people do when they hit that wall: I made my input images smaller. I dropped the input resolution, and suddenly the model ran faster. The detection got maybe a little worse, but it was barely noticeable, and I told myself it would be fine.
Unfortunately, resizing the image is a rookie move.
That day on the shuttle, I was what you call a Player. There are 2 kinds of engineers in this field:
Players and Pros
A Player can grab a dataset, pick an architecture, train a model and get a good result on the validation set. The notebook runs, the metrics look great, he's happy. That was me at the time.
A Pro not only does that, but can also ship models. The same model, they get it running on the real hardware, at real speed, next to ten other models, on a car that has to make decisions in real time. They know what to do when the GPU locks at a hundred percent, because resizing the image is the last thing on the list, not the first.
The Player plays with deep learning, the Pro deploys it. Almost everyone in this field is a Player, because that's the only thing the whole system trained you to be. University, tutorials, blogs, videos, courses, all of it stops the exact moment the model is trained and can run. In reality, this is exactly when the problems start.
If you currently are not able to deploy models to Deep Learning Environments, you are a Player. This is NOT a bad thing, and there are MANY startups and companies who WANT people who can only train and run models, and who have their own Deployment teams.
BUT:
I've recently spoken to 2 elite engineers of this Industry: one is the Head of AI Deployment of the world's largest autonomous delivery fleet. The other is the CTO of an off-road agriculture company; both told me the same thing:
"Today, learning Deployment matters MORE than learning the architectures."
When I shared this with clients, many told me how they thought they'd need to be a different profile, someone more technical, with tons of C++ skills. I've also heard phrases like "I do the models, deployment is someone else's job".
All of these are MYTHS.
Deployment is a sequence of skills you need to learn, the same way you learned neural networks, then backpropagation, then loss functions, optimizers, and libraries... you are now learning pruning, quantization, distillation, export, and GPU optimization.
2 Benefits of mastering optimization & deployment
BENEFIT 1: You get Control
You have the capability to deploy models anywhere. You have solutions when a model is too slow that are way beyond "GET A BETTER GPU" or "Resize the input". You finally build mastery of your job.
BENEFIT 2: You build RARE skills
Can you imagine if your resume listed skills like TensorRT deployment, GPU programming, ONNX, and Quantization? Wouldn't your profile look much more credible to the industry?
If you'd like to learn more about this skill, then meet my newest course...
