Why ROS?

For sure, it’s quite easy to find a tutorial on obstacle detection using a Raspberry Pi. So why bother including ROS?

The short answer: Obstacle detection for obstacle detection is pointless.

• Why would we draw 2D Bounding Boxes around pedestrians or vehicles if nothing is behind to interpret the data?
• Why would we detect obstacles if not to make a decision based on the presence or absence of an obstacle?

So what do you do?
You can print the detected bounding boxes, you can even print them in a file. But what then? No autonomous robot will ever read files to get the obstacles; it’s just not working.

For an autonomous system to work, you need something like ROS. ROS is the meta operating system behind at least half of the self-driving cars out there! I believe any aspiring self-driving car engineer has to learn about ROS, or the risk is simply to get rejected by half of the companies.

In this article, I will explain how to develop an embedded Computer Vision system that can play a real self-driving car recording (also called a bag), and add algorithms on this; all using a Raspberry Pi.

Robotic OS

In order to understand how to build a self-driving car, you need to understand the general concepts behind ROS.

ROS is using a Publisher/Subscriber architecture like the following.

• A ROS node is the equivalent of a function or a mini program. Its role is to publish information (obstacles, position, …) that we call messages. Each message is published on a topic.
• A Node that publishes information is called a Publisher. Those who listen to and use this information are called Subscribers. Generally, a node is both Subscriber and Publisher, except in specific cases (for example, the camera node is only a Publisher, it doesn’t subscribe to anything). 

This architecture makes it possible not to block the system if a module does not respond. Nodes publish no matter if there are other nodes listening.

• The set of nodes form a graph.

The course’s project is using ROS to detect obstacles in an image and publish bounding box information.

Here would be the typical architecture of such a project in a real self-driving car environment.

In this image, you can see:

• The sensors on the left are Publishers: They publish data like images, point clouds, ...
• The rest are Nodes. Most subscribe to sensor data, or to other nodes, and then publish for the next node. All until we reach the controller, in charge of generating a steering and an acceleration value for the car.

We know that ROS is perfect for our problem: To build a useful obstacle detection algorithm that can be integrated in a real autonomous robot. Our next problem will be to find an embedded platform to use ROS.

Choosing a ROS Platform— 

The Raspberry Pi has always found a community of edge Engineers who wish to build Do It Yourself projects.
What is it? It’s a mini-computer that is small, performant, and relatively cheap.

Raspberry Pi 4 vs. the rest

If we are to build a Do It Yourself project, or something like Edge Computer Vision, a lot of possibilities appear.

• For example, I recently wrote about VPUs and the Intel Movidius NCS. That’s a super powerful USB Key that can be plugged to a computer and provide a lot of processing power.
• More recently, the Nvidia Jetson Nano has been released and got a lot of attention.
• Soon, the OpenCV AI Kit will be there as a revolutionizing tool.

The last two computers have their own SDK; which means they won’t work right away.

Most Computer Vision and edge devices are very good when it comes to running an OpenCV program, but are they good when we have to work with Robotic OS, LiDAR point clouds, and GPIO sensors? Do they provide the possibility to plug any sensor and just make it work?

This is one of the reasons I looked into the Raspberry Pi. It’s running Linux, has tons of GPIO possibilities, and can be used immediately! It’s more of a tiny computer than the rest because it wasn’t built for AI.

It also has a lot of processing power, and the new 8Gb RAM version makes it super powerful! That, and the low price, are key elements.

Imagine students enrolled in my course want to buy it and try it themselves, a Jetson Nano would be a 200$ investment. The Raspberry Pi 4 costs about 100$, which was great for my use.

If you have the budget, you can probably go for a Jetson Nano or even better, but in our case, the Raspberry Pi 4 can totally work.

Raspberry Pi 4 — Prerequisites to work with ROS 

Operating System

If a self-driving car has 4 elements (Perception, Localization, Motion Planning, Control); then we must use ROS to make these elements work together and build a real autonomous system.

Once I had the Raspberry Pi 4, and once I put it in the delightful red and white box, I had to use it with ROS.

ROS is widely used, but there is a problem. ROS only runs with Ubuntu and Debian. The new version, ROS 2, works with Mac OS and Windows X, but not with Raspbian, the default Raspberry Pi OS.

For that reason, I had to change the OS and use Ubuntu 20.04 on my Raspberry Pi 4. I could install ROS Noetic on the Raspberry Pi 4.
This is where the fun begins!

32 vs 64 Bits

The Raspberry Pi website provides everything necessary to install Ubuntu, but it requires us to choose between a 32 bit version and a 64 bit version. Using the 64 bit version helps using the full RAM of the Rapsberry (8Gb) and not just 2³² bits (4Gb). In our case, it’s can be necessity.

Some problems came, a 64 bit Ubuntu is great, but it doesn’t work with VNC.

Remote Control

VNC (Virtual Network Computing) is the go-to option for remote control. 
I don’t want to plug a mouse and a keyboard on my Raspberry Pi. I want to plug my Raspberry Pi on my wifi box and use it on my Macbook Pro. Also, if students ever had to connect on it, it had to work.

Since VNC couldn’t work on Ubuntu 64 bits, I had to use another solution called Microsoft Remote Desktop that uses XRDP protocol.

We’re all set, we installed Ubuntu, ROS, and made sure we could fix any issue regarding the choices we made. We can now start to work with the project.

Building a ROS Project

The Project

So what can we expect from our project?

Well, what if we plugged a camera and performed obstacle detection? What if we plugged a LiDAR and performed Point Cloud Processing? In reality, anything is possible.

👉 In this project, I chose to focus on the camera aspect, but understand that we can do everything simultaneously.

Here’s the project architecture.

The first thing I did is to play a Bag File.

A Bag File is a recording of everything that is going on in a self-driving car. We can record camera images, LiDAR data, and many more, and then play it. In my case, I had access to LiDAR, camera, RADAR, and GPS data.

I played the bag, and displayed interesting data. It all ran using 85% of CPU and 1/8 Gb of RAM, leaving plenty of space for the algorithms.

Why a bag file?
When you use a Bag File, you replace the sensors. Instead of purchasing sensors and then going in the street to sense something, I just used someone else’s work and made it work on my computer.

Most Engineers work with Bag Files. 
They don’t go out every time they want to test something; they replay the events and see how their algorithms perform now.

Embedded Computer Vision

My first intuition to teach students how to use ROS is to make them build a node (program) that would:

• Subscribe to the camera information
• Publish the obstacles it found using a YOLOv4 algorithm.

All of that was possible, the students on the course learn to implement it. 

The course covers a lot of concepts and details to build the project: 

• How to use CV Bridge and avoid popular errors (you can literally loose days)? 
• How to build obstacle detection algorithms on ARM Processors (on Raspberry Pi and other embedded devices)? 
• How to publish custom messages and make sure we can compile in every situation?
• How to work in Python and C++ at the same time?
• and many more…

The course is called START WITH ROS: Build Real Robotics Applications on the edge.

The future of AI is tiny and on the edge.
I didn’t say it, Harvard did on their website. Cloud Computing has allowed us to learn a lot about AI and Deep Learning, but it did hide an important part: the hardware.

In an autonomous robot, understanding the sensors, the hardware, and the platforms you’re working on is beyond mandatory; it’s part of the job.

The course will teach you the fundamentals of Robotic OS, but also important Hardware concepts you must know as an aspiring Self-Driving Car Engineer. It has everything you need to get started, and to keep building great projects on it.