Foxglove: Why the future of Physical AI is collaborative

Ever heard of Spiderman's "Guy in the Chair"?. He's called Ned, and he's Peter Parker's best friend, but he's also the one that helps Spiderman and whispers him what to do when he's on a mission against Vulture and the other bad guys. Batman has Alfred. Iron Man has Jarvis. All superheroes has someone or something that supports them while they're busy fighting the villains. Don't they?

In Robotics, we may be fighting the Perception & Planning villains, but what is our support system helping us achieve that?

Often, it's tools.

Every time I go visit a company, for example for a seminar, they show me some of the internal tools they've built. Most of the time, it's things they're using every day and invested a lot in. Many times, I've noticed companies were putting their entire robotic infrastructure on a particular tool provider: Foxglove.

Foxglove is your guy in the chair. It's a platform you use for visualization, for testing, for data management, for robotic workshops... Companies like Wayve, Shield AI, Gatik, and NVIDIA use Foxglove every day; and in particular because of a specific feature I will call "Multi-Player Mode".

Before we get to this, I'd like to give you a quick intro to the platform, so let's see:

There are a few things that I learned from Roman, that I'd like to share with you:

• Can Foxglove replace Rviz?
• Why Multi-Player Robotics is the future of Physical AI
• What being a CTO really is about

I. Can Foxglove replace Rviz?

Traditionally, companies have been using ROS (Robotic OS) to build their self-driving cars, and its free visualizer Rviz. As great as Rviz can be, it has the many drawbacks a free and open product can have:

• Very simple viewer (only one 3D view, no side-by-side viewer with other plots etc...)
• Only on ROS with Linux (can't share a problem with managers who don't have ROS on their computer and are comfortable with it)
• Doesn't evolve (AI, robots, software, ...)
• Single player mode (no collaboration, insight sharing, comment, ...)
• No new or advanced features (data retrieval, querying, marking, ...)

Looking at it this way, rviz feels very 2017 like; but these days... wouldn't it feel normal to simply "ask" your data for some events and be able to retrieve them?

When working with free tools like Rviz, you do get a free visualizer, but there's so much you never get access to. Foxglove, on the other hand, has a team funded working every day on making these internal tools better.

The industry has a lot of discussions on "Saas vs Vibecoded", I think when you experience Foxglove, there is no comparison to be made. A tool like Rviz, or a custom visualizer can work for one task, but the depth and scope of what you can do (and what you will be able to do in the future) is unmatched.

A note on MCAP Files

Foxglove has built something called MCAP (em-cap). It's a file format that replaces the protobuf, ROS Bag, JSON, or other formats that is more universal. They benchmarked it against what ROS 2 was using at the time (SQLite-based bag files) and took the results to the ROS technical governance groups directly.

Results were so good MCAP became the default recording format in ROS 2. Run ros2 bag record today and you're using a Foxglove-designed format. This means when you go with a platform like Foxglove, you aren't following something, you're part of the leadership.

This brings me to where they lead too: Multi-Player Robotics.

II. Why Multi-Player Robotics is the future of Physical AI

Sebastian Thrun, known as godfather of self-driving cars, once shared this story of a major problem his team was stuck on when doing the DARPA Grand Challenge in 2005. Their self-driving car was able to drive through the desert for hours, perfectly following the destination. But during the long drives, the car sometimes detected a gigantic obstacle getting closer and closer and slammed the brakes.

Unexplainable.

Yet, it happened a certain number of times. So they ran the car again and again for hours to see the bug of the giant obstacle appearing in the middle of the desert. At some point, someone noticed this problem always happened after exactly a specific number of minutes, which led them to the solution (something about a timestamp refresh, from memory).

This would never happen with a tool like Foxglove, because we wouldn't need to re-run the car physically for hours and hours. What would happen is:

• We'd record the events on a file
• We'd replay the recording on the browser
• We'd share the problem with other engineers, some watching from other countries
• We'd simultaneously plot certain information, mark interesting comments until we debug the problem

Put differently, we wouldn't be in "Single-Player" mode, but in Multi-Player Robotics Mode.

The real strength of Foxglove is collaboration. They enable Multi-Player Mode for you. You can share a recording with someone else who works on a Windows computer and has no idea or desire to install your exact version of ROS and Ubuntu. They can collaborate with you, tag important moments, and do a range of tasks so you jointly visualize and solve the problem.

Roman talked the most about events. While reviewing data, an engineer drops a timestamped annotation right on the timeline. "The foot was out of alignment" to use my own example from the demo.

Teams that never move past a single laptop and local files don't hit this wall at all. It shows up the moment there's more than one person who needs to look at the same robot's brain.

III. What being a CTO is really about

As I'm running monthly interview with self-driving car and physical AI companies, I sometimes get the chance to be with executives, CEOs, CTOs, Director of Engineering. You can learn a lot from them.

One question I asked Roman was "What is different in a CTO rather than an engineer?"

He provided me a lot of interesting answers, but one quote he said in particular stuck with me.

He then went on explaining how he's spending A LOT of his time in Slack/Discord groups of Foxglove users, just seeing what people struggle with, need help with, ask questions about... in order to shape the product!

This is something engineers rarely do. A typical engineer, especially when young and inexperienced, would see tasks like support and validation as "beneath them". When you're in startup mode, you quickly learn than nothing is beneath you. I have changed tires, fixed windshields, and do all sorts of silly tasks an engineer isn't supposed to do because I was in a startup.

As a CTO, you need to care extremely about the market and can't work in a silo. You need people to get value out of it.

On that last "value" point, there is a complete video about it in the Behind-The-Scenes of our interview that I absolutely recommend about figuring out where the value is in a product.

Summary & Next Steps

• Physical AI isn't just algorithms, it's also tools. Countless companies develop tools internally for tasks like visualization or data management. Foxglove developed them for you (and keeps on improving them so you can focus on the algorithms)
• Rviz is a great visualization tool, but it has the drawbacks of a free visualizer. It's never improved, enhanced with new features, and stuck in a 2017 like world. On the other hand, a platform like Foxglove is continuously improved to live in its era.
• Foxglove pioneered the MCAP format, which is now the default file format in ROS 2. When you record a scene, everything is stored in MCAP file, which is a format directly visualizable on Foxglove browser.
• The future of Physical AI and Robotics happens in Multi-Player Mode. When using a tool like Foxglove, MPM allows you to share recordings with other on their browsers, mark events, and more generally collaborate on a problem rather than fixing it alone.
• A CTO has this tendency of caring for the customers that other roles may not have. They will spend a lot of time reading where users struggle to try and anticipate the next features. They are constantly in touch with their market; it's about the market first and the features second.

Interested in the full workshop?

Here's what you get in this training:

• You will understand how Foxglove works (you can add that to your resume) and have the skill of turning everything into a foxglove file (so any project, portfolio demo, dataset, anything… thanks to our bonus workshop).
• You'll be introduced to MCAP files, the now default format used in the industry, and have been through the SDK to build your own
• You will understand how CTOs think, and the 2/3 critical mind-shifts engineers lack that CTOs have
• You'll get valuable career and hiring insight from a working CTO (what it actually looks like to lead technically while staying hands-on, what he screens for when hiring engineers at an early-stage startup, and how he thinks about fundraising a $40M Series B)
• Plus a lot of other bonuses included…

Get access to this, PLUS my 2 past fragments here:

https://www.thinkautonomous.ai/the-edgeneers-land

(P.S: This offer may not be available by the time you read this, make sure to join the membership while it's live)