Over the years, developers have built an amazing suite of tools for debugging and introspecting the ROS system. Unfortunately, ROS packages also have a discoverability problem: unless you read about them in the tutorial or someone else tells you about them, you have no way of knowing they exist.

The first step to building a good ROS toolkit is taking care of your terminal situation, but that’s just the beginning. For good ROS debugging, you need to use tools that can analyze messages, show you the terminal outputs from various nodes, and just handle the general complexity that ROS brings to the table. Many of the best tools are installed by default with the ros-*-desktop-full standard installation, so they’re available to you on most ROS machines! I’ve compiled a list of my favorite under-utilized ROS tools below. Enjoy!

1. rqt_console

One tool I’m always surprised that people haven’t heard of is rqt_console. It comes standard with ros-*-desktop-full, and allows you to quickly filter messages by what node publishes them, what text they contain, their severity (such as only showing errors and ignoring all warnings), and more. If you are working with bugs that only appear when you have many nodes running, it can often be hard to pick out what the exact problems are and their associated error messages. By setting up rqt_console with heavy filtering, you can ensure that you only see the messages that matter. Run it with a simple rqt_console in the terminal.

The rqt_console gui with several recent messages displayed.

Houston, we have a warning. Source: ROS Wiki.

I find the interface for rqt_console is a bit hard to understand, because the icons on the various buttons don’t make much sense to me. But as you play around with the interface some more and read all the tooltips, how to use the tool should become clear. Read more on the ROS Wiki.

2. rqt_logger_level

rqt_console is great, but what if you want to see messages marked as Debug? By default, those messages won’t be shown at all, including in the terminal! The best way I know to get those pesky debug messages is to use rqt_logger_level. You can set the level of messages for each node and logger individually. You can also use this to block out noise from an overly talkative node by setting it to only produce “FATAL” messages (of which there should be very few!) Run it from the command line with rosrun rqt_logger_level rqt_logger_level.

This tool does not solve all your problems, by any means. Many times, even just one node will produce so much debug output that it still won’t be helpful. Also, each time a node restarts, you’ll have to use the GUI to set its logger level again. However, for dipping into the debug stream of a running node to view some generated values or low-level info, it can be quite nice. This tool is also not well-documented, but it’s pretty straightforward. Read what docs there are at the ROS Wiki.

3. rqt_launchtree

Launch files often end up nested three or four layers deep as you try to keep your code modular and clean. As you start calling launch files in MoveIt or the ROS navigation stack, the launch file include structure can get even more convoluted, making debugging a huge pain. RQT Launchtree is designed to help fix this problem. By parsing launch files, it allows you to drill down through multiple nested launch file includes to see exactly what’s going on. It has support for parameters and arguments as well, and I’ve used this tool to fix many a misconfigured launch file. You can install rqt_launchtree with a simple sudo apt install ros-${ROS_DISTRO}-rqt-launchtree, and see the wiki for more details.

Run it with rosrun rqt_launchtree rqt_launchtree.

A screenshot of the tree structure of a launch file in rqt_launchtree.

Look at that beautiful tree structure. Source: ROS Wiki.

4. rqt

If you liked the last three items on the list, you’re going to love RQT. RQT comes default with ros-*-desktop-full. The general-purpose rqt console (run from the terminal with just rqt) can be used to arrange any rqt-compatible GUI elements together. That includes other tools you may have used before, such as rqt_plot, and you can even add an RViz window or web browser as a panel. You can build your custom layout however you’d like, and then save it for reuse. This is a great way to build a quick-and-dirty control panel or status window for your robot. Check it out at the ROS Wiki page.

Screen shot of ROS GUI

Just one possible GUI configuration that’s possible with rqt. Source: http://wiki.ros.org/rqt.

5. Message Thief

The Message Thief is indispensable in a very specific circumstance–when you are working with a robot or ROS system you’ve never worked with before. Sometimes, if it’s hard to access the code or catkin workspace on the robot, it’s nearly impossible to communicate with the robot because it uses messages that you don’t have installed. Sam Pfeiffer, who wrote the package, told me about it when we were discussing working with lots of different robots. The message thief will create shadow packages on your own machine that match those that are published by other machines on the ROS system. This will allow you to actually work with those previously unknown messages, using tools likeĀ rostopic echo. It can even work with robots that use a different version of ROS! If you are a consultant or often find yourself working with robots you’ve never used before, check it out on GitHub.

6. Status Panel

Full disclosure: I wrote this plugin! This RViz display sits at the bottom or top of your RViz window and neatly displays String messages that are published to a certain topic. I’ve used this panel to show important information to a user without overloading them with a terminal full of unimportant messages, and it’s much prettier. The default topic is /status, but you can change it to listen to whatever topic you’d like. The panel also supports Qt’s rich text features, which means that by wrapping your strings with HTML and CSS markup, you can style the status text however you’d like.

The panel only shows one message at a time. Unfortunately, it’s not an official package, so you’ll have to get the package on GitHub and then add it to your own personal catkin workspace. But don’t let that stop you from giving it a try.

7. RViz Visual Tools

This package contains all the RViz markers you’ll ever want, and some you didn’t know you needed! Things like wireframe cubes and coordinate axes are huge timesavers to making useful and detailed debug visualizations. Even better, publishing all these new marker types is made easy by a bunch of wrapper functions. This package also contains tools for stepping through a debugger straight from RViz. Install it with sudo apt install ros-${ROS_DISTRO}-rviz-visual-tools and check out the README on GitHub for a detailed overview.

The many custom marker types that are a part of rviz_visual_tools

Look at all those beautiful custom RViz markers. Source: GitHub.

8. rosnode info

Ah, the humble rosnode command. Most commonly used for rosnode list or rosnode kill. If you went through all the ROS tutorials, you may have learned that you can also run rosnode info <node_name> to get a list of a node’s subscribed topics, advertised topics, and services. I recently re-discovered this command and have since put it to good use as a basic diagnostic tool. It’s much faster than pulling up another tool such as rqt_graph and hunting for your node’s connections (plus, it shows services too). The ROS tutorial on nodes has some more information.

9. rqt_top

I’m a fan of RQT packages, let’s keep this train rolling. rqt_top performs a similar function to the top command in the Linux terminal, showing what system resources each running node is currently using.

It’s great for finding which nodes are draining your battery or making your fans spin up (spoiler: it’s almost always your camera driver or perception algorithm). It’s also useful as a gut-check to do some basic profiling and make sure you haven’t built a program with huge memory leaks. rqt_top isn’t included in the ROS desktop installation, but it is a listed package, so get it with sudo apt install ros-${ROS_DISTRO}-rqt-top.

The legacy rostop gui, showing a list of processes and how many system resources they're using.

This is an older screenshot from this legacy page, but you get the idea.

 

10. roswtf

Perhaps the the most…creatively…named ROS tool, roswtf can be a lifesaver when you’re debugging complex ROS-specific issues, such as two nodes that aren’t communicating with each other, two nodes publishing to the same topic, or issues with your TF tree. It also diagnoses your environment variables and folder structure, although I’ve never had it find problems with my setup on that front. Try it out while your whole constellation of launch files and nodes is running, and see what it comes up with. Run it with roswtf, straight from the terminal.

Summary

I hope you learned about some new tools you can use improve your ROS-fu. I’ve enjoyed learning about and using these tools over the years, and they have made my ROS experience so much smoother.

Did I miss a tool you like to use? If so, let me know in the comments below.