This post shows how to add automated tests to ROS. If you want to get straight to the data, scroll to the bottom of the article for a cheat sheet that shows which testing framework to use depending on your code under test and your build system.
Why Write About Testing?
Between outdated tools, unclear documentation, and no good examples, learning how to write automated tests in ROS is hard.
If you search "ROS unit testing", there is a very nice ROS Wiki Page on why to test, but the how is still unclear to a lot of people. The second search result begins with a disclaimer that some of the documentation is out of date because
catkin changed things. Well,
catkin was released 6 years ago, in ROS Fuerte, but I guess the documentation still hasn't been updated. In fact,
catkin itself is old, and we now have
ament_cmake which was written for ROS2.
This post is a starting point for people who want to learn how to automate their ROS tests. It's not a complete guide, but I hope I can help clear up some of the confusion around this topic.
Know Your Testing Frameworks
Part of the reason it's so hard to get started on this topic is that there are lot of testing frameworks out there. Some are designed for C++, some for Python, and some for ROS.
Just like handy ROS tools, it's tough to find out about all of these frameworks on your own. It's not clear which ones are even compatible with your build system, much less which ones are the best to use.
I put a "cheat sheet" version of this data at the end of the article for easy lookup, but if you want details on each system, read on!
First, some automated tests require a running ROS system. Maybe you want to check that a service call returns correctly, or your node depends on other nodes while it's being tested.
catkin, rostest is your friend here.
.test files, which are like
roslaunch files, to set up an entire ROS system. This means a
roscore, a parameter server, and as many nodes as you want. Then it conducts your tests using another testing framework. So if you use
rostest, you ALSO have to use one of the options below. Be sure to add your rostest to CMakeLists.
ament, there are no good options if you want an already-working solution. You will have to use the new and poorly-documented system_tests to run tests against multiple ROS2 nodes. Or try using launch_testing to test that your launch file works as expected.
If you want to write your tests in C++, use gtest. Simple!
If you need to write your tests in Python, there are at least 6 decent testing frameworks. Only some are properly integrated with
catkin, you have three options:
unittest: Old but still very functional. There is a good guide on how to write a
unittesttest and add it to your CMakeLists over on the ROS Wiki.
nose: Nose is a newer framework that some people prefer to unittest. Note: nose is now in "maintenance mode" because there is a nose2, but nose2 is not integrated into catkin. Write your
nosetests as usual, then add the tests to your CMakeLists.
pytestis yet another Python testing framework. Here's a blog post by Alex Rössler on how to add these tests to your project.
ament_cmake, you can use nose (remember: not in active development) or pytest. But the ROS2 documentation is not very good so you may find that you need to browse the source code to learn how to add the proper commands to your CMakeLists. Here's the
ament_cmake source for nose and pytest.
Determine Your Code Under Test
Determine your code under test next.
Are you trying to test a Python library that does matrix and vector operations?
Or are you trying to run an integration test to make sure that a launch file sets up your C++ object detector to publish ROS messages to a
Think carefully about what part of the system is critical to test. You might find that you need two or even three types of test: testing the basic library, testing a single nodes' ROS interface, and testing many nodes talking to each other.
Write the Tests
Now that you know your code under test and your options for testing libraries, you are ready to write your tests and add them to the build system!
I hope this served as a handy reference for anyone looking to add automated tests to their ROS code. If you know of other test frameworks or tutorials I can link to to improve this tutorial, please leave a comment or contact me.
Here's a couple of tables with the information above, for quick lookup.
Testing using catkin
|Code Under Test||Testing Options|
|Python class or function, with no ROS||unittest, nose, pytest|
|C++ class or function, with no ROS||gtest|
|Python class or function with ROS1||rostest file + one of: gtest, unittest, or nose|
|C++ class or function with ROS1||rostest file + one of: gtest, unittest, or nose|
|roslaunch file (with ROS1 nodes written in C++, Python, or both)||rostest file + one of: gtest, unittest, or nose|
Testing using ament_cmake
|Code Under Test||Testing Options|
|Python class or function||pytest, nose|
|C++ class or function||gtest|
|Multiple ROS2 nodes||system_tests|
|ROS2 Launch File||launch_testing|