Performance benchmarking of multi-robot systems: Application to exploration

Performance benchmarking of multi-robot systems: Application to exploration

Zhi Yan
Luc Fabresse
Jannik Laval
Noury Bouraqadi

Département Informatique et Automatique, École des Mines de Douai, France

Laboratoire DISP, Université de Lyon, France

Corresponding Author Email:
30 April 2016
| Citation



Performance benchmarking has become an important topic within robotics. It is indeed, a critical way to compare different solutions under different conditions. In this paper, we focus on performance benchmarking of multi-robot systems which explore and map unknown terrains. We summarize metrics to objectively compare different algorithms that can be applied to collaborative multi-robot exploration. This work is also a first concrete step to address the general problem of objectively comparing different multi-robot coordination algorithms. We also identify parameters that impact robotic fleet performances. By varying these parameters, we can identify strengths and limits of an algorithm. We illustrate these contributions with realistic benchmark simulations of the frontier-based exploration strategy. The simulations were implemented in ROS (Robot Operating System).


performance benchmarking, multi-robot systems, collaborative exploration.

1. Introduction
2. État de l’art
3. Benchmarking et plan d’expérience
4. Plans d’expériences pour l’exploration multirobot
5. Banc d’essai
6. Illustration avec des simulations
7. Conclusion

Ce travail fait partie du projet Sucré qui est soutenu par la Région Nord Pas-de-Calais.


Amigoni F. (2008). Experimental evaluation of some exploration strategies for mobile robots. In Proceedings of icra’08, p. 2818–2823.

Bautin A., Simonin O., Charpillet F. (2012). MinPos : A novel frontier allocation algorithm for multi-robot exploration. In Proceedings of icira’12, p. 496–508.

Bonsignorio F., Messina E., Pobil A. P. del. (2014, march). Fostering progressin performance evaluation and benchmarking of robotic and automation systems. Robotics and Automation

Magazine, vol. 21, no 1, p. 22-25.

Burgard W., Moors M., Fox D., Simmons R. G., Thrun S. (2000, April). Collaborative multirobot exploration. In Proc. ICRA’00, p. 476-481.

Couceiro M. S., Vargas P. A., Rocha R. P., Ferreira N. M. (2014, February). Benchmark of swarm robotics distributed techniques in a search task. Robotics and Autonomous Systems, vol. 62, no 2, p. 200-213.

Doniec A., Bouraqadi N., Defoort M., Le V. T., Stinckwich S. (2009). Distributed constraint reasoning applied to multi-robot exploration. In Proceedings of ictai’09, p. 159-166.

Faigl J., Simonin O., Charpillet F. (2014). Comparison of task-allocation algorithms in frontierbased multi-robot exploration. In Proceedings of eumas’14.

Frank S., Listmann K., Haumann D., Willert V. (2010). Performance analysis for multi-robot exploration strategies. In proceedings of simpar’10, p. 399–410. Springer.

Grisetti G., Stachniss C., Burgard W. (2007). Improved techniques for grid mapping with rao-blackwellized particle filters. Robotics, IEEE Transactions on, vol. 23, no 1, p. 34–46.

Howard A. (2006, December). Multi-robot simultaneous localization and mapping using particle filters. The International Journal of Robotics Research, vol. 25, p. 1243-1256.

Lass R. N., Sultanik E. A., Regli W. C. (2009). Metrics for multiagent systems. In Performance evaluation and benchmarking of intelligent systems, p. 1–19. Springer.

Le V. T., Bouraqadi N., Stinckwich S., Moraru V., Doniec A. (2009, mai). Making networked robot connectivity-aware. In Proceedings of icra’09. Kobe, Japan.

Parker L. E. (2008). Multiple mobile robot systems. Springer Handbook of Robotics, p. 921–941.

Pobil A. P. del. (2006). Why do we need benchmarks in robotics research? In Proceedings of iros’06. Beijing, China.

Scrapper C., Madhavan R., Lakaemper R., Censi A., Godil A., Wagan A. et al. (2009). Quantitative assessment of robot-generated maps. In Performance evaluation and benchmarking of intelligent systems, p. 221–248. Springer.

Stachniss C. (2009). Robotic mapping and exploration. Springer.

Thrun S., Burgard W., Fox D. (2005). Probabilistic robotics. MIT press.

Yamauchi B. (1998). Frontier-based exploration using multiple robots. In Proceedings of agent’98.

Yan Z., Fabresse L., Laval J., Bouraqadi N. (2014, October). Team size optimization for multirobot exploration. In Proceedings of simpar’14. Bergamo, Italy.

Yan Z., Jouandeau N., Cherif A. A. (2013, December). A survey and analysis of multi-robot coordination. International Journal of Advanced Robotic Systems, vol. 10.

Zlot R., Stentz A., Dias M., Thayer S. (2002). Multi-robot exploration controlled by a market economy. In Proceedings of icra’02, vol. 3, p. 3016-3023.