13th German Conference on Multiagent System Technologies (MATES 2015)
Measuring and Comparing Scalability of Agent-based Simulation Frameworks
Fabian Lorig, Nils Dammenhayn, David-Johannes Müller, and Ingo J. Timm
Nowadays, a variety of toolkits and frameworks exists providing functionalities for supporting implementation and execution of simulation experiments. Yet, the choice of a suitable framework is difficult as unforeseen scalability issues may arise when extending agent models. Therefore, this paper aims at providing a method for analyzing and comparing agent-based simulation frameworks regarding their ability to scale simulation models and experiments. Based on performance metrics, standardized experiments are conducted while altering internal and external scaling parameters. As part of the study, four Java-based agent frameworks are analyzed and compared: Aimpulse Spectrum, JADE, MASON, and Repast.
As a first important observation we can remark that Aimpulse Spectrum can handle a very high number of agents. Experiments comprising of more than 10 million agents can be performed within less than 15 minutes using current standard hardware. Due to limitations of the other frameworks and for achieving comparability between the frameworks, experiments exceeding 1 million agents were not conducted.
Summarizing, [...] it can be said that Aimpulse Spectrum is the most performant framework regarding the different VM settings considered in this research. When instantiating 100.000 agents using 16 CPU cores, Aimpulse Spectrum is 100 times faster than JADE and about 15 times faster than MASON or Repast. While the execution time of Aimpulse Spectrum has not even doubled when increasing the amount of agents from 50.000 to 100.000, it has quadrupled for the other frameworks.
Reference: Lorig, F., Dammenhayn, N., Müller, D.-J., and Timm, I.J. (2015). Measuring and Comparing Scalability of Agent-based Simulation Frameworks. In Müller, J. P., Kaminka, G., Ketter, W., and Wagner, G. (eds.): 13th German Conference on Multiagent System Technologies (MATES 2015), Cottbus, Germany, September 28-October 2, 2015. LNCS 9433, Springer-Verlag, pp. 42-60.