A New Method to Evaluate Simulation Models: The Calibration Profile (CP) Algorithm

New publication, in JASSS : Journal of Artificial Societies and Social Simulation

Reuillon R., Schmitt C., De Aldama R., Mouret J.-B., 2015, « A New Method to Evaluate Simulation Models: The Calibration Profile (CP) Algorithm », JASSS : Journal of Artificial Societies and Social Simulation, Vol. 18, Issue 1, http://jasss.soc.surrey.ac.uk/18/1/12.html

Abstract. Models of social systems generally contain free parameters that cannot be evaluated directly from data. A calibration phase is therefore necessary to assess the capacity of the model to produce the expected dynamics. However, despite the high computational cost of this calibration it doesn’t produce a global picture of the relationship between the parameter space and the behaviour space of the model. The Calibration Profile (CP) algorithm is an innovative method extending the concept of automated calibration processes. It computes a profile that depicts the effect of each single parameter on the model behaviour, independently from the others. A 2-dimensional graph is thus produced exposing the impact of the parameter under study on the capacity of the model to produce expected dynamics. The first part of this paper is devoted to the formal description of the CP algorithm. In the second part,we apply it to an agent based geographical model (SimpopLocal). The analysis of the results brings to light novel insights on the model.

Key-words. Calibration Profile, Model Evaluation

Multilevel comparison of large urban systems

New publication, in Cybergeo, European Journal of Geography

Pumain D., Swerts E., Cottineau C., Vacchiani-Marcuzzo C., Ignazzi C.A., Bretagnolle A.,  Delisle F., Cura R., Lizzi L., Baffi S., 2015, « Multilevel comparison of large urban systems », Cybergeo : European Journal of Geography [En ligne], Systèmes, Modélisation, Géostatistiques, document 706, URL : http://cybergeo.revues.org/26730 ; DOI : 10.4000/cybergeo.26730

Abstract. For the first time the systems of cities in seven countries or regions among the largest in the world are made comparable through the building of spatio-temporally standardised statistical databases. We first explain the concept of a generic evolutionary urban unit (“city”) and its necessary adaptations to the information provided by each national statistical system. Second, the hierarchical structure and the urban growth process are compared at macro-scale for the seven countries with reference to Zipf’s and Gibrat’s model: in agreement with an evolutionary theory of urban systems, large similarities shape the hierarchical structure and growth processes in BRICS countries as well as in Europe and United States, despite their positions at different stages in the urban transition that explain some structural peculiarities. Third, the individual trajectories of some10,000 cities are mapped at micro-scale following a cluster analysis of their evolution over the last fifty years. A few common principles extracted from the evolutionary theory of urban systems can explain the diversity of these trajectories, including a specific pattern in their geographical repartition in the Chinese case. We conclude that the observations at macro-level when summarized as stylised facts can help in designing simulation models of urban systems whereas the urban trajectories identified at mico-level are consistent enough for constituting the basis of plausible future population projections.

Key-words. Urban systems, Zipf, Gibrat, Cities trajectories, BRICS

Half a billion simulations: Evolutionary algorithms and distributed computing for calibrating the SimpopLocal geographical model

New publication, in Environment and Planning B.

Schmitt C, Rey-Coyrehourcq S, Reuillon R, Pumain D, 2015, “Half a billion simulations: evolutionary algorithms and distributed computing for calibrating the SimpopLocal geographical model” Environment and Planning B: Planning and Design, 42(2), 300-315. http://www.envplan.com/abstract.cgi?id=b130064p

Abstract. Multiagent geographical models integrate very large numbers of spatial interactions. In order to validate these models a large amount of computing is necessary for their simulation and calibration. Here a new data-processing chain, including an automated calibration procedure, is tested on a computational grid using evolutionary algorithms. This is applied for the first time to a geographical model designed to simulate the evolution of an early urban settlement system. The method enables us to reduce the computing time and provides robust results. Using this method, we identify several parameter settings that minimize three objective functions that quantify how closely the model results match a reference pattern. As the values of each parameter in different settings are very close, this estimation considerably reduces the initial possible domain of variation of the parameters. Thus the model is a useful tool for further multiple applications in empirical historical situations.

Keywords: simulation model, multiagent system, calibration, evolutionary algorithm, geographical modelling, high-performance computing, model validation

Multi-agent modeling of urban growth distribution in systems of cities

A strong regularity in urban systems has long been identified : the hierarchical distribution of city sizes. Moreover, a closer observation of the evolution of this distribution shows that in the majority of city systems, there is a trend towards a more and more unequal distribution of city sizes. Why does the majority of urban systems show those strong regularities? What are the common growth processes involved? Several dynamic growth models have been proposed but no consensus has yet been reached because of the under-determination of models by those empirical laws. In this presentation we describe a new method of agent-based parsimonious modeling that we think can contribute to the identification of the common urban growth processes. This modeling method is based  on  intensive model exploration for quantitative evaluation of implemented mechanisms. The exploration tools were first developed for the evaluation of SimpopLocal, a model of the organization of urban systems when cities first emerged. The use of those exploration tools was then generalized into a modeling method that was applied for the first time with the construction of the MARIUS family of models which aims at reproducing the evolution of Soviet urbanisation between 1959 and 1989. Those two examples show how this new modeling method can help the construction of urban theories by helping the evaluation of assumptions made on urban processes.

Download (PDF, 3.51MB)

Communication at the seminar Quanturb, ISC-PIF (Paris), November 19th.

Clara Schmitt and Paul Chapron

Selective investments of foreign transnational firms in the French system of cities revealed by scaling laws

Many economic links connect the cities embedded in systems of cities (for example commercial links, financial ownership links…). They participate in the intrinsic complexity of these systems. The interactions and connections between places due to the economic stakeholders can substantially impact the shape and the dynamics of any system of cities. Especially, networks built up by transnational firms by the ownership of subsidiaries located beyond their territorial borders into foreign systems of cities, could shape the future of related cities. These foreign investors can provide these cities new jobs by the creation or extension of establishments but also sometimes weaken them through a massive control on their total employment, transnational investments being expected to be more volatile than national investments. A new source of data providing interesting insights on the effective location of these impacts on French cities is now available (see box below).

An original database

Three data bases were combined to assign the establishments (physical location of production, selling points) controlled by foreign capital in French cities (=“aires urbaines” i.e. SMAs):

  • the ORBIS data base (produced by Bureau van Dijk, augmented by Céline Rozenblat – UNIL – GeoDiverCity) that contains all financial linkages between companies into the 3,000 major groups worldwide,

  • the LIFI data base (produced by INSEE), a French data base similar to the previous one yet extended to smaller groups,

  • the CLAP data base (also produced by INSEE), to add the level of establishments (the most suitable one for our geographical approach, whereas ORBIS and LIFI are limited to companies).


A strong hierarchical effect…

Our first analyses outline the high concentration of foreign-controlled employment in the biggest cities of the system (figure 1). We could expect this result by observing these networks at the level of the companies, as it is known that their headquarters are more concentrated than their physical locations. Yet even when these jobs are located at the level of establishments, the foreign-controlled activities scale superlinearly (as indicated by the scaling law exponent higher than 1). Foreign-controlled employment is consequently much more concentrated in largest cities. The distribution of these establishments is not proportional to the size of the cities, as it benefits first and foremost to the biggest cities of the system.

Figure 1: Important role of the urban hierarchy in the distribution of foreign-controlled jobs in the French system of cities


and regional effects

Taking into account the power law relationship and the quality of fit which remains nevertheless medium, another major trend appear in our first analysis. The amount of foreign-controlled jobs in the smallest cities is highly variable, some of them being deeply invested by the transnational firms while some others are almost avoided. A more detailed study of this variability show strong regional effects in addition to the hierarchical ones (figure 2). Cities with higher amounts of foreign-controlled jobs than expected by the scaling law are almost always located in Northern and Eastern parts of the country or located near Paris; cities where this amount is lower than expected are located in Southern and Western parts of the country. Thus foreign investors adapt to former spatial trends of urban functional specialization in France.

Figure 2: Regional effects in the distribution of foreign-controlled jobs within the smallest cities of the system


Further analyses

As urban hierarchy doesn’t explain the distribution on its own, further analyses will investigate the effects of the economic specialization of cities, the closeness to bigger cities, or the shape of the networks where these cities are embedded or not. Our intention is to formulate some stylized facts identifying emerging properties and network dynamics that characterize the distribution of the foreign transnational firms into the French system of cities at an unprecedented level of detail. Besides classical multivariate analysis and networks analysis, scaling laws will constitute one of our main analysis tools, as they shed new light on the connection between urban functions, city sizes and economic innovation cycles (Pumain et al., 2006). This thesis, part of the GeoDiverCity program, will contribute an approach historicizing the scaling laws parameters relating them to the innovation cycles and the hierarchical diffusion of innovations theory in the system of cities.

Olivier Finance, PhD student under the supervision of Denise Pumain and Fabien Paulus.

ABM “MARIUS” replicates the evolutive structure of the former Soviet system of cities

MARIUS is a model of urban evolution in the post-Soviet space. It aims at simulating the socio-economic evolution of cities, the hierarchical and functional structuration of the geographical system resulting from their interactions at macro level, and their diversified trajectories over several decades at micro-level, while being as parsimonious as possible. For this purpose, we tackled the evaluation process alongside model-building, and prepared a path for progressively complexifying the model’s structure based on statistical findings 1. We start from a generic model that would suit other systems of cities and add mechanisms specific to the post-Soviet space when necessary (that is, if the generic mechanisms prove unable to match the observations).

The simplest model of urban evolution that satisfies criteria of proximity to data and realism of micro-dynamics is composed of a restricted set of mechanisms 2 :

– scaling laws for modeling the inequalities of production and consumption capacities in cities according to their size;

– a gravity model computing the interaction potentials of pairs of cities on an abstract market (summarizing exchanges of diversified goods, services, information and capital);

– an advantage modelled for cities exchanging large quantities of value with a large pool of partners, representing the spillovers of information and technology that are associated to interactions;

– a fixed cost for each transaction that constrains the potential network of cities’ interaction according to the profitability of each potential exchange;

– a mechanism of conversion between gains in wealth and evolution of population in the city that is non linear with city size.

The calibrated model that produces the best results is obtained after intensive parameter search with genetic algorithms (100 000 generations) distributed on a computer grid with the OpenMOLE 3 software.

Capture d’écran 2015-01-09 à 18.18.56Source : DARIUS 2014 4. GibratSimulator 5. Cities in the graphs are sorted by size. (each point represents one urban agglomeration)

This figure demonstrates that we were able to simulate with this model the degree and trend of hierarchization process among the 1145 cities of the Soviet Union from 1959 to 1989, much better than using a classical model (Gibrat’s) and by including strong interaction hypotheses (yet with 8 parameters instead of 2 in Gibrat’case). Moreover, this model results from a trial and error process of examining the model outputs, evaluating them against empirical data and theoretical findings, refining the evaluation criteria and finally complexifying the model in order to meet the requirements of this new evaluation method.

It would be unrealistic to pretend using the model for reconstructing the evolution during the last two decades that followed rather unpredictable political events. But it is likely that once having introduced exogeneously in the model the corresponding  bifurcation in economic rules, the model would help predicting the probable evolution of cities in countries of former Soviet Union in a rather accurate way.

Clémentine Cottineau, Paul Chapron and Romain Reuillon.


  1. Cottineau C., 2014, L’évolution des villes dans l’espace post-soviétique. Observation et modélisations, Doctoral dissertation, University Paris 1 Panthéon-Sorbonne.
  2. For a detailed description of the mechanisms’ implementation, see the ODD protocole of MARIUS and the source-code here : https://forge.iscpif.fr/projects/marius
  3. www.openmole.org
  4. Cottineau, Clémentine (2014): DARIUS Database. figshare. http://dx.doi.org/10.6084/m9.figshare.1108081
  5. Cura, Robin (2014) : http://shiny.parisgeo.cnrs.fr/gibrat/

OpenMOLE 1.0

 It’s finally there!

After 6 years of hard work we made it to 1.0 (Heroic Hippo). It combines stability and ergonomic interfaces to provide the best model exploration tool ever made. OpenMOLE is a workflow engine designed to leverage the computing power of distributed execution environments for naturally parallel processes. A process is said naturally parallel if the same computation runs many times for a set of different inputs, such as model experiment or data processing…

We are very proud of this success, that has been made possible thanks to the effort and support of a dynamic community of developers, users and also several academic funding.

This version has mostly focus on stabilization, with some novelties nevertheless:
* the distributed computing feature of OpenMOLE are now very robust and efficient,
* the evolutionary algorithm part has matured and is usable to explore models from the console API,
* a new distributed environments (SLURM) is now available.

In the next versions:
* the graphical interface will be multi-user, web based and run in the browser,
* new environments will be provided (Condor, SGE),
* advanced methods such as genetic algorithm will be usable from the graphical interface,
* it will be possible to generate console scripts from a graphical workflow,
* bootstrapping methods that quickly and automatically embed your models in OpenMOLE will be implemented.

The OpenMOLE team

European consultation on Global Systems Science

The FET (Future and Emergent Technologies) Unit of the European Commission proposes an online consultation to understand what the game-changing technologies of the next decades will be. It targets scientists, researchers, innovators, entrepreneurs and members of civil society in general and is open till 15 June 2014.

A particular consultation on “Global Systems Science” is launched with the question : “how can science support policy making on global challenges?”.

More details :