Beyond Corroboration: Strengthening Model Validation by Looking for Unexpected Patterns

diversityNew Paper, in PLoS ONE


Chérel G., Cottineau C., Reuillon R., 2015, « Beyond Corroboration: Strengthening Model Validation by Looking for Unexpected Patterns », PLoS ONE 10(9): e0138212. doi:10.1371/journal.pone.0138212

Abstract. Models of emergent phenomena are designed to provide an explanation to global-scale phenomena from local-scale processes. Model validation is commonly done by verifying that the model is able to reproduce the patterns to be explained. We argue that robust validation must not only be based on corroboration, but also on attempting to falsify the model, i.e. making sure that the model behaves soundly for any reasonable input and parameter values. We propose an open-ended evolutionary method based on Novelty Search to look for the diverse patterns a model can produce. The Pattern Space Exploration method was tested on a model of collective motion and compared to three common a priori sampling experiment designs. The method successfully discovered all known qualitatively different kinds of collective motion, and performed much better than the a priori sampling methods. The method was then applied to a case study of city system dynamics to explore the model’s predicted values of city hierarchisation and population growth. This case study showed that the method can provide insights on potential predictive scenarios as well as falsifiers of the model when the simulated dynamics are highly unrealistic.

Key-words. Simulation and Modelling, Evolutionary Algorithm, Population Growth, Space Exploration, Urbanisation, Complex Systems.

Multi-agent modeling of urban growth distribution in systems of cities

A strong regularity in urban systems has long been identified : the hierarchical distribution of city sizes. Moreover, a closer observation of the evolution of this distribution shows that in the majority of city systems, there is a trend towards a more and more unequal distribution of city sizes. Why does the majority of urban systems show those strong regularities? What are the common growth processes involved? Several dynamic growth models have been proposed but no consensus has yet been reached because of the under-determination of models by those empirical laws. In this presentation we describe a new method of agent-based parsimonious modeling that we think can contribute to the identification of the common urban growth processes. This modeling method is based  on  intensive model exploration for quantitative evaluation of implemented mechanisms. The exploration tools were first developed for the evaluation of SimpopLocal, a model of the organization of urban systems when cities first emerged. The use of those exploration tools was then generalized into a modeling method that was applied for the first time with the construction of the MARIUS family of models which aims at reproducing the evolution of Soviet urbanisation between 1959 and 1989. Those two examples show how this new modeling method can help the construction of urban theories by helping the evaluation of assumptions made on urban processes.

[gview file= »http://geodivercity.parisgeo.cnrs.fr/blog/wp-content/uploads/2015/01/prez-QuantUrb.pdf »]

Communication at the seminar Quanturb, ISC-PIF (Paris), November 19th.

Clara Schmitt and Paul Chapron

OpenMOLE 1.0

 It’s finally there!

After 6 years of hard work we made it to 1.0 (Heroic Hippo). It combines stability and ergonomic interfaces to provide the best model exploration tool ever made. OpenMOLE is a workflow engine designed to leverage the computing power of distributed execution environments for naturally parallel processes. A process is said naturally parallel if the same computation runs many times for a set of different inputs, such as model experiment or data processing…

We are very proud of this success, that has been made possible thanks to the effort and support of a dynamic community of developers, users and also several academic funding.

This version has mostly focus on stabilization, with some novelties nevertheless:
* the distributed computing feature of OpenMOLE are now very robust and efficient,
* the evolutionary algorithm part has matured and is usable to explore models from the console API,
* a new distributed environments (SLURM) is now available.

In the next versions:
* the graphical interface will be multi-user, web based and run in the browser,
* new environments will be provided (Condor, SGE),
* advanced methods such as genetic algorithm will be usable from the graphical interface,
* it will be possible to generate console scripts from a graphical workflow,
* bootstrapping methods that quickly and automatically embed your models in OpenMOLE will be implemented.

The OpenMOLE team

Intensive computing for the Social Sciences

Workshop organized around Intensive Computing in Social Sciences at the Complex Systems Institute in Paris, May 21st, from 10:00 am to 5:00 pm, 113 rue Nationale, Paris 13th, France.

Free but necessary registration : arnaud.banos@parisgeo.cnrs.fr

 

Preliminary Programme

> 10:00-10:30. Introduction : Patrice Bourdelais, Head of INSHS, CNRS

Presentation of the ERC GeoDiverCity : Denise Pumain, UMR Géographie-Cités

Presentation of the Complex Systems Institute : David Chavalarias, UMR CAMS

> 10:30- 11:30. OpenMOLE and its uses in Social Sciences

The OpenMOLE plateform, presentation and tutorial : Romain Reuillon and Mathieu Leclaire, ERC GeoDiverCity / ISC-PIF

A collective and interdisciplininary approach of geographical modeling around OpenMOLE : Paul Chapron, Clémentine Cottineau, Sébastien Rey Coyrehourcq, Clara Schmitt, ERC GeoDiverCity / UMR Géographie-Cités

> 11:30-12:30. Questions and discussion

> 2:00-4:00. « Regards croisés »

Big data and intensive computing : The Social Sciences in revolution ? : David Chavalarias, ISC-PIF

Equipex Matrice : Denis Peschanski, CNRS/Université Paris 1 (to be confirmed)

The « Run My Code » experiment, Christophe Perignon, HEC Paris (to be confirmed)

The MEXICO network : Robert Faivre, INRA

The interdisciplinary MAPS network : Marion Amalric, UMR CITERES

The SimTools network : Patrick Taillandier, UMR Idées

The research group GDR Modys : Xavier Rodier, UMR CITERES

> 4:00-5:00. Discussion : Modeling and intensive computing in Social Sciences

Michel Audiffren (GIS Réseau National des MSH), David Chavalarias (ISC-PIF), Guillaume Deffuant (IRSTEA), Catherine Garbay (IMAG), Michel Gollac (CREST), Thérèse Libourel (LIRMM), Anne Ruas (IFFSTAR), Françoise Thibault (Alliance Athena), Hervé Zwirn (CVT Alliance Athéna)

GeoDiverCity at AAG Annual Meeting 2014 in Tampa, Florida

Various contributions of GeoDiverCity team are scheduled for the Annual Meeting of the Association of American Geographers in Tampa, Florida :

Tuesday, 4/8/2014

> Session : 1654 European Research Council – Top European grants for brilliant minds from across the world, from 4:40 PM – 6:20 PM in Grand Salon C, Marriott, Second Floor. Organizer : Katja Meinke.

17:15-17:30    Denise Pumain, « ERC from an Advanced Grantee’s perspective. »

Wednesday, 4/9/2014

> Session : 2268 Urban systems and scaling laws: Functional diversity and urban economic trajectories, from 10:00 AM – 11:40 AM in Meeting Room 1, Marriott, Second Floor. Organizer : Céline Vacchiani-Marcuzzo.

10:00-10:20    Elfie Swerts, Céline Vacchiani-Marcuzzo, Fabien Paulus, « Scaling laws as a tool for characterising the functional evolution in urban systems »

10:20-10:40    Olivier Finance, « Transnational firms in the French system of cities and scaling laws »

> Session : 2239 Geosimulation Models 1: Methodological Advances, from 10:00 AM – 11:40 AM in Room 39, TCC, Fourth Floor. Organizers : Paul Torrens, Suzana Dragicevic, Andrew Crooks.

11:20-11:40    Mathieu Leclaire, Romain Reuillon, « Simpuzzle/Janet tools or how to build a step by step modular ABM ? »

> Session : 2539 Geosimulation Models 3 : Applications – Macro, from 2:40 PM – 4:20 PM in Room 39, TCC, Fourth Floor. Organizers : Paul Torrens, Suzana Dragicevic, Andrew Crooks.

14:00-14:20    Clémentine Cottineau, Paul Chapron, « Evaluation & Calibration for the comparison of ABMs of cities’ trajectories »

16:00-16:20  Denise Pumain, Clara Schmitt, Sébastien Rey-Coyrehourcq, Romain Reuillon, « Building and exploring an agent-based model with OpenMOLE »

GeoDiverCity open simulation platform

OpenMOLE has originally been developed in a generic way (and in particular with the cooperation of geographer modelers) to be scientific field independent. That is why dealing with geographical models within the Geodivercity program is straightforward. More specific features for geography will be added during the ERC depending on the needs of the modelers.

A first attempt is to build the Simprocess platform for the multi-level exploration of agent-based models of the Simpop type (PhD by Sébastien Rey-Coyrehourcq).

The next versions will include:

  •   new environments (remote servers through ssh, PBS clusters, cloud),
  •   a standardised serialisation format for workflows,
  •   an integration of cutting edge scientific method for model exploration (optimization, calibration, fitness landscape analysis, sensitivity analysis…),
  •   live visualization based on generic tools,
  •   many other good things!

http://www.openmole.org/

Romain Reuillon, Mathieu Leclaire, Sébastien Rey-Coyrehourcq, Hélène Mathian, Clara Schmitt and Arnaud Banos

The OpenMOLE software

OpenMOLE (version 0.5) is a piece of software for intensive scientific computing. OpenMOLE is the result of 4 years of daily work with model exploration issues in diverse fields (Human Sciences, Physics, Geography, Food-processing,…). It is 100% FOSS (Free and Open Source Software), 100% written in Scala.

It targets modelers who explore their models at wide scales. It makes it possible to generate automatically wide design of experiments (full-factorial, LHS, Saltelli…) and to take advantage of the computing power of massively parallel execution environments such as computing grids and clusters. It exposes a workflow formalism to design parallel processings in a natural way.

Strengths of OpenMOLE:

  •  a zero-deployment approach: the programs (C, C++, Python, Java, Scala, NetLogo, etc) are embedded at runtime and do not require installing software on execution machines,
  •  a small number of base concepts to handle (4):
    1. tasks (the executable components containing the model for instance);
    2. prototypes (types variables) which are transferred from one task to another;
    3. samplings (how to explore my model?);
    4. environments (where my jobs are executed?);
  •  a modular development for extending the platform with plug-ins in short time and based on OSGi,
  • an optimized and effortless access to grid resources (automatic resource discovery, eager submission, failure handling, management of data transfer …),
  •  a specific task to embed NetLogo models,
  •  a formal workflow validation prior to the execution based on a strongly typed dataflow,
  •  its scalability: it manages  up to hundreds of thousands of task executions from a laptop;
  •  a scripted interface as well as an ergonomic graphical user interface.

http://www.openmole.org/

Mathieu Leclaire and Romain Reuillon