Conflation of expert and crowd reference data to validate global binary thematic maps

With the unprecedented availability of satellite data and the rise of global binary maps, the collection of shared reference data sets should be fostered to allow systematic product benchmarking and validation. Authoritative global reference data are generally collected by experts with regional knowledge through photo-interpretation. During the last decade, crowdsourcing has emerged as an attractive alternative for rapid and relatively cheap data collection, beckoning the increasingly relevant question: can these two data sources be combined to validate thematic maps? In this article, we compared expert and crowd data and assessed their relative agreement for cropland identification, a land cover class often reported as difficult to map. Results indicate that observations from experts and volunteers could be partially conflated provided that several consistency checks are performed. We propose that conflation, i.e., replacement and augmentation of expert observations by crowdsourced observations, should be carried out both at the sampling and data analytics levels. The latter allows to evaluate the reliability of crowdsourced observations and to decide whether they should be conflated or discarded. We demonstrate that the standard deviation of crowdsourced contributions is a simple yet robust indicator of reliability which can effectively inform conflation. Following this criterion, we found that 70% of the expert observations could be crowdsourced with little to no effect on accuracy estimates, allowing a strategic reallocation of the spared expert effort to increase the reliability of the remaining 30% at no additional cost. Finally, we provide a collection of evidence-based recommendations for future hybrid reference data collection campaigns.

WALDNER Francois;  SCHUCKNECHT Anne;  LESIV Myroslava;  GALLEGO PINILLA Francisco;  SEE Linda;  PEREZ HOYOS Ana;  D'ANDRIMONT Raphael;  DE MAET Thomas;  LASO BAYAS Juan Carlos;  FRITZ Steffen;  LEO Olivier;  KERDILES Herve;  DIEZ Monica;  VAN TRICHT Kristof;  GILLIAMS Sven;  SHELESTOV Andrii;  LAVRENIUK Mykola;  SIMOES Marthareth;  FERRAZ Rodrigo;  BELLON Beatriz;  BEGUE Agnes;  HAZEU Gerard;  STONACEK Vaclav;  KOLOMAZNIK Jan;  MISUREC Jan;  VERON Santiago R;  DE ABBELLEYRA Diego;  PLOTNIKOV Dmityr;  MINGYONG Li;  SINGHA Mrinal;  PATIL Prashant;  ZHANG Miao;  DEFOURNEY Pierre; 

2018-12-11

ELSEVIER SCIENCE INC

JRC114258

0034-4257 (online),   

https://www.sciencedirect.com/science/article/pii/S0034425718305017?via%3Dihub#s0125,    https://publications.jrc.ec.europa.eu/repository/handle/JRC114258,   

10.1016/j.rse.2018.10.039 (online),