Web-based interactive analysis and visualisation of Earth Observation data at petabyte scale

The Copernicus programme of the European Union with its fleet of Sentinel satellites for monitoring land, ocean, and atmosphere with applications from environment monitoring to emergency response is generating Terabytes of free and open data on a daily basis. The Joint Research Centre (JRC) of the European Commission has developed a prototype Joint Earth Observation Data and Processing platform (JEODPP) to enable its knowledge production units to process and analyse global geospatial data at Petabyte scale in support to EU policy needs. In the framework of a collaboration between CERN and JRC, the EOS distributed file system enables high data throughput between the processing nodes and the storage servers. The performance of the JEODPP is analysed on use-cases in the context of interactive and batch processing based on docker containerisation and managed by the HTCondor workload manager. Web-based interactive analysis and visualisation of the Copernicus data on the EOS repository is obtained via Jupyter Notebooks connected to distributed backend processing servers. The process distribution and real-time visualisation of the results on interactive maps is achieved via custom interactive widget like IPyLeaflet used in the Jupyter notebooks. This way the data analysis capability of the JEODPP can be shared with internal or remote user groups.

BURGER Armin;  SOILLE Pierre; 

2017-08-16

CERN

JRC105666

https://zenodo.org/record/254064/files/book-of-abstracts-2017-01-17__16-PDF.pdf,    https://publications.jrc.ec.europa.eu/repository/handle/JRC105666,   

10.5281/zenodo.254064,