Data integration in a modular and parallel grid-computing workflow

Abstract

In the past decades a wide range of complex processes have been developed to solve specific geospatial data integration problems. As a drawback these complex processes are often not sufficiently transferable and interoperable. We propose modularisation of the whole data integration process into reusable, exchangeable, and multi-purpose web services to overcome these drawbacks. Both a high-level split of the process into subsequent modules such as pre-processing and feature matching is discussed as well as another fine-granular split within these modules. Thereby complex integration problems can be addressed by chaining selected services as part of a geo-processing workflow. Parallelization is needed for processing massive amounts of data or complex algorithms. In this paper the two concepts of task and data parallelization are compared and examples for their usage are given. The presented work provides vector data integration within grid-computing workflows of the German Spatial Data Infrastructure Grid (SDI-Grid) project.