In the paper “Data Pipeline Architecture for Serverless Platform”, Chinmaya Dehury, Pelle Jakovits and Satish Narayana Srirama from the University of Tartu, Vasilis Tountopoulos and Giorgos Giotis from Athens Technology Center S.A., propose a novel data pipeline architecture for a serverless platform for providing an environment to develop applications that can be broken into independently deployable, schedulable, scalable, and reusable modules and efficiently manage the flow of data between different environments.
Some key conclusions are:
- The architecture uses and extends the TOSCA specification for data pipeline based serverless applications.
- Apache NiFi is used as the underlined technology and Ansible as the automation engine for the implementation of the life-cycle of the serverless components.
- Different TOSCA nodes are proposed for consuming, publishing, and transforming data, including the utilization of remote serverless functions for analytical tasks.
- Development of additional TOSCA nodes for a variety of functionalities, such as node type to handle the data movement through a secure channel, encrypting the data only in case of a multi-cloud environment, etc. are the part of the future development plan of the proposed architecture.
- Last but not least, a set of necessary TOSCA node types will be developed for the implementation of the Viarota – an app that provides travelers with personalized touristic scheduling paths according to a number of mobility, geographical and user-based criteria– based on the data pipeline approach in order to support this application scenario in effectively managing the involved data pipelines and their movement across different cloud-based environments. To this point, we must mention that the RADON framework will be also exploited to enhance the Viarota solution.
You can find the paper “Data Pipeline Architecture for Serverless Platform” here.
The work in the paper “Data Pipeline Architecture for Serverless Platform” is partially funded by the European Union’s Horizon 2020 research and innovation project RADON (825040)