Vodafone, Europe’s leading multinational telecommunications company, undertook some years ago the migration of many of its services to Google Cloud Platform to co-develop new services and new functionalities in existing services.

One of Vodafone’s most widely used services for its data transformations is Cloud Data Fusion, a managed version of CDAP. However, one of the limitations of CDF/CDAP was the lack of a “push” type mechanism to receive near real time notifications about what is happening in each namespace.

The proposed solution allows CDF/CDAP to send these events in NRT to a Cloud Pub/Sub topic to which any service can subscribe to receive this data. CDAP/CDF now offers a NRT model to monitor the different pipelines’ executions along the different namespaces that a CDAP/CDF instance has, without pulling over and over the metrics API to get such information.