Page History

...

In this project, we provide a systematic way to real-time ingest DMaaP data to permanent storage.

DataLake's goals are:

1. To provide Provide a systematic way to real-time ingest DMaaP data to Couchbase, a distributed document-oriented database, and Druid, a data store designed for reallow-time OLAP latency OLAP analytics.
2. To serve Serve as a common data storage for other ONAP components as well, with easy access.
3. To provide Provide data-access APIs and ways for ONAP components and external systems (e.g. BSS/OSS) to consume the data.
4. To provide Provide sophisticated and ready-to-use data analytics tools that are built on the data.

Architecture

...

• Provide admin REST API for configuration and topic management. A topic can be configured to be exported to which data stores, with Couchbase and Druid supported initially. We may will support more distributed databases in the future if needed.

• Provide SDC/Design time framework UI for management, making use of the above admin REST API.

...

• Monitor selected topics, real-time pull the data and insert it into Couchbase, one table for each topic, with the same table name as the topic name.

• Data types JSON, XML, and YAML are auto converted into native store  schema. We may support additional formats. Data not in these formats is stored as a single string. 

• Provide REST API for data query, while applications can access the data through native API as well.

• Couchbase supports Spark direct running on it, which allow complicate analytics tools to be built. We may will develop Spark analytics applications if needed.

...