Project Name:
- Proposed name for the project:
DMaaP – Data movement as a platform
- Proposed name for the repository:
com.att.dmaap
Project description:
DMaaP is a premier platform for high performing and cost effective data movement services that transports and processes data from any source to any target with the format, quality, security, and concurrency required to serve the business and customer needs.
DMaaP consists of three major functional areas:
1. Data Filtering - the data preprocessing at the edge via data analytics and compression to reduce the data size needed to be processed.
2. Data Transport - the transport of data intra & inter data centers as well as internal and external to AT&T. The transport will support both file base and message base data movement. The Data Transport process needs to provide the ability to move data from any system to any system with minimal latency, guaranteed delivery and highly available solution that supports a self-subscription model that lowers initial cost and improves time to market.
3. Data Processing - the low latency and high throughput data transformation, aggregation, and analysis. The processing will be elastically scalable and fault-tolerant across data centers. The Data processing needs to provide the ability to process both batch and near real-time data.
Scope:
The DMaaP project will include the following functionality:
- Pub/sub messaging metaphor to broaden data processing opportunities. (refers to the label of the rendezvous point as a “topic”)
- A single solution for most event distribution needs to support a range of environments.
- Packaged to support deployment configurations ranging from a single container to a cluster of hosts.
- Horizontal scalability: Add servers to the cluster to add capacity.
- Durability: Hardware failure in the cluster should not impact service, and messages should never be lost.
- Durability: Consumers should not lose messages if they experience downtime.
- High throughput: Consumers must be able to distribute topic load across multiple hosts in a cluster.
- Easy integration via RESTful HTTP API:
- Implements a RESTful HTTP API for provisioning
- Implements a RESTful HTTP API for message transactions (i.e. pub, sub)
- Implements a RESTful HTTP API for transaction metrics
- Optionally registers with supporting network locations services which will allow client connections to nearest end point. (when there are multiple deployments in a network)
- Client authentication and authorization models:
- Supports multiple authentication and authorization models;
- Default model is “insecure” (i.e. not authenticated)
- Model is associated per topic or per some group of topics.
- Optionally supports a model that relies on external service for client authentication and authorization.
- Optionally supports replication of messages for selected topics to another instance of DMaaP MR.
- Optionally supports different underlying message bus technologies, but makes this transparent to clients.
- Standardized topic names
- Topic registry and discovery
- Recover partially delivered messages from the point of failure
Architecture Alignment:
- How does this project fit into the rest of the ONAP Architecture?
DMaaP has four components:
- Message Router (MR) -Message Router is a reliable, high-volume pub/sub messaging service with a RESTful HTTP API. It is intended to be deployed by Platform Service providers so that it is available to Platform clients as a web service. The service is initially built over Apache Kafka, with some important additional features which focus on integration with other Platform services such as provisioning and authorization.
- Data Router (DR) - The Data Routing System project is intended to provide a common framework by which data producers can make data available to data consumers and a way for potential consumers to find feeds with the data they require. The interface to DR is exposed as a RESTful web service known as the DR Publishing and Delivery API
- Data Movement Director (DMD) - A client to DMaaP platform to publish & subscribe data.
- Data Bus Controller - Provisioning API of the Data Movement Platform.
- Message Router (MR) -
- How does this align with external standards/specifications?
- APIs/Interfaces
- Information/data models
- Are there dependencies with other open source projects?
- APIs/Interfaces
- Integration Testing
- etc.
Resources:
- Primary Contact Person - Ram Koya (ATT), Jack Murray(ATT)
- Names, gerrit IDs, and company affiliations of the committers :
- Names and affiliations of any other contributors
- Project Roles (include RACI chart, if applicable)
Other Information:
- link to seed code (if applicable)
- Vendor Neutral
- if the proposal is coming from an existing proprietary codebase, have you ensured that all proprietary trademarks, logos, product names, etc., have been removed?
- Meets Board policy (including IPR)
Use the above information to create a key project facts section on your project page
Key Project Facts
Project Name:
- JIRA project name:
- JIRA project prefix:
Repo name:
Lifecycle State:
Primary Contact:
Project Lead:
mailing list tag [Should match Jira Project Prefix]
Committers:
foo@bar.com
baz@qux.com
*Link to TSC approval:
Link to approval of additional submitters: