Background
Developers of ONAP software components may consider utilizing the DMaaP services. Those software components are typically referred to as "publishers" (i.e. they introduce or produce the data) and "subscribers" (i.e. they read or consume the data), but collectively are referred to as "clients" of the DMaaP services. This article is intended to provide some guidance to developers of DMaaP clients.
Data Movement as a Platform, or "DMaaP" for short, is a nice abstraction that typically shows up as a single box on an architecture slide. But in reality, a developer of an ONAP application considering use of this Platform Service needs to know the details. The first decision for an ONAP application developer is whether they want to use DMaaP services or not. Assuming the choice to use DMaaP services, then an app developer needs to decide between Message Router and Data Router as the transport protocol. At its simplest, this is the difference between "small" messages distributed via a Topic versus "large" files distributed via a Feed, but there are other considerations that will be discussed.
Before an app can use a DMaaP, the service must be properly provisioned. We will discuss how this can be implemented as part of an ONAP deployment (to avoid manual steps).
And then, finally, data movement can occur between endpoints. Examples and conventions will be discussed.
Design Choices
Use DMaaP or not?
As our application components have become more granular due to the trends of microservice architectures and containers, there is a growing need to transport data between components. DMaaP is intended to standardize the implementation of this transport so that developers don't have to re-invent approaches, and can rely on a robust, feature rich platform service. At the same time, this approach does introduce some overhead, and subjects the client to resource constraints of a shared service.
Choosing between MR and DR
Some considerations for choosing between Message Router and Data Router:
| Attribute | Message Router | Data Router | Comments |
|---|---|---|---|
| Payload description | frequent, small (~KBs) messages | periodic, large sets of records (files) | |
| Max payload size | 1 MB | x GB | |
| Authentication Model | AAF Credentials via Basic Authorization | Local credentials via Basic Authentication | |
| Persistence | Yes (kafka data store on PV) | Yes (Linux file on PV) | |
| High Availability | Using ZK ensemble | Using horizontal scaling | |
| Data ordering | Possible | Not guaranteed | |
| Subscriber implementation | HTTP Client | HTTP Server | NOTE: the HTTP Server implementation implies the installation of a server SSL certificate. |
Some ways MR and DR are the same:
| Attribute | Message Router | Data Router | Comments |
|---|---|---|---|
| Bus Architecture | Yes | Yes | Both use "publisher" and "subscriber" terminology for end points |
| HTTP/HTTPS protocol | Yes | Yes | |
| Data Expiration | Yes | Yes | NOTE: different default values. MR: 7 days. DR: 24 hours. |
| Publisher implementation | HTTP Client | HTTP Client |
DMaaP Provisioning
Provisioning for Message Router Clients
The provisioning of Authenticated Topics and the subsequent publish/subscribe actions on that Authenticated Topic rely on the proper AAF Permissions to be granted to various Identities.
The following Identities are referenced:
- IdentityMR - the AAF Identity used by Message Router to access AAF and perform authorization checks.
- IdentityTopicMgr - the AAF Identity used by Bus Controller to access AAF and create Permissions and Roles, and grant those Permissions to Roles and Identities.
- IdentityPub - the AAF Identity used by the publisher of the Topic when accessing the MR API. The fully qualified Identity typically belongs to a namespace related to the application. i.e. IdentityPub@NamespacePub
- IdentitySub - the AAF Identity used by the subscriber of the Topic when accessing the MR API. The fully qualified Identity typically belongs to a namespace related to the application. i.e. IdentitySub@NamespaceSub
Pre-requisites:
- Identities created in AAF
- Credentials for Identities available in kubernetes values (probably as Secrets)
- IdentityTopicMgr has access permission granted for all Application Namespaces
4 Comments
Chris Lott
Please add recommendations for client software to poll & get data from a MR topic. Does DMaaP provide an abstraction layer, or do Java developers just use Kafka's org.apache.kafka.kafka-clients library?
sunil unnava
If you are looking for alternatives for Kafka client library, you may consider DMaaP Message Router (REST service that can be used to do the topic create and pub/sub). Refer the documentation for the API.
David Bainbridge
Is it possible to use a native kafka client and receive messages that are published via DMaap from other components? Essentially, I am attempting to understand how to integrate using Kafka, defined topics, and defined data encodings (formats), i.e. make my client as simple and lightweight as possible and rely on defined formats not APIs for data exchange.
sunil unnava
Kafka Client can be used to read the messages from a topic, irrespective of how the messages are published to the topic.
If you are planning to use the authenticated topics(Topics that require AAF permissions to read and write data), use the DMaaP BusController as it auto creates the topic and the required AAF permissions.
If you are planning to use the unauthenticated topic you can use the MR REST API or Kafka Client API to create the topic.
If you are using the Kafka Client , you can use an existing serializer/deserializer or create a new serializer or deserializer based on your requirements. If you are using Message Router for pub/sub it supports the plain text and json formats.