Page History

...

VES-HV Collector

...

(Option 1)

HV-VES collector has been proposed, based on a need to process high-volumes of data generated frequently by a large number of NFs.  It uses plain TCP connections. Connections are stream-based (as opposed to request-based) and long running. Payload is binary-encoded (currently using Google Protocol Buffers). 

Pros: 

• Designed to support high volume of data with minimal latency
• HV-VES uses direct connection to DMaaP’s Kafka. 

Cons:

• Added dependency on HV-VES DCAE components

Kafka

...

Interfacing using DMaaP client

...

(Option 2)

Message router is added as an additional layer on top of DMaaP to support message service API interaction with the ZooKeeper/Kafka. DmaapClient is a deliverable jar that can be used to interact with the DMaaP message Router api.

Pros:

• Designed to support REST calls to Kafka from both publishers and consumers.
• Pre Defined APIs in Message Router to create/view/delete a topic in Kafka and also to publish a message to a topic and subscribe to a topic.

Cons:

• Additional overhead as an additional layer Message Router would be added between CPS and Kafka. 

Kafka Direct interface without using DMaaP client: To be used in CPS (Option 3)

Pros:

• No additional layer used between CPS and DMaaP Kafka. 
• Spring boot enables easier configuration.
• CPS can make a direct interface with Kafka using spring-kafka. Spring-kafka used in CPS also provides support for Message-driven POJOs for publishing and subscribing events. CPS does not require message service API for interacting with Kafka.

Kafka configuration details needs to be added in the application yaml of both publisher(cps-core) and consumer(cps-temporal) of the events published to Kafka. These configuration should preferably be defined in application-helm.yaml included in the OOM charts to provide flexibility while deploying the application. 

Based on the encryption and authentication mechanism used, the required configurations could change. Hence it is suggested to use override files for configuring the required values according to the used environment.

Encryption and Authentication Listener Configuration

Supported security protocols are :

1.PLAINTEXT : Listener without any encryption or authentication. CPS application by default is configured to use PLAINTEXT both with testcontainers and docker-compose.

kafka:
    bootstrap-servers: ${KAFKA_BOOTSTRAP_SERVER}
    security:
        protocol: PLAINTEXT
    # to be added only in cps-core(producer)
    producer:
        client-id: cps-core
        value-serializer: org.springframework.kafka.support.serializer.JsonSerializer
    # to be added only in cps-temporal(consumer)

...

Kafka configuration details needs to be added in the application yaml of both publisher(cps-core) and consumer(cps-temporal) of the events published to Kafka. These configuration should preferably be defined in application-helm.yaml included in the OOM charts to provide flexibility while deploying the application. 

Based on the encryption and authentication mechanism used, the required configurations could change. Hence it is suggested to use override files for configuring the required values according to the used environment.

Encryption and Authentication Listener Configuration

Supported security protocols are :

1.PLAINTEXT : Listener without any encryption or authentication. CPS application by default is configured to use PLAINTEXT both with testcontainers and docker-compose.

kafka:
    bootstrap-servers: ${KAFKA_BOOTSTRAP_SERVER}
    security:
        protocol: PLAINTEXT
    # to be added only in cps-core(producer)
    producer:
        client-id: cps-core
        value-serializer: org.springframework.kafka.support.serializer.JsonSerializer
    # to be added only in cps-temporal(consumer)
    consumer:
        group-id: ${KAFKA_CONSUMER_GROUP_ID:cps-temporal-group}
        client-id: cps-temporal
        # Configures the Spring Kafka ErrorHandlingDeserializer that delegates to the 'real' deserializers
        # See https://docs.spring.io/spring-kafka/docs/2.5.11.RELEASE/reference/html/#error-handling-deserializer
        # and https://www.confluent.io/blog/spring-kafka-can-your-kafka-consumers-handle-a-poison-pill/
        key-deserializer: org.springframework.kafka.support.serializer.ErrorHandlingDeserializer
        value-deserializer: org.springframework.kafka.support.serializer.ErrorHandlingDeserializer
        properties:
             spring.deserializer.key.delegate.class: org.apache.kafka.common.serialization.StringDeserializer
             spring.deserializer.value.delegate.class: org.springframework.kafka.support.serializer.JsonDeserializer
             spring.json.value.default.type: org.onap.cps.event.model.CpsDataUpdatedEvent

...

spring:
    kafka:
        bootstrap-servers: 172.16.3.38:30490
        security:
            protocol: SASL_PLAINTEXT
        properties:
            sasl.mechanism: PLAIN
            sasl.jaas.config: org.apache.kafka.common.security.plain.PlainLoginModule required username=admin password=admin_secret;
            ssl.endpoint.identification.algorithm:
        consumer:
            group-id: ${KAFKA_CONSUMER_GROUP_ID:cps-temporal-group}
            # Configures the Spring Kafka ErrorHandlingDeserializer that delegates to the 'real' deserializers
            # See https://docs.spring.io/spring-kafka/docs/2.5.11.RELEASE/reference/html/#error-handling-deserializer
            # and https://www.confluent.io/blog/spring-kafka-can-your-kafka-consumers-handle-a-poison-pill/
            key-deserializer: org.springframework.kafka.support.serializer.ErrorHandlingDeserializer
            value-deserializer: org.springframework.kafka.support.serializer.ErrorHandlingDeserializer
            properties:
                spring.deserializer.key.delegate.class: org.apache.kafka.common.serialization.StringDeserializer
                spring.deserializer.value.delegate.class: org.springframework.kafka.support.serializer.JsonDeserializer
                spring.json.value.default.type: org.onap.cps.event.model.CpsDataUpdatedEvent

app:
    kafka:
        consumer:
            topic: ${KAFKA_CONSUMER_TOPIC:cps.cfg-state-events}KAFKA_CONSUMER_TOPIC:cps.cfg-state-events}

Note: AAF integration is not included in this documentation as there is already a Jira to handle the integration CPS-281 which is still under discussion.