Page History

...

Expand

title	Click here to see results...

NUMBER OF PRODUCERS	TOTAL MESSAGES PROCESSED	DIFFERENCE BETWEEN ALL MESSAGES AND SENT TO HV-VES	AVERAGE PROCESSING TIME IN HV-VES WITHOUT ROUTING [ms]	AVERAGE LATENCY TO HV-VES OUTPUT WITH ROUTING [ms]	PEAK INCOMING DATA RATE [MB/s]	PEAK PROCESSING MESSAGE QUEUE SIZE	PEAK CPU LOAD [%]	PEAK MEMORY USAGE [GB]	RESULTS PRESENTED IN GRAFANA
2	180000	0	0.026	8.9	1.6	11	3	0.35
2	180000	0	0.026	11.3	1.6	23	3.2	0.35

NUMBER OF PRODUCERS	TOTAL MESSAGES PROCESSED	DIFFERENCE BETWEEN ALL MESSAGES AND SENT TO HV-VES	AVERAGE PROCESSING TIME IN HV-VES WITHOUT ROUTING [s]	AVERAGE LATENCY TO HV-VES OUTPUT WITH ROUTING [ms]	PEAK INCOMING DATA RATE [MB/s]	PEAK PROCESSING MESSAGE QUEUE SIZE	PEAK CPU LOAD [%]	PEAK MEMORY USAGE [GB]	RESULTS PRESENTED IN GRAFANA
4	360000	0	0.022	41.3	3.2	130	4	0.35
4	360000	0	0.023	40.7	3.2	370	4.2	0.35

NUMBER OF PRODUCERS	TOTAL MESSAGES PROCESSED	DIFFERENCE BETWEEN ALL MESSAGES AND SENT TO HV-VES	AVERAGE PROCESSING TIME IN HV-VES WITHOUT ROUTING [ms]	AVERAGE LATENCY TO HV-VES OUTPUT WITH ROUTING [ms]	PEAK INCOMING DATA RATE [MB/s]	PEAK PROCESSING MESSAGE QUEUE SIZE	PEAK CPU LOAD [%]	PEAK MEMORY USAGE [GB]	RESULTS PRESENTED IN GRAFANA
6	359317	683	6240	15965	4.8	4500	4.3	1
6	359217	783	6490	16834	4.8	4200	5.8	1

...

No DMaaP Kafka SetUp

Conditions

...

Install Kafka Docker on Kubernetes

(based on: ultimate-guide-to-installing-kafka-docker-on-kuber)

Create config maps

Config maps are required by zookeeper and kafka-broker deployments.

Code Block

title	Create kafka-config-map

kubectl -n onap create cm kafka-config-map --from-file=kafka_server_jaas.conf

Code Block

title	kafka_server_jaas.conf
collapse	true

KafkaServer {
   org.apache.kafka.common.security.plain.PlainLoginModule required
   username="admin"
   password="admin_secret"
   user_admin="admin_secret";
};
Client {
   org.apache.zookeeper.server.auth.DigestLoginModule required
   username="kafka"
   password="kafka_secret";
};

...

Code Block

title	Create zk-config-map

kubectl -n onap create cm zk-config-map --from-file=zk_server_jaas.conf

Code Block

title	zk_server_jaas.conf
collapse	true

Server {
       org.apache.zookeeper.server.auth.DigestLoginModule required
       user_kafka="kafka_secret";
};

Create deployments

Code Block

title	Create zookeeper deployment

kubectl -n onap create -f zookeeper.yml

Code Block

language	yml
title	zookeeper.yml
collapse	true

---
kind: Deployment
apiVersion: extensions/v1beta1
metadata:
  name: zookeeper-deployment-1
spec:
  template:
    metadata:
      labels:
        app: zookeeper-1
    spec:
      containers:
      - name: zoo1
        image: digitalwonderland/zookeeper
        volumeMounts:
        - name: config-volume
          mountPath: "/home/ubuntu/kafkadepl/myconfig/zk_server_jaas.conf"
          subPath: zk_server_jaas.conf
        ports:
        - containerPort: 2181
        env:
        - name: ZOOKEEPER_ID
          value: "1"
        - name: ZOOKEEPER_SERVER_1
          value: zoo1
        - name: KAFKA_OPTS
          value: "-Djava.security.auth.login.config=/home/ubuntu/kafkadepl/myconfig/zk_server_jaas.conf -Dzookeeper.kerberos.removeHostFromPrincipal=true -Dzookeeper.kerberos.removeRealmFromPrincipal=true -Dzookeeper.authProvider.1=org.apache.zookeeper.server.auth.SASLAuthenticationProvider -Dzookeeper.requireClientAuthScheme=sasl"
        - name: SERVER_JVMFLAGS
          value: "-Djava.security.auth.login.config=/home/ubuntu/kafkadepl/myconfig/zk_server_jaas.conf -Dzookeeper.skipACL=yes"
        - name: ZOOKEEPER_AUTHPROVIDER_1
          value: "org.apache.zookeeper.server.auth.SASLAuthenticationProvider"
        - name: ZOOKEEPER_REQUIRECLIENTAUTHSCHEME
          value: "SASL"
      volumes:
      - name: config-volume
        configMap:
          name: zk-config-map
---
apiVersion: v1
kind: Service
metadata:
  name: zoo1
  labels:
    app: zookeeper-1
spec:
  ports:
  - name: client
    port: 2181
    protocol: TCP
  - name: follower
    port: 2888
    protocol: TCP
  - name: leader
    port: 3888
    protocol: TCP
  selector:
    app: zookeeper-1

...

Code Block

title	Create kafka-service

kubectl -n onap create -f kafka-service.yml

Code Block

language	yml
title	kafka-service.yml
collapse	true

---
apiVersion: v1
kind: Service
metadata:
  name: kafka-service
  labels:
    name: kafka
spec:
  ports:
  - port: 9092
    name: kafka-port
    protocol: TCP
  selector:
    app: kafka
    id: "0"
  type: LoadBalancer

...

Code Block

title	Create kafka-broker

kubectl -n onap create -f kafka-broker.yml

Code Block

language	yml
title	kafka-broker.yml
collapse	true

---
kind: Deployment
apiVersion: extensions/v1beta1
metadata:
  name: kafka-broker0
spec:
  template:
    metadata:
      labels:
        app: kafka
        id: "0"
    spec:
      containers:
      - name: kafka
        image: wurstmeister/kafka
        volumeMounts:
        - name: config-volume
          mountPath: "/home/ubuntu/kafkadepl/myconfig/kafka_server_jaas.conf"
          subPath: kafka_server_jaas.conf
        ports:
        - containerPort: 9092
        env:
        - name: KAFKA_ADVERTISED_PORT
          value: "30718"
        - name: KAFKA_ADVERTISED_HOST_NAME
          value: 10.43.155.122
        - name: KAFKA_ZOOKEEPER_CONNECT
          value: "zoo1:2181"
        - name: KAFKA_BROKER_ID
          value: "0"
        - name: KAFKA_DEFULT_REPLICATION_FACTOR
          value: "3"
        - name: KAFKA_AUTO_CREATE_TOPICS_ENABLE
          value: "true"
        - name: KAFKA_DELETE_TOPIC_ENABLE
          value: "true"
        - name: KAFKA_LISTENERS
          value: "INTERNAL_SASL_PLAINTEXT://0.0.0.0:9092"
        - name: KAFKA_ADVERTISED_LISTENERS
          value: "INTERNAL_SASL_PLAINTEXT://kafka-service:9092"
        - name: KAFKA_LISTENER_SECURITY_PROTOCOL_MAP
          value: "INTERNAL_SASL_PLAINTEXT:SASL_PLAINTEXT,EXTERNAL_SASL_PLAINTEXT:SASL_PLAINTEXT"
        - name: KAFKA_INTER_BROKER_LISTENER_NAME
          value: "INTERNAL_SASL_PLAINTEXT"
        - name: KAFKA_SASL_ENABLED_MECHANISMS
          value: "PLAIN"
        - name: KAFKA_SASL_MECHANISM_INTER_BROKER_PROTOCOL
          value: "PLAIN"
        - name: KAFKA_AUTHORIZER_CLASS_NAME
          value: "kafka.security.authorizer.AclAuthorizer"
        - name: KAFKA_ZOOKEEPER_SET_ACL
          value: "true"
        - name: KAFKA_OPTS
          value: "-Djava.security.auth.login.config=/home/ubuntu/kafkadepl/myconfig/kafka_server_jaas.conf"
        - name: KAFKA_ALLOW_EVERYONE_IF_NO_ACL_FOUND
          value: "true"
      volumes:
      - name: config-volume
        configMap:
          name: kafka-config-map

Verify that pods are up and running

Code Block

title	Verify zookeeper and broker

kubectl -n onap get pods | grep 'zookeeper-deployment-1\|broker0'

Code Block

title	Verify kafka service

kubectl -n onap get svc | grep kafka-service

If you need to change some variable or anything in a yml file, delete the current deployment, for example:

Code Block

title	Delete broker deployment

kubectl -n onap delete deploy kafka-broker0

And after modifying the file create a new deployment as described above.

Run the test

Modify tools/performance/cloud scripts to match the names in your deployments, described in the previous step. Here is a diff file:

tools.diff

Go to tools/performance/cloud and reboot the environment:

Code Block

title	Run the script

./reboot-test-environment.sh -v

Now you are ready to run the test.

Without DMaaP Kafka

Conditions

To gather tcpdump data another container was added to hv-ves deployment in kubernetes and producer-pod.yaml (command to get tcpdump data file: kubectl cp -n onap <pod-name>:/tcpdump.pcap -c tcpdump ./<pod-name>.pcap):

...

Space shortcuts

Page tree

Versions Compared

Old Version 9

New Version 10

Key

No DMaaP Kafka SetUp

Conditions

Install Kafka Docker on Kubernetes

Create config maps

Create deployments

Verify that pods are up and running

Run the test

Without DMaaP Kafka

Conditions