...
Expand |
---|
title | Click here to see results... |
---|
|
NUMBER OF PRODUCERS | TOTAL MESSAGES PROCESSED | DIFFERENCE BETWEEN ALL MESSAGES AND SENT TO HV-VES | AVERAGE PROCESSING TIME IN HV-VES WITHOUT ROUTING [ms] | AVERAGE LATENCY TO HV-VES OUTPUT WITH ROUTING [ms] | PEAK INCOMING DATA RATE [MB/s] | PEAK PROCESSING MESSAGE QUEUE SIZE | PEAK CPU LOAD [%] | PEAK MEMORY USAGE [GB] | RESULTS PRESENTED IN GRAFANA |
---|
2 | 180000 | 0 | 0.026 | 8.9 | 1.6 | 11 | 3 | 0.35 | | 2 | 180000 | 0 | 0.026 | 11.3 | 1.6 | 23 | 3.2 | 0.35 | ![](https://confluence.ext.net.nokia.com/download/thumbnails/983925650/Screenshot%20from%202020-04-23%2013-24-30.png?version=1&modificationDate=1587641506000&api=v2) |
NUMBER OF PRODUCERS | TOTAL MESSAGES PROCESSED | DIFFERENCE BETWEEN ALL MESSAGES AND SENT TO HV-VES | AVERAGE PROCESSING TIME IN HV-VES WITHOUT ROUTING [s] | AVERAGE LATENCY TO HV-VES OUTPUT WITH ROUTING [ms] | PEAK INCOMING DATA RATE [MB/s] | PEAK PROCESSING MESSAGE QUEUE SIZE | PEAK CPU LOAD [%] | PEAK MEMORY USAGE [GB] | RESULTS PRESENTED IN GRAFANA |
---|
4 | 360000 | 0 | 0.022 | 41.3 | 3.2 | 130 | 4 | 0.35 | ![](https://confluence.ext.net.nokia.com/download/thumbnails/983925650/Screenshot%20from%202020-04-23%2012-42-54.png?version=1&modificationDate=1587641670000&api=v2) ![](https://confluence.ext.net.nokia.com/download/thumbnails/983925650/Screenshot%20from%202020-04-23%2012-42-56.png?version=1&modificationDate=1587641670000&api=v2) ![](https://confluence.ext.net.nokia.com/download/thumbnails/983925650/Screenshot%20from%202020-04-23%2012-43-01.png?version=1&modificationDate=1587641673000&api=v2) | 4 | 360000 | 0 | 0.023 | 40.7 | 3.2 | 370 | 4.2 | 0.35 | |
NUMBER OF PRODUCERS | TOTAL MESSAGES PROCESSED | DIFFERENCE BETWEEN ALL MESSAGES AND SENT TO HV-VES | AVERAGE PROCESSING TIME IN HV-VES WITHOUT ROUTING [ms] | AVERAGE LATENCY TO HV-VES OUTPUT WITH ROUTING [ms] | PEAK INCOMING DATA RATE [MB/s] | PEAK PROCESSING MESSAGE QUEUE SIZE | PEAK CPU LOAD [%] | PEAK MEMORY USAGE [GB] | RESULTS PRESENTED IN GRAFANA |
---|
6 | 359317 | 683 | 6240 | 15965 | 4.8 | 4500 | 4.3 | 1 | | 6 | 359217 | 783 | 6490 | 16834 | 4.8 | 4200 | 5.8 | 1 | |
|
...
No DMaaP Kafka SetUp
Conditions
...
Install Kafka Docker on Kubernetes
(based on: ultimate-guide-to-installing-kafka-docker-on-kuber)
Create config maps
Config maps are required by zookeeper and kafka-broker deployments.
Code Block |
---|
title | Create kafka-config-map |
---|
|
kubectl -n onap create cm kafka-config-map --from-file=kafka_server_jaas.conf |
Code Block |
---|
title | kafka_server_jaas.conf |
---|
collapse | true |
---|
|
KafkaServer {
org.apache.kafka.common.security.plain.PlainLoginModule required
username="admin"
password="admin_secret"
user_admin="admin_secret";
};
Client {
org.apache.zookeeper.server.auth.DigestLoginModule required
username="kafka"
password="kafka_secret";
}; |
...
Code Block |
---|
title | Create zk-config-map |
---|
|
kubectl -n onap create cm zk-config-map --from-file=zk_server_jaas.conf |
Code Block |
---|
title | zk_server_jaas.conf |
---|
collapse | true |
---|
|
Server {
org.apache.zookeeper.server.auth.DigestLoginModule required
user_kafka="kafka_secret";
}; |
Create deployments
Code Block |
---|
title | Create zookeeper deployment |
---|
|
kubectl -n onap create -f zookeeper.yml |
Code Block |
---|
language | yml |
---|
title | zookeeper.yml |
---|
collapse | true |
---|
|
---
kind: Deployment
apiVersion: extensions/v1beta1
metadata:
name: zookeeper-deployment-1
spec:
template:
metadata:
labels:
app: zookeeper-1
spec:
containers:
- name: zoo1
image: digitalwonderland/zookeeper
volumeMounts:
- name: config-volume
mountPath: "/home/ubuntu/kafkadepl/myconfig/zk_server_jaas.conf"
subPath: zk_server_jaas.conf
ports:
- containerPort: 2181
env:
- name: ZOOKEEPER_ID
value: "1"
- name: ZOOKEEPER_SERVER_1
value: zoo1
- name: KAFKA_OPTS
value: "-Djava.security.auth.login.config=/home/ubuntu/kafkadepl/myconfig/zk_server_jaas.conf -Dzookeeper.kerberos.removeHostFromPrincipal=true -Dzookeeper.kerberos.removeRealmFromPrincipal=true -Dzookeeper.authProvider.1=org.apache.zookeeper.server.auth.SASLAuthenticationProvider -Dzookeeper.requireClientAuthScheme=sasl"
- name: SERVER_JVMFLAGS
value: "-Djava.security.auth.login.config=/home/ubuntu/kafkadepl/myconfig/zk_server_jaas.conf -Dzookeeper.skipACL=yes"
- name: ZOOKEEPER_AUTHPROVIDER_1
value: "org.apache.zookeeper.server.auth.SASLAuthenticationProvider"
- name: ZOOKEEPER_REQUIRECLIENTAUTHSCHEME
value: "SASL"
volumes:
- name: config-volume
configMap:
name: zk-config-map
---
apiVersion: v1
kind: Service
metadata:
name: zoo1
labels:
app: zookeeper-1
spec:
ports:
- name: client
port: 2181
protocol: TCP
- name: follower
port: 2888
protocol: TCP
- name: leader
port: 3888
protocol: TCP
selector:
app: zookeeper-1
|
...
Code Block |
---|
title | Create kafka-service |
---|
|
kubectl -n onap create -f kafka-service.yml |
Code Block |
---|
language | yml |
---|
title | kafka-service.yml |
---|
collapse | true |
---|
|
---
apiVersion: v1
kind: Service
metadata:
name: kafka-service
labels:
name: kafka
spec:
ports:
- port: 9092
name: kafka-port
protocol: TCP
selector:
app: kafka
id: "0"
type: LoadBalancer |
...
Code Block |
---|
|
kubectl -n onap create -f kafka-broker.yml |
Code Block |
---|
language | yml |
---|
title | kafka-broker.yml |
---|
collapse | true |
---|
|
---
kind: Deployment
apiVersion: extensions/v1beta1
metadata:
name: kafka-broker0
spec:
template:
metadata:
labels:
app: kafka
id: "0"
spec:
containers:
- name: kafka
image: wurstmeister/kafka
volumeMounts:
- name: config-volume
mountPath: "/home/ubuntu/kafkadepl/myconfig/kafka_server_jaas.conf"
subPath: kafka_server_jaas.conf
ports:
- containerPort: 9092
env:
- name: KAFKA_ADVERTISED_PORT
value: "30718"
- name: KAFKA_ADVERTISED_HOST_NAME
value: 10.43.155.122
- name: KAFKA_ZOOKEEPER_CONNECT
value: "zoo1:2181"
- name: KAFKA_BROKER_ID
value: "0"
- name: KAFKA_DEFULT_REPLICATION_FACTOR
value: "3"
- name: KAFKA_AUTO_CREATE_TOPICS_ENABLE
value: "true"
- name: KAFKA_DELETE_TOPIC_ENABLE
value: "true"
- name: KAFKA_LISTENERS
value: "INTERNAL_SASL_PLAINTEXT://0.0.0.0:9092"
- name: KAFKA_ADVERTISED_LISTENERS
value: "INTERNAL_SASL_PLAINTEXT://kafka-service:9092"
- name: KAFKA_LISTENER_SECURITY_PROTOCOL_MAP
value: "INTERNAL_SASL_PLAINTEXT:SASL_PLAINTEXT,EXTERNAL_SASL_PLAINTEXT:SASL_PLAINTEXT"
- name: KAFKA_INTER_BROKER_LISTENER_NAME
value: "INTERNAL_SASL_PLAINTEXT"
- name: KAFKA_SASL_ENABLED_MECHANISMS
value: "PLAIN"
- name: KAFKA_SASL_MECHANISM_INTER_BROKER_PROTOCOL
value: "PLAIN"
- name: KAFKA_AUTHORIZER_CLASS_NAME
value: "kafka.security.authorizer.AclAuthorizer"
- name: KAFKA_ZOOKEEPER_SET_ACL
value: "true"
- name: KAFKA_OPTS
value: "-Djava.security.auth.login.config=/home/ubuntu/kafkadepl/myconfig/kafka_server_jaas.conf"
- name: KAFKA_ALLOW_EVERYONE_IF_NO_ACL_FOUND
value: "true"
volumes:
- name: config-volume
configMap:
name: kafka-config-map
|
Verify that pods are up and running
Code Block |
---|
title | Verify zookeeper and broker |
---|
|
kubectl -n onap get pods | grep 'zookeeper-deployment-1\|broker0' |
Code Block |
---|
title | Verify kafka service |
---|
|
kubectl -n onap get svc | grep kafka-service |
If you need to change some variable or anything in a yml file, delete the current deployment, for example:
Code Block |
---|
title | Delete broker deployment |
---|
|
kubectl -n onap delete deploy kafka-broker0 |
And after modifying the file create a new deployment as described above.
Run the test
Modify tools/performance/cloud scripts to match the names in your deployments, described in the previous step. Here is a diff file:
tools.diff
Go to tools/performance/cloud and reboot the environment:
Code Block |
---|
|
./reboot-test-environment.sh -v |
Now you are ready to run the test.
Without DMaaP Kafka
Conditions
To gather tcpdump data another container was added to hv-ves deployment in kubernetes and producer-pod.yaml (command to get tcpdump data file: kubectl cp -n onap <pod-name>:/tcpdump.pcap -c tcpdump ./<pod-name>.pcap):
...