Table of Contents |
---|
References
CPS-1613 Explore How to Test Kafka in the CSITshttps://jira.onap.org/browse/CPS-1601
Assumptions
# | Issue | Notes | Decisions | |||
---|---|---|---|---|---|---|
1 | kafka container port issue | The exposed port of kafka container changed | from "19092"to | "9092 | " inin docker-compose.yml | The possible affected classes must be updatedAs per the technical discussion, there is no hard-coded port given in any java classes. |
2 | KAFKA_ADVERTISED_LISTENERS hostname resolution issue | The | hostname changed from "kafka" to "localhost" in docker-compose.ymlport of KAFKA_ADVERTISED_LISTENERS has been changed to 29092. | As per the technical discussion, there is no hard-coded port given in any java classes | The possible affected classes must be updated. |
Issues & Decisions
# | Issue | Notes | Decisions | |
---|---|---|---|---|
1 | kafka container port issue | Open Questions: 1- Can we change the port to | "9092"?9092 | Since docker-compose.yml is being used for local env. setup and CSIT, it is possible to change to 9092. |
2 | KAFKA_ADVERTISED_LISTENERS hostname resolution issue | Open Questions: 1- Can we change the | hostname to "localhost"?2- Is it possible to update /etc/hosts in ONAP Jenkins?port to 29092 | As per local tests run by the Team 2, and ONAP build, the required change has no negative effect. |
Overview
Can Robot Framework test Kafka clusters and events?
...
- ConfluentKafkaLibrary is a Robot Framework library to verify Kafka Event Streaming.
- Robotframework-ConfluentKafkaLibrary
- API for robotframework-ConfluentKafkaLibrary
Issues
1-In the current implementation of ConfluentKafkaLibrary, the consumer module connects to Kafka cluster from the port "9092".
...
Whereas, CPS's kafka container expose its port from "19092"
Code Block | ||||
---|---|---|---|---|
| ||||
kafka: image: confluentinc/cp-kafka:6.2.1 container_name: kafka ports: - "19092:19092" depends_on: - zookeeper environment: KAFKA_BROKER_ID: 1 KAFKA_ZOOKEEPER_CONNECT: zookeeper:2181 KAFKA_ADVERTISED_LISTENERS: PLAINTEXT://kafka:9092,CONNECTIONS_FROM_HOST://localhost:19092 KAFKA_LISTENER_SECURITY_PROTOCOL_MAP: PLAINTEXT:PLAINTEXT,CONNECTIONS_FROM_HOST:PLAINTEXT KAFKA_OFFSETS_TOPIC_REPLICATION_FACTOR: 1 |
...
In order to solve this conflict, the current port has been changed to 9092 in docker-compose.yml
Code Block | ||||
---|---|---|---|---|
| ||||
kafka:
image: confluentinc/cp-kafka:6.2.1
container_name: kafka
ports:
- '9092:9092'
depends_on:
- zookeeper
environment:
KAFKA_BROKER_ID: 1
KAFKA_ZOOKEEPER_CONNECT: zookeeper:2181
KAFKA_LISTENER_SECURITY_PROTOCOL_MAP: PLAINTEXT:PLAINTEXT,CONNECTIONS_FROM_HOST:PLAINTEXT
KAFKA_ADVERTISED_LISTENERS: PLAINTEXT://kafka:29092,CONNECTIONS_FROM_HOST://localhost:9092
KAFKA_OFFSETS_TOPIC_REPLICATION_FACTOR: 1 |
2-The ConfluentKafkaLibrary also try to access the KAFKA_ADVERTISED_LISTENERS (PLAINTEXT://kafka:9092) natively.
However, it gets fail because of hostname to IP address resolution issue. There are two is a possible solution for this problem.
a- Add an entry into /etc/hosts like 127.0.0.1 kafka
...
A change in the KAFKA_ADVERTISED_LISTENERS port to 29092
KAFKA_ADVERTISED_LISTENERS
...
: PLAINTEXT://
...
kafka:
...
29092,CONNECTIONS_FROM_HOST://localhost:
...
909