References
CPS-1613 Explore How to Test Kafka in the CSITs
Assumptions
| # | Issue | Notes | Decisions |
|---|---|---|---|
| 1 | kafka container port issue | The exposed port of kafka container changed from "19092" to "9092" in docker-compose.yml | The possible affected classes must be updated. |
| 2 | KAFKA_ADVERTISED_LISTENERS hostname resolution issue | The hostname changed from "kafka" to "localhost" in docker-compose.yml | The possible affected classes must be updated. |
Issues & Decisions
| # | Issue | Notes | Decisions |
|---|---|---|---|
| 1 | kafka container port issue | Open Questions: 1- Can we change the port to "9092"? | |
| 2 | KAFKA_ADVERTISED_LISTENERS hostname resolution issue | Open Questions: 1- Can we change the hostname to "localhost"? 2- Is it possible to update /etc/hosts in ONAP Jenkins? |
Overview
Can Robot Framework test Kafka clusters and events?
- In the latest version of Robot Framework (6.0.2), there is no support for Kafka Event Streaming.
Is there any third-party library that would aid in this?
- ConfluentKafkaLibrary is a Robot Framework library to verify Kafka Event Streaming.
- Robotframework-ConfluentKafkaLibrary
Issues
1-In the current implementation of ConfluentKafkaLibrary, the consumer module connects to Kafka cluster from the port "9092".
def __init__(
self,
server='127.0.0.1',
port='9092',
topics='',
group_id=None,
only_value=True,
**kwargs
):
Whereas, CPS's kafka container expose its port from "19092"
kafka:
image: confluentinc/cp-kafka:6.2.1
container_name: kafka
ports:
- "19092:19092"
depends_on:
- zookeeper
environment:
KAFKA_BROKER_ID: 1
KAFKA_ZOOKEEPER_CONNECT: zookeeper:2181
KAFKA_ADVERTISED_LISTENERS: PLAINTEXT://kafka:9092,CONNECTIONS_FROM_HOST://localhost:19092
KAFKA_LISTENER_SECURITY_PROTOCOL_MAP: PLAINTEXT:PLAINTEXT,CONNECTIONS_FROM_HOST:PLAINTEXT
KAFKA_OFFSETS_TOPIC_REPLICATION_FACTOR: 1
In order to solve this conflict, the current port has been changed to 9092 in docker-compose.yml
2-The ConfluentKafkaLibrary also try to access the KAFKA_ADVERTISED_LISTENERS (PLAINTEXT://kafka:9092) natively.
However, it gets fail because of hostname to IP address resolution issue. There are two possible solution for this problem.
a- Add an entry into /etc/hosts like 127.0.0.1 kafka
b- Change the KAFKA_ADVERTISED_LISTENERS into PLAINTEXT://localhost:9092,CONNECTIONS_FROM_HOST://localhost:19092
The second solution is being applied in the latest patch set.