References
CPS-1613 Explore How to Test Kafka in the CSITs
Assumptions
# | Issue | Notes | Decisions |
---|---|---|---|
1 | kafka container port issue | The exposed port of kafka container changed from "19092" to "9092" in docker-compose.yml | The possible affected classes must be updated. |
2 | KAFKA_ADVERTISED_LISTENERS hostname resolution issue | The hostname changed from "kafka" to "localhost" in docker-compose.yml | The possible affected classes must be updated. |
Issues & Decisions
# | Issue | Notes | Decisions |
---|---|---|---|
1 | kafka container port issue | ||
2 | KAFKA_ADVERTISED_LISTENERS hostname resolution issue |
Overview
Can Robot Framework test Kafka clusters and events?
- In the latest version of Robot Framework (6.0.2), there is no support for Kafka Event Streaming.
Is there any third-party library that would aid in this?
- ConfluentKafkaLibrary is a Robot Framework library to verify Kafka Event Streaming.
- Robotframework-ConfluentKafkaLibrary
Issues
1-In the current implementation of ConfluentKafkaLibrary, the consumer module connects to Kafka cluster from the port "9092".
def __init__( self, server='127.0.0.1', port='9092', topics='', group_id=None, only_value=True, **kwargs ):
Whereas, CPS's kafka container expose its port from "19092"
kafka: image: confluentinc/cp-kafka:6.2.1 container_name: kafka ports: - "19092:19092" depends_on: - zookeeper environment: KAFKA_BROKER_ID: 1 KAFKA_ZOOKEEPER_CONNECT: zookeeper:2181 KAFKA_ADVERTISED_LISTENERS: PLAINTEXT://kafka:9092,CONNECTIONS_FROM_HOST://localhost:19092 KAFKA_LISTENER_SECURITY_PROTOCOL_MAP: PLAINTEXT:PLAINTEXT,CONNECTIONS_FROM_HOST:PLAINTEXT KAFKA_OFFSETS_TOPIC_REPLICATION_FACTOR: 1
In order to solve this conflict, the current port has been changed to 9092 in docker-compose.yml
2-The ConfluentKafkaLibrary also try to access the KAFKA_ADVERTISED_LISTENERS (PLAINTEXT://kafka:9092) natively.
However, it gets fail because of hostname to IP address resolution issue. There are two possible solution for this problem.
a- Add an entry into /etc/hosts like 127.0.0.1 kafka
b- Change the KAFKA_ADVERTISED_LISTENERS into PLAINTEXT://localhost:9092,CONNECTIONS_FROM_HOST://localhost:19092
The second solution is being applied in the latest patch set.