Versions Compared

Key

  • This line was added.
  • This line was removed.
  • Formatting was changed.

...

https://github.com/strimzi/strimzi-kafka-operator/blob/main/documentation/api/io.strimzi.api.kafka.model.listener.arraylistener.GenericKafkaListener.adoc

Current Setup - no Ingress (Kohn):

  • External Access via Nodeports
    • onap-strimzi-kafka-external-bootstrap (30493)
    • onap-strimzi-kafka-0 (30490)
    • onap-strimzi-kafka-1 (30491)
    • onap-strimzi-kafka-2 (30492)
  • TLS termination on Kafka Pods

draw.io Diagram
bordertrue
diagramNameingres
simpleViewerfalse
linksauto
tbstyletop
lboxtrue
diagramWidth486
revision5


External Access to Kafka (DT implementation) in Jakarta/Kohn

  • External Access via Ingress (Traefik)
    • new TCP "EntryPoints" in Traefik Gateway for bootstrap and brokers
    • Update Pod "clienttls" ports (9093) to use "advertizedHost" and "advertizedPort"
    • NodePorts not used...
    • IngressRouteTCP entry to "internal" bootstrap service 
      • Use "tls passthough"
    • IngressRouteTCP entries to external broker ports

draw.io Diagram
bordertrue
diagramNameTrafik2Kafka
simpleViewerfalse
linksauto
tbstyletop
lboxtrue
diagramWidth887
revision3

Proposal for London (External Access via Ingress)

  • External Access via Ingress (istio-ingress)
    • new TLS ports on Ingress Gateway for bootstrap and brokers
  • Disable TLS on "external" broker ports 
  • Disable all Nodeports in Service definitions

...

draw.io Diagram
bordertrue
diagramNameIngress2Kafka
simpleViewerfalse
width
linksauto
tbstyletop
lboxtrue
diagramWidth857
revision6

Test steps on an existing ServiceMesh cluster

  1. Add custom ports to istio-ingressgateway service
    (https://www.dangtrinh.com/2019/09/how-to-open-custom-port-on-istio.html)
  2. Modify onap-strimzi-kafka pods and services to disable TLS and set advertizedHosts
  3. Add Ingress Gateway/VCs for onap_strimzi to istio-ingressgateway
  4. Create External Kafka User (optional)
  5. Test the external client access to Kafka


Add custom ports to istio-ingressgateway service


  • Export existing service definition
Code Block
languagebash
themeMidnight
titleAdd Custom ports
collapsetrue
1. Export existing service definition:
	kubectl -n istio-ingress get service istio-ingressgateway -o yaml > istio_ingressgateway.yaml
2. 
  • Check

...

  • existing

...

  • Nodeports

...

  • (The

...

  • range

...

  • of

...

  • valid

...

  • ports

...

  • is

...

  • 30000-32767) and choose 4 free ports (e.g. 30900, 30901,30902, 30910)
Code Block
languagebash
themeMidnight

	kubectl get svc -A |grep Load
	kubectl get svc -A |grep NodePortLoad
3.kubectl Chooseget 4svc free-A ports (e.g. 30900, 30901,30902, 30910)
4. Edit |grep NodePort 
  • Edit istio_ingressgateway.yaml

...

  • and add new ports
Code Block
languageyml
themeMidnight
  -  add:
  - port: 9010
    nodePort: 30910
    targetPort: 9010
    name: kafka-bootstrap
    protocol: TCP
  - port: 9000
    nodePort: 30900
    targetPort: 9000
    name: kafka-0
    protocol: TCP
  - port: 9001
    nodePort: 30901
    targetPort: 9001
    name: kafka-1
    protocol: TCP
  - port: 9002
    nodePort: 30902
    targetPort: 9002
    name: kafka-2
    protocol: TCP
5. 
  • Apply

...

  • changes:
Code Block
languagebash
themeMidnight

     kubectl apply -f ./istio_ingressgateway.yaml 

Modify onap-strimzi-kafka pods and services to disable TLS and set advertizedHosts

  • Login to the K8S Control Node and set the helm environment
Code Block
titleModify pods
collapsetrue
1.	helm Loginrepo toadd the K8S Control Node and set the helm environment
	helm repo add local http://127.0local http://127.0.0.1:8879
	helm plugin install --version v0.10.3 https://github.com/chartmuseum/helm-push.git
	git config --global --add safe.directory /opt/oom

2. 
  • Modify

...

  • the

...

  • onap-strimzi config
Code Block
languagebash
themeMidnight
  config
	cd /opt/oom/kubernetes
	vi strimzi/templates/strimzi-kafka.yaml
	Update "tls" and "authentication.type" of the "external" kafka listener:
    ---
      - name: external
        port: 9094
        type: nodeport
        tls: false
        authentication:
          type: {{ .Values.config.saslMechanism }}
        configuration:
          brokers:
            - broker: 0
              advertisedHost: kafka-api.simpledemo.onap.org
              advertisedPort: 9000
            - broker: 1
              advertisedHost: kafka-api.simpledemo.onap.org
              advertisedPort: 9001
            - broker: 2
              advertisedHost: kafka-api.simpledemo.onap.org
              advertisedPort: 9002

3. 
    • Apply

...

    • the

...

    • changes

...

    • to

...

    • onap-strimzi
Code Block
languagebash
themeMidnight

    make strimzi
    helm upgrade -i onap-strimzi local/strimzi --namespace onap --version 12.0.0 --values /opt/oom/kubernetes/onap/values.yaml --values /opt/oom/kubernetes/onap/resources/overrides/onap-all-ingress-istio.yaml --values /opt/oom/kubernetes/onap/resources/overrides/environment.yaml --values /home/ubuntu/oom/master/onap-overrides.yaml --timeout '900s'


Add Ingress Gateway/VCs for onap_strimzi to istio-ingressgateway

  • Create a file (e.g. kafka-ingress.yaml) Istio Ingress Gateway/VirtualService entries for the kafka-bootstrap-api and the brokers


Code Block
languageyml
themeMidnight
titleGW/VC
collapsetrue
apiVersion: networking.istio.io/v1beta1
kind: Gateway
metadata:
  name: kafka-bootstrap-api-gateway
spec:
  selector:
    istio: ingressgateway
  servers:
    - hosts:
        - kafka-bootstrap-api.simpledemo.onap.org
      port:
        name: tls-kafka-bootstrap
        number: 9010
        protocol: TLS
      tls:
        credentialName: ingress-tls-secret
        mode: SIMPLE
---
apiVersion: networking.istio.io/v1beta1
kind: VirtualService
metadata:
  name: kafka-bootstrap-api-service
spec:
  hosts:
    - kafka-bootstrap-api.simpledemo.onap.org
  gateways:
    - kafka-bootstrap-api-gateway
  tcp:
  - match:
    - port: 9010
    route:
    - destination:
        host: onap-strimzi-kafka-external-bootstrap
        port:
          number: 9094
---
apiVersion: networking.istio.io/v1beta1
kind: Gateway
metadata:
  name: kafka-api-gateway
spec:
  selector:
    istio: ingressgateway
  servers:
    - hosts:
        - kafka-api.simpledemo.onap.org
      port:
        name: tls-kafka-0
        number: 9000
        protocol: TLS
      tls:
        credentialName: ingress-tls-secret
        mode: SIMPLE
    - hosts:
        - kafka-api.simpledemo.onap.org
      port:
        name: tls-kafka-1
        number: 9001
        protocol: TLS
      tls:
        credentialName: ingress-tls-secret
        mode: SIMPLE
    - hosts:
        - kafka-api.simpledemo.onap.org
      port:
        name: tls-kafka-2
        number: 9002
        protocol: TLS
      tls:
        credentialName: ingress-tls-secret
        mode: SIMPLE
---
apiVersion: networking.istio.io/v1beta1
kind: VirtualService
metadata:
  name: kafka-0-api-service
spec:
  hosts:
    - kafka-api.simpledemo.onap.org
  gateways:
    - kafka-api-gateway
  tcp:
  - match:
    - port: 9000
    route:
    - destination:
        host: onap-strimzi-kafka-0
        port:
          number: 9094
---
apiVersion: networking.istio.io/v1beta1
kind: VirtualService
metadata:
  name: kafka-1-api-service
spec:
  hosts:
    - kafka-api.simpledemo.onap.org
  gateways:
    - kafka-api-gateway
  tcp:
  - match:
    - port: 9001
    route:
    - destination:
        host: onap-strimzi-kafka-1
        port:
          number: 9094
---
apiVersion: networking.istio.io/v1beta1
kind: VirtualService
metadata:
  name: kafka-2-api-service
spec:
  hosts:
    - kafka-api.simpledemo.onap.org
  gateways:
    - kafka-api-gateway
  tcp:
  - match:
    - port: 9002
    route:
    - destination:
        host: onap-strimzi-kafka-2
        port:
          number: 9094
  • Apply the file
Code Block
languagebash
themeMidnight
kubectl -n onap apply -f ./kafka-ingress.yaml ./kafka-ingress.yaml

Add Kafka User for external Access

  • Create kafka-user.yaml
Code Block
languagebash
themeMidnight
titletls-user.yaml
apiVersion: kafka.strimzi.io/v1beta2
kind: KafkaUser
metadata:
  labels:
    argocd.argoproj.io/instance: external-strimzi-kafka-user
    strimzi.io/cluster: onap-strimzi
  name: external-strimzi-kafka-user
  namespace: onap
spec:
  authentication:
    type: scram-sha-512
  authorization:
    acls:
      - resource:
          type: topic
          name: unauthenticated.VES_PERF3GPP_OUTPUT
          patternType: literal
        operation: Write
        host: "*"
      - resource:
          type: topic
          name: unauthenticated.VES_PERF3GPP_OUTPUT
          patternType: literal
        operation: Describe
        host: "*"
      - resource:
          type: topic
          name: unauthenticated.VES_NOTIFICATION_OUTPUT
          patternType: literal
        operation: Write
        host: "*"
      - resource:
          type: topic
          name: unauthenticated.VES_NOTIFICATION_OUTPUT
          patternType: literal
        operation: Describe
        host: "*"
      - resource:
          type: topic
          name: unauthenticated.SEC_3GPP_HEARTBEAT_OUTPUT
          patternType: literal
        operation: Write
        host: "*"
      - resource:
          type: topic
          name: unauthenticated.SEC_3GPP_HEARTBEAT_OUTPUT
          patternType: literal
        operation: Describe
        host: "*"
      - resource:
          type: topic
          name: unauthenticated.VES_MEASUREMENT_OUTPUT
          patternType: literal
        operation: Write
        host: "*"
      - resource:
          type: topic
          name: unauthenticated.VES_MEASUREMENT_OUTPUT
          patternType: literal
        operation: Describe
        host: "*"
    type: simple 
  • Apply kafka-user.yaml
Code Block
languagebash
themeMidnight
titleCreate user
kubectl apply -f kafka-user.yaml
  • List kafka users
Code Block
languagebash
themeMidnight
titleCheck/List new user
root@control01-daily-master-sm:/# kubectl -n onap get kafkauser
NAME                              CLUSTER        AUTHENTICATION   AUTHORIZATION   READY
external-strimzi-kafka-user       onap-strimzi   scram-sha-512    simple          True
onap-aai-sdc-list-user            onap-strimzi   scram-sha-512    simple          True
onap-cds-sdc-list-user            onap-strimzi   scram-sha-512    simple          True
onap-cps-kafka-user               onap-strimzi   scram-sha-512    simple          True
onap-dcae-hv-ves-kafka-user       onap-strimzi   scram-sha-512    simple          True
onap-mc-k8s-sdc-list-kafka-user   onap-strimzi   scram-sha-512    simple          True
onap-policy-kafka-user            onap-strimzi   scram-sha-512    simple          True
onap-sdc-be-kafka-user            onap-strimzi   scram-sha-512    simple          True
strimzi-kafka-admin               onap-strimzi   scram-sha-512    simple          True
  • List strimzi secrets
Code Block
languagebash
themeMidnight
titleList user secrets
oot@control01-daily-master-sm:/# kubectl -n onap get secret|grep strimzi
external-strimzi-kafka-user                                        Opaque                                2      2m7s
...
  • Get the user password

For each KafkaUser resource with scram-sha-512 auth, there will be a corresponding secret:

Code Block
languagebash
themeMidnight
titleGet the user secret
kubectl get secret external-strimzi-kafka-user -o jsonpath='{.data.password}' -n onap | base64 --decode
Ujl...lSD

Test the external client access to Kafka


  • Add hostnames to DNS (or /etc/hosts) by using the IP Address of the istio-ingressgateway LB 
Code Block
languagebash
themeMidnight
sudo vi /etc/hosts
----
10.32.240.14 kafka-bootstrap-api.simpledemo.onap.org
10.32.240.14 kafka-api.simpledemo.onap.org

...

  • Install KafkaCat
Code Block
languagebash
themeMidnight
sudo apt install kafkacat
  • Get the Metadata ( (use an existing Kafka User, e.g. Admin)here "external-strimzi-kafka-user"):
Code Block
languagebash
themeMidnight
collapsetrue
root@control01-daily-master-sm:/# kafkacat -L -b kafka-bootstrap-api.simpledemo.onap.org:9003 -X security.protocol=sasl_ssl -X enable.ssl.certificate.verification=false -X sasl.mechanisms=SCRAMmech-SHA-512 -X sasl.username=<strimzi-user>=external-strimzi-kafka-user -X sasl.password=<strimzi-password>hCv4IZ3Q6XLR -v
 
Metadata for all topics (from broker -1: sasl_ssl://kafka-bootstrap-api.simpledemo.onap.org:9003/bootstrap):
 3 brokers:
  broker 0 at kafka-api.simpledemo.onap.org:9000 (controller)
  broker 2 at kafka-api.simpledemo.onap.org:9002
  broker 1 at kafka-api.simpledemo.onap.org:9001
 33 topics:
  topic "org.onap.dmaap.mr.PNF_REGISTRATION" with 2 partitions:
    partition 0, leader 2, replicas: 2, isrs: 2
    partition 1, leader 1, replicas: 1, isrs: 1
  topic "SDC-DISTR-NOTIF-TOPIC-AUTO" with 6 partitions:
...
    • Get Topic Data (
  • (
    • use an existing Kafka User,
  • e.g. Admin)
    • here "external-strimzi-kafka-user"):
Code Block
languagebash
themeMidnight
collapsetrue
kafkacat -b kafka-bootstrap-api.simpledemo.onap.org:9003 -X security.protocol=sasl_ssl -X enable.ssl.certificate.verification=false -X sasl.mechanisms=SCRAM-SHA-512 -X sasl.username=external-strimzi-kafka-adminuser -X sasl.password=GzxcHZ29sUXb hCv4IZ3Q6XLR -C  -t unauthenticated.VES_NOTIFICATION_OUTPUT -v

{"event":{"commonEventHeader":{"startEpochMicrosec":8745745764578,"eventId":"FileReady_1797490e-10ae-4d48-9ea7-3d7d790b25e1","timeZoneOffset":"UTC+05.30","internalHeaderFields":{"collectorTimeStamp":"Tue, 12 06 2022 01:35:59 GMT"},"priority":"Normal","version":"4.0.1","reportingEntityName":"otenb5309","sequence":0,"domain":"notification","lastEpochMicrosec":8745745764578,"eventName":"Noti_RnNode-Ericsson_FileReady","vesEventListenerVersion":"7.0.1","sourceName":"oteNB5309"},"notificationFields":{"notificationFieldsVersion":"2.0","changeType":"FileReady","changeIdentifier":"PM_MEAS_FILES","arrayOfNamedHashMap":[{"name":"test.xml.gz","hashMap":{"location":"sftp://sftp:22/test.xml.gz","fileFormatType":"org.3GPP.32.435#measCollec","fileFormatVersion":"V10","compression":"gzip"}}]}}}
...



Tasks required for London:

  • Add Ingress-Gateway "custom port" configuration in OOM Documents
  • Extend _ingress.tpl to accept
    • external ports (here 9010,9000,...)
    • specific settings...
  • Modify onap-strimzi charts
    • Add ingress configuration
    • Update strimzi-kafka configuration to disable TLS in SM case