https://strimzi.io/docs/operators/latest/configuring.html

https://strimzi.io/docs/operators/latest/configuring.html#proc-accessing-kafka-using-ingress-str

https://strimzi.io/blog/2019/04/23/accessing-kafka-part-2/

https://github.com/strimzi/strimzi-kafka-operator/blob/main/documentation/api/io.strimzi.api.kafka.model.listener.arraylistener.GenericKafkaListener.adoc

Current Setup - no Ingress (Kohn):

  • External Access via Nodeports
    • onap-strimzi-kafka-external-bootstrap (30493)
    • onap-strimzi-kafka-0 (30490)
    • onap-strimzi-kafka-1 (30491)
    • onap-strimzi-kafka-2 (30492)
  • TLS termination on Kafka Pods


External Access to Kafka (DT implementation) in Jakarta/Kohn

  • External Access via Ingress (Traefik)
    • new TCP "EntryPoints" in Traefik Gateway for bootstrap and brokers
    • Update Pod "clienttls" ports (9093) to use "advertizedHost" and "advertizedPort"
    • NodePorts not used...
    • IngressRouteTCP entry to "internal" bootstrap service 
      • Use "tls passthough"
    • IngressRouteTCP entries to external broker ports

Proposal for London (External Access via Ingress)

  • External Access via Ingress (istio-ingress)
    • new TLS ports on Ingress Gateway for bootstrap and brokers
  • Disable TLS on "external" broker ports 
  • Disable all Nodeports in Service definitions

Need to check:
https://github.com/istio/istio/issues/20076


Test steps on an existing ServiceMesh cluster

  1. Add custom ports to istio-ingressgateway service
    (https://www.dangtrinh.com/2019/09/how-to-open-custom-port-on-istio.html)
  2. Modify onap-strimzi-kafka pods and services to disable TLS and set advertizedHosts
  3. Add Ingress Gateway/VCs for onap_strimzi to istio-ingressgateway
  4. Create External Kafka User (optional)
  5. Test the external client access to Kafka


Add custom ports to istio-ingressgateway service


  • Export existing service definition
Add Custom ports 
kubectl -n istio-ingress get service istio-ingressgateway -o yaml > istio_ingressgateway.yaml
  • Check existing Nodeports (The range of valid ports is 30000-32767) and choose 4 free ports (e.g. 30900, 30901,30902, 30910)
kubectl get svc -A |grep Load
kubectl get svc -A |grep NodePort
  • Edit istio_ingressgateway.yaml and add new ports
  - port: 9010
    nodePort: 30910
    targetPort: 9010
    name: kafka-bootstrap
    protocol: TCP
  - port: 9000
    nodePort: 30900
    targetPort: 9000
    name: kafka-0
    protocol: TCP
  - port: 9001
    nodePort: 30901
    targetPort: 9001
    name: kafka-1
    protocol: TCP
  - port: 9002
    nodePort: 30902
    targetPort: 9002
    name: kafka-2
    protocol: TCP
  • Apply changes:
kubectl apply -f ./istio_ingressgateway.yaml

Modify onap-strimzi-kafka pods and services to disable TLS and set advertizedHosts

  • Login to the K8S Control Node and set the helm environment
Modify pods 
	helm repo add local http://127.0.0.1:8879
	helm plugin install --version v0.10.3 https://github.com/chartmuseum/helm-push.git
	git config --global --add safe.directory /opt/oom
  • Modify the onap-strimzi config
 	cd /opt/oom/kubernetes
	vi strimzi/templates/strimzi-kafka.yaml
	Update "tls" and "authentication.type" of the "external" kafka listener:
    ---
      - name: external
        port: 9094
        type: nodeport
        tls: false
        authentication:
          type: {{ .Values.config.saslMechanism }}
        configuration:
          brokers:
            - broker: 0
              advertisedHost: kafka-api.simpledemo.onap.org
              advertisedPort: 9000
            - broker: 1
              advertisedHost: kafka-api.simpledemo.onap.org
              advertisedPort: 9001
            - broker: 2
              advertisedHost: kafka-api.simpledemo.onap.org
              advertisedPort: 9002
    • Apply the changes to onap-strimzi
make strimzi
helm upgrade -i onap-strimzi local/strimzi --namespace onap --version 12.0.0 --values /opt/oom/kubernetes/onap/values.yaml --values /opt/oom/kubernetes/onap/resources/overrides/onap-all-ingress-istio.yaml --values /opt/oom/kubernetes/onap/resources/overrides/environment.yaml --values /home/ubuntu/oom/master/onap-overrides.yaml --timeout '900s'


Add Ingress Gateway/VCs for onap_strimzi to istio-ingressgateway

  • Create a file (e.g. kafka-ingress.yaml) Istio Ingress Gateway/VirtualService entries for the kafka-bootstrap-api and the brokers


GW/VC 
apiVersion: networking.istio.io/v1beta1
kind: Gateway
metadata:
  name: kafka-bootstrap-api-gateway
spec:
  selector:
    istio: ingressgateway
  servers:
    - hosts:
        - kafka-bootstrap-api.simpledemo.onap.org
      port:
        name: tls-kafka-bootstrap
        number: 9010
        protocol: TLS
      tls:
        credentialName: ingress-tls-secret
        mode: SIMPLE
---
apiVersion: networking.istio.io/v1beta1
kind: VirtualService
metadata:
  name: kafka-bootstrap-api-service
spec:
  hosts:
    - kafka-bootstrap-api.simpledemo.onap.org
  gateways:
    - kafka-bootstrap-api-gateway
  tcp:
  - match:
    - port: 9010
    route:
    - destination:
        host: onap-strimzi-kafka-external-bootstrap
        port:
          number: 9094
---
apiVersion: networking.istio.io/v1beta1
kind: Gateway
metadata:
  name: kafka-api-gateway
spec:
  selector:
    istio: ingressgateway
  servers:
    - hosts:
        - kafka-api.simpledemo.onap.org
      port:
        name: tls-kafka-0
        number: 9000
        protocol: TLS
      tls:
        credentialName: ingress-tls-secret
        mode: SIMPLE
    - hosts:
        - kafka-api.simpledemo.onap.org
      port:
        name: tls-kafka-1
        number: 9001
        protocol: TLS
      tls:
        credentialName: ingress-tls-secret
        mode: SIMPLE
    - hosts:
        - kafka-api.simpledemo.onap.org
      port:
        name: tls-kafka-2
        number: 9002
        protocol: TLS
      tls:
        credentialName: ingress-tls-secret
        mode: SIMPLE
---
apiVersion: networking.istio.io/v1beta1
kind: VirtualService
metadata:
  name: kafka-0-api-service
spec:
  hosts:
    - kafka-api.simpledemo.onap.org
  gateways:
    - kafka-api-gateway
  tcp:
  - match:
    - port: 9000
    route:
    - destination:
        host: onap-strimzi-kafka-0
        port:
          number: 9094
---
apiVersion: networking.istio.io/v1beta1
kind: VirtualService
metadata:
  name: kafka-1-api-service
spec:
  hosts:
    - kafka-api.simpledemo.onap.org
  gateways:
    - kafka-api-gateway
  tcp:
  - match:
    - port: 9001
    route:
    - destination:
        host: onap-strimzi-kafka-1
        port:
          number: 9094
---
apiVersion: networking.istio.io/v1beta1
kind: VirtualService
metadata:
  name: kafka-2-api-service
spec:
  hosts:
    - kafka-api.simpledemo.onap.org
  gateways:
    - kafka-api-gateway
  tcp:
  - match:
    - port: 9002
    route:
    - destination:
        host: onap-strimzi-kafka-2
        port:
          number: 9094
  • Apply the file
kubectl -n onap apply -f ./kafka-ingress.yaml

Add Kafka User for external Access

  • Create kafka-user.yaml
tls-user.yaml
apiVersion: kafka.strimzi.io/v1beta2
kind: KafkaUser
metadata:
  labels:
    argocd.argoproj.io/instance: external-strimzi-kafka-user
    strimzi.io/cluster: onap-strimzi
  name: external-strimzi-kafka-user
  namespace: onap
spec:
  authentication:
    type: scram-sha-512
  authorization:
    acls:
      - resource:
          type: topic
          name: unauthenticated.VES_PERF3GPP_OUTPUT
          patternType: literal
        operation: Write
        host: "*"
      - resource:
          type: topic
          name: unauthenticated.VES_PERF3GPP_OUTPUT
          patternType: literal
        operation: Describe
        host: "*"
      - resource:
          type: topic
          name: unauthenticated.VES_NOTIFICATION_OUTPUT
          patternType: literal
        operation: Write
        host: "*"
      - resource:
          type: topic
          name: unauthenticated.VES_NOTIFICATION_OUTPUT
          patternType: literal
        operation: Describe
        host: "*"
      - resource:
          type: topic
          name: unauthenticated.SEC_3GPP_HEARTBEAT_OUTPUT
          patternType: literal
        operation: Write
        host: "*"
      - resource:
          type: topic
          name: unauthenticated.SEC_3GPP_HEARTBEAT_OUTPUT
          patternType: literal
        operation: Describe
        host: "*"
      - resource:
          type: topic
          name: unauthenticated.VES_MEASUREMENT_OUTPUT
          patternType: literal
        operation: Write
        host: "*"
      - resource:
          type: topic
          name: unauthenticated.VES_MEASUREMENT_OUTPUT
          patternType: literal
        operation: Describe
        host: "*"
    type: simple
  • Apply kafka-user.yaml
Create user
kubectl apply -f kafka-user.yaml
  • List kafka users
Check/List new user
root@control01-daily-master-sm:/# kubectl -n onap get kafkauser
NAME                              CLUSTER        AUTHENTICATION   AUTHORIZATION   READY
external-strimzi-kafka-user       onap-strimzi   scram-sha-512    simple          True
onap-aai-sdc-list-user            onap-strimzi   scram-sha-512    simple          True
onap-cds-sdc-list-user            onap-strimzi   scram-sha-512    simple          True
onap-cps-kafka-user               onap-strimzi   scram-sha-512    simple          True
onap-dcae-hv-ves-kafka-user       onap-strimzi   scram-sha-512    simple          True
onap-mc-k8s-sdc-list-kafka-user   onap-strimzi   scram-sha-512    simple          True
onap-policy-kafka-user            onap-strimzi   scram-sha-512    simple          True
onap-sdc-be-kafka-user            onap-strimzi   scram-sha-512    simple          True
strimzi-kafka-admin               onap-strimzi   scram-sha-512    simple          True
  • List strimzi secrets
List user secrets
oot@control01-daily-master-sm:/# kubectl -n onap get secret|grep strimzi
external-strimzi-kafka-user                                        Opaque                                2      2m7s
...
  • Get the user password

For each KafkaUser resource with scram-sha-512 auth, there will be a corresponding secret:

Get the user secret
kubectl get secret external-strimzi-kafka-user -o jsonpath='{.data.password}' -n onap | base64 --decode
Ujl...lSD

Test the external client access to Kafka


  • Add hostnames to DNS (or /etc/hosts) by using the IP Address of the istio-ingressgateway LB 
sudo vi /etc/hosts
----
10.32.240.14 kafka-bootstrap-api.simpledemo.onap.org
10.32.240.14 kafka-api.simpledemo.onap.org


  • Install KafkaCat
sudo apt install kafkacat
  • Get the Metadata (use an existing Kafka User, here "external-strimzi-kafka-user"):
root@control01-daily-master-sm:/# kafkacat -L -b kafka-bootstrap-api.simpledemo.onap.org:9003 -X security.protocol=sasl_ssl -X enable.ssl.certificate.verification=false -X sasl.mech-SHA-512 -X sasl.username=external-strimzi-kafka-user -X sasl.password=hCv4IZ3Q6XLR -v 
Metadata for all topics (from broker -1: sasl_ssl://kafka-bootstrap-api.simpledemo.onap.org:9003/bootstrap):
 3 brokers:
  broker 0 at kafka-api.simpledemo.onap.org:9000 (controller)
  broker 2 at kafka-api.simpledemo.onap.org:9002
  broker 1 at kafka-api.simpledemo.onap.org:9001
 33 topics:
  topic "org.onap.dmaap.mr.PNF_REGISTRATION" with 2 partitions:
    partition 0, leader 2, replicas: 2, isrs: 2
    partition 1, leader 1, replicas: 1, isrs: 1 ...
    • Get Topic Data (use an existing Kafka User, here "external-strimzi-kafka-user"):
kafkacat -b kafka-bootstrap-api.simpledemo.onap.org:9003 -X security.protocol=sasl_ssl -X enable.ssl.certificate.verification=false -X sasl.mechanisms=SCRAM-SHA-512 -X sasl.username=external-strimzi-kafka-user -X sasl.password=hCv4IZ3Q6XLR -C -t unauthenticated.VES_NOTIFICATION_OUTPUT -v

{"event":{"commonEventHeader":{"startEpochMicrosec":8745745764578,"eventId":"FileReady_1797490e-10ae-4d48-9ea7-3d7d790b25e1","timeZoneOffset":"UTC+05.30","internalHeaderFields":{"collectorTimeStamp":"Tue, 12 06 2022 01:35:59 GMT"},"priority":"Normal","version":"4.0.1","reportingEntityName":"otenb5309","sequence":0,"domain":"notification","lastEpochMicrosec":8745745764578,"eventName":"Noti_RnNode-Ericsson_FileReady","vesEventListenerVersion":"7.0.1","sourceName":"oteNB5309"},"notificationFields":{"notificationFieldsVersion":"2.0","changeType":"FileReady","changeIdentifier":"PM_MEAS_FILES","arrayOfNamedHashMap":[{"name":"test.xml.gz","hashMap":{"location":"sftp://sftp:22/test.xml.gz","fileFormatType":"org.3GPP.32.435#measCollec","fileFormatVersion":"V10","compression":"gzip"}}]}}}
...

Tasks required for London:

  • Add Ingress-Gateway "custom port" configuration in OOM Documents
  • Extend _ingress.tpl to accept
    • external ports (here 9010,9000,...)
    • specific settings...
  • Modify onap-strimzi charts
    • Add ingress configuration
    • Update strimzi-kafka configuration to disable TLS in SM case
  • No labels