This page will track all the issues and workaround or solutions to address them when Istio is deployed in ONAP with mTLS enabled.


Pod NameIssueWorkaroundComments
1message-router-kafkaUnable to connect to zookeeper

[2018-08-07 17:21:49,855] INFO Opening socket connection to server 10.42.2.218/10.42.2.218:2181. Will not attempt to authenticate using SASL (unknown error) (org.apache.zookeeper.ClientCnxn)
[2018-08-07 17:21:49,856] INFO Socket connection established to 10.42.2.218/10.42.2.218:2181, initiating session (org.apache.zookeeper.ClientCnxn)
[2018-08-07 17:21:49,857] WARN Session 0x0 for server 10.42.2.218/10.42.2.218:2181, unexpected error, closing socket connection and attempting reconnect (org.apache.zookeeper.ClientCnxn)
java.io.IOException: Packet len352518400 is out of range!

The error only occurs when dmaap is deployed with Istio. Without Istio, dmaap comes up fine.

This issue occurs both with mTLS enabled and when mTLS is disabled.

2message-routermessage-router-kafka is not ready
Depends on 1
3sdnc-dmaap-listenermessage-router is not ready
Depends on 2
4Http liveness probeMutual TLS can't work with K8S http/tcp liveness probe


If mutual TLS is enabled, http and tcp health checks from the kubelet will not work since they do not have Istio-issued certs.
5



6



7



8



9



  • No labels

5 Comments

  1. Srinivasa Addepalli

    Kiran Kamineni

    In case of headless service, there is no clusterIP. As I understand, ISTIO learns the services from service records. If cluster IP is not present, this learning does not happen well is my guess.  In these cases, in my view, it is required to register the IP addresses and ports explicitly with ISTIO.  Can you try following?

    • Figure out the IP addresses of all zookeper containers and also port on which zookeeper is listening on.
    • Let us give name of this service as zookeeper-service.
    • Using istioctl register command, register all IP addresses and port of zookeeper containers to zookeeper-service.  That, for each zooker container, do following
      • istioctl -n <namespace> zookeeper-service <IP address> <port>

    Also, can you list down the IP tables rules in both zookeeper container and the container that accessing zookeeper service?

    Srini

  2. Kiran Kamineni

    Huabing, I saw your comment on the dmaap. But, it is now deleted. Were you able to get it to work?

  3. Huabing Zhao

    All the dmaap pods have been spun up successfully, but I still found error messages in the docker logs.  It seems that connections to zookeepers succeed after a few retries.


    [2018-08-18 04:55:10,896] INFO Opening socket connection to server 192-168-54-182.message-router-zookeeper.onap.svc.cluster.local/192.168.54.182:2181
    . Will not attempt to authenticate using SASL (unknown error) (org.apache.zookeeper.ClientCnxn)
    [2018-08-18 04:55:10,903] WARN Session 0x0 for server null, unexpected error, closing socket connection and attempting reconnect (org.apache.zookeepe
    r.ClientCnxn)
    java.net.ConnectException: Connection refused
    at sun.nio.ch.SocketChannelImpl.checkConnect(Native Method)
    at sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:717)
    at org.apache.zookeeper.ClientCnxnSocketNIO.doTransport(ClientCnxnSocketNIO.java:361)
    at org.apache.zookeeper.ClientCnxn$SendThread.run(ClientCnxn.java:1141)
    [2018-08-18 04:55:12,006] INFO Opening socket connection to server 192-168-54-182.message-router-zookeeper.onap.svc.cluster.local/192.168.54.182:2181
    . Will not attempt to authenticate using SASL (unknown error) (org.apache.zookeeper.ClientCnxn)
    [2018-08-18 04:55:12,007] WARN Session 0x0 for server null, unexpected error, closing socket connection and attempting reconnect (org.apache.zookeepe
    r.ClientCnxn)
    java.net.ConnectException: Connection refused
    at sun.nio.ch.SocketChannelImpl.checkConnect(Native Method)
    at sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:717)
    at org.apache.zookeeper.ClientCnxnSocketNIO.doTransport(ClientCnxnSocketNIO.java:361)
    at org.apache.zookeeper.ClientCnxn$SendThread.run(ClientCnxn.java:1141)
    [2018-08-18 04:55:13,108] INFO Opening socket connection to server 192-168-54-182.message-router-zookeeper.onap.svc.cluster.local/192.168.54.182:2181
    . Will not attempt to authenticate using SASL (unknown error) (org.apache.zookeeper.ClientCnxn)
    [2018-08-18 04:55:13,109] INFO Socket connection established to 192-168-54-182.message-router-zookeeper.onap.svc.cluster.local/192.168.54.182:2181, i
    nitiating session (org.apache.zookeeper.ClientCnxn)
    [2018-08-18 04:55:13,145] INFO Session establishment complete on server 192-168-54-182.message-router-zookeeper.onap.svc.cluster.local/192.168.54.182
    :2181, sessionid = 0x1654b62b3840000, negotiated timeout = 6000 (org.apache.zookeeper.ClientCnxn)
    [2018-08-18 04:55:13,149] INFO [ZooKeeperClient] Connected. (kafka.zookeeper.ZooKeeperClient)
    [2018-08-18 04:55:13,574] INFO Cluster ID = 9aJfPXOuQw-7g0qnXjEHSQ (kafka.server.KafkaServer

    1. Kiran Kamineni

      Let me try to repro this on my side.

      I understand that you are using Istio 1.0

      Can you give me the versions you are using with docker and kubernetes? 

      I am trying it out with docker 17.03.2-ce and Kubernetes v1.9.7

  4. Huabing Zhao

    Sure,

    ubuntu@istio-k8s-master:~$ sudo docker version
    Client:
    Version: 17.03.2-ce
    API version: 1.27
    Go version: go1.6.2
    Git commit: f5ec1e2
    Built: Thu Jul 5 23:07:48 2018
    OS/Arch: linux/amd64

    Server:
    Version: 17.03.2-ce
    API version: 1.27 (minimum version 1.12)
    Go version: go1.6.2
    Git commit: f5ec1e2
    Built: Thu Jul 5 23:07:48 2018
    OS/Arch: linux/amd64
    Experimental: false

    ubuntu@istio-k8s-master:~$ sudo kubectl version
    Client Version: version.Info{Major:"1", Minor:"11", GitVersion:"v1.11.1", GitCommit:"b1b29978270dc22fecc592ac55d903350454310a", GitTreeState:"clean", BuildDate:"2018-07-17T18:53:20Z", GoVersion:"go1.10.3", Compiler:"gc", Platform:"linux/amd64"}
    Server Version: version.Info{Major:"1", Minor:"11", GitVersion:"v1.11.1", GitCommit:"b1b29978270dc22fecc592ac55d903350454310a", GitTreeState:"clean", BuildDate:"2018-07-17T18:43:26Z", GoVersion:"go1.10.3", Compiler:"gc", Platform:"linux/amd64"}