SDC FE not coming up . Connection refused to port 8181

 
1
0
-1

As part of SDC, SDC-FE is not coming up. All other containers are up & running. I also see a connection refused error.

Running health checks, please wait...
0
{
"cluster_name" : "SDC-ES-AUTO",
"status" : "green",
"timed_out" : false,
"number_of_nodes" : 1,
"number_of_data_nodes" : 1,
"active_primary_shards" : 1,
"active_shards" : 1,
"relocating_shards" : 0,
"initializing_shards" : 0,
"unassigned_shards" : 0,
"delayed_unassigned_shards" : 0,
"number_of_pending_tasks" : 0,
"number_of_in_flight_fetch" : 0,
"task_max_waiting_in_queue_millis" : 0,
"active_shards_percent_as_number" : 100.0
}
BE health-Check:
curl: (56) Recv failure: Connection reset by peer

FE health-Check:
curl: (7) Failed to connect to localhost port 8181: Connection refused


Could you please let me know where to find log files for FE, so that I can take a look at them and understand what is missing ?

  1. Viswanath KSP

    Thanks. I used docker logs <container ID> and here's what I found :

    DOWNLOAD: https://raw.githubusercontent.com/eclipse/jetty.project/master/jetty-server/src/test/config/etc/keystore?id=master to ${jetty.base}/etc/keystore
java.net.UnknownHostException: raw.githubusercontent.com
at java.net.AbstractPlainSocketImpl.connect(AbstractPlainSocketImpl.java:184)
at java.net.SocksSocketImpl.connect(SocksSocketImpl.java:392)
at java.net.Socket.connect(Socket.java:589)
at sun.security.ssl.SSLSocketImpl.connect(SSLSocketImpl.java:668)
at sun.security.ssl.BaseSSLSocketImpl.connect(BaseSSLSocketImpl.java:173)
at sun.net.NetworkClient.doConnect(NetworkClient.java:180)
at sun.net.www.http.HttpClient.openServer(HttpClient.java:432)
at sun.net.www.http.HttpClient.openServer(HttpClient.java:527)
at sun.net.www.protocol.https.HttpsClient.<init>(HttpsClient.java:264)
at sun.net.www.protocol.https.HttpsClient.New(HttpsClient.java:367)
at sun.net.www.protocol.https.AbstractDelegateHttpsURLConnection.getNewHttpClient(AbstractDelegateHttpsURLConnection.java:191)
at sun.net.www.protocol.http.HttpURLConnection.plainConnect0(HttpURLConnection.java:1138)
at sun.net.www.protocol.http.HttpURLConnection.plainConnect(HttpURLConnection.java:1032)
at sun.net.www.protocol.https.AbstractDelegateHttpsURLConnection.connect(AbstractDelegateHttpsURLConnection.java:177)
at sun.net.www.protocol.http.HttpURLConnection.getInputStream0(HttpURLConnection.java:1546)
at sun.net.www.protocol.http.HttpURLConnection.getInputStream(HttpURLConnection.java:1474)
at java.net.HttpURLConnection.getResponseCode(HttpURLConnection.java:480)
at sun.net.www.protocol.https.HttpsURLConnectionImpl.getResponseCode(HttpsURLConnectionImpl.java:338)
at org.eclipse.jetty.start.fileinits.UriFileInitializer.download(UriFileInitializer.java:75)
at org.eclipse.jetty.start.fileinits.UriFileInitializer.init(UriFileInitializer.java:60)
at org.eclipse.jetty.start.BaseBuilder.processFileResource(BaseBuilder.java:283)
at org.eclipse.jetty.start.BaseBuilder.processFileResources(BaseBuilder.java:375)
at org.eclipse.jetty.start.BaseBuilder.build(BaseBuilder.java:239)
at org.eclipse.jetty.start.Main.start(Main.java:407)
at org.eclipse.jetty.start.Main.main(Main.java:75)
WARNING: Failed to process all file resources.
- [UnknownHostException] raw.githubusercontent.com - /var/lib/jetty/etc/keystore
java.lang.RuntimeException: Failed to process all file resources.
- [UnknownHostException] raw.githubusercontent.com - /var/lib/jetty/etc/keystore
at org.eclipse.jetty.start.BaseBuilder.processFileResources(BaseBuilder.java:394)
at org.eclipse.jetty.start.BaseBuilder.build(BaseBuilder.java:239)
at org.eclipse.jetty.start.Main.start(Main.java:407)
at org.eclipse.jetty.start.Main.main(Main.java:75)


    This one is similar to sdc-FE docker container is not starting up while creating jetty modules but I'm not sure.

    Could someone help me resolve this Keystore issue?

  2. Josef Reisinger

    https://raw.githubusercontent.com/eclipse/jetty.project/master/jetty-server/src/test/config/etc/keystore?id=master to ${jetty.base}/etc/keystore java.net.UnknownHostException: raw.githubusercontent.com
    This looks pretty much like an DNS error: The container does not know how to resolve raw.githubusercontent.com into an IP address. This can have a couple of reason. You may start with 

    docker exec it sdc-FE ping raw.githubusercontent.com

    and see whether the ping is able to do the translation. I would guess not. Do the same on the VM running the sdc-FE container

    You may post your /etc/resolv.conf from the sdc-FE container as well as from the SDC VM. Also check that the DNS server configuration allows recursive - that might help if you do not have other DNS servers configured.

  3. Vitaly Emporopulo

    It does look like a firewall issue. I belies the link you mentioned describes a similar issue. Try to resolve this by running the containers with appropriate environment variables (https_proxy,  HTTPS_PROXY).

  4. Viswanath KSP

    Josef Reisinger Vitaly Emporopulo - I tried all possible combination in specifying https proxy. Unfortunately I'm behind corporate proxy and I cannot have a direct connection. Could you let me know why the container crashes ? If we can let the container running, I can generate Keystore files using start.jar --create-files option. But right now, as soon I start FE container, it crashes.

  5. Vitaly Emporopulo

    Unfortunately I don't have an environment to simulate your issue right now, bu I would probably do the following:

    1. Run the image while overriding its entry point (docker run --entrypoint "/bin/sh" <image>)
    2. Download the file keystore manually and inject it into the container.
    3. Save the container as a new image (docker commit)
    4. Run with the new image.

    Did you try the workaround suggested in sdc-FE docker container is not starting up while creating jetty modules?

    I must say that I too experience a lot of problems because of the corporate proxy/firewall.

  6. Viswanath KSP

    Vitaly Emporopulo - Even if I run FE image with /bin/bash entry point, container is crashing. I tried following.


     docker run --name "viswa" --entrypoint "/bin/sh" nexus3.onap.org:10001/openecomp/sdc-frontend:1.0-STAGING-latest

CONTAINER ID IMAGE COMMAND CREATED STATUS PORTS NAMES
71d699a45079 nexus3.onap.org:10001/openecomp/sdc-frontend:1.0-STAGING-latest "/bin/sh" 9 seconds ago Exited (0) 9 seconds ago viswa
  7. Vitaly Emporopulo

    docker run -ti --name "viswa" --entrypoint "/bin/bash" nexus3.onap.org:10001/openecomp/sdc-frontend:1.0-STAGING-latest

    Notice -ti to run in the interactive mode.

  8. Viswanath KSP

    Vitaly Emporopulo - Thanks for suggesting ti mode. However if I commit the image and re-use it, the entry point is always set to /bin/bash. 

    root@vm1-sdc:/opt# docker ps -a
CONTAINER ID IMAGE COMMAND CREATED STATUS PORTS NAMES
32a5375e3f9c nexus3.onap.org:10001/openecomp/sdc-frontend:1.0-STAGING-kspviswa "/bin/bash" 2 minutes ago Exited (0) 2 minutes ago sdc-FE
585c0c245cfc nexus3.onap.org:10001/openecomp/sdc-backend:latest "/root/startup.sh" 3 minutes ago Up 3 minutes 0.0.0.0:8080->8080/tcp, 0.0.0.0:8443->8443/tcp sdc-BE
4916f1176b14 nexus3.onap.org:10001/openecomp/sdc-kibana:latest "/root/startup.sh" 4 minutes ago Up 4 minutes 0.0.0.0:5601->5601/tcp sdc-kbn
1c9a3549c654 nexus3.onap.org:10001/openecomp/sdc-cassandra:latest "/root/startup.sh" 4 minutes ago Up 4 minutes 7000-7001/tcp, 0.0.0.0:9042->9042/tcp, 7199/tcp, 0.0.0.0:9160->9160/tcp sdc-cs
ad52890770a1 nexus3.onap.org:10001/openecomp/sdc-elasticsearch:latest "/root/startup.sh" 5 minutes ago Up 5 minutes 0.0.0.0:9200->9200/tcp, 0.0.0.0:9300->9300/tcp sdc-es
root@vm1-sdc:/opt# docker logs 32a5375e3f9c


    Could you please let me know how to re-use this image correctly ?

    BR,

    Viswa

  9. Vitaly Emporopulo

    Try to use the same trick - run the modified image with  

    --entrypoint "/root/startup.sh"

    This is really a workaround to get you started. It looks like support for proxies should be added to the docker images beyond the environment variables. I see Jetty uses a standard Java connection, so an image must allow passing proxy configuration as described in Java Networking and Proxies. I will raise an issue to get this fixed.

  10. Michael Lando

    Viswanath KSP where you successful?

CommentAdd your comment...

5 answers

  1.  
    1
    0
    -1
  2.  
    1
    0
    -1

    hi,

    the FE and BE dockers were never tested inside a corporate network.

    this is something that we will need to work on.

    but from your description, i think the issue is inside the VM you are using.

    i suggest you start any docker and try to ping from inside it to test that you have connectivity from the docker to the outside. 


      CommentAdd your comment...
    1.  
      1
      0
      -1

      Adding http environment variables are no use. If it fails at creating_jetty_modules then sdc-FE container will crash but sdc-BE may still work. after docker entry point container performs few Chef runs and it failing. Perhaps somehow we need to pass environment variables to java/jetty?

        CommentAdd your comment...
      1.  
        1
        0
        -1

        Is your BE container running? To begin with, check `docker logs <fe-container>` and `docker logs <be-container>`.


          CommentAdd your comment...
        1.  
          1
          0
          -1

          I just deleted my stack, but from my memory I would say it is under /data/logs/FE/<date-of-logfile>, likewise is BE (replace FE by BE in the path).

            CommentAdd your comment...