sdc-FE docker container is not starting up while creating jetty modules

 
3
2
1

Hi,

I am trying to install ONAP behind proxies and everything looks good so far. Made few changes to heat template to use proxies in docker and run-time environment. However I am having issues while creating sdc-FE docker container. 

Below are the docker failure logs. This will work without proxy. Can you help me to fix this problem?

I have tried passing env variables to docker run command but no use

docker run --name sdc-FE --env HOST_IP=<host_ip> --env http_proxy=<proxy_ip> --env https_proxy=<proxy_ip> --env ENVNAME="AUTO" --log-driver=json-file --log-opt max-size=100m --log-opt max-file=10 --ulimit memlock=-1:-1 --memory 2g --memory-swap=2g --ulimit nofile=4096:100000 --volume /etc/localtime:/etc/localtime:ro  --volume /data/logs/FE/:/var/lib/jetty/logs --volume /data/environments:/root/chef-solo/environments --publish 9443:9443 --publish 8181:8181 nexus3.onap.org:10001/openecomp/sdc-frontend:latest


Recipe: sdc-catalog-fe::FE_6_create_jetty_modules
* bash[create-jetty-modules] action run
[execute] INFO: logging initialised in ${jetty.base}/start.d/logging.ini
INFO: ssl initialised (transitively) in ${jetty.base}/start.d/ssl.ini
INFO: https initialised in ${jetty.base}/start.d/https.ini
DOWNLOAD: https://raw.githubusercontent.com/eclipse/jetty.project/master/jetty-server/src/test/config/etc/keystore?id=master to ${jetty.base}/etc/keystore




java.net.ConnectException: Connection timed out (Connection timed out)
at java.net.PlainSocketImpl.socketConnect(Native Method)
at java.net.AbstractPlainSocketImpl.doConnect(AbstractPlainSocketImpl.java:350)
at java.net.AbstractPlainSocketImpl.connectToAddress(AbstractPlainSocketImpl.java:206)
at java.net.AbstractPlainSocketImpl.connect(AbstractPlainSocketImpl.java:188)
at java.net.SocksSocketImpl.connect(SocksSocketImpl.java:392)
at java.net.Socket.connect(Socket.java:589)
at sun.security.ssl.SSLSocketImpl.connect(SSLSocketImpl.java:668)
at sun.security.ssl.BaseSSLSocketImpl.connect(BaseSSLSocketImpl.java:173)
at sun.net.NetworkClient.doConnect(NetworkClient.java:180)
at sun.net.www.http.HttpClient.openServer(HttpClient.java:432)
at sun.net.www.http.HttpClient.openServer(HttpClient.java:527)
at sun.net.www.protocol.https.HttpsClient.<init>(HttpsClient.java:264)
at sun.net.www.protocol.https.HttpsClient.New(HttpsClient.java:367)
at sun.net.www.protocol.https.AbstractDelegateHttpsURLConnection.getNewHttpClient(AbstractDelegateHttpsURLConnection.java:191)
at sun.net.www.protocol.http.HttpURLConnection.plainConnect0(HttpURLConnection.java:1138)
at sun.net.www.protocol.http.HttpURLConnection.plainConnect(HttpURLConnection.java:1032)
at sun.net.www.protocol.https.AbstractDelegateHttpsURLConnection.connect(AbstractDelegateHttpsURLConnection.java:177)
at sun.net.www.protocol.http.HttpURLConnection.getInputStream0(HttpURLConnection.java:1546)
at sun.net.www.protocol.http.HttpURLConnection.getInputStream(HttpURLConnection.java:1474)
at java.net.HttpURLConnection.getResponseCode(HttpURLConnection.java:480)
at sun.net.www.protocol.https.HttpsURLConnectionImpl.getResponseCode(HttpsURLConnectionImpl.java:338)
at org.eclipse.jetty.start.fileinits.UriFileInitializer.download(UriFileInitializer.java:75)
at org.eclipse.jetty.start.fileinits.UriFileInitializer.init(UriFileInitializer.java:60)
at org.eclipse.jetty.start.BaseBuilder.processFileResource(BaseBuilder.java:283)
at org.eclipse.jetty.start.BaseBuilder.processFileResources(BaseBuilder.java:375)
at org.eclipse.jetty.start.BaseBuilder.build(BaseBuilder.java:239)
at org.eclipse.jetty.start.Main.start(Main.java:407)
at org.eclipse.jetty.start.Main.main(Main.java:75)
WARNING: Failed to process all file resources.
- [ConnectException] Connection timed out (Connection timed out) - /var/lib/jetty/etc/keystore
java.lang.RuntimeException: Failed to process all file resources.
- [ConnectException] Connection timed out (Connection timed out) - /var/lib/jetty/etc/keystore
at org.eclipse.jetty.start.BaseBuilder.processFileResources(BaseBuilder.java:394)
at org.eclipse.jetty.start.BaseBuilder.build(BaseBuilder.java:239)
at org.eclipse.jetty.start.Main.start(Main.java:407)
at org.eclipse.jetty.start.Main.main(Main.java:75)

Usage: java -jar start.jar [options] [properties] [configs]
java -jar start.jar --help # for more information

================================================================================
Error executing action `run` on resource 'bash[create-jetty-modules]'
================================================================================

Mixlib::ShellOut::ShellCommandFailed
------------------------------------
Expected process to exit with [0], but received '247'
---- Begin output of "bash" "/tmp/jetty/chef-script20170510-13-1fcyeq3" ----
STDOUT:
STDERR: INFO: logging initialised in ${jetty.base}/start.d/logging.ini
INFO: ssl initialised (transitively) in ${jetty.base}/start.d/ssl.ini
INFO: https initialised in ${jetty.base}/start.d/https.ini
DOWNLOAD: https://raw.githubusercontent.com/eclipse/jetty.project/master/jetty-server/src/test/config/etc/keystore?id=master to ${jetty.base}/etc/keystore
java.net.ConnectException: Connection timed out (Connection timed out)
at java.net.PlainSocketImpl.socketConnect(Native Method)
at java.net.AbstractPlainSocketImpl.doConnect(AbstractPlainSocketImpl.java:350)
at java.net.AbstractPlainSocketImpl.connectToAddress(AbstractPlainSocketImpl.java:206)
at java.net.AbstractPlainSocketImpl.connect(AbstractPlainSocketImpl.java:188)
at java.net.SocksSocketImpl.connect(SocksSocketImpl.java:392)
at java.net.Socket.connect(Socket.java:589)
at sun.security.ssl.SSLSocketImpl.connect(SSLSocketImpl.java:668)
at sun.security.ssl.BaseSSLSocketImpl.connect(BaseSSLSocketImpl.java:173)
at sun.net.NetworkClient.doConnect(NetworkClient.java:180)
at sun.net.www.http.HttpClient.openServer(HttpClient.java:432)
at sun.net.www.http.HttpClient.openServer(HttpClient.java:527)
at sun.net.www.protocol.https.HttpsClient.<init>(HttpsClient.java:264)
at sun.net.www.protocol.https.HttpsClient.New(HttpsClient.java:367)
at sun.net.www.protocol.https.AbstractDelegateHttpsURLConnection.getNewHttpClient(AbstractDelegateHttpsURLConnection.java:191)
at sun.net.www.protocol.http.HttpURLConnection.plainConnect0(HttpURLConnection.java:1138)
at sun.net.www.protocol.http.HttpURLConnection.plainConnect(HttpURLConnection.java:1032)
at sun.net.www.protocol.https.AbstractDelegateHttpsURLConnection.connect(AbstractDelegateHttpsURLConnection.java:177)
at sun.net.www.protocol.http.HttpURLConnection.getInputStream0(HttpURLConnection.java:1546)
at sun.net.www.protocol.http.HttpURLConnection.getInputStream(HttpURLConnection.java:1474)
at java.net.HttpURLConnection.getResponseCode(HttpURLConnection.java:480)
at sun.net.www.protocol.https.HttpsURLConnectionImpl.getResponseCode(HttpsURLConnectionImpl.java:338)
at org.eclipse.jetty.start.fileinits.UriFileInitializer.download(UriFileInitializer.java:75)
at org.eclipse.jetty.start.fileinits.UriFileInitializer.init(UriFileInitializer.java:60)
at org.eclipse.jetty.start.BaseBuilder.processFileResource(BaseBuilder.java:283)
at org.eclipse.jetty.start.BaseBuilder.processFileResources(BaseBuilder.java:375)
at org.eclipse.jetty.start.BaseBuilder.build(BaseBuilder.java:239)
at org.eclipse.jetty.start.Main.start(Main.java:407)
at org.eclipse.jetty.start.Main.main(Main.java:75)
WARNING: Failed to process all file resources.
- [ConnectException] Connection timed out (Connection timed out) - /var/lib/jetty/etc/keystore
java.lang.RuntimeException: Failed to process all file resources.
- [ConnectException] Connection timed out (Connection timed out) - /var/lib/jetty/etc/keystore
at org.eclipse.jetty.start.BaseBuilder.processFileResources(BaseBuilder.java:394)
at org.eclipse.jetty.start.BaseBuilder.build(BaseBuilder.java:239)
at org.eclipse.jetty.start.Main.start(Main.java:407)
at org.eclipse.jetty.start.Main.main(Main.java:75)

Usage: java -jar start.jar [options] [properties] [configs]
java -jar start.jar --help # for more information
---- End output of "bash" "/tmp/jetty/chef-script20170510-13-1fcyeq3" ----
Ran "bash" "/tmp/jetty/chef-script20170510-13-1fcyeq3" returned 247

Resource Declaration:
---------------------
# In /root/chef-solo/cache/cookbooks/sdc-catalog-fe/recipes/FE_6_create_jetty_modules.rb

5: bash "create-jetty-modules" do
6: cwd "#{jetty_base}"
7: code <<-EOH
8: cd "#{jetty_base}"
9: java -jar "/#{jetty_home}"/start.jar --add-to-start=deploy
10: java -jar "/#{jetty_home}"/start.jar --add-to-startd=http,https,logging,setuid
11: EOH
12: not_if "ls /#{jetty_base}/start.d/https.ini"
13: end
14:

Compiled Resource:
------------------
# Declared in /root/chef-solo/cache/cookbooks/sdc-catalog-fe/recipes/FE_6_create_jetty_modules.rb:5:in `from_file'

bash("create-jetty-modules") do
action [:run]
default_guard_interpreter :default
command nil
backup 5
cwd "/var/lib/jetty"
returns 0
user nil
interpreter "bash"
declared_type :bash
cookbook_name "sdc-catalog-fe"
recipe_name "FE_6_create_jetty_modules"
code " cd \"/var/lib/jetty\"\n java -jar \"//usr/local/jetty\"/start.jar --add-to-start=deploy\n java -jar \"//usr/local/jetty\"/start.jar --add-to-startd=http,https,logging,setuid\n"
domain nil
not_if "ls //var/lib/jetty/start.d/https.ini"
end


  1. Catherine Lefevre

    Adding Michael Lando

  2. kranthi guttikonda

    A little bit help here would be appreciated. I see the same failure in SDC_BE as well but process didn't exit at that point. 

  3. kranthi guttikonda

    Hi @Dusan Rozman,

    Thanks for the reply and information. Appreciate your help. In my setup I believe jetty modules creation and download keystore file failures is because of environment proxies. I have updated Dockerfile but no use.

    I have tried docker mount option like below for sdc-BE

    sdc/sdc-os-chef/docker-compose.yml

    jettyBE:
    build: .
    container_name: "sdc-be"
    image: "ecomp-nexus:51212/ecomp/sdc-backend:1610.2.13"
    mem_limit: "3g"
    memswap_limit: "3g"
    logging:
    driver: "json-file"
    options:
    max-size: "100m"
    max-file: "10"
    ports:
    - "8080:8080"
    - "8443:8443"
    environment:
    - HOST_IP:HOSTIP
    - ENVNAME:DEP_ENV
    volumes:
    - /etc/localtime:/etc/localtime:ro
    - /data/ASDC/logs/BE:/var/lib/jetty/logs
    - /data/ASDC/environments:/root/chef-solo/environments
    - /root/keystore:/var/lib/jetty/etc/keystore
    ulimits:
    memlock: -1
    nproc: 65535
    nofile:
    soft: 100000
    hard: 100000

    I have keystore download at /root in docker host and mentioned mount point path. But this didn't work as well. Any ideas? Am I doing correct?


CommentAdd your comment...

2 answers

  1.  
    3
    2
    1

    I did the following to solve this issue

    Download the keystore file first to docker host VM

    wget https://raw.githubusercontent.com/eclipse/jetty.project/master/jetty-server/src/test/config/etc/keystore?id=master

    Delete the containers started by scripts already

    docker rm -f sdc-BE

    docker rm -f sdc-FE

    For sdc-BE 

    Start the sdc-BE container using following command

    docker run --name sdc-BE --env HOST_IP=<Host_IP> --env ENVNAME="AUTO" --log-driver=json-file --log-opt max-size=100m --log-opt max-file=10 --ulimit memlock=-1:-1 --memory 4g --memory-swap=4g --ulimit nofile=4096:100000 --volume /etc/localtime:/etc/localtime:ro --volume /data/logs/BE/:/var/lib/jetty/logs  --volume /data/environments:/root/chef-solo/environments --publish 8443:8443 --publish 8080:8080 nexus3.onap.org:10001/openecomp/sdc-backend:1.0-STAGING-latest

    Copy keystore file downloaded to container

    docker cp keystore sdc-BE:/var/lib/jetty/etc/

    Login to container bash

    docker exec -it sdc-BE bash

    update /root/startup.sh JAVA_OPTIONS (technically this will update /docker-entrypoint.sh

    sed -i '/^set -e/aJAVA_OPTIONS=\" -XX:MaxPermSize=256m -Xmx1500m -Dconfig.home=${JETTY_BASE}\/config -Dlog.home=${JETTY_BASE}\/logs -Dlogback.configurationFile=${JETTY_BASE}\/config\/catalog-be\/logback.xml -Dconfiguration.yaml=${JETTY_BASE}\/config\/catalog-be\/configuration.yaml -Dhttp.proxyHost=<http_proxy> -Dhttp.proxyPort=<http_proxy_port> -Dhttps.proxyHost=<https_proxy> -Dhttps.proxyPort=<https_port>\"' /docker-entrypoint.sh

    Exit from the shell

    docker commit sdc-BE nexus3.onap.org:10001/openecomp/sdc-backend-proxy

    stop and delete sdc-BE

    docker stop sdc-BE

    docker rm -f sdc-BE

    Now run the sdc-BE with latest image created in background

    docker run --detach --name sdc-BE --env HOST_IP=<Host_IP> --env ENVNAME="AUTO" --log-driver=json-file --log-opt max-size=100m --log-opt max-file=10 --ulimit memlock=-1:-1 --memory 4g --memory-swap=4g --ulimit nofile=4096:100000 --volume /etc/localtime:/etc/localtime:ro --volume /data/logs/BE/:/var/lib/jetty/logs  --volume /data/environments:/root/chef-solo/environments --publish 8443:8443 --publish 8080:8080 nexus3.onap.org:10001/openecomp/sdc-backend-proxy:latest


    Following the similar steps for sdc-FE container as well. 


    Run the health-checks should pass now and GUI SDC page will open


    root@vm1-sdc:~# /data/scripts/docker_health.sh
    {
    "cluster_name" : "SDC-ES-AUTO",
    "status" : "green",
    "timed_out" : false,
    "number_of_nodes" : 1,
    "number_of_data_nodes" : 1,
    "active_primary_shards" : 2,
    "active_shards" : 2,
    "relocating_shards" : 0,
    "initializing_shards" : 0,
    "unassigned_shards" : 0,
    "delayed_unassigned_shards" : 0,
    "number_of_pending_tasks" : 0,
    "number_of_in_flight_fetch" : 0,
    "task_max_waiting_in_queue_millis" : 0,
    "active_shards_percent_as_number" : 100.0
    }
    BE health-Check:
    {
    "sdcVersion": "1.0.0",
    "siteMode": "unknown",
    "componentsInfo": [
    {
    "healthCheckComponent": "BE",
    "healthCheckStatus": "UP",
    "version": "1.0.0",
    "description": "OK"
    },
    {
    "healthCheckComponent": "ES",
    "healthCheckStatus": "UP",
    "description": "OK"
    },
    {
    "healthCheckComponent": "TITAN",
    "healthCheckStatus": "UP",
    "description": "OK"
    },
    {
    "healthCheckComponent": "DE",
    "healthCheckStatus": "UP",
    "description": "OK"
    }
    ]
    }

    FE health-Check:
    {
    "sdcVersion": "1.0.0",
    "siteMode": "unknown",
    "componentsInfo": [
    {
    "healthCheckComponent": "BE",
    "healthCheckStatus": "UP",
    "version": "1.0.0",
    "description": "OK"
    },
    {
    "healthCheckComponent": "ES",
    "healthCheckStatus": "UP",
    "description": "OK"
    },
    {
    "healthCheckComponent": "TITAN",
    "healthCheckStatus": "UP",
    "description": "OK"
    },
    {
    "healthCheckComponent": "DE",
    "healthCheckStatus": "UP",
    "description": "OK"
    },
    {
    "healthCheckComponent": "FE",
    "healthCheckStatus": "UP",
    "version": "1.0.0",
    "description": "OK"
    }
    ]
    }

    check user existance: OK
    root@vm1-sdc:~#


    1. Eli Williams

      Thank you for this! It worked for me as well.

    CommentAdd your comment...
  2.  
    1
    0
    -1

    Hi kranthi guttikonda

    I had similar issue when I was trying to run the SDC container. I can share what I have encountered and learned and hope it can apply to your situation.


    DOWNLOAD: https://raw.githubusercontent.com/eclipse/jetty.project/master/jetty-server/src/test/config/etc/keystore?id=master to ${jetty.base}/etc/keystore
java.net.ConnectException: Connection timed out (Connection timed out)

    In this portion of the error we see SDC is trying to download 'keystore' file. Take a look inside your SDC-BE and SDC-FE container and go to /var/lib/jetty/etc/ directory. Inside here should be the file 'keystore'. In my case it was missing and from what I can gather from your error logs your sdc-be and sdc-fe are missing them as well. 


    So why is it missing? One of the start-up scripts (the docker-entry shell script, IIRC) is suppose to run 'start.jar' which downloads the keystore file. For some reason the download is not successful and results in the connection time out you are seeing in the above error message. I have played around inside the container doing a 'wget' and also trying to rerun the 'start.jar' file (with the '--create-files' option) with no success. The 'wget' for instance just hangs and results in a time-out. This is something I will need to investigate further as it may be timing out for several reasons.


    That said, my current work around was to locally (outside of the container) download the keystore file (using the download URL shown in the error log) and then place it in a persistent mount for both sdc-be and sdc-fe to use. This fixed my issue, however, this is just a temporary solution; I'll need to next look into resolving the time-out issue. If you find a solution or anyone else has any ideas or suggestions I would appreciate the feedback greatly.

    1. kranthi guttikonda

      Can you please help me to understand how to put the file into docker container while it running? I see that it stops immediately when I start it. I am having hard time to understand docker AUFS file system for persistent mount. Looks like from docker 1.10 it's no longer uses container id.

    2. Dusan Rozman


      At the moment, I am using Kubernetes to manage my containers and use it for volume mounts using hostPath: (https://kubernetes.io/docs/concepts/storage/volumes/#hostpath. However, the solution should be similar using docker files. You can read up a more in-depth response about docker mounts here: https://docs.docker.com/compose/compose-file/#volumes-for-services-swarms-and-stack-files

      Essentially you will just need to specify in your yaml where the file (in this case the Keystore file) is located on your docker host server. Example below:


      version: "3"
services:
  db:
    image: postgres:9.4
    volumes:
      - db-data:/var/lib/postgresql/data
    networks:
      - backend
    deploy:
      placement:
        constraints: [node.role == manager]


      If you do not wish to go with docker mounts. You can also use 'docker cp' (docker copy) to insert local files into containers (https://docs.docker.com/engine/reference/commandline/cp/#description). However, this would require the container to be up so you would need to continue your investigation on why the container crashes. 

    3. Viswanath KSP

      Am also stuck with same error. Awaiting resolution.

      +1


    CommentAdd your comment...