I did the following to solve this issue
Download the keystore file first to docker host VM
Delete the containers started by scripts already
docker rm -f sdc-BE
docker rm -f sdc-FE
For sdc-BE
Start the sdc-BE container using following command
docker run --name sdc-BE --env HOST_IP=<Host_IP> --env ENVNAME="AUTO" --log-driver=json-file --log-opt max-size=100m --log-opt max-file=10 --ulimit memlock=-1:-1 --memory 4g --memory-swap=4g --ulimit nofile=4096:100000 --volume /etc/localtime:/etc/localtime:ro --volume /data/logs/BE/:/var/lib/jetty/logs --volume /data/environments:/root/chef-solo/environments --publish 8443:8443 --publish 8080:8080 nexus3.onap.org:10001/openecomp/sdc-backend:1.0-STAGING-latest
Copy keystore file downloaded to container
docker cp keystore sdc-BE:/var/lib/jetty/etc/
Login to container bash
docker exec -it sdc-BE bash
update /root/startup.sh JAVA_OPTIONS (technically this will update /docker-entrypoint.sh
sed -i '/^set -e/aJAVA_OPTIONS=\" -XX:MaxPermSize=256m -Xmx1500m -Dconfig.home=${JETTY_BASE}\/config -Dlog.home=${JETTY_BASE}\/logs -Dlogback.configurationFile=${JETTY_BASE}\/config\/catalog-be\/logback.xml -Dconfiguration.yaml=${JETTY_BASE}\/config\/catalog-be\/configuration.yaml -Dhttp.proxyHost=<http_proxy> -Dhttp.proxyPort=<http_proxy_port> -Dhttps.proxyHost=<https_proxy> -Dhttps.proxyPort=<https_port>\"' /docker-entrypoint.sh
Exit from the shell
docker commit sdc-BE nexus3.onap.org:10001/openecomp/sdc-backend-proxy
stop and delete sdc-BE
docker stop sdc-BE
docker rm -f sdc-BE
Now run the sdc-BE with latest image created in background
docker run --detach --name sdc-BE --env HOST_IP=<Host_IP> --env ENVNAME="AUTO" --log-driver=json-file --log-opt max-size=100m --log-opt max-file=10 --ulimit memlock=-1:-1 --memory 4g --memory-swap=4g --ulimit nofile=4096:100000 --volume /etc/localtime:/etc/localtime:ro --volume /data/logs/BE/:/var/lib/jetty/logs --volume /data/environments:/root/chef-solo/environments --publish 8443:8443 --publish 8080:8080 nexus3.onap.org:10001/openecomp/sdc-backend-proxy:latest
Following the similar steps for sdc-FE container as well.
Run the health-checks should pass now and GUI SDC page will open
root@vm1-sdc:~# /data/scripts/docker_health.sh
{
"cluster_name" : "SDC-ES-AUTO",
"status" : "green",
"timed_out" : false,
"number_of_nodes" : 1,
"number_of_data_nodes" : 1,
"active_primary_shards" : 2,
"active_shards" : 2,
"relocating_shards" : 0,
"initializing_shards" : 0,
"unassigned_shards" : 0,
"delayed_unassigned_shards" : 0,
"number_of_pending_tasks" : 0,
"number_of_in_flight_fetch" : 0,
"task_max_waiting_in_queue_millis" : 0,
"active_shards_percent_as_number" : 100.0
}
BE health-Check:
{
"sdcVersion": "1.0.0",
"siteMode": "unknown",
"componentsInfo": [
{
"healthCheckComponent": "BE",
"healthCheckStatus": "UP",
"version": "1.0.0",
"description": "OK"
},
{
"healthCheckComponent": "ES",
"healthCheckStatus": "UP",
"description": "OK"
},
{
"healthCheckComponent": "TITAN",
"healthCheckStatus": "UP",
"description": "OK"
},
{
"healthCheckComponent": "DE",
"healthCheckStatus": "UP",
"description": "OK"
}
]
}
FE health-Check:
{
"sdcVersion": "1.0.0",
"siteMode": "unknown",
"componentsInfo": [
{
"healthCheckComponent": "BE",
"healthCheckStatus": "UP",
"version": "1.0.0",
"description": "OK"
},
{
"healthCheckComponent": "ES",
"healthCheckStatus": "UP",
"description": "OK"
},
{
"healthCheckComponent": "TITAN",
"healthCheckStatus": "UP",
"description": "OK"
},
{
"healthCheckComponent": "DE",
"healthCheckStatus": "UP",
"description": "OK"
},
{
"healthCheckComponent": "FE",
"healthCheckStatus": "UP",
"version": "1.0.0",
"description": "OK"
}
]
}
check user existance: OK
root@vm1-sdc:~#
Thank you for this! It worked for me as well.
Hi kranthi guttikonda
I had similar issue when I was trying to run the SDC container. I can share what I have encountered and learned and hope it can apply to your situation.
DOWNLOAD: https://raw.githubusercontent.com/eclipse/jetty.project/master/jetty-server/src/test/config/etc/keystore?id=master to ${jetty.base}/etc/keystore java.net.ConnectException: Connection timed out (Connection timed out)
In this portion of the error we see SDC is trying to download 'keystore' file. Take a look inside your SDC-BE and SDC-FE container and go to /var/lib/jetty/etc/ directory. Inside here should be the file 'keystore'. In my case it was missing and from what I can gather from your error logs your sdc-be and sdc-fe are missing them as well.
So why is it missing? One of the start-up scripts (the docker-entry shell script, IIRC) is suppose to run 'start.jar' which downloads the keystore file. For some reason the download is not successful and results in the connection time out you are seeing in the above error message. I have played around inside the container doing a 'wget' and also trying to rerun the 'start.jar' file (with the '--create-files' option) with no success. The 'wget' for instance just hangs and results in a time-out. This is something I will need to investigate further as it may be timing out for several reasons.
That said, my current work around was to locally (outside of the container) download the keystore file (using the download URL shown in the error log) and then place it in a persistent mount for both sdc-be and sdc-fe to use. This fixed my issue, however, this is just a temporary solution; I'll need to next look into resolving the time-out issue. If you find a solution or anyone else has any ideas or suggestions I would appreciate the feedback greatly.
Can you please help me to understand how to put the file into docker container while it running? I see that it stops immediately when I start it. I am having hard time to understand docker AUFS file system for persistent mount. Looks like from docker 1.10 it's no longer uses container id.
At the moment, I am using Kubernetes to manage my containers and use it for volume mounts using hostPath: (https://kubernetes.io/docs/concepts/storage/volumes/#hostpath. However, the solution should be similar using docker files. You can read up a more in-depth response about docker mounts here: https://docs.docker.com/compose/compose-file/#volumes-for-services-swarms-and-stack-files
Essentially you will just need to specify in your yaml where the file (in this case the Keystore file) is located on your docker host server. Example below:
version: "3" services: db: image: postgres:9.4 volumes: - db-data:/var/lib/postgresql/data networks: - backend deploy: placement: constraints: [node.role == manager]
If you do not wish to go with docker mounts. You can also use 'docker cp' (docker copy) to insert local files into containers (https://docs.docker.com/engine/reference/commandline/cp/#description). However, this would require the container to be up so you would need to continue your investigation on why the container crashes.
Am also stuck with same error. Awaiting resolution.
+1
Hi,
I am trying to install ONAP behind proxies and everything looks good so far. Made few changes to heat template to use proxies in docker and run-time environment. However I am having issues while creating sdc-FE docker container.
Below are the docker failure logs. This will work without proxy. Can you help me to fix this problem?
I have tried passing env variables to docker run command but no use
docker run --name sdc-FE --env HOST_IP=<host_ip> --env http_proxy=<proxy_ip> --env https_proxy=<proxy_ip> --env ENVNAME="AUTO" --log-driver=json-file --log-opt max-size=100m --log-opt max-file=10 --ulimit memlock=-1:-1 --memory 2g --memory-swap=2g --ulimit nofile=4096:100000 --volume /etc/localtime:/etc/localtime:ro --volume /data/logs/FE/:/var/lib/jetty/logs --volume /data/environments:/root/chef-solo/environments --publish 9443:9443 --publish 8181:8181 nexus3.onap.org:10001/openecomp/sdc-frontend:latest
Recipe: sdc-catalog-fe::FE_6_create_jetty_modules
* bash[create-jetty-modules] action run
[execute] INFO: logging initialised in ${jetty.base}/start.d/logging.ini
INFO: ssl initialised (transitively) in ${jetty.base}/start.d/ssl.ini
INFO: https initialised in ${jetty.base}/start.d/https.ini
DOWNLOAD: https://raw.githubusercontent.com/eclipse/jetty.project/master/jetty-server/src/test/config/etc/keystore?id=master to ${jetty.base}/etc/keystore
java.net.ConnectException: Connection timed out (Connection timed out)
at java.net.PlainSocketImpl.socketConnect(Native Method)
at java.net.AbstractPlainSocketImpl.doConnect(AbstractPlainSocketImpl.java:350)
at java.net.AbstractPlainSocketImpl.connectToAddress(AbstractPlainSocketImpl.java:206)
at java.net.AbstractPlainSocketImpl.connect(AbstractPlainSocketImpl.java:188)
at java.net.SocksSocketImpl.connect(SocksSocketImpl.java:392)
at java.net.Socket.connect(Socket.java:589)
at sun.security.ssl.SSLSocketImpl.connect(SSLSocketImpl.java:668)
at sun.security.ssl.BaseSSLSocketImpl.connect(BaseSSLSocketImpl.java:173)
at sun.net.NetworkClient.doConnect(NetworkClient.java:180)
at sun.net.www.http.HttpClient.openServer(HttpClient.java:432)
at sun.net.www.http.HttpClient.openServer(HttpClient.java:527)
at sun.net.www.protocol.https.HttpsClient.<init>(HttpsClient.java:264)
at sun.net.www.protocol.https.HttpsClient.New(HttpsClient.java:367)
at sun.net.www.protocol.https.AbstractDelegateHttpsURLConnection.getNewHttpClient(AbstractDelegateHttpsURLConnection.java:191)
at sun.net.www.protocol.http.HttpURLConnection.plainConnect0(HttpURLConnection.java:1138)
at sun.net.www.protocol.http.HttpURLConnection.plainConnect(HttpURLConnection.java:1032)
at sun.net.www.protocol.https.AbstractDelegateHttpsURLConnection.connect(AbstractDelegateHttpsURLConnection.java:177)
at sun.net.www.protocol.http.HttpURLConnection.getInputStream0(HttpURLConnection.java:1546)
at sun.net.www.protocol.http.HttpURLConnection.getInputStream(HttpURLConnection.java:1474)
at java.net.HttpURLConnection.getResponseCode(HttpURLConnection.java:480)
at sun.net.www.protocol.https.HttpsURLConnectionImpl.getResponseCode(HttpsURLConnectionImpl.java:338)
at org.eclipse.jetty.start.fileinits.UriFileInitializer.download(UriFileInitializer.java:75)
at org.eclipse.jetty.start.fileinits.UriFileInitializer.init(UriFileInitializer.java:60)
at org.eclipse.jetty.start.BaseBuilder.processFileResource(BaseBuilder.java:283)
at org.eclipse.jetty.start.BaseBuilder.processFileResources(BaseBuilder.java:375)
at org.eclipse.jetty.start.BaseBuilder.build(BaseBuilder.java:239)
at org.eclipse.jetty.start.Main.start(Main.java:407)
at org.eclipse.jetty.start.Main.main(Main.java:75)
WARNING: Failed to process all file resources.
- [ConnectException] Connection timed out (Connection timed out) - /var/lib/jetty/etc/keystore
java.lang.RuntimeException: Failed to process all file resources.
- [ConnectException] Connection timed out (Connection timed out) - /var/lib/jetty/etc/keystore
at org.eclipse.jetty.start.BaseBuilder.processFileResources(BaseBuilder.java:394)
at org.eclipse.jetty.start.BaseBuilder.build(BaseBuilder.java:239)
at org.eclipse.jetty.start.Main.start(Main.java:407)
at org.eclipse.jetty.start.Main.main(Main.java:75)
Usage: java -jar start.jar [options] [properties] [configs]
java -jar start.jar --help # for more information
================================================================================
Error executing action `run` on resource 'bash[create-jetty-modules]'
================================================================================
Mixlib::ShellOut::ShellCommandFailed
------------------------------------
Expected process to exit with [0], but received '247'
---- Begin output of "bash" "/tmp/jetty/chef-script20170510-13-1fcyeq3" ----
STDOUT:
STDERR: INFO: logging initialised in ${jetty.base}/start.d/logging.ini
INFO: ssl initialised (transitively) in ${jetty.base}/start.d/ssl.ini
INFO: https initialised in ${jetty.base}/start.d/https.ini
DOWNLOAD: https://raw.githubusercontent.com/eclipse/jetty.project/master/jetty-server/src/test/config/etc/keystore?id=master to ${jetty.base}/etc/keystore
java.net.ConnectException: Connection timed out (Connection timed out)
at java.net.PlainSocketImpl.socketConnect(Native Method)
at java.net.AbstractPlainSocketImpl.doConnect(AbstractPlainSocketImpl.java:350)
at java.net.AbstractPlainSocketImpl.connectToAddress(AbstractPlainSocketImpl.java:206)
at java.net.AbstractPlainSocketImpl.connect(AbstractPlainSocketImpl.java:188)
at java.net.SocksSocketImpl.connect(SocksSocketImpl.java:392)
at java.net.Socket.connect(Socket.java:589)
at sun.security.ssl.SSLSocketImpl.connect(SSLSocketImpl.java:668)
at sun.security.ssl.BaseSSLSocketImpl.connect(BaseSSLSocketImpl.java:173)
at sun.net.NetworkClient.doConnect(NetworkClient.java:180)
at sun.net.www.http.HttpClient.openServer(HttpClient.java:432)
at sun.net.www.http.HttpClient.openServer(HttpClient.java:527)
at sun.net.www.protocol.https.HttpsClient.<init>(HttpsClient.java:264)
at sun.net.www.protocol.https.HttpsClient.New(HttpsClient.java:367)
at sun.net.www.protocol.https.AbstractDelegateHttpsURLConnection.getNewHttpClient(AbstractDelegateHttpsURLConnection.java:191)
at sun.net.www.protocol.http.HttpURLConnection.plainConnect0(HttpURLConnection.java:1138)
at sun.net.www.protocol.http.HttpURLConnection.plainConnect(HttpURLConnection.java:1032)
at sun.net.www.protocol.https.AbstractDelegateHttpsURLConnection.connect(AbstractDelegateHttpsURLConnection.java:177)
at sun.net.www.protocol.http.HttpURLConnection.getInputStream0(HttpURLConnection.java:1546)
at sun.net.www.protocol.http.HttpURLConnection.getInputStream(HttpURLConnection.java:1474)
at java.net.HttpURLConnection.getResponseCode(HttpURLConnection.java:480)
at sun.net.www.protocol.https.HttpsURLConnectionImpl.getResponseCode(HttpsURLConnectionImpl.java:338)
at org.eclipse.jetty.start.fileinits.UriFileInitializer.download(UriFileInitializer.java:75)
at org.eclipse.jetty.start.fileinits.UriFileInitializer.init(UriFileInitializer.java:60)
at org.eclipse.jetty.start.BaseBuilder.processFileResource(BaseBuilder.java:283)
at org.eclipse.jetty.start.BaseBuilder.processFileResources(BaseBuilder.java:375)
at org.eclipse.jetty.start.BaseBuilder.build(BaseBuilder.java:239)
at org.eclipse.jetty.start.Main.start(Main.java:407)
at org.eclipse.jetty.start.Main.main(Main.java:75)
WARNING: Failed to process all file resources.
- [ConnectException] Connection timed out (Connection timed out) - /var/lib/jetty/etc/keystore
java.lang.RuntimeException: Failed to process all file resources.
- [ConnectException] Connection timed out (Connection timed out) - /var/lib/jetty/etc/keystore
at org.eclipse.jetty.start.BaseBuilder.processFileResources(BaseBuilder.java:394)
at org.eclipse.jetty.start.BaseBuilder.build(BaseBuilder.java:239)
at org.eclipse.jetty.start.Main.start(Main.java:407)
at org.eclipse.jetty.start.Main.main(Main.java:75)
Usage: java -jar start.jar [options] [properties] [configs]
java -jar start.jar --help # for more information
---- End output of "bash" "/tmp/jetty/chef-script20170510-13-1fcyeq3" ----
Ran "bash" "/tmp/jetty/chef-script20170510-13-1fcyeq3" returned 247
Resource Declaration:
---------------------
# In /root/chef-solo/cache/cookbooks/sdc-catalog-fe/recipes/FE_6_create_jetty_modules.rb
5: bash "create-jetty-modules" do
6: cwd "#{jetty_base}"
7: code <<-EOH
8: cd "#{jetty_base}"
9: java -jar "/#{jetty_home}"/start.jar --add-to-start=deploy
10: java -jar "/#{jetty_home}"/start.jar --add-to-startd=http,https,logging,setuid
11: EOH
12: not_if "ls /#{jetty_base}/start.d/https.ini"
13: end
14:
Compiled Resource:
------------------
# Declared in /root/chef-solo/cache/cookbooks/sdc-catalog-fe/recipes/FE_6_create_jetty_modules.rb:5:in `from_file'
bash("create-jetty-modules") do
action [:run]
default_guard_interpreter :default
command nil
backup 5
cwd "/var/lib/jetty"
returns 0
user nil
interpreter "bash"
declared_type :bash
cookbook_name "sdc-catalog-fe"
recipe_name "FE_6_create_jetty_modules"
code " cd \"/var/lib/jetty\"\n java -jar \"//usr/local/jetty\"/start.jar --add-to-start=deploy\n java -jar \"//usr/local/jetty\"/start.jar --add-to-startd=http,https,logging,setuid\n"
domain nil
not_if "ls //var/lib/jetty/start.d/https.ini"
end
Adding Michael Lando
A little bit help here would be appreciated. I see the same failure in SDC_BE as well but process didn't exit at that point.
Hi @Dusan Rozman,
Thanks for the reply and information. Appreciate your help. In my setup I believe jetty modules creation and download keystore file failures is because of environment proxies. I have updated Dockerfile but no use.
I have tried docker mount option like below for sdc-BE
sdc/sdc-os-chef/docker-compose.yml
jettyBE:
build: .
container_name: "sdc-be"
image: "ecomp-nexus:51212/ecomp/sdc-backend:1610.2.13"
mem_limit: "3g"
memswap_limit: "3g"
logging:
driver: "json-file"
options:
max-size: "100m"
max-file: "10"
ports:
- "8080:8080"
- "8443:8443"
environment:
- HOST_IP:HOSTIP
- ENVNAME:DEP_ENV
volumes:
- /etc/localtime:/etc/localtime:ro
- /data/ASDC/logs/BE:/var/lib/jetty/logs
- /data/ASDC/environments:/root/chef-solo/environments
- /root/keystore:/var/lib/jetty/etc/keystore
ulimits:
memlock: -1
nproc: 65535
nofile:
soft: 100000
hard: 100000
I have keystore download at /root in docker host and mentioned mount point path. But this didn't work as well. Any ideas? Am I doing correct?