DCAE-CDAP installing stuck

 
1
0
-1

I am now using DCAE 1.0.0 to deploy dcae component on Openstack, I modify the corresponding deploy file in dcae controller docker(hosts, ip-assignment, vm-templates, tenants)  and 5 VMs are created.

The VM vm-docker-host and vm-postgresql are deploy successfully.

However, the cdap vm installation is not correct: cdap00 and cdap02 vm stuck at installing small hadoop(staring nodemanager ), while cdap01 runs the dcae_install.sh successfully.

Before I met this problem dcae-repo-missing-file , does it affect the small hadoop installing step? 

Is there any solution to start cdap service?


This is the tail of dcae_install.log in cdap00

ZooKeeper JMX enabled by default
Using config: /etc/zookeeper/conf/zoo.cfg
Starting zookeeper ... STARTED
starting zkfc, logging to /opt/data/log/hdfs/hadoop-hdfs-zkfc-zldcz1oscdap00.out
Exception in thread "main" org.apache.hadoop.HadoopIllegalArgumentException: HA is not enabled for this namenode.
	at org.apache.hadoop.hdfs.tools.DFSZKFailoverController.create(DFSZKFailoverController.java:121)
	at org.apache.hadoop.hdfs.tools.DFSZKFailoverController.main(DFSZKFailoverController.java:179)
17/08/31 09:03:54 INFO namenode.NameNode: STARTUP_MSG: 
/************************************************************
STARTUP_MSG: Starting NameNode
STARTUP_MSG:   user = hdfs
STARTUP_MSG:   host = zldcz1oscdap00.dcae.simpledemo.openecomp.org/10.0.0.14
STARTUP_MSG:   args = [-format, -force]
STARTUP_MSG:   version = 2.7.1.2.4.3.0-227
STARTUP_MSG:   classpath = *****
STARTUP_MSG:   build = git@github.com:hortonworks/hadoop.git -r 7e73cf651746a25cc06f303750bf26fa02652e8a; compiled by 'jenkins' on 2016-09-08T11:12Z
STARTUP_MSG:   java = 1.8.0_131
************************************************************/
17/08/31 09:03:54 INFO namenode.NameNode: registered UNIX signal handlers for [TERM, HUP, INT]
17/08/31 09:03:54 INFO namenode.NameNode: createNameNode [-format, -force]
17/08/31 09:03:54 WARN common.Util: Path /opt/data/hadoop/hdfs/namenode should be specified as a URI in configuration files. Please update hdfs configuration.
17/08/31 09:03:54 WARN common.Util: Path /opt/data/hadoop/hdfs/namenode should be specified as a URI in configuration files. Please update hdfs configuration.
Formatting using clusterid: CID-0a584bb2-684a-4444-9305-ace5643a94cb
17/08/31 09:03:54 WARN common.Storage: set restore failed storage to true
17/08/31 09:03:54 INFO namenode.FSNamesystem: No KeyProvider found.
17/08/31 09:03:54 INFO namenode.FSNamesystem: Enabling async auditlog
17/08/31 09:03:54 INFO namenode.FSNamesystem: fsLock is fair:false
17/08/31 09:03:54 INFO blockmanagement.HeartbeatManager: Setting heartbeat recheck interval to 30000 since dfs.namenode.stale.datanode.interval is less than dfs.namenode.heartbeat.recheck-interval
17/08/31 09:03:54 INFO blockmanagement.DatanodeManager: dfs.block.invalidate.limit=1000
17/08/31 09:03:54 INFO blockmanagement.DatanodeManager: dfs.namenode.datanode.registration.ip-hostname-check=true
17/08/31 09:03:54 INFO blockmanagement.BlockManager: dfs.namenode.startup.delay.block.deletion.sec is set to 000:01:00:00.000
17/08/31 09:03:54 INFO blockmanagement.BlockManager: The block deletion will start around 2017 Aug 31 10:03:54
17/08/31 09:03:54 INFO util.GSet: Computing capacity for map BlocksMap
17/08/31 09:03:54 INFO util.GSet: VM type       = 64-bit
17/08/31 09:03:54 INFO util.GSet: 2.0% max memory 1011.3 MB = 20.2 MB
17/08/31 09:03:54 INFO util.GSet: capacity      = 2^21 = 2097152 entries
17/08/31 09:03:55 INFO blockmanagement.BlockManager: dfs.block.access.token.enable=true
17/08/31 09:03:55 INFO blockmanagement.BlockManager: dfs.block.access.key.update.interval=600 min(s), dfs.block.access.token.lifetime=600 min(s), dfs.encrypt.data.transfer.algorithm=null
17/08/31 09:03:55 INFO blockmanagement.BlockManager: defaultReplication         = 3
17/08/31 09:03:55 INFO blockmanagement.BlockManager: maxReplication             = 50
17/08/31 09:03:55 INFO blockmanagement.BlockManager: minReplication             = 1
17/08/31 09:03:55 INFO blockmanagement.BlockManager: maxReplicationStreams      = 2
17/08/31 09:03:55 INFO blockmanagement.BlockManager: replicationRecheckInterval = 3000
17/08/31 09:03:55 INFO blockmanagement.BlockManager: encryptDataTransfer        = false
17/08/31 09:03:55 INFO blockmanagement.BlockManager: maxNumBlocksToLog          = 1000
17/08/31 09:03:55 INFO namenode.FSNamesystem: fsOwner             = hdfs (auth:SIMPLE)
17/08/31 09:03:55 INFO namenode.FSNamesystem: supergroup          = hdfs
17/08/31 09:03:55 INFO namenode.FSNamesystem: isPermissionEnabled = true
17/08/31 09:03:55 INFO namenode.FSNamesystem: HA Enabled: false
17/08/31 09:03:55 INFO namenode.FSNamesystem: Append Enabled: true
17/08/31 09:03:55 INFO util.GSet: Computing capacity for map INodeMap
17/08/31 09:03:55 INFO util.GSet: VM type       = 64-bit
17/08/31 09:03:55 INFO util.GSet: 1.0% max memory 1011.3 MB = 10.1 MB
17/08/31 09:03:55 INFO util.GSet: capacity      = 2^20 = 1048576 entries
17/08/31 09:03:55 INFO namenode.FSDirectory: ACLs enabled? false
17/08/31 09:03:55 INFO namenode.FSDirectory: XAttrs enabled? true
17/08/31 09:03:55 INFO namenode.FSDirectory: Maximum size of an xattr: 16384
17/08/31 09:03:55 INFO namenode.NameNode: Caching file names occuring more than 10 times
17/08/31 09:03:55 INFO util.GSet: Computing capacity for map cachedBlocks
17/08/31 09:03:55 INFO util.GSet: VM type       = 64-bit
17/08/31 09:03:55 INFO util.GSet: 0.25% max memory 1011.3 MB = 2.5 MB
17/08/31 09:03:55 INFO util.GSet: capacity      = 2^18 = 262144 entries
17/08/31 09:03:55 INFO namenode.FSNamesystem: dfs.namenode.safemode.threshold-pct = 1.0
17/08/31 09:03:55 INFO namenode.FSNamesystem: dfs.namenode.safemode.min.datanodes = 0
17/08/31 09:03:55 INFO namenode.FSNamesystem: dfs.namenode.safemode.extension     = 30000
17/08/31 09:03:55 INFO metrics.TopMetrics: NNTop conf: dfs.namenode.top.window.num.buckets = 10
17/08/31 09:03:55 INFO metrics.TopMetrics: NNTop conf: dfs.namenode.top.num.users = 10
17/08/31 09:03:55 INFO metrics.TopMetrics: NNTop conf: dfs.namenode.top.windows.minutes = 1,5,25
17/08/31 09:03:55 INFO namenode.FSNamesystem: Retry cache on namenode is enabled
17/08/31 09:03:55 INFO namenode.FSNamesystem: Retry cache will use 0.03 of total heap and retry cache entry expiry time is 600000 millis
17/08/31 09:03:55 INFO util.GSet: Computing capacity for map NameNodeRetryCache
17/08/31 09:03:55 INFO util.GSet: VM type       = 64-bit
17/08/31 09:03:55 INFO util.GSet: 0.029999999329447746% max memory 1011.3 MB = 310.7 KB
17/08/31 09:03:55 INFO util.GSet: capacity      = 2^15 = 32768 entries
17/08/31 09:03:55 INFO namenode.FSImage: Allocated new BlockPoolId: BP-1115782778-10.0.0.14-1504170235159
17/08/31 09:03:55 INFO common.Storage: Storage directory /opt/data/hadoop/hdfs/namenode has been successfully formatted.
17/08/31 09:03:55 INFO namenode.FSImageFormatProtobuf: Saving image file /opt/data/hadoop/hdfs/namenode/current/fsimage.ckpt_0000000000000000000 using no compression
17/08/31 09:03:55 INFO namenode.FSImageFormatProtobuf: Image file /opt/data/hadoop/hdfs/namenode/current/fsimage.ckpt_0000000000000000000 of size 336 bytes saved in 0 seconds.
17/08/31 09:03:55 INFO namenode.NNStorageRetentionManager: Going to retain 1 images with txid >= 0
17/08/31 09:03:55 INFO util.ExitUtil: Exiting with status 0
17/08/31 09:03:55 INFO namenode.NameNode: SHUTDOWN_MSG: 
/************************************************************
SHUTDOWN_MSG: Shutting down NameNode at zldcz1oscdap00.dcae.simpledemo.openecomp.org/10.0.0.14
************************************************************/
starting namenode, logging to /opt/data/log/hdfs/hadoop-hdfs-namenode-zldcz1oscdap00.out
starting datanode, logging to /opt/data/log/hdfs/hadoop-hdfs-datanode-zldcz1oscdap00.out
starting nodemanager, logging to /opt/data/log/hadoop/yarn/yarn/yarn-yarn-nodemanager-zldcz1oscdap00.out


This is the tail of dcae_install.log in cdap01

cdap-cli is already the newest version (3.5.5-1).
cdap-gateway is already the newest version (3.5.5-1).
cdap-kafka is already the newest version (3.5.5-1).
cdap-master is already the newest version (3.5.5-1).
cdap-security is already the newest version (3.5.5-1).
cdap-ui is already the newest version (3.5.5-1).
0 upgraded, 0 newly installed, 0 to remove and 35 not upgraded.
ZooKeeper JMX enabled by default
Using config: /etc/zookeeper/conf/zoo.cfg
Starting zookeeper ... STARTED
starting zkfc, logging to /opt/data/log/hdfs/hadoop-hdfs-zkfc-zldcz1oscdap01.out
Exception in thread "main" org.apache.hadoop.HadoopIllegalArgumentException: HA is not enabled for this namenode.
	at org.apache.hadoop.hdfs.tools.DFSZKFailoverController.create(DFSZKFailoverController.java:121)
	at org.apache.hadoop.hdfs.tools.DFSZKFailoverController.main(DFSZKFailoverController.java:179)
starting datanode, logging to /opt/data/log/hdfs/hadoop-hdfs-datanode-zldcz1oscdap01.out
starting resourcemanager, logging to /opt/data/log/hadoop/yarn/yarn/yarn-yarn-resourcemanager-zldcz1oscdap01.out
starting nodemanager, logging to /opt/data/log/hadoop/yarn/yarn/yarn-yarn-nodemanager-zldcz1oscdap01.out
starting master, logging to /opt/data/log/hbase/hbase-hbase-master-zldcz1oscdap01.out
starting regionserver, logging to /opt/data/log/hbase/hbase-hbase-regionserver-zldcz1oscdap01.out
+ sudo su dcae -c '/opt/app/dcae-controller-service-cdap-cluster-manager/bin/manager.sh config'
+ sudo su dcae -c '/opt/app/dcae-controller-service-cdap-cluster-manager/bin/manager.sh restart'
Dcae Controller (no pidfile) is NOT running
Dcae Controller (pid 27595) is running
+ '[' '!' -e /home/dcae ']'
+ case runtime in
+ A1=org.openecomp.dcae.controller:dcae-controller-service-dmaap-drsub-manager:1.0.0:zip:runtime
+ mvn -s /tmp/settings.xml -q org.apache.maven.plugins:maven-dependency-plugin:2.10:copy -Dartifact=org.openecomp.dcae.controller:dcae-controller-service-dmaap-drsub-manager:1.0.0:zip:runtime -Dmaven.wagon.http.ssl.insecure=true -Dmaven.wagon.http.ssl.allowall=true -DoutputDirectory=/opt/app/dcae-controller-service-dmaap-drsub-manager
Aug 31, 2017 9:23:37 AM org.apache.maven.wagon.providers.http.httpclient.impl.execchain.RetryExec execute
INFO: I/O exception (org.apache.maven.wagon.providers.http.httpclient.NoHttpResponseException) caught when processing request to {s}->https://nexus.onap.org:443: The target server failed to respond
Aug 31, 2017 9:23:37 AM org.apache.maven.wagon.providers.http.httpclient.impl.execchain.RetryExec execute
INFO: Retrying request to {s}->https://nexus.onap.org:443
+ case zip in
+ cd /opt/app/dcae-controller-service-dmaap-drsub-manager
+ unzip -o dcae-controller-service-dmaap-drsub-manager-1.0.0-runtime.zip
Archive:  dcae-controller-service-dmaap-drsub-manager-1.0.0-runtime.zip
    lib/ncomp-sirius-manager-agent-model-1.0.0.jar  
....
+ chown -R dcae:dcae /opt/app/dcae-controller-service-dmaap-drsub-manager
+ mkdir /home/dcae/.ssh
+ chmod og-rwx /home/dcae/.ssh
+ chown -R dcae:dcae /home/dcae/.ssh
+ touch /home/dcae/.ssh/authorized_keys
+ chmod og-rwx /home/dcae/.ssh/authorized_keys
+ chown -R dcae:dcae /home/dcae/.ssh/authorized_keys
+ echo ssh-rsa ****


This is the tail of dcae_install.log in cdap02

cdap-cli is already the newest version (3.5.5-1).
cdap-gateway is already the newest version (3.5.5-1).
cdap-kafka is already the newest version (3.5.5-1).
cdap-master is already the newest version (3.5.5-1).
cdap-security is already the newest version (3.5.5-1).
cdap-ui is already the newest version (3.5.5-1).
0 upgraded, 0 newly installed, 0 to remove and 35 not upgraded.
ZooKeeper JMX enabled by default
Using config: /etc/zookeeper/conf/zoo.cfg
Starting zookeeper ... STARTED
starting zkfc, logging to /opt/data/log/hdfs/hadoop-hdfs-zkfc-zldcz1oscdap02.out
Exception in thread "main" org.apache.hadoop.HadoopIllegalArgumentException: HA is not enabled for this namenode.
	at org.apache.hadoop.hdfs.tools.DFSZKFailoverController.create(DFSZKFailoverController.java:121)
	at org.apache.hadoop.hdfs.tools.DFSZKFailoverController.main(DFSZKFailoverController.java:179)
starting datanode, logging to /opt/data/log/hdfs/hadoop-hdfs-datanode-zldcz1oscdap02.out
starting nodemanager, logging to /opt/data/log/hadoop/yarn/yarn/yarn-yarn-nodemanager-zldcz1oscdap02.out



    CommentAdd your comment...

    1 answer

    1.  
      1
      0
      -1

      Hi You Li I have not tried with 1.0.0 and I may not be able to provide a solution but I can help to understand perhaps

      I see hadoop runes multiple processes. So, it will take a bit time to come up. Try connection htp://<cdap_vm2_ip>:9999 from a browser. If it directly displays the dashboard then CDAP is working. If it redirects to login page then something is wrong.

      How many interface does the CDAP VM contains? If you are using two interface setup then you may have to change the interface configuration file for eth1 and make it as static instead DHDP and then you can do a reboot of all three CDAP VMs. After the reboot try to browse htp://<cdap_vm2_ip>:9999

      DCAE Controller Development Guide

      To check the health status of deployment please refer

      https://wiki.onap.org/questions/8226517/dcae-controller-startup.sh-stalls-after-initiating-vm-docker-host-1

      You may have to export environment variables from /opt/app/dcae-controller/config.yaml before running health checks from dcae controller container


        CommentAdd your comment...