...
  Work to align the logs, ship them, consolidate, provide for analysis, backup and restore them will be done from a developer perspective for the Dublin release.
...
Features
R1: Provide for consistent logs across ONAP that allow for cross pod indexing
R2: Validate/Implement log shipping to the existing ELK stack
Analysis
Existing ONAP Logging State - Infrastructure and Schema
...
Tech
...
Key words are: out of the box, reuse components, iterative changes - not major spec changes.
There is one primary use case
| Provide for transaction tracing via search or dashboard for distributed transactions across ONAP components - enabled by the logs shipped to the ELK stack - in real time. |
Logging Alignment before enhancement
Plan is to take the portal/sdk library - align the rest of onap to this library and then add marker/mdc (labels and key/value pair) support as required in phase 2
Work will include 1) verify pipeline 2) verify logback format
Open Standards
Look at a standard that will allow us to use standard parsing libraries all the way to eventual AI/ML by adopting a standard like open tracing.
Infrastructure Improvements
including filebeat sidecar anti-pattern/replacement, elk stack upgrade, deployment template, library language scope (python, swift...)
Monitoring
Investigate out of the box prometheus monitoring
S3P
backup/restore, security, log format monitoring, elk dashboards
3rd party consumption alignment
Lower in priority will be to align onap with requirements of Acumos and ECOMP for example - the focus will be on alignment of ONAP components only - to the primary goal of providing a transaction tracing system via the ELK stack.
Analysis
Existing ONAP Logging State - Infrastructure and Schema
| Pod | Container/service | Tech | URLs | logback | library | dependencies | |||||
|---|---|---|---|---|---|---|---|---|---|---|---|
| aaf |
| ||||||||||
| aai |
| ||||||||||
| appc |
| ||||||||||
Deployment Profile
| Type | VMs | Total RAM vCores HD | VM Flavor | K8S/Rancher Idle RAM | Deployed | Deployed ONAP RAM | Pods | Max vCores | Idle vCores | HD/VM | HD NFS only | IOPS | Date | Notes deployment post 75min |
|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| Full Cluster (14 + 1) - recommended | 15 | 224G 112 vC 100G/VM | 16G, 8 vCores C5.2xLarge | 187Gb | 102Gb | 248 total 241 onap 217 up 0 error 24 config | 18 | 6+G master | 8.1G | 20181106 | ||||
| Single VM (possible - not recommended) | 1 | 432G 64 vC 120G | 256G+ 32+ vCores | Rancher: 13G Kubernetes: 8G Top: 10G | 151Gb | 141Gb | 240 total | 55 | 22 | 108 of 128G | n/a | Max: 550/sec Idle: 220/sec | 20181105 | Tested on 432G/64vCore azure VM - R 1.6.22 K8S 1.11 |
| Developer 1-n pods | 1 | 16G | 16/32G 4-16 vCores | 14Gb | 10Gb | 120+G | n/a | AAI+robot only |
Example 15 node (1 master + 14 nodes) OOM Deployment
empty
With ONAP deployed
Throughput and Volumetrics
Cloudwatch CPU Average
Specific to logging - we have a problem on any VM that contains AAI - the logstash container is being saturated there - see the 30+ percent VM -Â
| Jira | ||||||
|---|---|---|---|---|---|---|
|
NFS Throughput for /dockerdata-nfs
Cloudwatch Network In Max
Details: 20181106:1800 EDT master
| Code Block | ||
|---|---|---|
| ||
ubuntu@ip-172-31-40-250:~$ kubectl get pods --all-namespaces | wc -l
248
ubuntu@ip-172-31-40-250:~$ kubectl get pods --all-namespaces | grep onap | wc -l
241
ubuntu@ip-172-31-40-250:~$ kubectl get pods --all-namespaces | grep onap | grep -E '1/1|2/2' | wc -l
217
ubuntu@ip-172-31-40-250:~$ kubectl get pods --all-namespaces | grep onap | grep -E '0/|1/2' | wc -l
24
ubuntu@ip-172-31-40-250:~$ kubectl get pods --all-namespaces -o wide | grep onap | grep -E '0/|1/2'
onap onap-aaf-aaf-sms-preload-lvqx9 0/1 Completed 0 4h 10.42.75.71 ip-172-31-37-59.us-east-2.compute.internal <none>
onap onap-aaf-aaf-sshsm-distcenter-ql5f8 0/1 Completed 0 4h 10.42.75.223 ip-172-31-34-207.us-east-2.compute.internal <none>
onap onap-aaf-aaf-sshsm-testca-7rzcd 0/1 Completed 0 4h 10.42.18.37 ip-172-31-34-111.us-east-2.compute.internal <none>
onap onap-aai-aai-graphadmin-create-db-schema-26pfs 0/1 Completed 0 4h 10.42.14.14 ip-172-31-37-59.us-east-2.compute.internal <none>
onap onap-aai-aai-traversal-update-query-data-qlk7w 0/1 Completed 0 4h 10.42.88.122 ip-172-31-36-163.us-east-2.compute.internal <none>
onap onap-contrib-netbox-app-provisioning-gmmvj 0/1 Completed 0 4h 10.42.111.99 ip-172-31-41-229.us-east-2.compute.internal <none>
onap onap-contrib-netbox-app-provisioning-n6fw4 0/1 Error 0 4h 10.42.21.12 ip-172-31-36-163.us-east-2.compute.internal <none>
onap onap-contrib-netbox-app-provisioning-nc8ww 0/1 Error 0 4h 10.42.109.156 ip-172-31-41-110.us-east-2.compute.internal <none>
onap onap-contrib-netbox-app-provisioning-xcxds 0/1 Error 0 4h 10.42.152.223 ip-172-31-39-138.us-east-2.compute.internal <none>
onap onap-dmaap-dmaap-dr-node-6496d8f55b-jfvrm 0/1 Init:0/1 28 4h 10.42.95.32 ip-172-31-38-194.us-east-2.compute.internal <none>
onap onap-dmaap-dmaap-dr-prov-86f79c47f9-tldsp 0/1 CrashLoopBackOff 59 4h 10.42.76.248 ip-172-31-34-207.us-east-2.compute.internal <none>
onap onap-oof-music-cassandra-job-config-7mb5f 0/1 Completed 0 4h 10.42.38.249 ip-172-31-41-110.us-east-2.compute.internal <none>
onap onap-oof-oof-has-healthcheck-rpst7 0/1 Completed 0 4h 10.42.241.223 ip-172-31-39-138.us-east-2.compute.internal <none>
onap onap-oof-oof-has-onboard-5bd2l 0/1 Completed 0 4h 10.42.205.75 ip-172-31-38-194.us-east-2.compute.internal <none>
onap onap-portal-portal-db-config-qshzn 0/2 Completed 0 4h 10.42.112.46 ip-172-31-45-152.us-east-2.compute.internal <none>
onap onap-portal-portal-db-config-rk4m2 0/2 Init:Error 0 4h 10.42.57.79 ip-172-31-38-194.us-east-2.compute.internal <none>
onap onap-sdc-sdc-be-config-backend-2vw2q 0/1 Completed 0 4h 10.42.87.181 ip-172-31-39-138.us-east-2.compute.internal <none>
onap onap-sdc-sdc-be-config-backend-k57lh 0/1 Init:Error 0 4h 10.42.148.79 ip-172-31-45-152.us-east-2.compute.internal <none>
onap onap-sdc-sdc-cs-config-cassandra-vgnz2 0/1 Completed 0 4h 10.42.111.187 ip-172-31-34-111.us-east-2.compute.internal <none>
onap onap-sdc-sdc-es-config-elasticsearch-lkb9m 0/1 Completed 0 4h 10.42.20.202 ip-172-31-39-138.us-east-2.compute.internal <none>
onap onap-sdc-sdc-onboarding-be-cassandra-init-7zv5j 0/1 Completed 0 4h 10.42.218.1 ip-172-31-41-229.us-east-2.compute.internal <none>
onap onap-sdc-sdc-wfd-be-workflow-init-q8t7z 0/1 Completed 0 4h 10.42.255.91 ip-172-31-41-30.us-east-2.compute.internal <none>
onap onap-vid-vid-galera-config-4f274 0/1 Completed 0 4h 10.42.80.200 ip-172-31-33-223.us-east-2.compute.internal <none>
onap onap-vnfsdk-vnfsdk-init-postgres-lf659 0/1 Completed 0 4h 10.42.238.204 ip-172-31-38-194.us-east-2.compute.internal <none>
ubuntu@ip-172-31-40-250:~$ kubectl get pods --all-namespaces -o wide
NAMESPACE NAME READY STATUS RESTARTS AGE IP NODE NOMINATED NODE
kube-system heapster-7b48b696fc-kxjlx 1/1 Running 0 5h 10.42.237.56 ip-172-31-37-24.us-east-2.compute.internal <none>
kube-system kube-dns-6655f78c68-wzmvh 3/3 Running 0 5h 10.42.18.171 ip-172-31-37-24.us-east-2.compute.internal <none>
kube-system kubernetes-dashboard-6f54f7c4b-58k4h 1/1 Running 0 5h 10.42.205.56 ip-172-31-37-24.us-east-2.compute.internal <none>
kube-system monitoring-grafana-7877679464-9ssz5 1/1 Running 0 5h 10.42.252.38 ip-172-31-37-24.us-east-2.compute.internal <none>
kube-system monitoring-influxdb-64664c6cf5-h4bxt 1/1 Running 0 5h 10.42.226.121 ip-172-31-37-24.us-east-2.compute.internal <none>
kube-system tiller-deploy-6f4745cbcf-q6kxx 1/1 Running 0 5h 10.42.12.224 ip-172-31-37-24.us-east-2.compute.internal <none>
onap dep-config-binding-service-68d9cd6f4c-9n2nf 2/2 Running 0 4h 10.42.157.143 ip-172-31-41-30.us-east-2.compute.internal <none>
onap dep-dcae-datafile-collector-77956c65df-8mfwc 2/2 Running 0 4h 10.42.110.106 ip-172-31-41-30.us-east-2.compute.internal <none>
onap dep-dcae-hv-ves-collector-55844fb468-9zdss 2/2 Running 0 4h 10.42.231.39 ip-172-31-45-152.us-east-2.compute.internal <none>
onap dep-dcae-prh-74977b5fb4-n47b6 2/2 Running 0 4h 10.42.115.19 ip-172-31-39-138.us-east-2.compute.internal <none>
onap dep-dcae-snmptrap-collector-69f99dc589-f7ffl 1/1 Running 0 4h 10.42.173.221 ip-172-31-41-229.us-east-2.compute.internal <none>
onap dep-dcae-tca-analytics-84fbfbc894-xzq8c 2/2 Running 0 4h 10.42.28.146 ip-172-31-38-194.us-east-2.compute.internal <none>
onap dep-dcae-ves-collector-98f8ccb5c-tghs4 2/2 Running 0 4h 10.42.207.99 ip-172-31-34-111.us-east-2.compute.internal <none>
onap dep-deployment-handler-76cf6d665-tvcwp 2/2 Running 0 4h 10.42.11.59 ip-172-31-37-59.us-east-2.compute.internal <none>
onap dep-holmes-engine-mgmt-9889496c5-2jbnb 1/1 Running 27 4h 10.42.186.10 ip-172-31-36-163.us-east-2.compute.internal <none>
onap dep-holmes-rule-mgmt-9b49769b5-n6hqj 1/1 Running 27 4h 10.42.185.154 ip-172-31-39-138.us-east-2.compute.internal <none>
onap dep-inventory-5c9f5bcc6d-lzqvh 1/1 Running 0 4h 10.42.68.75 ip-172-31-37-59.us-east-2.compute.internal <none>
onap dep-policy-handler-544dcdfd75-vpsm7 2/2 Running 0 4h 10.42.233.2 ip-172-31-38-194.us-east-2.compute.internal <none>
onap dep-pstg-write-f67b57947-mkbpq 1/1 Running 0 4h 10.42.185.41 ip-172-31-39-138.us-east-2.compute.internal <none>
onap dep-service-change-handler-66b78745c-995ks 1/1 Running 3 4h 10.42.92.139 ip-172-31-36-163.us-east-2.compute.internal <none>
onap onap-aaf-aaf-cm-7fcc8548cc-nrmsf 1/1 Running 0 4h 10.42.226.234 ip-172-31-45-152.us-east-2.compute.internal <none>
onap onap-aaf-aaf-cs-5755ccf8f4-tz6fn 1/1 Running 0 4h 10.42.181.148 ip-172-31-42-33.us-east-2.compute.internal <none>
onap onap-aaf-aaf-fs-7f8f9dcd74-27gfd 1/1 Running 0 4h 10.42.118.145 ip-172-31-41-30.us-east-2.compute.internal <none>
onap onap-aaf-aaf-gui-bfd9c7fd6-qk6zw 1/1 Running 0 4h 10.42.201.9 ip-172-31-41-229.us-east-2.compute.internal <none>
onap onap-aaf-aaf-hello-6447444665-5t24n 1/1 Running 0 4h 10.42.150.107 ip-172-31-41-110.us-east-2.compute.internal <none>
onap onap-aaf-aaf-locate-845c5867db-2q8qv 1/1 Running 0 4h 10.42.190.184 ip-172-31-39-138.us-east-2.compute.internal <none>
onap onap-aaf-aaf-oauth-68bc85dbd4-7qv7w 1/1 Running 0 4h 10.42.233.42 ip-172-31-38-95.us-east-2.compute.internal <none>
onap onap-aaf-aaf-service-69fb68d68-qhgqc 1/1 Running 0 4h 10.42.108.135 ip-172-31-38-194.us-east-2.compute.internal <none>
onap onap-aaf-aaf-sms-7497c9bfcc-dtc8s 1/1 Running 0 4h 10.42.85.30 ip-172-31-42-33.us-east-2.compute.internal <none>
onap onap-aaf-aaf-sms-preload-lvqx9 0/1 Completed 0 4h 10.42.75.71 ip-172-31-37-59.us-east-2.compute.internal <none>
onap onap-aaf-aaf-sms-quorumclient-0 1/1 Running 0 4h 10.42.30.16 ip-172-31-36-163.us-east-2.compute.internal <none>
onap onap-aaf-aaf-sms-quorumclient-1 1/1 Running 0 4h 10.42.164.247 ip-172-31-41-110.us-east-2.compute.internal <none>
onap onap-aaf-aaf-sms-quorumclient-2 1/1 Running 0 4h 10.42.232.51 ip-172-31-38-95.us-east-2.compute.internal <none>
onap onap-aaf-aaf-sms-vault-0 2/2 Running 1 4h 10.42.242.253 ip-172-31-33-223.us-east-2.compute.internal <none>
onap onap-aaf-aaf-sshsm-distcenter-ql5f8 0/1 Completed 0 4h 10.42.75.223 ip-172-31-34-207.us-east-2.compute.internal <none>
onap onap-aaf-aaf-sshsm-testca-7rzcd 0/1 Completed 0 4h 10.42.18.37 ip-172-31-34-111.us-east-2.compute.internal <none>
onap onap-aai-aai-695546c99-l4vd4 1/1 Running 0 4h 10.42.61.157 ip-172-31-41-229.us-east-2.compute.internal <none>
onap onap-aai-aai-babel-c489c944c-l6fxm 2/2 Running 0 4h 10.42.151.176 ip-172-31-42-33.us-east-2.compute.internal <none>
onap onap-aai-aai-cassandra-0 1/1 Running 0 4h 10.42.60.200 ip-172-31-34-207.us-east-2.compute.internal <none>
onap onap-aai-aai-cassandra-1 1/1 Running 1 4h 10.42.102.208 ip-172-31-45-152.us-east-2.compute.internal <none>
onap onap-aai-aai-cassandra-2 1/1 Running 0 4h 10.42.168.115 ip-172-31-36-163.us-east-2.compute.internal <none>
onap onap-aai-aai-champ-54f7986b6b-6qqk8 2/2 Running 0 4h 10.42.233.99 ip-172-31-41-30.us-east-2.compute.internal <none>
onap onap-aai-aai-data-router-5b777d798c-hkt9p 2/2 Running 0 4h 10.42.254.152 ip-172-31-41-229.us-east-2.compute.internal <none>
onap onap-aai-aai-elasticsearch-c9bf9dbf6-flljw 1/1 Running 0 4h 10.42.136.54 ip-172-31-41-110.us-east-2.compute.internal <none>
onap onap-aai-aai-gizmo-5f8bf54f6f-4sj45 2/2 Running 0 4h 10.42.128.212 ip-172-31-39-138.us-east-2.compute.internal <none>
onap onap-aai-aai-graphadmin-789c67bb8b-kd9kw 2/2 Running 0 4h 10.42.196.56 ip-172-31-38-95.us-east-2.compute.internal <none>
onap onap-aai-aai-graphadmin-create-db-schema-26pfs 0/1 Completed 0 4h 10.42.14.14 ip-172-31-37-59.us-east-2.compute.internal <none>
onap onap-aai-aai-modelloader-695fd4bf48-5gdmt 2/2 Running 0 4h 10.42.188.20 ip-172-31-38-194.us-east-2.compute.internal <none>
onap onap-aai-aai-resources-765d6f6678-xg4jv 2/2 Running 0 4h 10.42.224.167 ip-172-31-34-111.us-east-2.compute.internal <none>
onap onap-aai-aai-search-data-559b8dbc7f-k549g 2/2 Running 0 4h 10.42.110.212 ip-172-31-33-223.us-east-2.compute.internal <none>
onap onap-aai-aai-sparky-be-75658695f5-wjkzw 2/2 Running 0 4h 10.42.7.139 ip-172-31-45-152.us-east-2.compute.internal <none>
onap onap-aai-aai-spike-f854cf86d-fdx7j 2/2 Running 0 4h 10.42.175.92 ip-172-31-42-33.us-east-2.compute.internal <none>
onap onap-aai-aai-traversal-5fcc7cc847-mrqtb 2/2 Running 0 4h 10.42.174.143 ip-172-31-39-138.us-east-2.compute.internal <none>
onap onap-aai-aai-traversal-update-query-data-qlk7w 0/1 Completed 0 4h 10.42.88.122 ip-172-31-36-163.us-east-2.compute.internal <none>
onap onap-appc-appc-0 2/2 Running 0 4h 10.42.106.203 ip-172-31-37-59.us-east-2.compute.internal <none>
onap onap-appc-appc-ansible-server-78784978b6-jrz2b 1/1 Running 0 4h 10.42.232.15 ip-172-31-38-95.us-east-2.compute.internal <none>
onap onap-appc-appc-cdt-d654974f6-bxhkw 1/1 Running 0 4h 10.42.170.154 ip-172-31-38-194.us-east-2.compute.internal <none>
onap onap-appc-appc-db-0 1/1 Running 0 4h 10.42.215.183 ip-172-31-41-110.us-east-2.compute.internal <none>
onap onap-appc-appc-db-1 1/1 Running 1 4h 10.42.113.105 ip-172-31-41-30.us-east-2.compute.internal <none>
onap onap-appc-appc-db-2 1/1 Running 1 4h 10.42.245.153 ip-172-31-38-95.us-east-2.compute.internal <none>
onap onap-appc-appc-dgbuilder-6bdc7bd56b-ssb2p 1/1 Running 0 4h 10.42.188.237 ip-172-31-37-59.us-east-2.compute.internal <none>
onap onap-clamp-clamp-8cd95c6b4-kbmxg 2/2 Running 0 4h 10.42.149.44 ip-172-31-38-95.us-east-2.compute.internal <none>
onap onap-clamp-clamp-dash-es-5566796d7c-b2mbd 1/1 Running 0 4h 10.42.51.26 ip-172-31-36-163.us-east-2.compute.internal <none>
onap onap-clamp-clamp-dash-kibana-646fccf75d-vf68k 1/1 Running 0 4h 10.42.173.69 ip-172-31-34-207.us-east-2.compute.internal <none>
onap onap-clamp-clamp-dash-logstash-7bfc956564-xhlj5 1/1 Running 0 4h 10.42.38.41 ip-172-31-33-223.us-east-2.compute.internal <none>
onap onap-clamp-clampdb-6797447d-bkrbr 1/1 Running 0 4h 10.42.37.63 ip-172-31-45-152.us-east-2.compute.internal <none>
onap onap-cli-cli-5498964cf9-ndk74 1/1 Running 0 4h 10.42.251.86 ip-172-31-41-229.us-east-2.compute.internal <none>
onap onap-consul-consul-849447d678-8677q 1/1 Running 0 4h 10.42.56.217 ip-172-31-41-110.us-east-2.compute.internal <none>
onap onap-consul-consul-server-0 1/1 Running 0 4h 10.42.69.75 ip-172-31-34-111.us-east-2.compute.internal <none>
onap onap-consul-consul-server-1 1/1 Running 0 4h 10.42.156.89 ip-172-31-33-223.us-east-2.compute.internal <none>
onap onap-consul-consul-server-2 1/1 Running 0 4h 10.42.69.41 ip-172-31-38-194.us-east-2.compute.internal <none>
onap onap-contrib-netbox-app-6d4d768d9d-gksvt 1/1 Running 0 4h 10.42.179.105 ip-172-31-38-95.us-east-2.compute.internal <none>
onap onap-contrib-netbox-app-provisioning-gmmvj 0/1 Completed 0 4h 10.42.111.99 ip-172-31-41-229.us-east-2.compute.internal <none>
onap onap-contrib-netbox-app-provisioning-n6fw4 0/1 Error 0 4h 10.42.21.12 ip-172-31-36-163.us-east-2.compute.internal <none>
onap onap-contrib-netbox-app-provisioning-nc8ww 0/1 Error 0 4h 10.42.109.156 ip-172-31-41-110.us-east-2.compute.internal <none>
onap onap-contrib-netbox-app-provisioning-xcxds 0/1 Error 0 4h 10.42.152.223 ip-172-31-39-138.us-east-2.compute.internal <none>
onap onap-contrib-netbox-nginx-54c979ff7f-nk5kk 1/1 Running 0 4h 10.42.244.179 ip-172-31-34-111.us-east-2.compute.internal <none>
onap onap-contrib-netbox-postgres-784b5d5fd9-2wdf7 1/1 Running 0 4h 10.42.175.55 ip-172-31-36-163.us-east-2.compute.internal <none>
onap onap-dcaegen2-dcae-bootstrap-7bb4b6bc57-gvz47 1/1 Running 0 4h 10.42.74.208 ip-172-31-37-59.us-east-2.compute.internal <none>
onap onap-dcaegen2-dcae-cloudify-manager-9cb7f4c66-7h77h 1/1 Running 0 4h 10.42.250.189 ip-172-31-36-163.us-east-2.compute.internal <none>
onap onap-dcaegen2-dcae-db-0 1/1 Running 0 4h 10.42.230.118 ip-172-31-33-223.us-east-2.compute.internal <none>
onap onap-dcaegen2-dcae-db-1 1/1 Running 0 4h 10.42.29.230 ip-172-31-37-24.us-east-2.compute.internal <none>
onap onap-dcaegen2-dcae-healthcheck-7b664b566c-w5w4h 1/1 Running 0 4h 10.42.9.151 ip-172-31-34-111.us-east-2.compute.internal <none>
onap onap-dcaegen2-dcae-pgpool-688d65b66d-jpv8x 1/1 Running 0 4h 10.42.177.56 ip-172-31-42-33.us-east-2.compute.internal <none>
onap onap-dcaegen2-dcae-pgpool-688d65b66d-srm5q 1/1 Running 0 4h 10.42.164.252 ip-172-31-34-207.us-east-2.compute.internal <none>
onap onap-dcaegen2-dcae-redis-0 1/1 Running 0 4h 10.42.32.17 ip-172-31-45-152.us-east-2.compute.internal <none>
onap onap-dcaegen2-dcae-redis-1 1/1 Running 0 4h 10.42.52.99 ip-172-31-34-207.us-east-2.compute.internal <none>
onap onap-dcaegen2-dcae-redis-2 1/1 Running 0 4h 10.42.45.144 ip-172-31-41-229.us-east-2.compute.internal <none>
onap onap-dcaegen2-dcae-redis-3 1/1 Running 0 4h 10.42.79.151 ip-172-31-37-59.us-east-2.compute.internal <none>
onap onap-dcaegen2-dcae-redis-4 1/1 Running 0 4h 10.42.110.141 ip-172-31-39-138.us-east-2.compute.internal <none>
onap onap-dcaegen2-dcae-redis-5 1/1 Running 0 4h 10.42.253.134 ip-172-31-38-95.us-east-2.compute.internal <none>
onap onap-dmaap-dbc-pg-0 1/1 Running 0 4h 10.42.166.51 ip-172-31-33-223.us-east-2.compute.internal <none>
onap onap-dmaap-dbc-pg-1 1/1 Running 0 4h 10.42.230.243 ip-172-31-37-59.us-east-2.compute.internal <none>
onap onap-dmaap-dbc-pgpool-c5f8498-lr854 1/1 Running 0 4h 10.42.35.8 ip-172-31-41-229.us-east-2.compute.internal <none>
onap onap-dmaap-dbc-pgpool-c5f8498-m84f5 1/1 Running 0 4h 10.42.240.60 ip-172-31-38-194.us-east-2.compute.internal <none>
onap onap-dmaap-dmaap-bus-controller-59c96d6b8f-nlc4t 1/1 Running 0 4h 10.42.240.232 ip-172-31-37-59.us-east-2.compute.internal <none>
onap onap-dmaap-dmaap-dr-db-557c66dc9d-4ltgp 1/1 Running 1 4h 10.42.210.116 ip-172-31-41-110.us-east-2.compute.internal <none>
onap onap-dmaap-dmaap-dr-node-6496d8f55b-jfvrm 0/1 Init:0/1 28 4h 10.42.95.32 ip-172-31-38-194.us-east-2.compute.internal <none>
onap onap-dmaap-dmaap-dr-prov-86f79c47f9-tldsp 0/1 CrashLoopBackOff 59 4h 10.42.76.248 ip-172-31-34-207.us-east-2.compute.internal <none>
onap onap-dmaap-message-router-557cb765-sc4pb 1/1 Running 0 4h 10.42.157.140 ip-172-31-45-152.us-east-2.compute.internal <none>
onap onap-dmaap-message-router-kafka-7b4d45b44c-7g5pz 1/1 Running 0 4h 10.42.218.166 ip-172-31-34-111.us-east-2.compute.internal <none>
onap onap-dmaap-message-router-zookeeper-5bc7588665-pz9x5 1/1 Running 0 4h 10.42.129.130 ip-172-31-34-207.us-east-2.compute.internal <none>
onap onap-esr-esr-gui-6c5ccd59d6-xzh46 1/1 Running 0 4h 10.42.89.47 ip-172-31-41-30.us-east-2.compute.internal <none>
onap onap-esr-esr-server-5f967d4767-nl5xc 2/2 Running 0 4h 10.42.146.185 ip-172-31-41-30.us-east-2.compute.internal <none>
onap onap-log-log-elasticsearch-7c5b74569f-g64hw 1/1 Running 0 4h 10.42.54.225 ip-172-31-33-223.us-east-2.compute.internal <none>
onap onap-log-log-kibana-84b875cc6b-gdzp8 1/1 Running 0 4h 10.42.56.186 ip-172-31-36-163.us-east-2.compute.internal <none>
onap onap-log-log-logstash-7hmrz 1/1 Running 0 4h 10.42.200.38 ip-172-31-36-163.us-east-2.compute.internal <none>
onap onap-log-log-logstash-8cb9c 1/1 Running 0 4h 10.42.162.152 ip-172-31-34-207.us-east-2.compute.internal <none>
onap onap-log-log-logstash-8pqrp 1/1 Running 0 4h 10.42.97.239 ip-172-31-41-110.us-east-2.compute.internal <none>
onap onap-log-log-logstash-bvq5r 1/1 Running 0 4h 10.42.241.246 ip-172-31-42-33.us-east-2.compute.internal <none>
onap onap-log-log-logstash-cckqk 1/1 Running 0 4h 10.42.222.107 ip-172-31-38-194.us-east-2.compute.internal <none>
onap onap-log-log-logstash-gq826 1/1 Running 0 4h 10.42.129.134 ip-172-31-33-223.us-east-2.compute.internal <none>
onap onap-log-log-logstash-kzvbv 1/1 Running 0 4h 10.42.195.222 ip-172-31-39-138.us-east-2.compute.internal <none>
onap onap-log-log-logstash-mdbtv 1/1 Running 0 4h 10.42.54.127 ip-172-31-41-229.us-east-2.compute.internal <none>
onap onap-log-log-logstash-n78q5 1/1 Running 0 4h 10.42.22.101 ip-172-31-37-24.us-east-2.compute.internal <none>
onap onap-log-log-logstash-ns76c 1/1 Running 0 4h 10.42.124.239 ip-172-31-37-59.us-east-2.compute.internal <none>
onap onap-log-log-logstash-pvsnw 1/1 Running 0 4h 10.42.164.130 ip-172-31-45-152.us-east-2.compute.internal <none>
onap onap-log-log-logstash-q86z9 1/1 Running 0 4h 10.42.73.74 ip-172-31-41-30.us-east-2.compute.internal <none>
onap onap-log-log-logstash-skr52 1/1 Running 0 4h 10.42.0.150 ip-172-31-38-95.us-east-2.compute.internal <none>
onap onap-log-log-logstash-zjkph 1/1 Running 0 4h 10.42.47.5 ip-172-31-34-111.us-east-2.compute.internal <none>
onap onap-msb-kube2msb-5c79ddd89f-4m524 1/1 Running 0 4h 10.42.86.56 ip-172-31-34-111.us-east-2.compute.internal <none>
onap onap-msb-msb-consul-6949bd46f4-gsm59 1/1 Running 0 4h 10.42.19.136 ip-172-31-33-223.us-east-2.compute.internal <none>
onap onap-msb-msb-discovery-86c7b945f9-ffxtx 2/2 Running 0 4h 10.42.27.240 ip-172-31-45-152.us-east-2.compute.internal <none>
onap onap-msb-msb-eag-5f86f89c4f-4pr2t 2/2 Running 0 4h 10.42.36.35 ip-172-31-42-33.us-east-2.compute.internal <none>
onap onap-msb-msb-iag-56cdd4c87b-4xtz7 2/2 Running 0 4h 10.42.152.29 ip-172-31-41-30.us-east-2.compute.internal <none>
onap onap-multicloud-multicloud-867df449f8-gwkdw 2/2 Running 0 4h 10.42.87.4 ip-172-31-37-24.us-east-2.compute.internal <none>
onap onap-multicloud-multicloud-azure-5db4ddd44d-cpphs 2/2 Running 0 4h 10.42.84.246 ip-172-31-37-24.us-east-2.compute.internal <none>
onap onap-multicloud-multicloud-ocata-c557948df-5sgfs 2/2 Running 0 4h 10.42.39.16 ip-172-31-41-110.us-east-2.compute.internal <none>
onap onap-multicloud-multicloud-pike-8ff99dfb6-zmrw8 2/2 Running 0 4h 10.42.176.68 ip-172-31-39-138.us-east-2.compute.internal <none>
onap onap-multicloud-multicloud-vio-866cd4bf9b-g2zlx 2/2 Running 0 4h 10.42.64.228 ip-172-31-38-95.us-east-2.compute.internal <none>
onap onap-multicloud-multicloud-windriver-7775ddc4b9-4278w 2/2 Running 0 4h 10.42.87.19 ip-172-31-38-194.us-east-2.compute.internal <none>
onap onap-nbi-nbi-7595d4dbc-9m95t 1/1 Running 1 4h 10.42.94.19 ip-172-31-41-30.us-east-2.compute.internal <none>
onap onap-nbi-nbi-mariadb-65f556b8b7-fgjzx 1/1 Running 0 4h 10.42.195.127 ip-172-31-34-207.us-east-2.compute.internal <none>
onap onap-nbi-nbi-mongo-0 1/1 Running 0 4h 10.42.184.20 ip-172-31-37-59.us-east-2.compute.internal <none>
onap onap-oof-cmso-db-0 1/1 Running 0 4h 10.42.193.70 ip-172-31-37-59.us-east-2.compute.internal <none>
onap onap-oof-music-cassandra-0 1/1 Running 1 4h 10.42.189.19 ip-172-31-42-33.us-east-2.compute.internal <none>
onap onap-oof-music-cassandra-1 1/1 Running 0 4h 10.42.163.171 ip-172-31-33-223.us-east-2.compute.internal <none>
onap onap-oof-music-cassandra-2 1/1 Running 0 4h 10.42.195.0 ip-172-31-34-207.us-east-2.compute.internal <none>
onap onap-oof-music-cassandra-job-config-7mb5f 0/1 Completed 0 4h 10.42.38.249 ip-172-31-41-110.us-east-2.compute.internal <none>
onap onap-oof-music-tomcat-7974f8748c-4pgg8 1/1 Running 0 4h 10.42.118.22 ip-172-31-38-95.us-east-2.compute.internal <none>
onap onap-oof-music-tomcat-7974f8748c-dxdds 1/1 Running 0 4h 10.42.159.58 ip-172-31-41-110.us-east-2.compute.internal <none>
onap onap-oof-music-tomcat-7974f8748c-jh7jg 1/1 Running 0 4h 10.42.64.7 ip-172-31-45-152.us-east-2.compute.internal <none>
onap onap-oof-oof-6dbf897c86-8d6bx 1/1 Running 0 4h 10.42.20.170 ip-172-31-45-152.us-east-2.compute.internal <none>
onap onap-oof-oof-cmso-service-7d54b6b478-djvdm 1/1 Running 0 4h 10.42.192.55 ip-172-31-41-229.us-east-2.compute.internal <none>
onap onap-oof-oof-has-api-774f9458cb-mtfg8 1/1 Running 0 4h 10.42.208.73 ip-172-31-36-163.us-east-2.compute.internal <none>
onap onap-oof-oof-has-controller-7998c7f7b6-vg6nz 1/1 Running 0 4h 10.42.91.250 ip-172-31-36-163.us-east-2.compute.internal <none>
onap onap-oof-oof-has-data-5dffb4c7df-qmjzz 1/1 Running 0 4h 10.42.213.36 ip-172-31-34-207.us-east-2.compute.internal <none>
onap onap-oof-oof-has-healthcheck-rpst7 0/1 Completed 0 4h 10.42.241.223 ip-172-31-39-138.us-east-2.compute.internal <none>
onap onap-oof-oof-has-onboard-5bd2l 0/1 Completed 0 4h 10.42.205.75 ip-172-31-38-194.us-east-2.compute.internal <none>
onap onap-oof-oof-has-reservation-59759db679-dz4cs 1/1 Running 0 4h 10.42.48.244 ip-172-31-34-111.us-east-2.compute.internal <none>
onap onap-oof-oof-has-solver-5bfc76f46-hfc88 1/1 Running 0 4h 10.42.225.129 ip-172-31-33-223.us-east-2.compute.internal <none>
onap onap-oof-zookeeper-0 1/1 Running 0 4h 10.42.171.61 ip-172-31-37-59.us-east-2.compute.internal <none>
onap onap-oof-zookeeper-1 1/1 Running 0 4h 10.42.251.101 ip-172-31-34-111.us-east-2.compute.internal <none>
onap onap-oof-zookeeper-2 1/1 Running 0 4h 10.42.65.144 ip-172-31-37-24.us-east-2.compute.internal <none>
onap onap-policy-brmsgw-86dd8fc4f7-xtzk9 1/1 Running 0 4h 10.42.56.63 ip-172-31-41-30.us-east-2.compute.internal <none>
onap onap-policy-drools-0 1/1 Running 0 4h 10.42.75.29 ip-172-31-41-110.us-east-2.compute.internal <none>
onap onap-policy-nexus-b7f484bdb-gmn76 1/1 Running 0 4h 10.42.45.249 ip-172-31-41-229.us-east-2.compute.internal <none>
onap onap-policy-pap-6555659676-77nl7 2/2 Running 0 4h 10.42.87.216 ip-172-31-36-163.us-east-2.compute.internal <none>
onap onap-policy-pdp-0 2/2 Running 0 4h 10.42.63.230 ip-172-31-39-138.us-east-2.compute.internal <none>
onap onap-policy-policy-apex-pdp-0 1/1 Running 0 4h 10.42.73.87 ip-172-31-38-95.us-east-2.compute.internal <none>
onap onap-policy-policy-distribution-79544b78c7-hk5mk 1/1 Running 0 4h 10.42.104.145 ip-172-31-37-59.us-east-2.compute.internal <none>
onap onap-policy-policydb-79cdf7cb7c-jb82c 1/1 Running 0 4h 10.42.98.252 ip-172-31-38-194.us-east-2.compute.internal <none>
onap onap-pomba-pomba-aaictxbuilder-684d5977cd-sj8dm 2/2 Running 0 4h 10.42.170.23 ip-172-31-34-207.us-east-2.compute.internal <none>
onap onap-pomba-pomba-contextaggregator-779745878d-4s8c9 1/1 Running 0 4h 10.42.154.6 ip-172-31-34-111.us-east-2.compute.internal <none>
onap onap-pomba-pomba-data-router-59bb4fd899-dc5zc 1/1 Running 0 4h 10.42.171.88 ip-172-31-38-194.us-east-2.compute.internal <none>
onap onap-pomba-pomba-elasticsearch-7b8bc5f864-5mjqk 1/1 Running 0 4h 10.42.111.181 ip-172-31-33-223.us-east-2.compute.internal <none>
onap onap-pomba-pomba-kibana-6dd8dbb68b-xltcm 1/1 Running 0 4h 10.42.181.201 ip-172-31-45-152.us-east-2.compute.internal <none>
onap onap-pomba-pomba-networkdiscovery-d47f68c88-j2zw4 2/2 Running 0 4h 10.42.34.101 ip-172-31-41-30.us-east-2.compute.internal <none>
onap onap-pomba-pomba-networkdiscoveryctxbuilder-6978dc6c79-6z7zm 2/2 Running 0 4h 10.42.116.177 ip-172-31-41-229.us-east-2.compute.internal <none>
onap onap-pomba-pomba-sdcctxbuilder-fcbc7f96b-nj2c9 1/1 Running 0 4h 10.42.87.59 ip-172-31-41-110.us-east-2.compute.internal <none>
onap onap-pomba-pomba-search-data-5b4d8f7dc6-k7qlx 2/2 Running 0 4h 10.42.37.226 ip-172-31-39-138.us-east-2.compute.internal <none>
onap onap-pomba-pomba-servicedecomposition-99b445978-6crkz 2/2 Running 0 4h 10.42.46.61 ip-172-31-41-110.us-east-2.compute.internal <none>
onap onap-pomba-pomba-validation-service-5c8454484-t4kls 1/1 Running 0 4h 10.42.235.206 ip-172-31-38-194.us-east-2.compute.internal <none>
onap onap-portal-portal-app-67bc49d88-tpxbq 2/2 Running 0 4h 10.42.25.53 ip-172-31-33-223.us-east-2.compute.internal <none>
onap onap-portal-portal-cassandra-75d4746d76-jn78x 1/1 Running 0 4h 10.42.128.240 ip-172-31-36-163.us-east-2.compute.internal <none>
onap onap-portal-portal-db-7b768ffcc-92q8b 1/1 Running 1 4h 10.42.187.56 ip-172-31-34-207.us-east-2.compute.internal <none>
onap onap-portal-portal-db-config-qshzn 0/2 Completed 0 4h 10.42.112.46 ip-172-31-45-152.us-east-2.compute.internal <none>
onap onap-portal-portal-db-config-rk4m2 0/2 Init:Error 0 4h 10.42.57.79 ip-172-31-38-194.us-east-2.compute.internal <none>
onap onap-portal-portal-sdk-59fbd4bd9d-6j9x9 2/2 Running 0 4h 10.42.217.29 ip-172-31-41-229.us-east-2.compute.internal <none>
onap onap-portal-portal-widget-79587fdf56-5dh2c 1/1 Running 0 4h 10.42.12.249 ip-172-31-33-223.us-east-2.compute.internal <none>
onap onap-portal-portal-zookeeper-7c9444bfb5-9bvwq 1/1 Running 0 4h 10.42.113.205 ip-172-31-37-59.us-east-2.compute.internal <none>
onap onap-robot-robot-86c5f986b7-kd257 1/1 Running 0 4h 10.42.246.138 ip-172-31-36-163.us-east-2.compute.internal <none>
onap onap-sdc-sdc-be-7fdc576bfb-6q5bx 2/2 Running 0 4h 10.42.142.158 ip-172-31-34-111.us-east-2.compute.internal <none>
onap onap-sdc-sdc-be-config-backend-2vw2q 0/1 Completed 0 4h 10.42.87.181 ip-172-31-39-138.us-east-2.compute.internal <none>
onap onap-sdc-sdc-be-config-backend-k57lh 0/1 Init:Error 0 4h 10.42.148.79 ip-172-31-45-152.us-east-2.compute.internal <none>
onap onap-sdc-sdc-cs-5b9bb5756d-28zhd 1/1 Running 1 4h 10.42.232.24 ip-172-31-37-59.us-east-2.compute.internal <none>
onap onap-sdc-sdc-cs-config-cassandra-vgnz2 0/1 Completed 0 4h 10.42.111.187 ip-172-31-34-111.us-east-2.compute.internal <none>
onap onap-sdc-sdc-es-8bdf86659-bvgj6 1/1 Running 0 4h 10.42.21.223 ip-172-31-36-163.us-east-2.compute.internal <none>
onap onap-sdc-sdc-es-config-elasticsearch-lkb9m 0/1 Completed 0 4h 10.42.20.202 ip-172-31-39-138.us-east-2.compute.internal <none>
onap onap-sdc-sdc-fe-65c95cf8bf-2944f 2/2 Running 0 4h 10.42.193.193 ip-172-31-34-207.us-east-2.compute.internal <none>
onap onap-sdc-sdc-kb-79bfffc4cf-6p5qn 1/1 Running 0 4h 10.42.59.138 ip-172-31-34-111.us-east-2.compute.internal <none>
onap onap-sdc-sdc-onboarding-be-84466885d5-pwqpn 2/2 Running 0 4h 10.42.163.142 ip-172-31-45-152.us-east-2.compute.internal <none>
onap onap-sdc-sdc-onboarding-be-cassandra-init-7zv5j 0/1 Completed 0 4h 10.42.218.1 ip-172-31-41-229.us-east-2.compute.internal <none>
onap onap-sdc-sdc-wfd-be-54f49cb9f-8wmkz 1/1 Running 0 4h 10.42.69.253 ip-172-31-34-207.us-east-2.compute.internal <none>
onap onap-sdc-sdc-wfd-be-workflow-init-q8t7z 0/1 Completed 0 4h 10.42.255.91 ip-172-31-41-30.us-east-2.compute.internal <none>
onap onap-sdc-sdc-wfd-fe-585bd84f8d-grwwh 2/2 Running 0 4h 10.42.110.244 ip-172-31-38-95.us-east-2.compute.internal <none>
onap onap-sdnc-controller-blueprints-7c8f9dbfb7-tq4f5 1/1 Running 0 4h 10.42.200.245 ip-172-31-37-59.us-east-2.compute.internal <none>
onap onap-sdnc-controller-blueprints-db-0 1/1 Running 0 4h 10.42.207.149 ip-172-31-41-30.us-east-2.compute.internal <none>
onap onap-sdnc-nengdb-0 1/1 Running 0 4h 10.42.211.255 ip-172-31-41-229.us-east-2.compute.internal <none>
onap onap-sdnc-network-name-gen-f4ff8b99f-8bxbh 1/1 Running 0 4h 10.42.107.56 ip-172-31-41-110.us-east-2.compute.internal <none>
onap onap-sdnc-sdnc-0 2/2 Running 0 4h 10.42.220.124 ip-172-31-33-223.us-east-2.compute.internal <none>
onap onap-sdnc-sdnc-ansible-server-dffcf4c8b-jdpmh 1/1 Running 0 4h 10.42.84.202 ip-172-31-39-138.us-east-2.compute.internal <none>
onap onap-sdnc-sdnc-db-0 2/2 Running 0 4h 10.42.45.121 ip-172-31-36-163.us-east-2.compute.internal <none>
onap onap-sdnc-sdnc-dgbuilder-5496ff5f86-5l27s 1/1 Running 0 4h 10.42.17.254 ip-172-31-38-95.us-east-2.compute.internal <none>
onap onap-sdnc-sdnc-dmaap-listener-54f5c69c7c-bxp2n 1/1 Running 0 4h 10.42.42.177 ip-172-31-41-229.us-east-2.compute.internal <none>
onap onap-sdnc-sdnc-portal-8ff4bb696-j9xz5 1/1 Running 0 4h 10.42.250.3 ip-172-31-38-95.us-east-2.compute.internal <none>
onap onap-sdnc-sdnc-ueb-listener-5d5749f879-2pw5j 1/1 Running 0 4h 10.42.93.48 ip-172-31-38-194.us-east-2.compute.internal <none>
onap onap-sniro-emulator-sniro-emulator-5bdc8cc4cb-qjbk8 1/1 Running 0 4h 10.42.207.251 ip-172-31-37-59.us-east-2.compute.internal <none>
onap onap-so-so-7f5f98949b-w85kc 1/1 Running 0 4h 10.42.110.129 ip-172-31-39-138.us-east-2.compute.internal <none>
onap onap-so-so-bpmn-infra-74fcf888b4-9tnfc 1/1 Running 0 4h 10.42.60.90 ip-172-31-37-24.us-east-2.compute.internal <none>
onap onap-so-so-catalog-db-adapter-5b8cb47dc-j44bt 1/1 Running 0 4h 10.42.222.142 ip-172-31-36-163.us-east-2.compute.internal <none>
onap onap-so-so-mariadb-5b5bd695c4-jdx8v 1/1 Running 0 4h 10.42.249.209 ip-172-31-37-59.us-east-2.compute.internal <none>
onap onap-so-so-monitoring-855456948b-fr24q 1/1 Running 0 4h 10.42.14.54 ip-172-31-34-207.us-east-2.compute.internal <none>
onap onap-so-so-openstack-adapter-c44cd98dc-pk2gb 1/1 Running 1 4h 10.42.25.240 ip-172-31-34-111.us-east-2.compute.internal <none>
onap onap-so-so-request-db-adapter-745dc6677-z7tcx 1/1 Running 0 4h 10.42.186.49 ip-172-31-45-152.us-east-2.compute.internal <none>
onap onap-so-so-sdc-controller-67f8cbbc7c-kdfmr 1/1 Running 0 4h 10.42.23.215 ip-172-31-41-30.us-east-2.compute.internal <none>
onap onap-so-so-sdnc-adapter-5cbc5f8dd8-6k22h 1/1 Running 0 4h 10.42.14.50 ip-172-31-41-229.us-east-2.compute.internal <none>
onap onap-so-so-vfc-adapter-6c5996d748-5h2jj 1/1 Running 0 4h 10.42.205.174 ip-172-31-41-110.us-east-2.compute.internal <none>
onap onap-uui-uui-d5ff69677-xkmq2 1/1 Running 0 4h 10.42.157.9 ip-172-31-38-194.us-east-2.compute.internal <none>
onap onap-uui-uui-server-7cc8bcb956-c28g5 1/1 Running 0 4h 10.42.245.99 ip-172-31-38-95.us-east-2.compute.internal <none>
onap onap-vfc-vfc-catalog-66dd84cc97-htgsr 2/2 Running 0 4h 10.42.248.74 ip-172-31-37-59.us-east-2.compute.internal <none>
onap onap-vfc-vfc-db-65d8fd5567-25z45 1/1 Running 0 4h 10.42.181.50 ip-172-31-37-24.us-east-2.compute.internal <none>
onap onap-vfc-vfc-ems-driver-6f5fd887db-hrh7q 1/1 Running 0 4h 10.42.28.44 ip-172-31-36-163.us-east-2.compute.internal <none>
onap onap-vfc-vfc-generic-vnfm-driver-7964b7f4cc-nd2d2 2/2 Running 1 4h 10.42.172.200 ip-172-31-34-207.us-east-2.compute.internal <none>
onap onap-vfc-vfc-huawei-vnfm-driver-6c5676758d-vs8fh 2/2 Running 0 4h 10.42.58.24 ip-172-31-34-111.us-east-2.compute.internal <none>
onap onap-vfc-vfc-juju-vnfm-driver-67b9c4bb4c-fhv45 2/2 Running 0 4h 10.42.10.58 ip-172-31-41-30.us-east-2.compute.internal <none>
onap onap-vfc-vfc-multivim-proxy-5f6f455bfd-tg7m2 1/1 Running 0 4h 10.42.51.118 ip-172-31-41-229.us-east-2.compute.internal <none>
onap onap-vfc-vfc-nokia-v2vnfm-driver-6f7646f69d-22p88 1/1 Running 0 4h 10.42.158.31 ip-172-31-41-110.us-east-2.compute.internal <none>
onap onap-vfc-vfc-nokia-vnfm-driver-7c9d59597f-gkmct 2/2 Running 0 4h 10.42.184.15 ip-172-31-39-138.us-east-2.compute.internal <none>
onap onap-vfc-vfc-nslcm-c8478dcc8-cnlvd 2/2 Running 0 4h 10.42.144.183 ip-172-31-38-95.us-east-2.compute.internal <none>
onap onap-vfc-vfc-resmgr-6d7d7bc78b-f5v88 2/2 Running 0 4h 10.42.255.238 ip-172-31-37-24.us-east-2.compute.internal <none>
onap onap-vfc-vfc-vnflcm-fb656d967-p6wh2 2/2 Running 0 4h 10.42.56.73 ip-172-31-36-163.us-east-2.compute.internal <none>
onap onap-vfc-vfc-vnfmgr-66d5cd85f7-ld4v4 2/2 Running 0 4h 10.42.121.241 ip-172-31-34-207.us-east-2.compute.internal <none>
onap onap-vfc-vfc-vnfres-7757d4b486-bkqqz 2/2 Running 0 4h 10.42.100.120 ip-172-31-34-111.us-east-2.compute.internal <none>
onap onap-vfc-vfc-workflow-68f85b8fcf-mxgwl 1/1 Running 0 4h 10.42.72.65 ip-172-31-45-152.us-east-2.compute.internal <none>
onap onap-vfc-vfc-workflow-engine-7d76bb4b56-tj6ks 1/1 Running 0 4h 10.42.48.13 ip-172-31-33-223.us-east-2.compute.internal <none>
onap onap-vfc-vfc-zte-sdnc-driver-58d4fc6d9f-hk4d9 1/1 Running 0 4h 10.42.245.72 ip-172-31-41-30.us-east-2.compute.internal <none>
onap onap-vfc-vfc-zte-vnfm-driver-fc669cc8d-64k6j 2/2 Running 1 4h 10.42.68.226 ip-172-31-41-229.us-east-2.compute.internal <none>
onap onap-vid-vid-5cb464799b-fw69j 2/2 Running 0 4h 10.42.21.193 ip-172-31-41-110.us-east-2.compute.internal <none>
onap onap-vid-vid-galera-config-4f274 0/1 Completed 0 4h 10.42.80.200 ip-172-31-33-223.us-east-2.compute.internal <none>
onap onap-vid-vid-mariadb-galera-0 1/1 Running 0 4h 10.42.40.225 ip-172-31-45-152.us-east-2.compute.internal <none>
onap onap-vnfsdk-vnfsdk-6b45bfffb4-phcwx 1/1 Running 0 4h 10.42.32.180 ip-172-31-38-95.us-east-2.compute.internal <none>
onap onap-vnfsdk-vnfsdk-init-postgres-lf659 0/1 Completed 0 4h 10.42.238.204 ip-172-31-38-194.us-east-2.compute.internal <none>
onap onap-vnfsdk-vnfsdk-pgpool-6874f7b695-69njk 1/1 Running 0 4h 10.42.239.14 ip-172-31-36-163.us-east-2.compute.internal <none>
onap onap-vnfsdk-vnfsdk-pgpool-6874f7b695-pjmwp 1/1 Running 0 4h 10.42.171.177 ip-172-31-39-138.us-east-2.compute.internal <none>
onap onap-vnfsdk-vnfsdk-postgres-0 1/1 Running 0 4h 10.42.106.183 ip-172-31-37-59.us-east-2.compute.internal <none>
onap onap-vnfsdk-vnfsdk-postgres-1 1/1 Running 0 4h 10.42.47.66 ip-172-31-41-229.us-east-2.compute.internal <none>
ubuntu@ip-172-31-40-250:~$ kubectl get nodes -o wide
NAME STATUS ROLES AGE VERSION INTERNAL-IP EXTERNAL-IP OS-IMAGE KERNEL-VERSION CONTAINER-RUNTIME
ip-172-31-33-223.us-east-2.compute.internal Ready <none> 5h v1.11.2-rancher1 18.222.148.116 18.222.148.116 Ubuntu 16.04.1 LTS 4.4.0-1049-aws docker://17.3.2
ip-172-31-34-111.us-east-2.compute.internal Ready <none> 5h v1.11.2-rancher1 3.16.37.170 3.16.37.170 Ubuntu 16.04.1 LTS 4.4.0-1049-aws docker://17.3.2
ip-172-31-34-207.us-east-2.compute.internal Ready <none> 5h v1.11.2-rancher1 18.225.32.201 18.225.32.201 Ubuntu 16.04.1 LTS 4.4.0-1049-aws docker://17.3.2
ip-172-31-36-163.us-east-2.compute.internal Ready <none> 5h v1.11.2-rancher1 13.58.189.251 13.58.189.251 Ubuntu 16.04.1 LTS 4.4.0-1049-aws docker://17.3.2
ip-172-31-37-24.us-east-2.compute.internal Ready <none> 5h v1.11.2-rancher1 18.224.180.26 18.224.180.26 Ubuntu 16.04.1 LTS 4.4.0-1049-aws docker://17.3.2
ip-172-31-37-59.us-east-2.compute.internal Ready <none> 5h v1.11.2-rancher1 18.191.248.14 18.191.248.14 Ubuntu 16.04.1 LTS 4.4.0-1049-aws docker://17.3.2
ip-172-31-38-194.us-east-2.compute.internal Ready <none> 4h v1.11.2-rancher1 18.217.45.91 18.217.45.91 Ubuntu 16.04.1 LTS 4.4.0-1049-aws docker://17.3.2
ip-172-31-38-95.us-east-2.compute.internal Ready <none> 4h v1.11.2-rancher1 52.15.39.21 52.15.39.21 Ubuntu 16.04.1 LTS 4.4.0-1049-aws docker://17.3.2
ip-172-31-39-138.us-east-2.compute.internal Ready <none> 4h v1.11.2-rancher1 18.224.199.40 18.224.199.40 Ubuntu 16.04.1 LTS 4.4.0-1049-aws docker://17.3.2
ip-172-31-41-110.us-east-2.compute.internal Ready <none> 4h v1.11.2-rancher1 18.223.151.180 18.223.151.180 Ubuntu 16.04.1 LTS 4.4.0-1049-aws docker://17.3.2
ip-172-31-41-229.us-east-2.compute.internal Ready <none> 5h v1.11.2-rancher1 18.218.252.13 18.218.252.13 Ubuntu 16.04.1 LTS 4.4.0-1049-aws docker://17.3.2
ip-172-31-41-30.us-east-2.compute.internal Ready <none> 4h v1.11.2-rancher1 3.16.113.3 3.16.113.3 Ubuntu 16.04.1 LTS 4.4.0-1049-aws docker://17.3.2
ip-172-31-42-33.us-east-2.compute.internal Ready <none> 5h v1.11.2-rancher1 13.59.2.86 13.59.2.86 Ubuntu 16.04.1 LTS 4.4.0-1049-aws docker://17.3.2
ip-172-31-45-152.us-east-2.compute.internal Ready <none> 4h v1.11.2-rancher1 18.219.56.50 18.219.56.50 Ubuntu 16.04.1 LTS 4.4.0-1049-aws docker://17.3.2
ubuntu@ip-172-31-40-250:~$ kubectl top nodes
NAME CPU(cores) CPU% MEMORY(bytes) MEMORY%
ip-172-31-33-223.us-east-2.compute.internal 852m 10% 13923Mi 90%
ip-172-31-34-111.us-east-2.compute.internal 1160m 14% 11643Mi 75%
ip-172-31-34-207.us-east-2.compute.internal 1101m 13% 7981Mi 51%
ip-172-31-36-163.us-east-2.compute.internal 656m 8% 13377Mi 87%
ip-172-31-37-24.us-east-2.compute.internal 401m 5% 8543Mi 55%
ip-172-31-37-59.us-east-2.compute.internal 711m 8% 10873Mi 70%
ip-172-31-38-194.us-east-2.compute.internal 1136m 14% 8195Mi 53%
ip-172-31-38-95.us-east-2.compute.internal 1195m 14% 9127Mi 59%
ip-172-31-39-138.us-east-2.compute.internal 296m 3% 10870Mi 70%
ip-172-31-41-110.us-east-2.compute.internal 2586m 32% 10950Mi 71%
ip-172-31-41-229.us-east-2.compute.internal 159m 1% 9138Mi 59%
ip-172-31-41-30.us-east-2.compute.internal 180m 2% 9862Mi 64%
ip-172-31-42-33.us-east-2.compute.internal 1573m 19% 6352Mi 41%
ip-172-31-45-152.us-east-2.compute.internal 1579m 19% 10633Mi 69%
ubuntu@ip-172-31-40-250:~/oom/kubernetes/robot$ sudo ./ete-k8s.sh onap health
./ete-k8s.sh: 22: [: onap: unexpected operator
./ete-k8s.sh: 22: [: health: unexpected operator
+ export NAMESPACE=onap
Testsuites
==============================================================================
Testsuites.Health-Check :: Testing ecomp components are available via calls.
==============================================================================
Basic A&AI Health Check | PASS |
------------------------------------------------------------------------------
Basic AAF Health Check | PASS |
------------------------------------------------------------------------------
Basic AAF SMS Health Check | PASS |
------------------------------------------------------------------------------
Basic APPC Health Check | PASS |
------------------------------------------------------------------------------
Basic CLI Health Check | PASS |
------------------------------------------------------------------------------
Basic CLAMP Health Check | PASS |
------------------------------------------------------------------------------
Basic DCAE Health Check | PASS |
------------------------------------------------------------------------------
Basic DMAAP Data Router Health Check [ WARN ] Retrying (Retry(total=2, connect=None, read=None, redirect=None, status=None)) after connection broken by 'NewConnectionError('<urllib3.connection.HTTPConnection object at 0x7f536960ea50>: Failed to establish a new connection: [Errno 111] Connection refused',)': /internal/fetchProv
[ WARN ] Retrying (Retry(total=1, connect=None, read=None, redirect=None, status=None)) after connection broken by 'NewConnectionError('<urllib3.connection.HTTPConnection object at 0x7f536972e450>: Failed to establish a new connection: [Errno 111] Connection refused',)': /internal/fetchProv
[ WARN ] Retrying (Retry(total=0, connect=None, read=None, redirect=None, status=None)) after connection broken by 'NewConnectionError('<urllib3.connection.HTTPConnection object at 0x7f5369617090>: Failed to establish a new connection: [Errno 111] Connection refused',)': /internal/fetchProv
| FAIL |
ConnectionError: HTTPConnectionPool(host='dmaap-dr-node.onap', port=8080): Max retries exceeded with url: /internal/fetchProv (Caused by NewConnectionError('<urllib3.connection.HTTPConnection object at 0x7f5369d21850>: Failed to establish a new connection: [Errno 111] Connection refused',))
------------------------------------------------------------------------------
Basic DMAAP Message Router Health Check | PASS |
------------------------------------------------------------------------------
Basic External API NBI Health Check | PASS |
------------------------------------------------------------------------------
Basic Log Elasticsearch Health Check | PASS |
------------------------------------------------------------------------------
Basic Log Kibana Health Check | PASS |
------------------------------------------------------------------------------
Basic Log Logstash Health Check | PASS |
------------------------------------------------------------------------------
Basic Microservice Bus Health Check | PASS |
------------------------------------------------------------------------------
Basic Multicloud API Health Check | PASS |
------------------------------------------------------------------------------
Basic Multicloud-ocata API Health Check | PASS |
------------------------------------------------------------------------------
Basic Multicloud-pike API Health Check |
...
| Code Block | ||
|---|---|---|
| ||
aaf-cm
aaf-fs
aaf-gui
aaf-hello
aaf-locate
aaf-oauth
aaf-service
aaf-sms-vault-0
|
...
| Code Block | ||
|---|---|---|
| ||
aai-babel
aai-champ
aai-data-router
aai-gizmo
aai-graphadmin
aai-modelloader
aai-resources
aai-search-data
aai-sparky-be
aai-spike
aai-traversal
|
...
| Code Block | ||
|---|---|---|
| ||
appc-0
|
ref
| Code Block | ||
|---|---|---|
| ||
20181031 ubuntu@a-osn-cd:~/oom/kubernetes$ kubectl get pods --all-namespaces | grep /2 onap onap-clamp-clamp-9dfd65f8f-rhdlc | PASS | ------------------------------------------------------------------------------ Basic Multicloud-titanium_cloud API Health Check 0/2 Init:0/1 0| 6m onap onap-dmaap-message-router-kafka-74f9bff668-h6flz PASS | ------------------------------------------------------------------------------ Basic Multicloud-vio API Health Check 0/1 Init:0/2 0 | PASS 6m onap onap-esr-esr-server-6d7dd55794-rp9zz | ------------------------------------------------------------------------------ Basic OOF-Homing Health Check 0/2 Pending | 0 6m onap onap-msb-msb-discovery-59d48c65b-k68cj 0/2 Init:0/1 0 6m onap onap-msb-msb-eag-76857bcc54-8zbpp PASS | ------------------------------------------------------------------------------ Basic OOF-SNIRO Health Check 0/2 Pending | 0 6m onap onap-msb-msb-iag-59697db845-mlpz8 PASS | ------------------------------------------------------------------------------ Basic OOF-CMSO Health Check 0/2 Pending | 0 6m onap onap-multicloud-multicloud-5765cd745-bb57j 0/2 Pending 0 6m onap onap-multicloud-multicloud-azure-6c99454479-4457s PASS | ------------------------------------------------------------------------------ Basic Policy Health Check 0/2 Pending 0 6m onap | onap-multicloud-multicloud-ocata-79cc9dfb76-w6bx4 0/2 Pending 0 6m onap onap-multicloud-multicloud-pike-547d878bb5-d9cjj PASS | ------------------------------------------------------------------------------ Basic Pomba AAI-context-builder Health Check 0/2 Pending | 0 6m onap onap-multicloud-multicloud-vio-67979f99bc-gvzxz 0/2PASS | ------------------------------------------------------------------------------ Basic Pomba SDC-context-builder Health Check Pending 0 | 6m onap onap-multicloud-multicloud-windriver-748868744d-69sjr 0/2 Pending 0 PASS | ------------------------------------------------------------------------------ Basic Pomba Network-discovery-context-builder Health Check 6m onap | onap-policy-pap-5b95866fff-6wxln PASS | ------------------------------------------------------------------------------ Basic Portal Health Check 0/2 Pending | 0 5m onap onap-policy-pdp-0 PASS | ------------------------------------------------------------------------------ Basic SDC Health Check 0/2 Init:0/1 0 5m onap onap-pomba-pomba-aaictxbuilder-5d45d589-nnxck (DMaaP:UP)| PASS | ------------------------------------------------------------------------------ Basic SDNC Health Check 0/2 Pending 0 5m onap | onap-pomba-pomba-networkdiscovery-8487cc6667-2j6wq 0/2 Pending 0PASS | ------------------------------------------------------------------------------ Basic SO Health Check 5m onap onap-pomba-pomba-networkdiscoveryctxbuilder-c9c5c8f98-2svzw 0/2 Pending 0 | PASS 5m onap onap-pomba-pomba-search-data-7548469497-9pjq4 0/2 Pending 0 5m onap onap-pomba-pomba-servicedecomposition-8985f9849-2cdl7 0/2| ------------------------------------------------------------------------------ Basic UseCaseUI API Health Check Pending 0 5m onap | onap-portal-portal-app-5cc7b45bd9-45c6c 0/2 PASS | ------------------------------------------------------------------------------ Basic VFC catalog API Health Check Init:0/1 0 5m onap | onap-portal-portal-db-config-dwrrt PASS | ------------------------------------------------------------------------------ Basic VFC emsdriver API Health Check 0/2 Pending | 0 5m onap onap-portal-portal-sdk-5bb84db56c-x4zcj PASS | ------------------------------------------------------------------------------ Basic VFC gvnfmdriver API Health Check 0/2 Init:0/1 0 | 5m onap onap-sdc-sdc-be-68d86d55cf-pxf66 PASS | ------------------------------------------------------------------------------ Basic VFC huaweivnfmdriver API Health Check 0/2 | PASS | ------------------------------------------------------------------------------ Basic VFC jujuvnfmdriver PendingAPI Health Check 0 5m onap onap-sdc-sdc-fe-67ccb88464-9qbwj | PASS | ------------------------------------------------------------------------------ Basic VFC multivimproxy API Health Check 0/2 Pending | 0 5m onap onap-sdc-sdc-onboarding-be-67b4bc94b-2gscq PASS | ------------------------------------------------------------------------------ Basic VFC nokiavnfmdriver API Health Check 0/2 Pending | 0 5m onap onap-sdc-sdc-wfd-fe-87d9cd457-26crq PASS | ------------------------------------------------------------------------------ Basic VFC nokiav2driver API Health Check 0/2 Pending | 0 5m onap onap-sdnc-sdnc-0 PASS | ------------------------------------------------------------------------------ Basic VFC nslcm API Health Check 0/2 Pending 0| 5m onap onap-sdnc-sdnc-db-0 0/2PASS | ------------------------------------------------------------------------------ Basic VFC resmgr API Health Check Pending 0 5m onap | onap-vfc-vfc-catalog-85bd56b986-c6zld PASS | ------------------------------------------------------------------------------ Basic VFC vnflcm API Health Check 0/2 Pending | 0 4m onap onap-vfc-vfc-generic-vnfm-driver-dc4864b96-c9b6h PASS | ------------------------------------------------------------------------------ Basic VFC vnfmgr API Health Check 0/2 Pending 0 | PASS 4m onap onap-vfc-vfc-huawei-vnfm-driver-589c6c8467-swdqt | ------------------------------------------------------------------------------ Basic VFC vnfres API Health Check 0/2 Pending 0 | PASS 4m onap onap-vfc-vfc-juju-vnfm-driver-7cb5dc655d-mkb7l 0/2 Pending 0 4m onap onap-vfc-vfc-nokia-vnfm-driver-57c4f5d9cd-jh9mb | ------------------------------------------------------------------------------ Basic VFC workflow API Health Check 0/2 ContainerCreating 0 4m onap | onap-vfc-vfc-nslcm-675c748bb7-t6rfz 0/2 Init:0/1 0 4m onap onap-vfc-vfc-resmgr-599fb9c6f7-64djz PASS | ------------------------------------------------------------------------------ Basic VFC ztesdncdriver API Health Check 0/2 Pending | 0 4m onap onap-vfc-vfc-vnflcm-75cc76d968-9dblx PASS | ------------------------------------------------------------------------------ Basic VFC ztevnfmdriver API Health Check 0/2 Pending | 0 4m onap onap-vfc-vfc-vnfmgr-5f866c5458-4h56p PASS | ------------------------------------------------------------------------------ Basic VID Health Check 0/2 Init:0/1 0 4m onap | onap-vfc-vfc-vnfres-579459469f-f2kns PASS | ------------------------------------------------------------------------------ Basic VNFSDK Health Check 0/2 Pending 0 | 4m onap onap-vfc-vfc-zte-vnfm-driver-5dd4b87cd7-ppj42 PASS | ------------------------------------------------------------------------------ Basic Holmes Rule Management API Health Check 0/2 Pending | FAIL | 502 0 4m onap onap-vid-vid-599d9c54f9-79vfl != 200 ------------------------------------------------------------------------------ Basic Holmes Engine Management API Health Check | FAIL | 502 0/2 Pending 0 4m!= 200 ------------------------------------------------------------------------------ Testsuites.Health-Check :: Testing ecomp components are available ... | FAIL | 51 critical tests, 48 passed, 3 failed 51 tests total, 48 passed, 3 failed |
ONAP Module Dependencies - Compile and Runtime
...