LOG-770 - Getting issue details... STATUS

2 Types of test - Logs make it to the ELK stack, and logs are using the current spec library

pending: documentation, security waivers continued..., 

The ELK stack tests  - are all or nothing - as in if the log-ls service on my end is up for AAI is is up for CLAMP as well for example

ONAP deployment Model

Diagram not yet complete - i am in the M's

Component Tests


Projects NameOOMPOD

Clone Date
GMT-0

Test DateTest DetailsNotes
Health Check

PASS

all20181101:010020181123:0100Healthcheck

live server

http://jenkins.onap.info/job/oom-cd-master/4233/console


http://kibana.onap.info:5601/app/kibana#/dashboard/AWAtvpS63NTXK5mX2kuS


CD system: OK

http://jenkins.onap.info/job/oom-cd-master/4700/console

16:34:14 kube-system   heapster-7b48b696fc-hbb86                                      1/1       Running   0          2h

16:34:15 kube-system   kube-dns-6655f78c68-zqnkh                                      3/3       Running   0          2h

16:34:15 kube-system   kubernetes-dashboard-6f54f7c4b-b9phq                           1/1       Running   0          2h

16:34:15 kube-system   monitoring-grafana-7877679464-6jxbk                            1/1       Running   0          2h

16:34:15 kube-system   monitoring-influxdb-64664c6cf5-6swxb                           1/1       Running   0          2h

16:34:15 kube-system   tiller-deploy-6f4745cbcf-247xx                                 1/1       Running   0          2h

16:34:15 onap          onap-dmaap-dbc-pg-0                                            1/1       Running   0          13m

16:34:15 onap          onap-dmaap-dbc-pg-1                                            1/1       Running   0          11m

16:34:15 onap          onap-dmaap-dbc-pgpool-c5f8498-n9xc5                            1/1       Running   0          13m

16:34:15 onap          onap-dmaap-dbc-pgpool-c5f8498-wwskh                            1/1       Running   0          13m

16:34:15 onap          onap-dmaap-dmaap-bus-controller-557dc8c59c-rr5pr               1/1       Running   0          13m

16:34:15 onap          onap-dmaap-dmaap-dr-db-576f7968b8-z474s                        1/1       Running   0          13m

16:34:15 onap          onap-dmaap-dmaap-dr-node-7647f9d6d8-bp796                      1/1       Running   0          13m

16:34:15 onap          onap-dmaap-dmaap-dr-prov-f4d84869f-zlpwq                       1/1       Running   0          13m

16:34:15 onap          onap-dmaap-message-router-76f4799d-h44dm                       1/1       Running   0          13m

16:34:15 onap          onap-dmaap-message-router-kafka-757cf5cfb5-h8cpj               1/1       Running   0          13m

16:34:15 onap          onap-dmaap-message-router-zookeeper-5fb67dfdb5-tlhc4           1/1       Running   0          13m

16:34:15 onap          onap-log-log-elasticsearch-9798b655f-szr9j                     1/1       Running   0          13m

16:34:15 onap          onap-log-log-kibana-6b89fd4858-22brs                           1/1       Running   0          13m

16:34:15 onap          onap-log-log-logstash-8676887cd4-7nff7                         1/1       Running   0          13m

16:34:15 onap          onap-log-log-logstash-8676887cd4-f5xkj                         1/1       Running   0          13m

16:34:15 onap          onap-log-log-logstash-8676887cd4-fg8b9                         1/1       Running   0          13m

16:34:15 onap          onap-log-log-logstash-8676887cd4-xt289                         1/1       Running   0          13m

16:34:15 onap          onap-log-log-logstash-8676887cd4-zpt9t                         1/1       Running   0          13m

16:34:15 onap          onap-pomba-pomba-aaictxbuilder-8565f7d75c-6rfn5                2/2       Running   0          13m

16:34:15 onap          onap-pomba-pomba-contextaggregator-d9f888c4-7bkrs              1/1       Running   0          13m

16:34:15 onap          onap-pomba-pomba-data-router-66c5544446-526hc                  1/1       Running   0          13m

16:34:15 onap          onap-pomba-pomba-elasticsearch-5c4f8c6b5b-w5t49                1/1       Running   0          13m

16:34:15 onap          onap-pomba-pomba-kibana-849cb74588-2k2hv                       1/1       Running   0          13m

16:34:15 onap          onap-pomba-pomba-networkdiscovery-556bb9d7b8-59vnn             2/2       Running   0          13m

16:34:15 onap          onap-pomba-pomba-networkdiscoveryctxbuilder-6ff7d49464-s5wrv   2/2       Running   0          13m

16:34:15 onap          onap-pomba-pomba-sdcctxbuilder-7799ffccc9-qz5tq                1/1       Running   0          13m

16:34:15 onap          onap-pomba-pomba-search-data-9896cf798-7kfdt                   2/2       Running   0          13m

16:34:15 onap          onap-pomba-pomba-servicedecomposition-5f8bb9d6f6-wxzsw         2/2       Running   0          13m

16:34:15 onap          onap-pomba-pomba-validation-service-79b8d5cb9f-zstqr           1/1       Running   0          13m

16:34:15 onap          onap-robot-robot-64886c7774-lxkg7                              1/1       Running   0          13m

AAI - LOGN/A - expected


Library formatN/A because the component is not currently using a spec library (using their own format) - see Logging Dublin Scope

AAI - LOG

PASS


2018101720181017Logs reach ELK stack

CLAMP - LOGN/A - expected


Library formatN/A because the component is not currently using a spec library (using their own format) - see Logging Dublin Scope
CLAMP - LOGN/Aclamp2018103120181031Logs don't reach ELK stack

Cannot login to get test data - followed Control Loop Flows and Models for Casablanca

used

https://127.0.0.1:30258/designer/index.html

LOG-902 - Getting issue details... STATUS

APPC - LOGN/A - expected


Library formatN/A because the component is not currently using a spec library (using their own format) - see Logging Dublin Scope
APPC - LOGN/Aappc-020181031
Logs reach ELK stack

not able to get data from rest calls below


http://cd.onap.cloud:30230/restconf/config/network-topology:network-topology/topology/topology-netconf
{
    "timestamp": 1541079758835,
    "status": 404,
    "error": "Not Found",
    "message": "No message available",
    "path": "/restconf/config/network-topology:network-topology/topology/topology-netconf"
}

trying

https://onap.readthedocs.io/en/latest/submodules/appc.git/docs/APPC%20OAM%20API%20Guide/APPC%20OAM%20API%20Guide.html#oam-commands

DCAE - LOGN/A - expected

20181101Library formatN/A because the component is not currently using a spec library (using their own format) - see Logging Dublin Scope
DCAE - LOGPASS
2018103120181101Logs reach ELK stack

1 of n

DMAAP-LOGN/A




MSB - LOGN/A - expected

20181101Library formatN/A because the component is not currently using a spec library (using their own format) - see Logging Dublin Scope
MSB - LOGPASS
2018103120181101Logs reach ELK stack

Policy - LOGN/A - expected


Library formatN/A because the component is not currently using a spec library (using their own format) - see Logging Dublin Scope
Policy - LOGPASS

2018101720181017Logs reach ELK stack

POMBA - LOGN/A - expected


Library formatN/A because the component is not currently using a spec library (using their own format) - see Logging Dublin Scope
POMBA - LOG

PASS


2018101720181017Logs reach ELK stack

PORTAL - LOGN/A - expected


Library formatN/A because the component is not currently using a spec library (using their own format) - see Logging Dublin Scope
Portal - LOG

PASS



2018101720181017Logs reach ELK stack

SDC - LOGN/A - expected


Library formatN/A because the component is not currently using a spec library (using their own format) - see Logging Dublin Scope
SDC - LOG

PASS


2018101720181017Logs reach ELK stack

SDNC - LOGN/A - expected


Library formatN/A because the component is not currently using a spec library (using their own format) - see Logging Dublin Scope
SDNC - LOG

PASS



2018103120181101Logs reach ELK stack

SO - LOGN/A - expected


Library formatN/A because the component is not currently using a spec library (using their own format) - see Logging Dublin Scope
SO - LOG

N/A - expected

Filebeat sidecars missing from SO refactor - fix in progress Michael O'Brien


20181017
Logs reach ELK stack

SO-1109 - Getting issue details... STATUS

SO-1110 - Getting issue details... STATUS

VFC - LOGN/A - expected


Library formatN/A because the component is not currently using a spec library (using their own format) - see Logging Dublin Scope
VFC - LOGpending
20181017
Logs reach ELK stackno data - need a closed loop vCPE test - too much time right now to set this up
VID - LOGN/A - expected


Library formatN/A because the component is not currently using a spec library (using their own format) - see Logging Dublin Scope
VID - LOG

PASS



2018101720181017Logs reach ELK stack

1 Comment

  1. ref

    20181031
    ubuntu@a-osn-cd:~/oom/kubernetes$ kubectl get pods --all-namespaces | grep /2
    onap          onap-aaf-aaf-cm-7f7494fffd-fmwxp                              0/1       Init:0/2            0          7m
    onap          onap-aaf-aaf-fs-7c7b4966d7-h5p9m                              0/1       Init:0/2            0          7m
    onap          onap-aaf-aaf-gui-5b9cf459bf-9hlx2                             0/1       Init:0/2            0          7m
    onap          onap-aaf-aaf-hello-d5c4f8dd-zbfms                             0/1       Init:0/2            0          7m
    onap          onap-aaf-aaf-locate-58bdf576f-jlnsc                           0/1       Init:0/2            0          7m
    onap          onap-aaf-aaf-oauth-86c8d8687f-lfntl                           0/1       Init:0/2            0          7m
    onap          onap-aaf-aaf-service-66689b78f5-gj7tb                         0/1       Init:1/2            0          7m
    onap          onap-aaf-aaf-sms-vault-0                                      0/2       ContainerCreating   0          7m
    onap          onap-aai-aai-babel-5b44f78ff4-8mdpw                           0/2       ContainerCreating   0          6m
    onap          onap-aai-aai-champ-64bf5d7c59-f9kj9                           0/2       Init:0/1            0          6m
    onap          onap-aai-aai-data-router-b74db89d8-76s8v                      0/2       Init:0/1            0          6m
    onap          onap-aai-aai-gizmo-9fc98788d-wjkf5                            0/2       ContainerCreating   0          6m
    onap          onap-aai-aai-graphadmin-7c4c8f66bf-f6f9w                      0/2       Init:0/1            0          6m
    onap          onap-aai-aai-modelloader-7f56f4b6cf-lsclh                     0/2       ContainerCreating   0          6m
    onap          onap-aai-aai-resources-64c8b8c4c4-vmvl9                       0/2       Init:0/1            0          6m
    onap          onap-aai-aai-search-data-648975d74-5g8ks                      0/2       ContainerCreating   0          6m
    onap          onap-aai-aai-sparky-be-c8bfdcbcb-tcm9m                        0/2       Init:0/1            0          6m
    onap          onap-aai-aai-spike-77f64cfc5c-wms6r                           0/2       Init:0/1            0          6m
    onap          onap-aai-aai-traversal-64fff6d57f-hgwgd                       0/2       Init:0/1            0          6m
    onap          onap-appc-appc-0                                              0/2       Init:0/1            0          6m
    onap          onap-clamp-clamp-9dfd65f8f-rhdlc                              0/2       Init:0/1            0          6m
    onap          onap-dmaap-message-router-kafka-74f9bff668-h6flz              0/1       Init:0/2            0          6m
    onap          onap-esr-esr-server-6d7dd55794-rp9zz                          0/2       Pending             0          6m
    onap          onap-msb-msb-discovery-59d48c65b-k68cj                        0/2       Init:0/1            0          6m
    onap          onap-msb-msb-eag-76857bcc54-8zbpp                             0/2       Pending             0          6m
    onap          onap-msb-msb-iag-59697db845-mlpz8                             0/2       Pending             0          6m
    onap          onap-multicloud-multicloud-5765cd745-bb57j                    0/2       Pending             0          6m
    onap          onap-multicloud-multicloud-azure-6c99454479-4457s             0/2       Pending             0          6m
    onap          onap-multicloud-multicloud-ocata-79cc9dfb76-w6bx4             0/2       Pending             0          6m
    onap          onap-multicloud-multicloud-pike-547d878bb5-d9cjj              0/2       Pending             0          6m
    onap          onap-multicloud-multicloud-vio-67979f99bc-gvzxz               0/2       Pending             0          6m
    onap          onap-multicloud-multicloud-windriver-748868744d-69sjr         0/2       Pending             0          6m
    onap          onap-policy-pap-5b95866fff-6wxln                              0/2       Pending             0          5m
    onap          onap-policy-pdp-0                                             0/2       Init:0/1            0          5m
    onap          onap-pomba-pomba-aaictxbuilder-5d45d589-nnxck                 0/2       Pending             0          5m
    onap          onap-pomba-pomba-networkdiscovery-8487cc6667-2j6wq            0/2       Pending             0          5m
    onap          onap-pomba-pomba-networkdiscoveryctxbuilder-c9c5c8f98-2svzw   0/2       Pending             0          5m
    onap          onap-pomba-pomba-search-data-7548469497-9pjq4                 0/2       Pending             0          5m
    onap          onap-pomba-pomba-servicedecomposition-8985f9849-2cdl7         0/2       Pending             0          5m
    onap          onap-portal-portal-app-5cc7b45bd9-45c6c                       0/2       Init:0/1            0          5m
    onap          onap-portal-portal-db-config-dwrrt                            0/2       Pending             0          5m
    onap          onap-portal-portal-sdk-5bb84db56c-x4zcj                       0/2       Init:0/1            0          5m
    onap          onap-sdc-sdc-be-68d86d55cf-pxf66                              0/2       Pending             0          5m
    onap          onap-sdc-sdc-fe-67ccb88464-9qbwj                              0/2       Pending             0          5m
    onap          onap-sdc-sdc-onboarding-be-67b4bc94b-2gscq                    0/2       Pending             0          5m
    onap          onap-sdc-sdc-wfd-fe-87d9cd457-26crq                           0/2       Pending             0          5m
    onap          onap-sdnc-sdnc-0                                              0/2       Pending             0          5m
    onap          onap-sdnc-sdnc-db-0                                           0/2       Pending             0          5m
    onap          onap-vfc-vfc-catalog-85bd56b986-c6zld                         0/2       Pending             0          4m
    onap          onap-vfc-vfc-generic-vnfm-driver-dc4864b96-c9b6h              0/2       Pending             0          4m
    onap          onap-vfc-vfc-huawei-vnfm-driver-589c6c8467-swdqt              0/2       Pending             0          4m
    onap          onap-vfc-vfc-juju-vnfm-driver-7cb5dc655d-mkb7l                0/2       Pending             0          4m
    onap          onap-vfc-vfc-nokia-vnfm-driver-57c4f5d9cd-jh9mb               0/2       ContainerCreating   0          4m
    onap          onap-vfc-vfc-nslcm-675c748bb7-t6rfz                           0/2       Init:0/1            0          4m
    onap          onap-vfc-vfc-resmgr-599fb9c6f7-64djz                          0/2       Pending             0          4m
    onap          onap-vfc-vfc-vnflcm-75cc76d968-9dblx                          0/2       Pending             0          4m
    onap          onap-vfc-vfc-vnfmgr-5f866c5458-4h56p                          0/2       Init:0/1            0          4m
    onap          onap-vfc-vfc-vnfres-579459469f-f2kns                          0/2       Pending             0          4m
    onap          onap-vfc-vfc-zte-vnfm-driver-5dd4b87cd7-ppj42                 0/2       Pending             0          4m
    onap          onap-vid-vid-599d9c54f9-79vfl                                 0/2       Pending             0          4m
    
    
    
    
    ubuntu@a-osn-cd:~$ kubectl get pods --all-namespaces | grep -E '0/|1/2'
    onap          onap-aaf-aaf-sms-preload-8bx5m                                0/1       Completed          0          10h
    onap          onap-aaf-aaf-sshsm-distcenter-crpdz                           0/1       Completed          0          10h
    onap          onap-aaf-aaf-sshsm-testca-gxnqw                               0/1       Completed          0          9h
    onap          onap-aaf-aaf-sshsm-testca-tx4sc                               0/1       Init:Error         0          10h
    onap          onap-aai-aai-champ-64bf5d7c59-f9kj9                           1/2       Running            0          10h
    onap          onap-aai-aai-df9746c8-pg2t9                                   0/1       Init:0/1           5          10h
    onap          onap-aai-aai-graphadmin-create-db-schema-gvbkf                0/1       Init:Error         0          10h
    onap          onap-aai-aai-graphadmin-create-db-schema-rmrzr                0/1       Completed          0          9h
    onap          onap-aai-aai-sparky-be-c8bfdcbcb-tcm9m                        0/2       Init:0/1           1          10h
    onap          onap-aai-aai-traversal-64fff6d57f-hgwgd                       1/2       CrashLoopBackOff   77         10h
    onap          onap-aai-aai-traversal-update-query-data-9kmdm                0/1       Init:0/1           53         10h
    onap          onap-contrib-netbox-app-provisioning-4ptzx                    0/1       Init:Error         0          10h
    onap          onap-contrib-netbox-app-provisioning-h6lm7                    0/1       Completed          0          9h
    onap          onap-dmaap-dmaap-dr-node-f9565b67-r5flc                       0/1       Init:0/1           59         10h
    onap          onap-dmaap-dmaap-dr-prov-65c7d74996-6t8gd                     0/1       CrashLoopBackOff   117        10h
    onap          onap-oof-music-cassandra-job-config-ngxf7                     0/1       Init:Error         0          10h
    onap          onap-oof-music-cassandra-job-config-wp9s2                     0/1       Completed          0          9h
    onap          onap-oof-oof-has-healthcheck-j4dx9                            0/1       Completed          0          10h
    onap          onap-oof-oof-has-onboard-q468q                                0/1       Completed          0          10h
    onap          onap-portal-portal-db-config-dwrrt                            0/2       Init:Error         0          10h
    onap          onap-portal-portal-db-config-wwdk7                            0/2       Completed          0          9h
    onap          onap-sdc-sdc-be-config-backend-7qh6w                          0/1       Init:Error         0          10h
    onap          onap-sdc-sdc-be-config-backend-rqj6z                          0/1       Completed          0          9h
    onap          onap-sdc-sdc-be-config-backend-rww2q                          0/1       Init:Error         0          9h
    onap          onap-sdc-sdc-be-config-backend-vrj8z                          0/1       Init:Error         0          9h
    onap          onap-sdc-sdc-cs-config-cassandra-29s44                        0/1       Completed          0          9h
    onap          onap-sdc-sdc-cs-config-cassandra-8lhs8                        0/1       Error              0          9h
    onap          onap-sdc-sdc-cs-config-cassandra-tfm4p                        0/1       Init:Error         0          10h
    onap          onap-sdc-sdc-es-config-elasticsearch-pfghc                    0/1       Init:Error         0          10h
    onap          onap-sdc-sdc-es-config-elasticsearch-w4ks6                    0/1       Completed          0          9h
    onap          onap-sdc-sdc-onboarding-be-cassandra-init-8kbnc               0/1       Init:Error         0          10h
    onap          onap-sdc-sdc-onboarding-be-cassandra-init-ntzzs               0/1       Init:Error         0          9h
    onap          onap-sdc-sdc-onboarding-be-cassandra-init-wllwm               0/1       Completed          0          9h
    onap          onap-sdc-sdc-wfd-be-workflow-init-2f5zv                       0/1       Init:Error         0          9h
    onap          onap-sdc-sdc-wfd-be-workflow-init-45dms                       0/1       Completed          0          9h
    onap          onap-sdc-sdc-wfd-be-workflow-init-8rzh9                       0/1       Init:Error         0          10h
    onap          onap-vid-vid-galera-config-h6w4n                              0/1       Init:Error         0          10h
    onap          onap-vid-vid-galera-config-v4s4x                              0/1       Completed          0          9h
    onap          onap-vnfsdk-vnfsdk-init-postgres-ctbqf                        0/1       Init:Error         0          10h
    onap          onap-vnfsdk-vnfsdk-init-postgres-pcbf4                        0/1       Init:Error         0          9h
    onap          onap-vnfsdk-vnfsdk-init-postgres-qtzkq                        0/1       Completed          0          9h
    ubuntu@a-osn-cd:~$ kubectl get pods --all-namespaces | grep -E '1/1|2/2'
    kube-system   heapster-6cfb49f776-q299j                                     1/1       Running            0          10h
    kube-system   kubernetes-dashboard-6f4c8b9cd5-dndb6                         1/1       Running            0          10h
    kube-system   monitoring-grafana-76f5b489d5-jtdgv                           1/1       Running            0          10h
    kube-system   monitoring-influxdb-6fc88bd58d-jf7bs                          1/1       Running            0          10h
    kube-system   tiller-deploy-8b6c5d4fb-9sg9m                                 1/1       Running            0          10h
    onap          dep-config-binding-service-756ddc9575-s557d                   2/2       Running            0          9h
    onap          dep-dcae-datafile-collector-7c88976b58-mrz5n                  2/2       Running            0          9h
    onap          dep-dcae-hv-ves-collector-769cc87fb-dwb96                     2/2       Running            0          9h
    onap          dep-dcae-prh-54df9f7dd8-sq8pm                                 2/2       Running            0          9h
    onap          dep-dcae-snmptrap-collector-58f7dcf-9nrk9                     1/1       Running            0          9h
    onap          dep-dcae-tca-analytics-7cf8bb79bd-56bhk                       2/2       Running            0          9h
    onap          dep-dcae-ves-collector-69b77696f8-l4b7d                       2/2       Running            0          9h
    onap          dep-deployment-handler-8896d8896-pxhr9                        2/2       Running            0          9h
    onap          dep-holmes-engine-mgmt-84b49b7b68-lbbb4                       1/1       Running            44         8h
    onap          dep-holmes-rule-mgmt-6878db67cf-jdtpq                         1/1       Running            45         9h
    onap          dep-inventory-7d4dfcdffb-nhbp6                                1/1       Running            0          9h
    onap          dep-policy-handler-c66cf69bd-6r8b4                            2/2       Running            0          9h
    onap          dep-pstg-write-77c89cb8c4-h6xnh                               1/1       Running            0          9h
    onap          dep-service-change-handler-5d4c4c756f-f6mtq                   1/1       Running            0          9h
    onap          onap-aaf-aaf-cm-7f7494fffd-fmwxp                              1/1       Running            0          10h
    onap          onap-aaf-aaf-cs-5cdd4b4c54-8xzkv                              1/1       Running            0          10h
    onap          onap-aaf-aaf-fs-7c7b4966d7-h5p9m                              1/1       Running            0          10h
    onap          onap-aaf-aaf-gui-5b9cf459bf-9hlx2                             1/1       Running            0          10h
    onap          onap-aaf-aaf-hello-d5c4f8dd-zbfms                             1/1       Running            0          10h
    onap          onap-aaf-aaf-locate-58bdf576f-jlnsc                           1/1       Running            0          10h
    onap          onap-aaf-aaf-oauth-86c8d8687f-lfntl                           1/1       Running            0          10h
    onap          onap-aaf-aaf-service-66689b78f5-gj7tb                         1/1       Running            0          10h
    onap          onap-aaf-aaf-sms-5447ccf794-qq64l                             1/1       Running            0          10h
    onap          onap-aaf-aaf-sms-quorumclient-0                               1/1       Running            0          10h
    onap          onap-aaf-aaf-sms-quorumclient-1                               1/1       Running            0          10h
    onap          onap-aaf-aaf-sms-quorumclient-2                               1/1       Running            0          9h
    onap          onap-aaf-aaf-sms-vault-0                                      2/2       Running            1          10h
    onap          onap-aai-aai-babel-5b44f78ff4-8mdpw                           2/2       Running            0          10h
    onap          onap-aai-aai-cassandra-0                                      1/1       Running            7          10h
    onap          onap-aai-aai-data-router-b74db89d8-76s8v                      2/2       Running            0          10h
    onap          onap-aai-aai-elasticsearch-56f6f9d69c-8gt25                   1/1       Running            0          10h
    onap          onap-aai-aai-gizmo-9fc98788d-wjkf5                            2/2       Running            0          10h
    onap          onap-aai-aai-graphadmin-7c4c8f66bf-f6f9w                      2/2       Running            0          10h
    onap          onap-aai-aai-modelloader-7f56f4b6cf-lsclh                     2/2       Running            0          10h
    onap          onap-aai-aai-resources-64c8b8c4c4-vmvl9                       2/2       Running            0          10h
    onap          onap-aai-aai-search-data-648975d74-5g8ks                      2/2       Running            0          10h
    onap          onap-aai-aai-spike-77f64cfc5c-wms6r                           2/2       Running            0          10h
    onap          onap-appc-appc-0                                              2/2       Running            0          10h
    onap          onap-appc-appc-ansible-server-5c9bb8f65-nktpx                 1/1       Running            0          10h
    onap          onap-appc-appc-cdt-987bb87bf-6mplx                            1/1       Running            0          10h
    onap          onap-appc-appc-db-0                                           1/1       Running            0          10h
    onap          onap-appc-appc-db-1                                           1/1       Running            8          9h
    onap          onap-appc-appc-db-2                                           1/1       Running            0          9h
    onap          onap-appc-appc-dgbuilder-6877786fb5-4kvvt                     1/1       Running            0          10h
    onap          onap-clamp-clamp-9dfd65f8f-rhdlc                              2/2       Running            0          10h
    onap          onap-clamp-clamp-dash-es-7b5f7987cb-cbhbl                     1/1       Running            0          10h
    onap          onap-clamp-clamp-dash-kibana-6657c47b4f-fzbr7                 1/1       Running            0          10h
    onap          onap-clamp-clamp-dash-logstash-65d75f898c-t6h5s               1/1       Running            0          10h
    onap          onap-clamp-clampdb-8455fbc86c-hgchw                           1/1       Running            1          10h
    onap          onap-cli-cli-77684df54d-fnrt8                                 1/1       Running            0          10h
    onap          onap-consul-consul-58b4cd6fc7-5m8sh                           1/1       Running            0          10h
    onap          onap-consul-consul-server-0                                   1/1       Running            0          10h
    onap          onap-consul-consul-server-1                                   1/1       Running            0          10h
    onap          onap-consul-consul-server-2                                   1/1       Running            0          9h
    onap          onap-contrib-netbox-app-5c674bfbfc-4vjmm                      1/1       Running            0          10h
    onap          onap-contrib-netbox-nginx-6c8b494d88-k45ls                    1/1       Running            0          10h
    onap          onap-contrib-netbox-postgres-b76d44ff8-8gk9n                  1/1       Running            0          10h
    onap          onap-dcaegen2-dcae-bootstrap-6b54548796-x6qhn                 1/1       Running            0          10h
    onap          onap-dcaegen2-dcae-cloudify-manager-7d8bffc78c-6h2xr          1/1       Running            0          10h
    onap          onap-dcaegen2-dcae-db-0                                       1/1       Running            0          10h
    onap          onap-dcaegen2-dcae-db-1                                       1/1       Running            0          9h
    onap          onap-dcaegen2-dcae-healthcheck-74c5f8f984-7m4cg               1/1       Running            0          10h
    onap          onap-dcaegen2-dcae-pgpool-6bf57b654f-28tv9                    1/1       Running            0          10h
    onap          onap-dcaegen2-dcae-pgpool-6bf57b654f-t5bm2                    1/1       Running            0          10h
    onap          onap-dcaegen2-dcae-redis-0                                    1/1       Running            0          10h
    onap          onap-dcaegen2-dcae-redis-1                                    1/1       Running            0          9h
    onap          onap-dcaegen2-dcae-redis-2                                    1/1       Running            0          9h
    onap          onap-dcaegen2-dcae-redis-3                                    1/1       Running            0          9h
    onap          onap-dcaegen2-dcae-redis-4                                    1/1       Running            0          9h
    onap          onap-dcaegen2-dcae-redis-5                                    1/1       Running            0          9h
    onap          onap-dmaap-dbc-pg-0                                           1/1       Running            0          10h
    onap          onap-dmaap-dbc-pg-1                                           1/1       Running            0          9h
    onap          onap-dmaap-dbc-pgpool-65966bf95f-95f8k                        1/1       Running            0          10h
    onap          onap-dmaap-dbc-pgpool-65966bf95f-tjtml                        1/1       Running            0          10h
    onap          onap-dmaap-dmaap-bus-controller-fc45ff664-w2klc               1/1       Running            0          10h
    onap          onap-dmaap-dmaap-dr-db-7f6d8f56ff-zrwct                       1/1       Running            1          10h
    onap          onap-dmaap-message-router-6b886c44bd-qsbn5                    1/1       Running            0          10h
    onap          onap-dmaap-message-router-kafka-74f9bff668-h6flz              1/1       Running            0          10h
    onap          onap-dmaap-message-router-zookeeper-79f5556fb8-td7n4          1/1       Running            0          10h
    onap          onap-esr-esr-gui-5857c86679-xrmk9                             1/1       Running            0          10h
    onap          onap-esr-esr-server-6d7dd55794-rp9zz                          2/2       Running            0          10h
    onap          onap-log-log-elasticsearch-7f87cb44db-sxgs4                   1/1       Running            0          10h
    onap          onap-log-log-kibana-7d94fc69d9-pn9sr                          1/1       Running            0          10h
    onap          onap-log-log-logstash-5pdbx                                   1/1       Running            0          10h
    onap          onap-msb-kube2msb-5b66b44fb6-2lcnc                            1/1       Running            0          10h
    onap          onap-msb-msb-consul-699c9cb779-gwlqn                          1/1       Running            0          10h
    onap          onap-msb-msb-discovery-59d48c65b-k68cj                        2/2       Running            0          10h
    onap          onap-msb-msb-eag-76857bcc54-8zbpp                             2/2       Running            0          10h
    onap          onap-msb-msb-iag-59697db845-mlpz8                             2/2       Running            0          10h
    onap          onap-multicloud-multicloud-5765cd745-bb57j                    2/2       Running            0          10h
    onap          onap-multicloud-multicloud-azure-6c99454479-4457s             2/2       Running            0          10h
    onap          onap-multicloud-multicloud-ocata-79cc9dfb76-w6bx4             2/2       Running            0          10h
    onap          onap-multicloud-multicloud-pike-547d878bb5-d9cjj              2/2       Running            0          10h
    onap          onap-multicloud-multicloud-vio-67979f99bc-gvzxz               2/2       Running            0          10h
    onap          onap-multicloud-multicloud-windriver-748868744d-69sjr         2/2       Running            0          10h
    onap          onap-nbi-nbi-5486c89944-gqgqt                                 1/1       Running            9          10h
    onap          onap-nbi-nbi-mariadb-cfc89888d-hx8n2                          1/1       Running            0          10h
    onap          onap-nbi-nbi-mongo-0                                          1/1       Running            0          10h
    onap          onap-oof-music-cassandra-0                                    1/1       Running            7          10h
    onap          onap-oof-music-cassandra-1                                    1/1       Running            0          9h
    onap          onap-oof-music-cassandra-2                                    1/1       Running            0          9h
    onap          onap-oof-music-tomcat-7c6f8967d-8n8t7                         1/1       Running            0          10h
    onap          onap-oof-music-tomcat-7c6f8967d-mdhcl                         1/1       Running            0          10h
    onap          onap-oof-music-tomcat-7c6f8967d-xbb8d                         1/1       Running            0          10h
    onap          onap-oof-oof-55cbc5b854-crmdd                                 1/1       Running            0          10h
    onap          onap-oof-oof-has-api-577b8496ff-nmkkz                         1/1       Running            0          10h
    onap          onap-oof-oof-has-controller-d5b6b5978-txdgg                   1/1       Running            0          10h
    onap          onap-oof-oof-has-data-5d6d7bc98f-nxq7r                        1/1       Running            0          10h
    onap          onap-oof-oof-has-reservation-546c8bd6d9-dcs58                 1/1       Running            0          10h
    onap          onap-oof-oof-has-solver-76d6bfcf76-bxjct                      1/1       Running            0          10h
    onap          onap-oof-zookeeper-0                                          1/1       Running            0          10h
    onap          onap-oof-zookeeper-1                                          1/1       Running            0          9h
    onap          onap-oof-zookeeper-2                                          1/1       Running            0          9h
    onap          onap-policy-brmsgw-5d5fc5575b-fgzdw                           1/1       Running            0          10h
    onap          onap-policy-drools-0                                          1/1       Running            0          10h
    onap          onap-policy-nexus-fd77f7c-vmwmn                               1/1       Running            0          10h
    onap          onap-policy-pap-5b95866fff-6wxln                              2/2       Running            0          10h
    onap          onap-policy-pdp-0                                             2/2       Running            0          10h
    onap          onap-policy-policy-apex-pdp-0                                 1/1       Running            0          10h
    onap          onap-policy-policy-distribution-d754f6b9d-rfbx2               1/1       Running            0          10h
    onap          onap-policy-policydb-5cdb75fc86-8l92w                         1/1       Running            0          10h
    onap          onap-pomba-pomba-aaictxbuilder-5d45d589-nnxck                 2/2       Running            0          10h
    onap          onap-pomba-pomba-contextaggregator-6d96d8bcbb-rq8hn           1/1       Running            0          10h
    onap          onap-pomba-pomba-data-router-d58466bb4-b2z9w                  1/1       Running            0          10h
    onap          onap-pomba-pomba-elasticsearch-5f579756f5-wldjh               1/1       Running            0          10h
    onap          onap-pomba-pomba-kibana-7ccb9f548f-rbsgl                      1/1       Running            0          10h
    onap          onap-pomba-pomba-networkdiscovery-8487cc6667-2j6wq            2/2       Running            0          10h
    onap          onap-pomba-pomba-networkdiscoveryctxbuilder-c9c5c8f98-2svzw   2/2       Running            0          10h
    onap          onap-pomba-pomba-sdcctxbuilder-79b495b9b7-dt5jw               1/1       Running            0          10h
    onap          onap-pomba-pomba-search-data-7548469497-9pjq4                 2/2       Running            0          10h
    onap          onap-pomba-pomba-servicedecomposition-8985f9849-2cdl7         2/2       Running            0          10h
    onap          onap-pomba-pomba-validation-service-558479f7c7-fltvn          1/1       Running            0          10h
    onap          onap-portal-portal-app-5cc7b45bd9-45c6c                       2/2       Running            0          10h
    onap          onap-portal-portal-cassandra-54c74d8b57-7gfl6                 1/1       Running            0          10h
    onap          onap-portal-portal-db-dbd5986fd-mlp66                         1/1       Running            0          10h
    onap          onap-portal-portal-sdk-5bb84db56c-x4zcj                       2/2       Running            0          10h
    onap          onap-portal-portal-widget-7fbb5f4bb7-48p8g                    1/1       Running            0          10h
    onap          onap-portal-portal-zookeeper-6d98ff566-mwckl                  1/1       Running            0          10h
    onap          onap-robot-robot-646d8546d4-g4cpw                             1/1       Running            0          10h
    onap          onap-sdc-sdc-be-68d86d55cf-pxf66                              2/2       Running            0          10h
    onap          onap-sdc-sdc-cs-68f65d56cc-wj4kq                              1/1       Running            0          10h
    onap          onap-sdc-sdc-es-5b7ddbf999-qv5ts                              1/1       Running            0          10h
    onap          onap-sdc-sdc-fe-67ccb88464-9qbwj                              2/2       Running            0          10h
    onap          onap-sdc-sdc-kb-6848b4f8db-66b99                              1/1       Running            0          10h
    onap          onap-sdc-sdc-onboarding-be-67b4bc94b-2gscq                    2/2       Running            0          10h
    onap          onap-sdc-sdc-wfd-be-865b945ff4-nmnd5                          1/1       Running            0          10h
    onap          onap-sdc-sdc-wfd-fe-87d9cd457-26crq                           2/2       Running            0          10h
    onap          onap-sdnc-controller-blueprints-c9c94b7f8-klrnz               1/1       Running            0          10h
    onap          onap-sdnc-controller-blueprints-db-0                          1/1       Running            0          10h
    onap          onap-sdnc-nengdb-0                                            1/1       Running            0          10h
    onap          onap-sdnc-network-name-gen-76684d7898-f2dlg                   1/1       Running            0          10h
    onap          onap-sdnc-sdnc-0                                              2/2       Running            0          10h
    onap          onap-sdnc-sdnc-ansible-server-6b45bfb765-xjjml                1/1       Running            0          10h
    onap          onap-sdnc-sdnc-db-0                                           2/2       Running            0          10h
    onap          onap-sdnc-sdnc-dgbuilder-74db46865c-rzvlz                     1/1       Running            0          10h
    onap          onap-sdnc-sdnc-dmaap-listener-6644995cd4-db7jn                1/1       Running            0          10h
    onap          onap-sdnc-sdnc-portal-76d9f6c887-5mzx2                        1/1       Running            0          10h
    onap          onap-sdnc-sdnc-ueb-listener-6b98f9c978-kqrn7                  1/1       Running            0          10h
    onap          onap-sniro-emulator-sniro-emulator-dc95bb557-54z87            1/1       Running            0          10h
    onap          onap-so-so-5f95f6b7b7-xjcfn                                   1/1       Running            0          10h
    onap          onap-so-so-bpmn-infra-5bdf868455-slbcz                        1/1       Running            0          10h
    onap          onap-so-so-catalog-db-adapter-7bfc746b86-dv9j4                1/1       Running            0          10h
    onap          onap-so-so-mariadb-7cfcd8d54f-n5qz2                           1/1       Running            0          10h
    onap          onap-so-so-monitoring-6bf4f6f989-nv7bw                        1/1       Running            0          10h
    onap          onap-so-so-openstack-adapter-774d56bd8b-dnjnh                 1/1       Running            3          10h
    onap          onap-so-so-request-db-adapter-5f5bd44f5b-4vh29                1/1       Running            0          10h
    onap          onap-so-so-sdc-controller-db4d7f648-pvv8x                     1/1       Running            0          10h
    onap          onap-so-so-sdnc-adapter-d489cf65c-d6dh7                       1/1       Running            0          10h
    onap          onap-so-so-vfc-adapter-b87475979-7c6d2                        1/1       Running            0          10h
    onap          onap-uui-uui-6f96f5994b-d7wwl                                 1/1       Running            0          10h
    onap          onap-uui-uui-server-7f97585b9-7pwrp                           1/1       Running            0          10h
    onap          onap-vfc-vfc-catalog-85bd56b986-c6zld                         2/2       Running            6          10h
    onap          onap-vfc-vfc-db-84cdd56b7b-cd8fk                              1/1       Running            0          10h
    onap          onap-vfc-vfc-ems-driver-57fb449dcf-2p4p7                      1/1       Running            10         10h
    onap          onap-vfc-vfc-generic-vnfm-driver-dc4864b96-c9b6h              2/2       Running            9          10h
    onap          onap-vfc-vfc-huawei-vnfm-driver-589c6c8467-swdqt              2/2       Running            0          10h
    onap          onap-vfc-vfc-juju-vnfm-driver-7cb5dc655d-mkb7l                2/2       Running            0          10h
    onap          onap-vfc-vfc-multivim-proxy-c5c6794cf-2zsjx                   1/1       Running            0          10h
    onap          onap-vfc-vfc-nokia-v2vnfm-driver-5ddc5cff47-7xfkq             1/1       Running            0          10h
    onap          onap-vfc-vfc-nokia-vnfm-driver-57c4f5d9cd-jh9mb               2/2       Running            0          10h
    onap          onap-vfc-vfc-nslcm-675c748bb7-t6rfz                           2/2       Running            6          10h
    onap          onap-vfc-vfc-resmgr-599fb9c6f7-64djz                          2/2       Running            0          10h
    onap          onap-vfc-vfc-vnflcm-75cc76d968-9dblx                          2/2       Running            5          10h
    onap          onap-vfc-vfc-vnfmgr-5f866c5458-4h56p                          2/2       Running            6          10h
    onap          onap-vfc-vfc-vnfres-579459469f-f2kns                          2/2       Running            6          10h
    onap          onap-vfc-vfc-workflow-99f46966d-hf866                         1/1       Running            0          10h
    onap          onap-vfc-vfc-workflow-engine-7db744c68f-b4dx5                 1/1       Running            0          10h
    onap          onap-vfc-vfc-zte-sdnc-driver-6d5f4f6cdd-9gjj4                 1/1       Running            0          10h
    onap          onap-vfc-vfc-zte-vnfm-driver-5dd4b87cd7-ppj42                 2/2       Running            10         10h
    onap          onap-vid-vid-599d9c54f9-79vfl                                 2/2       Running            0          10h
    onap          onap-vid-vid-mariadb-galera-0                                 1/1       Running            0          10h
    onap          onap-vnfsdk-vnfsdk-84c876c9b5-6lbg7                           1/1       Running            0          10h
    onap          onap-vnfsdk-vnfsdk-pgpool-7bcf75bb9b-9rr4z                    1/1       Running            0          10h
    onap          onap-vnfsdk-vnfsdk-pgpool-7bcf75bb9b-fv2dh                    1/1       Running            0          10h
    onap          onap-vnfsdk-vnfsdk-postgres-0                                 1/1       Running            0          10h
    onap          onap-vnfsdk-vnfsdk-postgres-1                                 1/1       Running            0          9h