20181025: update: originally we decided to continue the casablanca version v111 of ONAP Application Logging Specification v1.2 (Casablanca) in Dublin but after constructive feedback/pushback from one small part of the AT&T based Acumos community and feedback/pushback from part of the AT&T based ECOMP community based on initial pulls into ECOMP and some feedback from 2 high level projects in ONAP - I recommend refocusing on only one requirement - a common ONAP first spec. Creating a 3-way spec that satisfies Acumos, ECOMP and ONAP can no longer be our focus due to the limited time and very limited resources of the logging spec/ELK deployment work a couple of us need to do. I have received tremendous support and spec/JIRA work from Lorraine Welch, Spondon Dey, Shishir Thakore and Luke Parker (all the marker/mdc work and slf4j library - and originating the project spec/elk stack). Creating a combined specification that meets the needs of private AT&T ECOMP that consumes Open Source ONAP, public Open Source Acumos that consumes parts of ONAP and even ONAP itself with it's several Logging implementations across Portal, SO, AAI is not really working or will take too long to accept a new library that everyone can accept. Since I am the current PTL, I have decided that an "ONAP first" direction is required. Any implementation/spec changes will need to be prototyped in public first and accepted directly into affected projects as we change the spec - not as an exercise after we have finalized the spec. I recommend that the way forward as suggested by the core of my coworkers in Acumos is to take an existing library and slowly iterate that library to match an "ONAP" specific spec that includes markers and MDC's and only if possible keep the older formats by the ONAP consuming projects. I will start from the bottom up for Dublin - using the portal/sdk library and adjust it in sync with any spec changes in parallel (no spec changes without first prototyping them in existing use cases by users of the portal/sdk library). For the E release we can increase adoption to the rest of the components not currently using portal/sdk. What is critical this time around is working with the team(s) affected by any adjustment to their library first. Secondary consideration will be given to non-opensource consumers of this specification inside corporations that privately wrap ONAP. This will mean becoming part of the public portal and policy weekly meetings as a de-facto sub-project. The first order of work will be infrastructure (shipping logs to ELK) and then an alignment to the existing portal/sdk logback.xml spec - before we start changing things. so we can start mining the ELK logs before we adjust them. |
Statement of Work
Work to align the logs, ship them, consolidate, provide for analysis, backup and restore them will be done from a developer perspective for the Dublin release.
Requirements
R1: Provide for consistent logs across ONAP that allow for cross pod indexing
R2: Validate/Implement log shipping to the existing ELK stack
Analysis
Existing ONAP Logging State - Infrastructure and Schema
ref
20181031 ubuntu@a-osn-cd:~/oom/kubernetes$ kubectl get pods --all-namespaces | grep /2 onap onap-aaf-aaf-cm-7f7494fffd-fmwxp 0/1 Init:0/2 0 7m onap onap-aaf-aaf-fs-7c7b4966d7-h5p9m 0/1 Init:0/2 0 7m onap onap-aaf-aaf-gui-5b9cf459bf-9hlx2 0/1 Init:0/2 0 7m onap onap-aaf-aaf-hello-d5c4f8dd-zbfms 0/1 Init:0/2 0 7m onap onap-aaf-aaf-locate-58bdf576f-jlnsc 0/1 Init:0/2 0 7m onap onap-aaf-aaf-oauth-86c8d8687f-lfntl 0/1 Init:0/2 0 7m onap onap-aaf-aaf-service-66689b78f5-gj7tb 0/1 Init:1/2 0 7m onap onap-aaf-aaf-sms-vault-0 0/2 ContainerCreating 0 7m onap onap-aai-aai-babel-5b44f78ff4-8mdpw 0/2 ContainerCreating 0 6m onap onap-aai-aai-champ-64bf5d7c59-f9kj9 0/2 Init:0/1 0 6m onap onap-aai-aai-data-router-b74db89d8-76s8v 0/2 Init:0/1 0 6m onap onap-aai-aai-gizmo-9fc98788d-wjkf5 0/2 ContainerCreating 0 6m onap onap-aai-aai-graphadmin-7c4c8f66bf-f6f9w 0/2 Init:0/1 0 6m onap onap-aai-aai-modelloader-7f56f4b6cf-lsclh 0/2 ContainerCreating 0 6m onap onap-aai-aai-resources-64c8b8c4c4-vmvl9 0/2 Init:0/1 0 6m onap onap-aai-aai-search-data-648975d74-5g8ks 0/2 ContainerCreating 0 6m onap onap-aai-aai-sparky-be-c8bfdcbcb-tcm9m 0/2 Init:0/1 0 6m onap onap-aai-aai-spike-77f64cfc5c-wms6r 0/2 Init:0/1 0 6m onap onap-aai-aai-traversal-64fff6d57f-hgwgd 0/2 Init:0/1 0 6m onap onap-appc-appc-0 0/2 Init:0/1 0 6m onap onap-clamp-clamp-9dfd65f8f-rhdlc 0/2 Init:0/1 0 6m onap onap-dmaap-message-router-kafka-74f9bff668-h6flz 0/1 Init:0/2 0 6m onap onap-esr-esr-server-6d7dd55794-rp9zz 0/2 Pending 0 6m onap onap-msb-msb-discovery-59d48c65b-k68cj 0/2 Init:0/1 0 6m onap onap-msb-msb-eag-76857bcc54-8zbpp 0/2 Pending 0 6m onap onap-msb-msb-iag-59697db845-mlpz8 0/2 Pending 0 6m onap onap-multicloud-multicloud-5765cd745-bb57j 0/2 Pending 0 6m onap onap-multicloud-multicloud-azure-6c99454479-4457s 0/2 Pending 0 6m onap onap-multicloud-multicloud-ocata-79cc9dfb76-w6bx4 0/2 Pending 0 6m onap onap-multicloud-multicloud-pike-547d878bb5-d9cjj 0/2 Pending 0 6m onap onap-multicloud-multicloud-vio-67979f99bc-gvzxz 0/2 Pending 0 6m onap onap-multicloud-multicloud-windriver-748868744d-69sjr 0/2 Pending 0 6m onap onap-policy-pap-5b95866fff-6wxln 0/2 Pending 0 5m onap onap-policy-pdp-0 0/2 Init:0/1 0 5m onap onap-pomba-pomba-aaictxbuilder-5d45d589-nnxck 0/2 Pending 0 5m onap onap-pomba-pomba-networkdiscovery-8487cc6667-2j6wq 0/2 Pending 0 5m onap onap-pomba-pomba-networkdiscoveryctxbuilder-c9c5c8f98-2svzw 0/2 Pending 0 5m onap onap-pomba-pomba-search-data-7548469497-9pjq4 0/2 Pending 0 5m onap onap-pomba-pomba-servicedecomposition-8985f9849-2cdl7 0/2 Pending 0 5m onap onap-portal-portal-app-5cc7b45bd9-45c6c 0/2 Init:0/1 0 5m onap onap-portal-portal-db-config-dwrrt 0/2 Pending 0 5m onap onap-portal-portal-sdk-5bb84db56c-x4zcj 0/2 Init:0/1 0 5m onap onap-sdc-sdc-be-68d86d55cf-pxf66 0/2 Pending 0 5m onap onap-sdc-sdc-fe-67ccb88464-9qbwj 0/2 Pending 0 5m onap onap-sdc-sdc-onboarding-be-67b4bc94b-2gscq 0/2 Pending 0 5m onap onap-sdc-sdc-wfd-fe-87d9cd457-26crq 0/2 Pending 0 5m onap onap-sdnc-sdnc-0 0/2 Pending 0 5m onap onap-sdnc-sdnc-db-0 0/2 Pending 0 5m onap onap-vfc-vfc-catalog-85bd56b986-c6zld 0/2 Pending 0 4m onap onap-vfc-vfc-generic-vnfm-driver-dc4864b96-c9b6h 0/2 Pending 0 4m onap onap-vfc-vfc-huawei-vnfm-driver-589c6c8467-swdqt 0/2 Pending 0 4m onap onap-vfc-vfc-juju-vnfm-driver-7cb5dc655d-mkb7l 0/2 Pending 0 4m onap onap-vfc-vfc-nokia-vnfm-driver-57c4f5d9cd-jh9mb 0/2 ContainerCreating 0 4m onap onap-vfc-vfc-nslcm-675c748bb7-t6rfz 0/2 Init:0/1 0 4m onap onap-vfc-vfc-resmgr-599fb9c6f7-64djz 0/2 Pending 0 4m onap onap-vfc-vfc-vnflcm-75cc76d968-9dblx 0/2 Pending 0 4m onap onap-vfc-vfc-vnfmgr-5f866c5458-4h56p 0/2 Init:0/1 0 4m onap onap-vfc-vfc-vnfres-579459469f-f2kns 0/2 Pending 0 4m onap onap-vfc-vfc-zte-vnfm-driver-5dd4b87cd7-ppj42 0/2 Pending 0 4m onap onap-vid-vid-599d9c54f9-79vfl 0/2 Pending 0 4m