...
Discuss shared library approach to logging (Java only for now)
Logging DevOps
ONAP Development#KubernetesDevOps
Logging Framework Deployment
Triage Log Capture
The ELK stack containers are under the onap-log namespace in the OOM deployment of ONAP. They should be started by default - if not you can start them manually.
Kibana is on port 30253, Elasticsearch is on port 30254
| Code Block |
|---|
# look for log containers
root@ip-172-31-82-46:~# kubectl get pods --all-namespaces -a | grep onap-log
onap-log elasticsearch-2934588800-st9j3 1/1 Running 0 3h
onap-log kibana-3372627750-ff8rv 1/1 Running 0 3h
onap-log logstash-1708188010-703pk 1/1 Running 0 3h
# start the ELK stack if required
root@ip-172-31-82-46:~/oom/kubernetes/oneclick# ./createAll.bash -n onap -a log
# check access ports (30254 and 30253)
root@ip-172-31-82-46:~/oom/kubernetes/oneclick# kubectl get services --all-namespaces -a | grep onap-log
onap-log elasticsearch 10.43.86.120 <nodes> 9200:30254/TCP
onap-log kibana 10.43.165.215 <nodes> 5601:30253/TCP
onap-log logstash 10.43.72.107 <none> 5044/TCP
# check for pods with filebeat containers (will be 2 per pod)
root@kos1001:/dockerdata-nfs/onap/robot# kubectl get pods --all-namespaces -a | grep 2/2
onap-aai aai-resources-338473047-8k6vr 2/2 Running 0 7h
onap-aai aai-traversal-2033243133-6cr9v 2/2 Running 0 7h
onap-aai model-loader-service-3356570452-25fjp 2/2 Running 0 7h
onap-aai search-data-service-2366687049-jt0nb 2/2 Running 0 7h
onap-aai sparky-be-3141964573-f2mhr 2/2 Running 0 7h
onap-appc appc-1335254431-v1pcs 2/2 Running 0 7h
onap-mso mso-3911927766-bmww7 2/2 Running 0 7h
onap-policy drools-2302173499-t0zmt 2/2 Running 0 7h
onap-policy pap-1954142582-vsrld 2/2 Running 0 7h
onap-policy pdp-4137191120-qgqnj 2/2 Running 0 7h
onap-portal portalapps-4168271938-4kp32 2/2 Running 0 7h
onap-portal portaldb-2821262885-0t32z 2/2 Running 0 7h
onap-sdc sdc-be-2986438255-sdqj6 2/2 Running 0 7h
onap-sdc sdc-fe-1573125197-7j3gp 2/2 Running 0 7h
onap-sdnc sdnc-3858151307-w9h7j 2/2 Running 0 7h
onap-vid vid-server-1837290631-x4ttc 2/2 Running 0 7h
|
Triage Log Capture
following are procedures to determine the state of logs traversing from the container under use via the filebeat, logstash, elasticsearch pipeline.
I would start with the deployment.yaml and verify the filebeat section is the same as other working pods that are listed in the last check I did for pairwise testing. I would expect logs from for example portal in portal-app and portal-sdk – at least error logs.
| Jira | ||||||
|---|---|---|---|---|---|---|
|
After this I would check the logstash service to see if it is receiving logs its port – I’ll add instructions to the wiki.
We can also check the docker pv for the 2 portal containers – the emptydir should be in /var/lib/docker – this is a secondary place we can pickup/verify the logs besides filebeatThe following are procedures to determine the state of logs traversing from the container under use via the filebeat, logstash, elasticsearch pipeline.
Originating Container
Filebeat Sidecar Container
Filebeat sidecar container setup and configuration in OOM
Logstash DaemonSet
Elasticsearch container
...
/dockerdata-nfs/onap/sdc/logs/ASDC/ASDC-BE/
| Code Block | ||
|---|---|---|
| ||
ubuntu@ip-172-31-15-18:/dockerdata-nfs/onap$ ls -la aai/data-router/logs/AAI-DR/ -rw-r--r-- 1 root root 0 Aug 16 01:11 audit.log -rw-r--r-- 1 root root 0 Aug 16 01:11 debug.log -rw-r--r-- 1 root root 106459 Aug 17 00:01 error.2018-08-16.log.zip -rw-r--r-- 1 root root 5562166 Aug 17 19:50 error.log -rw-r--r-- 1 root root 536 Aug 16 01:12 metrics.log |
ELK Logs
Training Videos
| Video | Details | ||||||
|---|---|---|---|---|---|---|---|
| Configuring a new ONAP install's ELK stack |
...