...
see:
Jira | ||||||
---|---|---|---|---|---|---|
|
Jira | ||||||
---|---|---|---|---|---|---|
|
Logging Framework Deployment
The ELK stack containers are under the onap-log namespace in the OOM deployment of ONAP. They should be started by default - if not you can start them manually.
...
Quickstart - getting your ELK Dashboard up
Logging Analytics Dashboards (Kibana)
Enabling the ELK stack
Target the VM in your cluster that is running log-kibana - in the live CD system for example it is usually on http://master3.onap.info:30253/
The index needs to be enabled - this is currently being automated in
Jira | ||||||
---|---|---|---|---|---|---|
|
You will be able to search on logs - separate from also using the kibana dashoards.
Troubleshooting:
# check indices to verify data exists in Elasticsearch
root@k8s:~# curl -u elastic:changeme http://127.0.0.1:30254/_cat/indices
red open .monitoring-es-6-2017.10.04 X9nA9PHsR92u9VA1EjvrjA 1 1
yellow open .monitoring-es-6-2017.10.05 GEWfbYV8Qu632ILcOaGufg 1 1 70825 324 49mb 49mb
yellow open onaplogs-2017.10.05 qOErjm_zR1yJbES2A94GuA 5 1 417450 0 146.9mb 146.9mb
yellow open .triggered_watches sJWEmGgYRmuOsOa0ouWgaw 1 1 0 0 365.7kb 365.7kb
yellow open onaplogs-2017.10.04 NoEhHdK3ToeUC2KGqo3XlA 5 1 270744 0 103.1mb 103.1mb
yellow open .monitoring-alerts-6 84pDDo6HQOGIqgJQ5nMriA 1 1 1 0 6.3kb 6.3kb
yellow open .watcher-history-3-2017.10.04 lZ_1PBP-RiuevffPV3a1-g 1 1 2612 0 2.6mb 2.6mb
yellow open .watcher-history-3-2017.10.05 IDLSTtl9Thq0pTuTyzngNQ 1 1 4544 0 3.2mb 3.2mb
yellow open .kibana WWyQNR5HTzCRsqEQkPR3YA 1 1 1 0 3.2kb 3.2kb
yellow open .watches vS6TCNdiTwSL9JJOx_N8QQ 1 1 4 0 63.4kb 63.4kb
Configuration
search on log* and select a "time filter field name" to @timestamp
onap | split rows | aggregation=terms | field=source.keyword | size=100, play
# filter by aai
October 5th 2017, 11:10:39.087 source:/var/log/onap/aai/aai-ml/error.log offset:605,688 DistributionClientResultImpl .responseStatus:ASDC_SERVER_PROBLEM, responseMessage=ASDC server problem] 2017-10-05T15:10:37.393Z input_type:log message: MDLSVC2001E|MDLSVC2001E Unable to register with ASDC: Failed to initialize distribution client: ASDC server problem| type:log Logger:org.onap.aai.modelloader.service.SdcConnectionJob Timestamp:October 5th 2017, 11:10:37.347 Thread:[Timer-0] INFO org.openecomp.sdc.impl.DistributionClientImpl DistributionClient - init 2017-10-05T15:10:37.393Z [Timer-0] ERROR org.openecomp.sdc.http.AsdcConnectorClient status from ASDC is org.openecomp.sdc.http.HttpAsdcResponse@2c523e67 2017-10-05T15:10:37.393Z [Timer-0] ERROR org.open
ONAP Kibana Dashboard
Logging Design
Repositories
Repo | Directory | Details |
---|---|---|
oom | Deployment yamls | |
oom | configs | |
logging-analytics | docker image configs | |
logging-analytics | reference logback.xml configs | |
(various) aai.... | runtime logback.xml/log4j configs |
Logging API
Discuss shared library approach to logging (Java only for now)
Logging DevOps
ONAP Development#KubernetesDevOps
Logging Framework Deployment
Triage Log Capture
The following are procedures to determine the state of logs traversing from the container under use via the filebeat, logstash, elasticsearch pipeline.
I would start with the deployment.yaml and verify the filebeat section is the same as other working pods that are listed in the last check I did for pairwise testing. I would expect logs from for example portal in portal-app and portal-sdk – at least error logs.
Jira | ||||||
---|---|---|---|---|---|---|
|
After this I would check the logstash service to see if it is receiving logs its port – I’ll add instructions to the wiki.
We can also check the docker pv for the 2 portal containers – the emptydir should be in /var/lib/docker – this is a secondary place we can pickup/verify the logs besides filebeat.
Originating Container
Filebeat Sidecar Container
Filebeat sidecar container setup and configuration in OOM
Logstash DaemonSet
Elasticsearch container
Kibana container
Enabling Debug Logs
Via URL
MSO | Debuglog successfully enabled. |
Via Logback.xml
Logging Access
Robot logs
after a
...
/dockerdata-nfs/onap/sdc/logs/ASDC/ASDC-BE/
ELK Logs
Configuring Logging Analytics
Login to the kibana dashboard on the ONAP host machine (http://dev.onap.org:30253)
$elasticsearch_username: "elastic"
$elasticsearch_password: "changeme"
...
Code Block | ||
---|---|---|
| ||
ubuntu@ip-172-31-15-18:/dockerdata-nfs/onap$ ls -la aai/data-router/logs/AAI-DR/
-rw-r--r-- 1 root root 0 Aug 16 01:11 audit.log
-rw-r--r-- 1 root root 0 Aug 16 01:11 debug.log
-rw-r--r-- 1 root root 106459 Aug 17 00:01 error.2018-08-16.log.zip
-rw-r--r-- 1 root root 5562166 Aug 17 19:50 error.log
-rw-r--r-- 1 root root 536 Aug 16 01:12 metrics.log |
ELK Logs
Training Videos
Video | Details | ||||||
---|---|---|---|---|---|---|---|
| Configuring a new ONAP install's ELK stack |