Skip to end of metadata
Go to start of metadata

see:  LOG-66 - Getting issue details... STATUS    LOG-49 - Getting issue details... STATUS

Quickstart - getting your ELK Dashboard up

Logging Analytics Dashboards (Kibana)

Enabling the ELK stack

Target the VM in your cluster that is running log-kibana - in the live CD system for example it is usually on http://master3.onap.info:30253/

The index needs to be enabled - this is currently being automated in  LOG-152 - Getting issue details... STATUS  for now you can flip the following dropdown.

You will be able to search on logs - separate from also using the kibana dashoards.


Troubleshooting:

# check indices to verify data exists in Elasticsearch
root@k8s:~# curl -u elastic:changeme http://127.0.0.1:30254/_cat/indices
red open .monitoring-es-6-2017.10.04 X9nA9PHsR92u9VA1EjvrjA 1 1
yellow open .monitoring-es-6-2017.10.05 GEWfbYV8Qu632ILcOaGufg 1 1 70825 324 49mb 49mb
yellow open onaplogs-2017.10.05 qOErjm_zR1yJbES2A94GuA 5 1 417450 0 146.9mb 146.9mb
yellow open .triggered_watches sJWEmGgYRmuOsOa0ouWgaw 1 1 0 0 365.7kb 365.7kb
yellow open onaplogs-2017.10.04 NoEhHdK3ToeUC2KGqo3XlA 5 1 270744 0 103.1mb 103.1mb
yellow open .monitoring-alerts-6 84pDDo6HQOGIqgJQ5nMriA 1 1 1 0 6.3kb 6.3kb
yellow open .watcher-history-3-2017.10.04 lZ_1PBP-RiuevffPV3a1-g 1 1 2612 0 2.6mb 2.6mb
yellow open .watcher-history-3-2017.10.05 IDLSTtl9Thq0pTuTyzngNQ 1 1 4544 0 3.2mb 3.2mb
yellow open .kibana WWyQNR5HTzCRsqEQkPR3YA 1 1 1 0 3.2kb 3.2kb
yellow open .watches vS6TCNdiTwSL9JJOx_N8QQ 1 1 4 0 63.4kb 63.4kb


Configuration
search on log* and select a "time filter field name" to @timestamp


onap | split rows | aggregation=terms | field=source.keyword | size=100, play


http://k8s:30253/app/kibana#/discover?_g=()&_a=(columns:!(_source),index:'onap*',interval:auto,query:(query_string:(analyze_wildcard:!t,query:'*')),sort:!('@timestamp',desc))

# filter by aai
October 5th 2017, 11:10:39.087 source:/var/log/onap/aai/aai-ml/error.log offset:605,688 DistributionClientResultImpl .responseStatus:ASDC_SERVER_PROBLEM, responseMessage=ASDC server problem] 2017-10-05T15:10:37.393Z input_type:log message: MDLSVC2001E|MDLSVC2001E Unable to register with ASDC: Failed to initialize distribution client: ASDC server problem| type:log Logger:org.onap.aai.modelloader.service.SdcConnectionJob Timestamp:October 5th 2017, 11:10:37.347 Thread:[Timer-0] INFO org.openecomp.sdc.impl.DistributionClientImpl DistributionClient - init 2017-10-05T15:10:37.393Z [Timer-0] ERROR org.openecomp.sdc.http.AsdcConnectorClient status from ASDC is org.openecomp.sdc.http.HttpAsdcResponse@2c523e67 2017-10-05T15:10:37.393Z [Timer-0] ERROR org.open

ONAP Kibana Dashboard

Logging Design

Repositories

RepoDirectoryDetails
oom
Deployment yamls
oom
configs
logging-analytics
docker image configs
logging-analytics
reference logback.xml configs
(various) aai....
runtime logback.xml/log4j configs


Logging API

Discuss shared library approach to logging (Java only for now)

Logging DevOps

ONAP Development#KubernetesDevOps

Logging Framework Deployment

Triage Log Capture

The following are procedures to determine the state of logs traversing from the container under use via the filebeat, logstash, elasticsearch pipeline.

I would start with the deployment.yaml and verify the filebeat section is the same as other working pods that are listed in the last check I did for pairwise testing.  I would expect logs from for example portal in portal-app and portal-sdk – at least error logs.

LOG-230 - Getting issue details... STATUS

After this I would check the logstash service to see if it is receiving logs its port – I’ll add instructions to the wiki.

We can also check the docker pv for the 2 portal containers – the emptydir should be in /var/lib/docker – this is a secondary place we can pickup/verify the logs besides filebeat.

Originating Container

Filebeat Sidecar Container

Filebeat sidecar container setup and configuration in OOM

Logstash DaemonSet

Elasticsearch container

Kibana container

Enabling Debug Logs

Via URL


Via Logback.xml


Logging Access

Robot logs


after a

/dockerdata-nfs/onap/robot# ./demo-k8s.sh init_customer

http://host:30209/logs/demo/InitCustomer/report.html

user:robot pass:robot

Host VM logs

/dockerdata-nfs/onap/aai/aai-traversal/logs/

/dockerdata-nfs/onap/sdc/logs/ASDC/ASDC-BE/

ubuntu@ip-172-31-15-18:/dockerdata-nfs/onap$ ls -la aai/data-router/logs/AAI-DR/
-rw-r--r-- 1 root root       0 Aug 16 01:11 audit.log
-rw-r--r-- 1 root root       0 Aug 16 01:11 debug.log
-rw-r--r-- 1 root root  106459 Aug 17 00:01 error.2018-08-16.log.zip
-rw-r--r-- 1 root root 5562166 Aug 17 19:50 error.log
-rw-r--r-- 1 root root     536 Aug 16 01:12 metrics.log


ELK Logs




Training Videos

VideoDetails
Configuring a new ONAP install's ELK stack


References



  • No labels

1 Comment

  1. TODO: LOOK AT

    public class PreAaiAjscInterceptor implements AjscInterceptor {
    	public boolean allowOrReject(HttpServletRequest req, HttpServletResponse resp, Map<?, ?> paramMap)
    		LoggingContext.requestId(req.getHeader("X-TransactionId"));
    		LoggingContext.partnerName(req.getHeader("X-FromAppId"));
    
    public class AAIAppServletContextListener implements ServletContextListener {
    	private static final EELFLogger LOGGER = EELFManager.getInstance().getLogger(AAIAppServletContextListener.class.getName());	public void contextInitialized(ServletContextEvent arg0) {
    		System.setProperty("org.openecomp.aai.serverStarted", "false");
    	LOGGER.info("***AAI Server initialization started...");