Notes
- Short meeting - 14 min - 8 attendees - we agreed to meet on thu 17th to finalize the logging spec.
- JIRA board - https://jira.onap.org/secure/RapidBoard.jspa?rapidView=53&view=planning&selectedIssue=LOG-68&epics=visible
- review Karen's log fix in AAI https://gerrit.onap.org/r/#/c/43321/2
- Logging spec finalization
- Good meeting last thu - final meeting this thu - for 1100EDT - zoom.us/j/7939937123
- https://lists.onap.org/pipermail/onap-discuss/2018-April/009080.html
- LOG healthcheck Thanks Neal Chatterley OOM-914 - Getting issue details... STATUS - to add to http://jenkins.onap.info/job/oom-cd-master/
- LOG fix sonar error after oparent 1.1.1 upgrade - Michael O'Brien in progress
- LOG java test coverage - Michael O'Brien - in progress
- LOG python test coverage - anyone
- Showcase:
- ONAP Culprit Locator Thanks Ray Chang, Neal Chatterley and J. Ram Balasubramanian and team
- SDC Onboarding Failure/Success Monitoring Thanks Ray Chang, Neal Chatterley and team
- RC0 release planning
- Meetings daily with integration team at 1100EDT - join - https://zoom.us/j/4466668888
- Today at 11 on the docker slowdown and proxy workaround for 1100EDT - LF meet on https://zoom.us/j/371947800 - send any nexus3 issue mails to onap-discuss@lists.onap.org and helpdesk@onap.org
- reference links in OOM-942 - Getting issue details... STATUS
Kibana pod is down - OOM-957 - Getting issue details... STATUS
Periodically one of the higher containers in a dependency tree will not get restarted in time to pick up running child containers - usually this is the kibana container
Fix this or "any" container by deleting the container in question and kubernetes will bring another one up.
root@a-onap-auto-20180412-ref:~# kubectl get services --all-namespaces | grep log onap dev-vfc-catalog ClusterIP 10.43.210.8 <none> 8806/TCP 5d onap log-es NodePort 10.43.77.87 <none> 9200:30254/TCP 5d onap log-es-tcp ClusterIP 10.43.159.93 <none> 9300/TCP 5d onap log-kibana NodePort 10.43.41.102 <none> 5601:30253/TCP 5d onap log-ls NodePort 10.43.180.165 <none> 5044:30255/TCP 5d onap log-ls-http ClusterIP 10.43.13.180 <none> 9600/TCP 5d root@a-onap-auto-20180412-ref:~# kubectl get pods --all-namespaces | grep log onap dev-log-elasticsearch-66cdc4f855-wmpkz 1/1 Running 0 5d onap dev-log-kibana-5b6f86bcb4-drpzq 0/1 Running 1076 5d onap dev-log-logstash-6d9fdccdb6-ngq2f 1/1 Running 0 5d root@a-onap-auto-20180412-ref:~# kubectl delete pod dev-log-kibana-5b6f86bcb4-drpzq -n onap pod "dev-log-kibana-5b6f86bcb4-drpzq" deleted root@a-onap-auto-20180412-ref:~# kubectl get pods --all-namespaces | grep log onap dev-log-elasticsearch-66cdc4f855-wmpkz 1/1 Running 0 5d onap dev-log-kibana-5b6f86bcb4-drpzq 0/1 Terminating 1076 5d onap dev-log-kibana-5b6f86bcb4-gpn2m 0/1 Pending 0 12s onap dev-log-logstash-6d9fdccdb6-ngq2f 1/1 Running 0 5d
1 Comment
Michael O'Brien
The log healthcheck is not fully complete unless it also verifies functionality - not just that the ELK containers are up.
LOG-223 - Getting issue details... STATUS
I recommend we put a filebeat container alongside robot so that when healtchecks are run - logs will show up in the ELK stack that we can verify - will log a JIRA shortly