WIP
This page is the design and reference site for the Logging RI. The following topics are being provided to the rest of the ONAP community.
Questions on onap-discuss from the teams
https://lists.onap.org/pipermail/onap-discuss/2018-January/007415.html
Latest Filebeat/ELK implementation by Clamp for reference - https://gerrit.onap.org/r/#/c/48961/ on
The goals of the logging project are the following
provide for standardized logs across all ONAP components
provide ELK framework for log capture, indexing and presentation/search
provide framework for requestID tracking of distributed transactions (correlation ID's in p.162 of Building Microservices 2015 by Sam Newman) via HTTP ETags
Project | repo | Dependencies | Artifact | Docker |
---|---|---|---|---|
logging-reference | logging-reference | pom | ||
logging-demo | logging-demo | WAR | ||
logging-mock-service | logging-mock-service | logging-library | WAR | |
Logging Library | logging-library | JAR | ||
Kubernetes demo spec | kubernetes/log-mock-demo | yaml, config | ||
Kubernetes service spec | kubernetes/log-mock-service | yaml, config | ||
logging-docker-root | pom | |||
logging-docker-demo | logging-demo | DockerFile | ||
logging-docker-service | logging-mock-service | DockerFile |
WIP (as of 20180121) - will notify when complete
Goto your IDE workspace
Clone the logging repo
https://gerrit.onap.org/r/#/admin/projects/logging-analytics
https://git.onap.org/logging-analytics/tree/
goto the reference folder and load the pom.xml project
obrienbiometrics:wse_sts392b michaelobrien$ git clone ssh://michaelobrien@gerrit.onap.org:29418/logging-analytics Cloning into 'logging-analytics'... Warning: Permanently added '[gerrit.onap.org]:29418,[198.145.29.92]:29418' (RSA) to the list of known hosts. remote: Total 374 (delta 0), reused 374 (delta 0) Receiving objects: 100% (374/374), 1001.91 KiB | 1.22 MiB/s, done. Resolving deltas: 100% (120/120), done. obrienbiometrics:wse_sts392b michaelobrien$ cd logging-analytics/reference/ obrienbiometrics:reference michaelobrien$ ls\ logging-demo logging-library logging-mock-service pom.xml # build the project to verify obrienbiometrics:reference michaelobrien$ mvn clean install -U -DskipTests=true -Dmaven.test.skip=true -Dmaven.javadoc.skip=true -Dadditionalparam=-Xdoclint:none [INFO] logging-reference .................................. SUCCESS [ 3.069 s] [INFO] logging-library .................................... SUCCESS [ 1.313 s] [INFO] logging-mock-service ............................... SUCCESS [ 0.395 s] [INFO] logging-demo ....................................... SUCCESS [ 2.329 s] [INFO] ------------------------------------------------------------------------ [INFO] BUILD SUCCESS [INFO] ------------------------------------------------------------------------ [INFO] Total time: 9.232 s [INFO] Finished at: 2018-01-21T09:29:15-05:00 |
Please use the latest 1.0.0 from https://github.com/att/EELF - nexus artifacts pending
app | project | repo | version |
---|---|---|---|
aai | aai-common | 1.0.0 | |
aai | aai-core | 1.0.0 | |
aai | aai-service (deprecated) | 0.0.1 (not in use) | |
aai | aai-traversal | 1.0.0 | |
aai | ajsc-aai | 1.0.0 | |
aai | eelf-logging | logging-service | 0.0.1 (not in use) |
appc | (pax.logging = wrapper on log4j, slf4j, commons-logging 1.2 and logback 1.1.1) needs this RI - provide a demo to the appc team | 0.0.1 (upgrade?) | |
ccsdk | nbapi | ccsdk/platform/nbapi | 1.0.0 |
dmaap | buscontroller | dmaap/buscontroller | 0.0.1 |
ui | dcae_dmaap_webapp | dmaapbc | 1.0.0 |
ui | dmaap-bc-app-common | 1.0.0 | |
ui | dmaap-bc-app-parent | 1.0.0 | |
policy | common-modules/ | common | 1.0.0 |
policy | drools-pdp | 1.0.0 | |
portal | ecompportal-be-common | portal | <eelf.version>1.0.0 |
policy | feature-eelf | drools-pdp | 1.0.0 |
policy | PolicyEngineSuite/ONAP-logging | common | 0.0.1 (not in use) |
policy | ONAP-SDK-APP | engine | 0.0.1 (not in use) |
sdc | ? | ||
so | common | so | 0.0.1 (upgrade ?) |
vid | epsdk-app-onap | vid | 1.0.0 |
vid | vid-app-common | vid | 1.0.0 |
open-o | |||
multicloud | python based |
Projects with a filebeat container as of 20170114
ubuntu@ip-172-31-49-200:~$ kubectl get pods --all-namespaces | grep 2/2 onap-aai aai-resources-1039856271-3ph1d 2/2 Running 0 37m onap-aai aai-traversal-143081090-f4cls 2/2 Running 0 37m onap-aai model-loader-service-4144225433-0jr21 2/2 Running 0 37m onap-aai search-data-service-3842430948-53j1w 2/2 Running 0 37m onap-aai sparky-be-4222608366-966p4 2/2 Running 0 37m onap-appc appc-1828810488-mhsf7 2/2 Running 0 37m onap-mso mso-736294529-c2p8v 2/2 Running 0 37m onap-policy drools-2600956298-dp7bq 2/2 Running 0 37m onap-policy pap-466625067-n23bp 2/2 Running 0 37m onap-policy pdp-2354817903-kmxv5 2/2 Running 0 37m onap-portal portalapps-1783099045-79lcq 2/2 Running 0 37m onap-sdc sdc-be-2336519847-p6p5l 2/2 Running 0 37m onap-sdc sdc-fe-2862673798-xcpnk 2/2 Running 0 37m onap-sdnc sdnc-0 2/2 Running 0 37m onap-sdnc sdnc-dbhost-0 2/2 Running 0 37m onap-vid vid-server-3026751708-rpmq4 2/2 Running 0 37m |
https://gerrit.onap.org/r/#/admin/projects/logging-analytics
git clone ssh://michaelobrien@gerrit.onap.org:29418/logging-analytics
see
obrienbiometrics:logging-analytics michaelobrien$ git status On branch master Your branch is up-to-date with 'origin/master'. Changes to be committed: (use "git reset HEAD <file>..." to unstage) new file: reference/logging-demo/.gitignore new file: reference/logging-demo/pom.xml new file: reference/logging-demo/src/main/java/org/onap/logging/LoggingReference.java new file: reference/logging-library/.gitignore new file: reference/logging-library/pom.xml new file: reference/logging-library/src/main/java/org/onap/logging/LoggingReference.java new file: reference/pom.xml Untracked files: (use "git add <file>..." to include in what will be committed) reference/.project reference/.settings/ reference/logging-demo/.checkstyle reference/logging-demo/.classpath reference/logging-demo/.project reference/logging-demo/.settings/ reference/logging-library/.checkstyle reference/logging-library/.classpath reference/logging-library/.project reference/logging-library/.settings/ reference/target/ obrienbiometrics:logging-analytics michaelobrien$ git commit -m "Initial logging lib/demo proj struct" [master 9e42247] Initial logging lib/demo proj struct 7 files changed, 306 insertions(+) create mode 100644 reference/logging-demo/.gitignore create mode 100644 reference/logging-demo/pom.xml create mode 100644 reference/logging-demo/src/main/java/org/onap/logging/LoggingReference.java create mode 100644 reference/logging-library/.gitignore create mode 100644 reference/logging-library/pom.xml create mode 100644 reference/logging-library/src/main/java/org/onap/logging/LoggingReference.java create mode 100644 reference/pom.xml obrienbiometrics:logging-analytics michaelobrien$ git review -s obrienbiometrics:logging-analytics michaelobrien$ git commit -s --amend [master c11cd50] Initial logging lib/demo proj struct Date: Sun Dec 31 13:30:04 2017 -0500 7 files changed, 306 insertions(+) create mode 100644 reference/logging-demo/.gitignore create mode 100644 reference/logging-demo/pom.xml create mode 100644 reference/logging-demo/src/main/java/org/onap/logging/LoggingReference.java create mode 100644 reference/logging-library/.gitignore create mode 100644 reference/logging-library/pom.xml create mode 100644 reference/logging-library/src/main/java/org/onap/logging/LoggingReference.java create mode 100644 reference/pom.xml obrienbiometrics:logging-analytics michaelobrien$ git review Warning: Permanently added '[gerrit.onap.org]:29418,[198.145.29.92]:29418' (RSA) to the list of known hosts. remote: Processing changes: new: 1, refs: 1, done remote: remote: New Changes: remote: https://gerrit.onap.org/r/27195 Initial logging lib/demo proj struct remote: To ssh://gerrit.onap.org:29418/logging-analytics * [new branch] HEAD -> refs/publish/master |
For eclipse you will need to make sure the spring nature in the .classpath is set correctly - in order for maven build to also work in eclipse for war projects - this will show up in the project build path
<classpathentry exported="true" kind="con" path="org.eclipse.m2e.MAVEN2_CLASSPATH_CONTAINER"> <attributes> <attribute name="maven.pomderived" value="true"/> <attribute name="org.eclipse.jst.component.nondependency" value=""/> <attribute name="org.eclipse.jst.component.dependency" value="/WEB-INF/lib"/> </attributes> </classpathentry> |
Figuring out the reason for the different dependencies
aai-service, ajsc-aai, common-modules/common-logging
<dependency>
<groupId>com.att.eelf</groupId>
<artifactId>eelf-core</artifactId>
<version>0.0.1</version>
</dependency>
aai-common, eelf-logging, ONAP-logging, ONAP-SDK-APP
<dependency>
<groupId>com.att.eelf</groupId>
<artifactId>eelf-core</artifactId>
<version>1.0.0</version>
</dependency>
Passing UUID based requestID's via rest calls
https://en.wikipedia.org/wiki/HTTP_ETag
PV for logback.xml,
container for mock-log-nbi
container for mock-log-proc (the scalable TX substrate)
FROM tomcat:8.0.48-jre8 COPY target/logging-demo-1.2.0-SNAPSHOT.war /usr/local/tomcat/webapps/logging-demo.war |
build.sh
Building, tagging and pushing the docker image so we can consume it in Kubernetes
mkdir target cp ../../logging-demo/target/*.war target docker build -t obrienlabs/logging-demo-nbi -f DockerFile . docker images | grep logging-demo-nbi docker tag obrienlabs/logging-demo-nbi obrienlabs/logging-demo-nbi:1 docker login docker push obrienlabs/logging-demo-nbi:1 #docker run -d -it --rm -p 8888:8080 obrienlabs/logging-demo-nbi:latest |
obrienbiometrics:docker michaelobrien$ vi DockerFile obrienbiometrics:docker michaelobrien$ docker build -t mock-logging-tomcat -f DockerFile . Sending build context to Docker daemon 18.04MB Step 1/2 : FROM tomcat:8.0.48-jre8 ---> e072422ca96f Step 2/2 : COPY logging-demo-1.2.0-SNAPSHOT.war /usr/local/tomcat/webapps/logging-demo.war ---> f0d234622af6 Successfully built f0d234622af6 Successfully tagged mock-logging-tomcat:latest obrienbiometrics:docker michaelobrien$ docker images | grep mock-logging mock-logging-tomcat latest f0d234622af6 9 seconds ago 576MB obrienbiometrics:docker michaelobrien$ docker run -d -it --rm -p 8888:8080 mock-logging-tomcat:latest ef5bdffb0e0aef9f185a51be2a915eae3d72fdb9a2125e1bb373516137c205b4 obrienbiometrics:docker michaelobrien$ docker ps CONTAINER ID IMAGE COMMAND CREATED STATUS PORTS NAMES ef5bdffb0e0a mock-logging-tomcat:latest "catalina.sh run" 5 seconds ago Up 4 seconds 0.0.0.0:8888->8080/tcp adoring_hawking obrienbiometrics:docker michaelobrien$ docker logs -f adoring_hawking 20-Jan-2018 04:30:26.092 INFO [localhost-startStop-1] org.springframework.web.context.ContextLoader.initWebApplicationContext Root WebApplicationContext: initialization completed in 1352 ms 20-Jan-2018 04:30:26.592 INFO [localhost-startStop-1] org.apache.catalina.startup.HostConfig.deployWAR Deployment of web application archive /usr/local/tomcat/webapps/logging-demo.war has finished in 3,652 ms 20-Jan-2018 04:30:26.841 INFO [main] org.apache.catalina.startup.Catalina.start Server startup in 3943 ms ^C obrienbiometrics:docker michaelobrien$ curl http://localhost:8888/logging-demo/rest/read/test testing: org.onap.logging.ApplicationService@433552a obrienbiometrics:docker michaelobrien$ docker build -t obrienlabs/mock-logging-tomcat -f DockerFile . Sending build context to Docker daemon 18.04MB Step 1/2 : FROM tomcat:8.0.48-jre8 ---> e072422ca96f Step 2/2 : COPY logging-demo-1.2.0-SNAPSHOT.war /usr/local/tomcat/webapps/logging-demo.war ---> Using cache ---> f0d234622af6 Successfully built f0d234622af6 Successfully tagged obrienlabs/mock-logging-tomcat:latest obrienbiometrics:docker michaelobrien$ docker tag obrienlabs/mock-logging-tomcat obrienlabs/mock-logging-tomcat:1 obrienbiometrics:docker michaelobrien$ docker login Login with your Docker ID to push and pull images from Docker Hub. If you don't have a Docker ID, head over to https://hub.docker.com to create one. Username (michaelobrien): obrienlabs Password: Login Succeeded obrienbiometrics:docker michaelobrien$ docker push obrienlabs/mock-logging-tomcat:1 The push refers to a repository [docker.io/obrienlabs/mock-logging-tomcat] 8bb560dc792d: Pushed 7a5faefa0b46: Pushed d1a0f91a1ad8: Pushed c4d99fd86e94: Pushed 04178c827c65: Pushed c571db474d75: Pushed 93742a1ae069: Pushed 7ffe9646653b: Pushed 6ef532e39c1e: Pushed a4d7b0ac0438: Pushed 06f4de5fefea: Pushed 851f3e348c69: Pushed e27a10675c56: Pushed 1: digest: sha256:a583f08001b21635e74e205f130ffbd23d933032686345be50f39a46b537e11b size: 3048 run obrienbiometrics:docker michaelobrien$ docker run -d -it --rm -p 8888:8080 obrienlabs/mock-logging-tomcat:1 316023a8abed5911000a1c366b7c897a33abcd7e9e70f48a4e2c2611f4c98d93 obrienbiometrics:docker michaelobrien$ curl http://localhost:8888/logging-demo/rest/read/test testing: org.onap.logging.ApplicationService@75f0964b obrienbiometrics:docker michaelobrien$ test on separate machine ubuntu@ip-172-31-52-161:~$ docker ps CONTAINER ID IMAGE COMMAND CREATED STATUS PORTS NAMES ubuntu@ip-172-31-52-161:~$ docker run -d -it -p 8888:8080 obrienlabs/mock-logging-tomcat:1 Unable to find image 'obrienlabs/mock-logging-tomcat:1' locally 1: Pulling from obrienlabs/mock-logging-tomcat 723254a2c089: Pull complete abe15a44e12f: Pull complete 409a28e3cc3d: Pull complete a9511c68044a: Pull complete 9d1b16e30bc8: Pull complete 0fc5a09c9242: Pull complete d34976006493: Pull complete 3b70003f0c10: Pull complete bc7887582e2e: Pull complete d2ab4f165865: Pull complete 7a0257d5eec9: Pull complete a23d5c1d9825: Pull complete 42f73552ced4: Pull complete Digest: sha256:a583f08001b21635e74e205f130ffbd23d933032686345be50f39a46b537e11b Status: Downloaded newer image for obrienlabs/mock-logging-tomcat:1 4262ca54a570a177185b646ec92b440f37250f7ee2922e018aec5d226c5b4f2a ubuntu@ip-172-31-52-161:~$ docker ps CONTAINER ID IMAGE COMMAND CREATED STATUS PORTS NAMES 4262ca54a570 obrienlabs/mock-logging-tomcat:1 "catalina.sh run" 9 seconds ago Up 8 seconds 0.0.0.0:8888->8080/tcp kickass_hoover ubuntu@ip-172-31-52-161:~$ curl http://dev.onap.info:8888/logging-demo/rest/read/test testing: org.onap.logging.ApplicationService@86a2339 |
Periodically one of the higher containers in a dependency tree will not get restarted in time to pick up running child containers - usually this is the kibana container
Fix this or "any" container by deleting the container in question and kubernetes will bring another one up.
root@a-onap-auto-20180412-ref:~# kubectl get services --all-namespaces | grep log onap dev-vfc-catalog ClusterIP 10.43.210.8 <none> 8806/TCP 5d onap log-es NodePort 10.43.77.87 <none> 9200:30254/TCP 5d onap log-es-tcp ClusterIP 10.43.159.93 <none> 9300/TCP 5d onap log-kibana NodePort 10.43.41.102 <none> 5601:30253/TCP 5d onap log-ls NodePort 10.43.180.165 <none> 5044:30255/TCP 5d onap log-ls-http ClusterIP 10.43.13.180 <none> 9600/TCP 5d root@a-onap-auto-20180412-ref:~# kubectl get pods --all-namespaces | grep log onap dev-log-elasticsearch-66cdc4f855-wmpkz 1/1 Running 0 5d onap dev-log-kibana-5b6f86bcb4-drpzq 0/1 Running 1076 5d onap dev-log-logstash-6d9fdccdb6-ngq2f 1/1 Running 0 5d onap dev-vfc-catalog-7d89bc8b9d-vxk74 2/2 Running 0 5d root@a-onap-auto-20180412-ref:~# kubectl delete pod dev-log-kibana-5b6f86bcb4-drpzq -n onap pod "dev-log-kibana-5b6f86bcb4-drpzq" deleted root@a-onap-auto-20180412-ref:~# kubectl get pods --all-namespaces | grep log onap dev-log-elasticsearch-66cdc4f855-wmpkz 1/1 Running 0 5d onap dev-log-kibana-5b6f86bcb4-drpzq 0/1 Terminating 1076 5d onap dev-log-kibana-5b6f86bcb4-gpn2m 0/1 Pending 0 12s onap dev-log-logstash-6d9fdccdb6-ngq2f 1/1 Running 0 5d onap dev-vfc-catalog-7d89bc8b9d-vxk74 2/2 Running 0 5d |
WIP
Deploy a Kubernetes cluster as normal via (No ONAP yet, just the K8S cluster) - ONAP on Kubernetes#InstallDocker/Rancher/Helm/Kubectl
Run the config and log pods
# clone oom git clone ssh://michaelobrien@gerrit.onap.org:29418/oom # clone logging-analytics git clone ssh://michaelobrien@gerrit.onap.org:29418/logging-analytics # ./createConfig.sh -n onap ./createAll.bash -n onap -a log # temp create config dirs ubuntu@ip-172-31-52-161:~/oom/kubernetes/oneclick$ sudo mkdir /dockerdata-nfs/onap/logdemo ubuntu@ip-172-31-52-161:~/oom/kubernetes/oneclick$ sudo mkdir /dockerdata-nfs/onap/logdemo/eteshare # copy the logdemo folder into the oom folder root@obriensystemsu0:~/logging-analytics/reference/logging-kubernetes# cp -R logdemo ~/oom/kubernetes/ # make sure the 3 ELK log containers are up root@obriensystemsu0:~/oom/kubernetes/oneclick# ./createAll.bash -n onap -a log root@obriensystemsu0:~/oom/kubernetes/oneclick# kubectl get pods --all-namespaces -a onap-log elasticsearch-3138115406-mlwjf 1/1 Running 0 12m onap-log kibana-3372627750-260f2 1/1 Running 0 12m onap-log logstash-1708188010-bhlqf 1/1 Running 0 12m # create the log demo root@obriensystemsu0:~/oom/kubernetes/oneclick# ./createAll.bash -n onap -a logdemo ********** Creating instance 1 of ONAP with port range 30200 and 30399 ********** Creating ONAP: ********** Creating deployments for logdemo ********** Creating namespace ********** namespace "onap-logdemo" created Creating service account ********** clusterrolebinding "onap-logdemo-admin-binding" created Creating registry secret ********** secret "onap-docker-registry-key" created Creating deployments and services ********** NAME: onap-logdemo LAST DEPLOYED: Mon Jan 29 11:25:44 2018 NAMESPACE: onap STATUS: DEPLOYED RESOURCES: ==> v1/Service NAME TYPE CLUSTER-IP EXTERNAL-IP PORT(S) AGE logdemo NodePort 10.43.26.45 <none> 8080:30299/TCP 0s ==> v1beta1/Deployment NAME DESIRED CURRENT UP-TO-DATE AVAILABLE AGE logdemo 1 1 1 0 0s ==> v1/Pod(related) NAME READY STATUS RESTARTS AGE logdemo-3133092390-g4qx7 0/1 ContainerCreating 0 0s #check root@obriensystemsu0:~/oom/kubernetes/oneclick# kubectl get pods --all-namespaces | grep logdemo onap-logdemo logdemo-3133092390-g4qx7 1/1 Running 0 5m root@obriensystemsu0:~/oom/kubernetes/oneclick# kubectl get services --all-namespaces -o wide | grep logdemo onap-logdemo logdemo NodePort 10.43.26.45 <none> 8080:30299/TCP 6m app=logdemo root@obriensystemsu0:~/logging-analytics/reference# kubectl -n onap-logdemo logs -f logdemo-3133092390-g4qx7 29-Jan-2018 17:28:03.456 INFO [main] org.apache.catalina.startup.Catalina.start Server startup in 11584 ms # test in-container and out-container ubuntu@ip-172-31-52-161:~/oom/kubernetes/oneclick$ kubectl -n onap-logdemo exec -it logdemo-4091951930-zwd53 bash root@logdemo-4091951930-zwd53:/usr/local/tomcat# ls webapps/ ROOT docs examples host-manager logging-demo logging-demo.war manager root@logdemo-4091951930-zwd53:/usr/local/tomcat# curl http://127.0.0.1:8080/logging-demo/rest/read/test testing: org.onap.demo.logging.ApplicationService@7a4fa9b outside ubuntu@ip-172-31-52-161:~/oom/kubernetes/oneclick$ curl http://dev.onap.info:30299/logging-demo/rest/read/test testing: org.onap.demo.logging.ApplicationService@52772382 |
http://dev.onap.info:30299/logging-demo/rest/read/test
"kind": "Deployment", "apiVersion": "extensions/v1beta1", "metadata": { "name": "mock-logging-tomcat", "namespace": "default", "selfLink": "/apis/extensions/v1beta1/namespaces/default/deployments/mock-logging-tomcat", "uid": "740b40f3-fda1-11e7-a4b4-02e171d3f6c2", "resourceVersion": "677", "generation": 1, "creationTimestamp": "2018-01-20T05:19:13Z", "labels": { "app": "mock-logging-tomcat", "version": "1" }, "annotations": { "deployment.kubernetes.io/revision": "1" } }, "spec": { "replicas": 1, "selector": { "matchLabels": { "app": "mock-logging-tomcat", "version": "1" } }, "template": { "metadata": { "name": "mock-logging-tomcat", "creationTimestamp": null, "labels": { "app": "mock-logging-tomcat", "version": "1" } }, "spec": { "containers": [ { "name": "mock-logging-tomcat", "image": "obrienlabs/mock-logging-tomcat:1", "resources": {}, "terminationMessagePath": "/dev/termination-log", "terminationMessagePolicy": "File", "imagePullPolicy": "IfNotPresent", "securityContext": { "privileged": false } } ], "restartPolicy": "Always", "terminationGracePeriodSeconds": 30, "dnsPolicy": "ClusterFirst", "securityContext": {}, "schedulerName": "default-scheduler" } }, "strategy": { "type": "RollingUpdate", "rollingUpdate": { "maxUnavailable": 1, "maxSurge": 1 } } }, "status": { "observedGeneration": 1, "replicas": 1, "updatedReplicas": 1, "readyReplicas": 1, "availableReplicas": 1, "conditions": [ { "type": "Available", "status": "True", "lastUpdateTime": "2018-01-20T05:19:13Z", "lastTransitionTime": "2018-01-20T05:19:13Z", "reason": "MinimumReplicasAvailable", "message": "Deployment has minimum availability." } ] } } |
Depending on how you set your port overrides for Tomcat 8.x
http://localhost:8988/logging-demo/rest/read/test
http://127.0.0.1:8080/logging-demo/rest/read/test
git clone https://github.com/att/EELF.git
https://tomcat.apache.org/download-80.cgi
https://spring.io/tools/sts/all
https://www.sitepoint.com/inside-java-9-part-ii/#performanceimprovements