...
https://jira.onap.org/secure/RapidBoard.jspa?rapidView=143&view=planning.nodetail&epics=visible
Attendees
Luke Parker , Prudence Au , Michael O'Brien , Sanjay, Lorraine Welch
Agenda
- Meetings every 2 weeks now - next meeting the 19th
- Need to accomidate Luke Parker - who is up at 3 am right now in Australia - will be 1 in the morning until April - then 2am
- ideas: Better to be up at 1-3 than 8am - keep at 11 for now
- For NA - stays at 11 - but I will verify if we are on UTC
- DST is coming this weekend - we should agree on the meeting time - Luke does not go DST for 3 more weeks
- Michael: status no progress on logging related jiras involving dublin scope (code/tech) last week
- Passed M2
- Discussed in TSC and PTL meets - issue related to manifest vs k8s values.yaml overrides for casablanca and CMR releases -
Jira server ONAP JIRA serverId 425b2b0a-557c-3c0c-b515-579789cceedb key TSC-86 - TSC 2019-02-28
Jira server ONAP JIRA serverId 425b2b0a-557c-3c0c-b515-579789cceedb key OOM-1560
- In architecture review for M3 - (Addressed comments) Logging Dublin M3 Architecture Review
- Working more for OOM lately - Created RKE script to replace rancher - pending work is to test HA - https://gerrit.onap.org/r/#/c/79067/ for
Jira server ONAP JIRA serverId 425b2b0a-557c-3c0c-b515-579789cceedb key OOM-1670 - Assisting with ONS April demo prep - 2 or more ONAP installations working together - one as an edge cloud
- Focused on onap devops - primarily for 2 customers deploying ONAP
- 5G training next week
- April 3-5 ONS work
- k8s infrastructure and
- Logging will be a part of the demo - we will show the kibana logs during a vFW operation in an i-frame for the master and edge ONAP deploys
- PTL position is open - let me know if anyone would like to take it.
- Also quickly discussed idea of rolling LOG into OOM - should revisit
- Pending committer for pomba - Committer Request for Trevor Tait and Pierre Rioux - Committer Request for Pierre Rioux[Logging-Analytics] - will send out reviews once stats are completed - TSC review will be needed
- TODO I need to check MSB issues on onap-discuss -
- Spondon - import the dashboard - fix the one on the wiki - Lorraine Welch export to json first ONAP Culprit Locator
- Lorraine Welch question about changing the index for the db (Grafana for example) - @Timestamp timestamp for example - TODO - merge the 2 indexes -
- Q) Sanjay: (opentracking related - picking up errors on stdout (err, out)
- how:
- (also review from 2 weeks ago - Lorraine Welch System.out standardout - review in terms of not using a hardcoded file appender - in terms of syslogs
- see https://12factor.net/logs)
- log retention on pod failure - those not mapped to PVs - including the current EmptyDir link between the sidecar - etherial PV - logs only end up in the elk stack
- Look into PV/PVC structure for log retention instead of the default emptydir
- https://git.onap.org/logging-analytics/tree/reference/logging-kubernetes/logdemonode/charts/logdemonode/templates/deployment.yaml#n114
- see PV/PVC example .
logs can go to the /dockerdatanfs) - https://git.onap.org/oom/tree/kubernetes/portal/charts/portal-mariadb/templates/deployment.yaml#n84
- Topic: File retention
- Q: rotation schedule, size of retention - what role does the PV on the ELK stack play compared to the individual source pod PVs
- @Luke's example - discuss with Mike Elliott - review a pending upstream contribution
- https://git.onap.org/logging-analytics/tree/reference/provider/helm/logback/chart/resources/logback.xml#n10
- variables set in
- https://git.onap.org/logging-analytics/tree/reference/provider/helm/logback/chart/values.yaml#n14'
- Review any issues with multiple providers in SDNC and APPC
- Lorraine Welch check https://www.loomsystems.com/blog/single-post/2017/01/30/a-comparison-of-fluentd-vs-logstash-log-collector
Attendees
Items
| PRI | Title | Responsible | Status
| In Dublin | Last Worked on | Start | Notes | ||||||||||||||||||||||||||||||||||||||
|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| Security Vulnerability template | Ongoing | IN | 20190122 | ||||||||||||||||||||||||||||||||||||||||||
| M1 template |
| IN | 20190124 | 20190122 | |||||||||||||||||||||||||||||||||||||||||
| ONS NA 2019 April Talk proposal |
| IN | 20190122 | pending 20190208 decision
| |||||||||||||||||||||||||||||||||||||||||
| Use manifest generation over raw oom values.yaml docker image tag names |
| IN | 20190124 | 20190117 | pending documentation in RTD Team, In the TSC it was decided to treat the diff between oom and the manifest by always running the manifest generated yaml in your deployments – you will not need to do this for master work – just for Casablanca and RC0-2 work
Working out the details in https://jira.onap.org/browse/LOG-929 /michael | ||||||||||||||||||||||||||||||||||||||||
| S3P Logging compliance TSC/PTL |
| IN | 20190115 | 20190114 | |||||||||||||||||||||||||||||||||||||||||
El-Alto 1.4 logging spec change - plan only todo merge with Dave's below |
| IN | 20190122 | ||||||||||||||||||||||||||||||||||||||||||
Dublin Scope Planning |
| IN | 20190124 |
| |||||||||||||||||||||||||||||||||||||||||
| RTD documentation |
| IN | 20180129 | Attending Thu 1130-1230 meets | |||||||||||||||||||||||||||||||||||||||||
restart log4j format and files example |
| IN | 201901115 | 20190108 | https://gerrit.onap.org/r/#/c/62405/ for
and
| ||||||||||||||||||||||||||||||||||||||||
| Work with portal/sdk library | Michael O'Brien |
| IN | 20190129 | 20190115 | Update: 20190129 - Existing eclipse environ for the RI being retrofitted At the pom stage bringing in the jar via portal/sdk in use by aai, dmaap, sdk, vid (vid link into so maybe?)
epic
Jira
review investigation in Log Streaming Compliance and API#ExistingLibraryResearch\ Luke Parker discussion need to use the portal library in an initiating project for tx processing working likely with the SO team - via the work we are doing for them in https://gerrit.onap.org/r/#/c/69947/ (check the original spec - ODL specific - check appc/sdnc use of ccsdk) | |||||||||||||||||||||||||||||||||||||||
| New Committers |
| 20190115 | We have room for 2-5 committers and will be reviewing the list Logging Enhancements Project Proposal#KeyProjectFacts add your details to Logging Committer Promotion Requests 20190129 status - waiting on contributor documentation from each contributor | ||||||||||||||||||||||||||||||||||||||||||
| OPNFV/ONAP Paris |
| 20190108 | https://ddfplugfest19.sched.com/ Tue-Thu Clover Gambia on prior https://zoom.us/j/115579117 - 7 hours ahead https://ddfplugfest19.sched.com/event/K1Gy/opnfv-clover-utilizing-cloud-native-technologies-for-nfv | ||||||||||||||||||||||||||||||||||||||||||
| Security badging |
| IN | 20190129 | Need to restart this | |||||||||||||||||||||||||||||||||||||||||
| Security Vulnerabilities |
| IN | 20190129 | lower - but for M4 | |||||||||||||||||||||||||||||||||||||||||
s3p Secure https endpoints LOG + POMBA | for Jason Hunt |
| IN | Discussion on whether we need to lock down the nodeport exposed ports Can key off POMBA work already done todo: get s3p page | |||||||||||||||||||||||||||||||||||||||||
Format compliance - working with AAI team + perf |
| IN | 20190115 | 20181101 |
https://gerrit.onap.org/r/#/c/75702/ (+) 20190109 #6 on 2018-12-20 AAI Developers Meeting around
Discussion with @Sanjay Agraharam and [~pau2882] on checking how cassandra is running on the vm and if debug levels are on should be verified use labelling to split aai-cs and ls - no DaemonSet Michael O'Brien to reduce core count for ls to 1 from 3
edited 2019-01-10 AAI Developers Meeting for the 10th | ||||||||||||||||||||||||||||||||||||||||
| AAI team - 2 types of logging AOP/non-AOP |
| IN | 20181101 | #22 on 2018-12-20 AAI Developers Meeting | |||||||||||||||||||||||||||||||||||||||||
| Logging requests from Vendors |
|
#15,19 and 37 on SP priorities for Dublin | |||||||||||||||||||||||||||||||||||||||||||
| LOG Streaming compliance |
| IN | Log Streaming Compliance and API
and | ||||||||||||||||||||||||||||||||||||||||||
opentracing via | IN (planning/POC for sure) | 20190123 | @Sanjay discuss integration - out of band processing -
see zipkin arch https://zipkin.io/pages/architecture.html possibly tie both as a client of es ? Tie in to ONS NA 2019 April demo booth for LF | ||||||||||||||||||||||||||||||||||||||||||
| discussion - remove | 20190108 | discuss tick/tock logging spec behaviour - cassablanca implemented in dublin, dublin implemented in elalto
| |||||||||||||||||||||||||||||||||||||||||||
| Log Checker | OUT | 20190109 | MIke to review with Horace | ||||||||||||||||||||||||||||||||||||||||||
| Search Guard |
| Maybe | 20180109 |
| |||||||||||||||||||||||||||||||||||||||||
| spec changes for Dublin | IN (planning) | 20190109 | 20190109 | Dublin spec changes for Elalto environment name release name check mail for reply Michael O'Brien Prudence Au proposal of renaming the log file name itself for the release ie: 3.0.0-ONAP - will discuss later for next week | |||||||||||||||||||||||||||||||||||||||||
Cluster logging behaviour S3P | IN | server name in clustered environments - I will add the details and the Jira right after this meet | |||||||||||||||||||||||||||||||||||||||||||
LOG ELK stack indexing/dashboards with Prometheus below |
| IN | 20190123 | ||||||||||||||||||||||||||||||||||||||||||
Casablanca 3.0.1 work until 10th Jan Including POMBA | DONE | 20190122 | 20190113 |
revert Jira for data-router off TSC-92 - pending merge of https://gerrit.onap.org/r/#/c/75999/ | |||||||||||||||||||||||||||||||||||||||||
LOG openlab tenant devops cluster creation/testing | Done pending vFW | IN | We have 2 clusters a 1+4 and 1+13 used for testing deployments and running the vFW | ||||||||||||||||||||||||||||||||||||||||||
| Wiki edits, RTD review |
| IN | Requiring Updates, Merges or Marked Deprecated | ||||||||||||||||||||||||||||||||||||||||||
| Metric Streaming and Prometheus |
| IN | 20181207 |
work with Vaibhav Chopra
@Sanjay - note the prom chart assumes a k8s environment - what about bare metal | |||||||||||||||||||||||||||||||||||||||||
Finish SO filebeat additions |
| IN | 20181207 |
| |||||||||||||||||||||||||||||||||||||||||
| Finish LOG common charts |
| OUT to El-alto | 20190123 | 20181207 | James MacNider - bring in Prianka's common eLK charts and use them in Clamp, LOG, SDC, POMBA https://gerrit.onap.org/r/#/c/64767/
rever to El-alto under
| ||||||||||||||||||||||||||||||||||||||||
| Team Members Thank you and review |
| IN |
| ||||||||||||||||||||||||||||||||||||||||||
| del | Review last 4 weeks since |
|
| ||||||||||||||||||||||||||||||||||||||||||
| TSC/PTL meet actions |
| ||||||||||||||||||||||||||||||||||||||||||||
| OOM transfer chart ownership to teams LOG is part of poc |
| IN | 20190107 | Starting - will have a training session - will send out any meetings to onap-discuss We may have the same symlink repo folder like we do for doc Last discussed TSC 20180109 | |||||||||||||||||||||||||||||||||||||||||
| OOM Deployment priority base platform includes LOG |
| OUT todo review with Mike Elliott | 20190123 | 20181207 | Q2) priority of system level containers like the ELK stack - OOM has a common services JIRA - DMaaP, AAF - TODO get JIRA - make sure log is in this! There is a cd.sh retrofit that sequences the pods in order for deployment stability - this will be phased out when tiered deployment comes in
https://gerrit.onap.org/r/#/c/75422/ via ONAP Development#WorkingwithJSONPath
| ||||||||||||||||||||||||||||||||||||||||
| k8s manifest or oom values.yaml for docker tags - truth |
| 20190123 |
TODO: paste TSC review - manifest is truth RTD doc link to run the script to get the yaml override | ||||||||||||||||||||||||||||||||||||||||||
| Nexus3 routing slowdown |
| 20181222 |
20181217-22
| ||||||||||||||||||||||||||||||||||||||||||
| LOG compliance diagram/exercise | @Sanjay Agraharam |
| 20181205 | Log Streaming Compliance and API part of prometheus work now Sanjay - diagram FB must be split between using AOP and AOP+spec compliant - only this one should be green | |||||||||||||||||||||||||||||||||||||||||
| Ongoing | |||||||||||||||||||||||||||||||||||||||||||||
| CI/CD pipeline TSC poc |
| 20180107 |
LOG is part of this poc |
20190115 - casablanca cherry pick in queue logstack 5 to 3 and 1