Action point follow-up
- Lukasz Rajewski prepare a demo on test automation managed by Lukasz..why it could not be integrated in CI for the moment (reflexion for Istanbul)
- Morgan Richomme sent a mail to PTL on good practice. Reminder that no patch in gate oldes than x months is a criteria for milestones..it is clearly not respected.
- Lasse Kaihlavirta plan a 20' section for a full session
- Admin
- committers
- baseline images
- repo cleanup
- Status on CSIT Lasse Kaihlavirta : CSIT status after Honolulu.pptx
- CPS = Showcase example, everything is fine
- CCSDK previous showcase, still requires distributtion of docker builds, not possible to rebuild locally
- DMaaP (dmaap/buscontroller moved), OPTF (optf/odf and optf/has already moved), SDNC (almost finished but still some failures) transition in progress
- AAI mumtitenancy done outside ONAP planned to be incoprorated into aai/resources and aai/traversal => use of scala+gatling (examples based on robot so far)
- Policy: CSIT moved to reir repositories but tests do not follow reco (test after merge, not project specific, jjb fully specific
- Still some active projects with old design (DCAEGEN2, Modeling, Multicloud, OOM-platform-cert-service, SO)
- Usecase #action Morgan Richomme dig a little bit to understand if it corresponds to active use cases
- VFC and VID do not seem really maintained
- VNFSDK: only images for frankfurt => not maintained
- CSIT tests disabled (aaf, appc, clamp, music)
- clamp moved to SDC => what does it mean from a CSIT perspective
- Demo vFW CNF Lukasz Rajewski Konrad Bańka
- slide deck: #action Lukasz Rajewski vFW CNF Use Case Demo
- use case partially automated but would require a KUD (test cannot use a ns of the existing k8s hosting ONAP)
- As the use case is maintained and includes regularly feature changes, it woudl be interesting to include it in CI..to show that it is red/green as part of the use cases (more than smoke tests)
- #action Morgan Richomme Konrad Bańka Lukasz Rajewski estimate what is missing for integration in CI
- Open question and where the status of the xNF shall be available for ONAP admin..not clear status of the service/vnf/module was available in VID, some test status was available in SDC but not sure to know where the info on the status of the xNF would be the more accurate #action Morgan Richomme send a mail to the community to get feedback on this xNF status page
- AoB
- Action point follow-up
- AP1: Morgan Richomme review Integration simu and release versions if possible + update doc accordingly
- not done yet wait for release of the xtesting dockers
- AP2: all move or closed the JIRA in honolulu
- done
- AP3: report your feedback on honolulu lesson learned
- to be done
- AP4: participants indicate their top 3 priorities for honolulu (https://etherpad.anuket.io/p/onap_integration_istanbul)
- done
- AP1: Morgan Richomme review Integration simu and release versions if possible + update doc accordingly
- Admin
- Admin status (PTL, election, INFO.yaml)
- congratulations Bartosz Gardziejewski elected as integration committer
- congratulations Bartosz Gardziejewski elected as integration committer
- Admin status (PTL, election, INFO.yaml)
- Morgan Richomme send a reminder for Alexander currently 4/4 but more than for 4 votes needed
- all the INFO.yaml actions will be performed once
- PTL no candidate => escalation done to TSC
- xtesting docker release
- new way, need to twist the scripts but could be OK. Need to fix guilin CI chain first (issue due to new pbr value)
- baseline images: wait for seccom reco for istanbul
- Morgan Richomme contact Seccom to prepare baseline images. user-98b79 indicates that we need to discuss offline especially for the python image.
- Honolulu
- regression on multicloud-k8s / pnf-macro
- - MULTICLOUD-1348Getting issue details... STATUS issue due to helm format, no error with the helm charts sued for Lukasz's use case. Fix done, docker images must be regenerated
- action Lukasz Rajewski prepare a demo on test automation managed by Lukasz..why it could not be integrated in CI for the moment (reflexion for Istanbul)
- master-weekly
- tern
- should be fully OK next week
- last results : https://logs.onap.org/onap-integration/weekly/onap_weekly_pod4_master/2021-05/07_22-21/security/tern/index.html
- user-98b79 indicates that anothe resource from Saùmsung is working on the topic to improve the images. No feedback from the PTL.Morgan Richomme indicates that he reports the issue to PTL during PTL meeting. Not normal to have no feedback (e.g. CLI improvement patch in gate for a long time), David McBride sent a mail to user-67d6f #action morgan sent a mail to PTL on good practice. Reminder that no patch in gate oldes than x months is a criteria for milestones..it is clearly not respected.
- stability tests
- update on MariaDB tuning => reduce the replica of non core components/removal of appc/... see stability_follow-up 2-2.pdf
- redimensioning is enough to avoid DB crash but recommendation to TSC to reduce replica for non core components 15 instances of Maria DB => 15x30 Go of additional RAM needed to avoid crash in load conditions...9 nodes cassandra...
- stability tests are now performed on each weekly => results are here: https://logs.onap.org/onap-integration/weekly/onap_weekly_pod4_master/2021-05/07_22-21/stability_tests/
- choice of the figures for weekly
- start with instantiation to avoid lots of models in SDC
- 10 instantiation in // during 24h (basic_vm / a la carte bpmn)
- then 5 onboarding in // during 24h
- needs to add basic_vm_macro if stability reached on first instantiation
- DB tuning
- suggestions from Samuel Liard to modify Maria Db and Cassandra DB default configuration: https://gerrit.onap.org/r/c/oom/+/121229/10
- all integration committers are using teh default DB values on their lab, no specific tuning, only the nb of compoentns may be reduced (because not used)
- update on MariaDB tuning => reduce the replica of non core components/removal of appc/... see stability_follow-up 2-2.pdf
- tern
- CSIT status after Honolulu (we may have to reserve some time for this so not necessarily today): Integration
- introduction to the status of CSIT at the end of Honolulu
- #action Lasse Kaihlavirta plan a 20' section for a full session
- regression on multicloud-k8s / pnf-macro
- Istanbul
- New functional tests
- Do we have resources / Shall we consider the existing list and select some to automate?
- scale out
- vfw CL
- Software upgrade
- MDONS
- 5G slicing
- ...
- pythonsdk
- Current Open issues: https://gitlab.com/Orange-OpenSource/lfn/onap/python-onapsdk/-/issues
- onaptests priorities
- update pnf_macro with Eli's simulator wrapper?
- to be reprogrammed whith Michał Jagiełło Andreas Geissler Illia Halych
- Do we have resources / Shall we consider the existing list and select some to automate?
- Backup and Restore
- tooling: Velero? other solutions? Any feedback
- Scenario? what can we automate?
- #action Morgan Richomme send a mail to the community/task force to converge on a scenario of tests that coudl be used for istanbul
- Operations
- Upgrade of a pod / chart
- Stability
- include a test with basic_vm_macro
- tuning of the DB
- Resiliency
- Integration of a k8s chaos framework?
- New functional tests
- AoB
- if ready Krzysztof Kuzmicki could also make a demo of the simulator
- Action point follow-up
- Morgan Richomme review Integration simu and release versions if possible + update doc accordingly
- not done yet..several issues on LF (SSO, Nexus, gerrit,..)
- Morgan Richomme create a JIRA for basic_vm_macro
- not needed last results with CCSDK fix look good
- user-98b79 finalize push to LF back end
- test in progress on weekly master..config is good
- Morgan Richomme create resiliency chapter in doc and initiate Jira for confirmed issue (appears at least 2 times), restart of a controller still to be done to complete the test.
- done see next section
- Krzysztof Kuzmicki create a wiki page in Istanbul page for robot refactoring follow-up
- Krzysztof Kuzmicki create a wiki page in Istanbul page for DCAE migration
- Morgan Richomme review Integration simu and release versions if possible + update doc accordingly
- Admin
- INFO.yaml updated needed (NS repo have no yaml + status of the project)
- Commiter Promotion Request for Bartosz Gardziejewski: results 10 Yes on 10
- Commiter Promotion Request for Alexander Mazuruk: results 4 Yes on 4
- PTL election: call for self nomination sent
- all operations on INFO.yaml will be done at the same time
- note Lasse Kaihlavirta indicated that he also will step down in 1 month
- Honolulu
- status:board review https://jira.onap.org/secure/RapidBoard.jspa?rapidView=229
- few JIRA still open # action all move or closed the JIRA in honolulu
- release testsuite 1.8.0 done
- just merged
- doc updated
- done
- Remaining work on tests
- macro issues Lukasz Rajewski
stability/resiliency tests
Maria DB galera troubleshooting
- honolulu_sign-off.pdf
- stability_follow-up.pdf
- Honolulu: Lessons Learned
- #action report your feedback on honolulu lesson learned
- status:board review https://jira.onap.org/secure/RapidBoard.jspa?rapidView=229
- Istanbul
- New Jira board created: https://jira.onap.org/secure/RapidBoard.jspa?rapidView=233
- brainstorm on new use cases for Istanbul open => https://etherpad.anuket.io/p/onap_integration_istanbul
- participants indicate their top 3 priorities for honolulu
- AoB
- Morgan Richomme review Integration simu and release versions if possible + update doc accordingly
- all move or closed the JIRA in honolulu
- report your feedback on honolulu lesson learned
- Action point follow-up
- Morgan Richomme review Integration simu and release versions if possible + update doc accordingly
- not done, will be done after doc, stability and resiliency testing #reaction
- Lukasz Rajewski properly tag SO bugs to reference them as blocking in Integration blicking table
- SO bugs do not appear in Honolulu Integration Blocking Points => to be crosscheck with Lukasz Rajewski
- if SDNC-1515 merged and pnf-simu OK in honoluluMorgan Richomme add pnf-macro to CI
- honolulu daily shows that the basic_vm_macro issue is not really fully fixed => #action Morgan Richomme create a JIRA for basoc_vm_macro
- Morgan Richomme grant access to Christophe Closset to check the status of SDC/DB
- done see section on SDC stability test
- Morgan Richomme retry and wait for the 5 minutes (Eviction timeout) then investigate on the networking issue
- done, Jira completed
- Morgan Richomme check ansible image availability for chained-ci
- doneAndreas Geissler gave a new try, still issue but not related to teh image. open question: shall we also reference the image in the CI page of the release?
- Morgan Richomme review Integration simu and release versions if possible + update doc accordingly
- Admin
- INFO.yaml updated needed (NS repo have no yaml + status of the project)
- Commiter Promotion Request for Bartosz Gardziejewski: please vote before end of the week. BTW if other committer proposal => feel free to suggest (to replace Pawel, Marcin and partly Thierry)
- Honolulu
- status:board review https://jira.onap.org/secure/RapidBoard.jspa?rapidView=229
- release testsuite 1.8.0 under built
- in progress https://gerrit.onap.org/r/c/testsuite/+/120960, once merged, submit a patch to OOM
- Remaining work on tests
- macro issues Lukasz Rajewski
- tern: test run in CI but results not pushed yet to LF Backend => WIP to manage properly the push to lf backend. On last honolulu and master-weekly, artifacts were produced and pushed manually #action user-98b79 finalize push to LF back end
stability/resiliency tests
Resiliency test
- worker node restart (thanks Bartek Grzybowski for the full analysis) => how do we follow, shall we create Jiras?
- test done, eviction looks good. some issues detected on some pods but not always trivial to reproduce may depend on the associated evicted pods. sometimes Init error, sometimes pod Running but exception..sometimes first eviction look good, but second seems to be fail... different cases referenced in Honolulu Resiliency and Backup and Restore test
- for SDC issue, the restart may be due to teh fact that the SDC is trying to recreate tables that already exist in cassandra
- #action Morgan Richomme create resiliency chapter in doc and initiate Jira for confirmed issue (appears at least 2 times), restart of a controller still to be done to complete the test.
stability tests
- SDC tests: 72h 5 parallel on boarding - INT-1912SDC Stability test: success rate drops after ~ 500 onboarding Open
- Christophe Closset reports that it was useful to detect new issues, especially the continuous time increasing that my be lead to teh DB and/or a middle ware layer (explicit exception seen)
- plan to integrate such test in weekly as part of benchmark test
- instantiation tests
- 48h 5 parallel basic_vm (initial issue on SDNC (fixed by restart) then regular timeout value then internal openstack issues - INT-1918Stability tests: // instantiation Open
- more timeout than in guilin, need to be confirmed with a 1 test (no parallelization), mariadbgalera issue observed on some replica
- 72h 1 test basic_vm => problem with mariadb galera, results will not be as good as in guilin run
- issue with mariadbgalera after 24h (but tests run after the 48h tests..) once finished (thurday morning => reinstallation of honolulu weekly to reproduce the tests with the last dockers
- mariadb galera workaround reported by Bell to OOM, seems to improve the overall quality of gate but trigger an new issue on CDS. tradefoff between redundancy and efficiency not clear
- documentation update started but not finished => https://gerrit.onap.org/r/c/integration/+/120236
- estimation 2-3 days still needed to compelte the doc / release note + testsuite integration
- Istanbul
- New Jira board created: https://jira.onap.org/secure/RapidBoard.jspa?rapidView=233
- robot refacroting..it eems we have a plan (Krzysztof Kuzmicki )
- discussion by mail, #action Krzysztof Kuzmicki crerate a wiki page in Istanbul page
- migration of DCAE from cloudify to helm chart..first analysis by Krzysztof Kuzmicki , impacts to be planned on several tests
- discussion by mail, #action Krzysztof Kuzmicki crerate a wiki page in Istanbul page
- New CSIT (AAI, CDS,..)
- AAI is folowwing the SDC way (Maven based), for CDS it shoudl be possible to reuse what Lasse Kaihlavirta suggested. AAI team contacted SDC
- brainstorm on new use cases for Istanbul open => https://etherpad.anuket.io/p/onap_integration_istanbul
- all review the page, it is an etherpad..so do not refrain yourself..put everything you have in mind
- AoB
- Lasse Kaihlavirta closed the CSIT refactoring EPIC
- Lasse Kaihlavirta also disabled useless SO CSIT tests (with very old images) - the Jira was closed without action by Seshu Kumar Mudiganti so we remove some jobs as they are meaningless (test with casablanca images)
- Lasse Kaihlavirta add a browser cleanup on robot healthcheck (vid) to save resource, it will be integrated in 1.8.0
- Action point follow-up
- Illia Halych review the official simu page (pythonsdk wrapper)
- done, no example for Honolulu, to be completed in Istanbul
- Morgan Richomme refine the information about the conditions of the resiliency tests of worker restart -
- See Honolulu topic
- Lasse Kaihlavirta Offline discussions with Lasse about AAI CSIT
- Done
- Krzysztof Kuzmicki Next week proposal about what we can do with robot
- Postponed to next week
- Krzysztof Kuzmicki - INT-1907browser_setup.robot does not provide proper teardown Open no teardown for browser-based checks - high consumption of resources at the end
- used by VID test, workaround to be confirmed
- Maybe we need more information about DMaaP simulator use and to write it better in the simulator doc → Reported by Lasse Kaihlavirta
Globally we are not well structured on the simulator, on the release note of version X we should list all the versions of the simulators we used for the validation. It shall be done by the use case teams that developed a simu and:or the integration team. The difficulty ois to get the full view on the simulators, some are hidden in repositories, some are unmaintained,...# action Morgan Richomme review Integration simu and release versions if possible + update doc accordingly
- Need of consolidation of versioning of images of simulators used in CSIT → Reported by Lasse Kaihlavirta
- sure it is a bit messy...and impossible to maintain, integration can cleanup time to time..but the best way for functional tests => bring functional tests including simu in their own repo. And if the simu can be used more widely create a dedicate repo under integration/simulators..
- Illia Halych review the official simu page (pythonsdk wrapper)
- Admin
- INFO.yaml updated needed (NS repo have no yaml + status of the project)
- strange INFO.yaml is needed to create the repo => to be xchecked. Morgan Richomme indicates to Thomas Kulik that it will be done with the next INFO update (needed as at least 2 committers announced they will stop after Honolulu)
- thanks to Pawel and Marcin for their contributions and all the best for the next challenges
- INFO.yaml updated needed (NS repo have no yaml + status of the project)
- Honolulu
- status:board review https://jira.onap.org/secure/RapidBoard.jspa?rapidView=229
- Remaining work on tests
- pnf_macro => OK on daily guilin, what do we do regarding CI integration
kudos to Michał Jagiełło - first pythonsdk tests with simulator - let's wait for the merge of Dan fix for SDNC-1515, if OK #action Morgan Richomme add pnf-macro to CI
- stil SO bugs preventing Lukasz Rajewski to complete his test .. deja-vu?
- #action Lukasz Rajewski properly tag SO bugs to reference them as blocking in Integration blicking table
- tern: test run in CI
- test results available (launched manually) => https://logs.onap.org/onap-integration/weekly/onap_weekly_pod4_master/2021-04/19_18-36/security/tern/index.html
- next CI run shall be good..wait and see
stability/resiliency tests
I started the first resiliency tests: - TEST-308[Resiliency] Evaluate ONAP behavior on k8s worker node restart In Progress , replay done, mail sent to the community (network issue?)
- behavor different on Nokia (pod stuck for ever in Terminating state when stopping a working) and Samsung RKE2 cluster, continue discussion on the ticket Bartek Grzybowski indicates that we shall be careful especially with statefulset, it coudl explain some issues
- #action morgan retry and wait for the 5 minutes (Eviction timeout) then investigate on the networking issue
stability tests
- SDC tests: 72h 5 parallel on boarding started on the 20th of April..wait and see...
- First tests were all OK but duration ontinuously increases. No error rate is high. Wait for the end of the tests to get the graphs.
- #action Morgan Richomme grant access to Christophe Closset to check the status of SDC/DB
- instantiation tests to be planned after onboarding tests
- SDC tests: 72h 5 parallel on boarding started on the 20th of April..wait and see...
consolidation of versions
better exception catching + integration of the GUI => still action Morgan
removal of the submodule: done
- pnf_macro => OK on daily guilin, what do we do regarding CI integration
- documentation update started but not finished => https://gerrit.onap.org/r/c/integration/+/120236
- release testsuite 1.8.0 to be done?
- next week if no objection
- simulators: whuch dockers are available in nexus today? could we clarify the build chain?
- see previous topic
- Istanbul
- New Jira board created: https://jira.onap.org/secure/RapidBoard.jspa?rapidView=233
- discussion on future tests
- follow up integration AAI tests in CSIT
- CDS in CSIT?
- AoB
- Andreas Geissler mentioned that his CI chains were affected by the non availability of an ansible image used in the runner => could be a problem, need to release also such images somewhere to avoid such issues #action Morgan Richomme check ansible image availability for chained-ci (2.7.13)
- issue also on Orange chains last week (build chain of ansible image was broken, chain fixed)
- Christophe Closset raises a question on python2.7 in jenkins CI and in robot. Partly addressed by the initiative of robot refactoring. note xtesting dockers also still using python2.7 due to dependency to python-utils that has not evolved for a long time.
- Andreas Geissler mentioned that his CI chains were affected by the non availability of an ansible image used in the runner => could be a problem, need to release also such images somewhere to avoid such issues #action Morgan Richomme check ansible image availability for chained-ci (2.7.13)
- Morgan Richomme review Integration simu and release versions if possible + update doc accordingly
- Lukasz Rajewski properly tag SO bugs to reference them as blocking in Integration blicking table
- if SDNC-1515 merged and pnf-simu OK in honoluluMorgan Richomme add pnf-macro to CI
- Morgan Richomme grant access to Christophe Closset to check the status of SDC/DB
- Morgan Richomme retry and wait for the 5 minutes (Eviction timeout) then investigate on the networking issue
- Morgan Richomme check ansible image availability for chained-ci
- Action point follow-up
- Michał Jagiełło pnf_macro add processing for pnf_macro to wait for a clean startup of the simulator (currently time based, and it seems that it takes more time than expected)
- in progress
- Michał Jagiełło complete basic_vm_macro
- done
- all review the wiki page and add the links of the GUI you are using
- all review/complete the wiki on process to integrate new external tests in CSIT or CI
- Krzysztof Kuzmicki complete the official simulator page with NS illustration
- documentation is done
- Illia Halych review the official simu page (pythonsdk wrapper)
- no feedback
- all review your Jira ticket move to instanbul or close
- Morgan Richomme review the patch dealing with sumbmodule removal + cherry + adapt xtesting-onap
- done
- Morgan Richomme complete wiki instanbul page to track this AAI scenario
- done, thanks to Lasse Kaihlavirta to ahve provided more examples.
- Michał Jagiełło pnf_macro add processing for pnf_macro to wait for a clean startup of the simulator (currently time based, and it seems that it takes more time than expected)
- Admin
- good occasion to discuss on the future organization and the future PTL..
- instanbul renaming (instead of istambul) done in seccom and oparent repo
- Honolulu
- status:board review https://jira.onap.org/secure/RapidBoard.jspa?rapidView=229&selectedIssue=INT-1842
- Daily Honolulu CI/CD chain created
- still sometimes some issues at SDNC startup ( - SDNC-1512Getting issue details... STATUS , Dan Timoney already added check mechanism but it seems that there are still issue in mariadbgalera, workaround (restart of SDNC seems to be OK)
- basic_vm_macro does not work due to SDNC ( - SDNC-1515Getting issue details... STATUS )
- basic_vm and basic_cnf fail randomly - mostly due to SDC problem or timeout in SO - this looks less stable than in guilin
- observed problem with config-assign/config-deploy in vFW-CNF integration test - SO-3626Getting issue details... STATUS - we need to cover config-assign/config-deploy in our smoke tests
- to sum up: we have two major problems with service-macro-create
- Remaining work on tests
basic_clamp: OK
basic_vm_macro: OK show a regression in SDNC ( - SDNC-1515Getting issue details... STATUS )
pnf_macro => Michał Jagiełłois working on the latest comments and needs further tests
refactoring of 5gbulkpm => OK (additional patch required submitted by Krzysztof Kuzmicki and cherry-picked)
tern: test declared in CI => we went further on the weekly-honolulu (https://gitlab.com/Orange-OpenSource/lfn/onap/xtesting-onap/-/jobs/1165577708), ansible-playbook was missing on the runner...fix merged (we need to use image with already ansible installed) ...to be continued but at least it is properly triggered
stability tests
consolidation of versions
better exception catching
integration of the GUI part to generate an html reporting (for the moment I usually replay manually the tests, get the results and apply the GUI processing manually)
removal of the submodule: test in progress (seems there was an import in the simu step), once done we could review our docker build (to remove the developer mode) but we shall not forget to adapt xtesting-onap according ly (/Src/onaptests/src/onaptests => /usr/lib/python3.8/sites-available/onaptests => action morgan for the week end..to avoid breaking everything before my PTO..
- documentation update started => https://gerrit.onap.org/r/c/integration/+/120236
- release testsuite 1.8.0 to be done
- Istanbul
- discussion on future tests
- follow up integration AAI tests in CSIT
- AOB
- Illia Halych review the official simu page (pythonsdk wrapper)
- Morgan Richommerefine the information about the conditions of the resiliency tests of worker restart -
- Lasse Kaihlavirta Offline discussions with Lasse about AAI CSIT
- Krzysztof Kuzmicki Next week proposal about what we can do with robot
- Krzysztof Kuzmicki - INT-1907Getting issue details... STATUS no teardown for browser-based checks - high consumption of resources at the end
- Maybe we need more information about DMaaP simulator use and to write it better in the simulator doc → Reported by Lasse Kaihlavirta
- Need of consolidation of versioning of images of simulators used in CSIT → Reported by Lasse Kaihlavirta
- Action point follow-up
- Michał Jagiełło pnf_macro add processing for pnf_macro to wait for a clean startup of the simulator (currently time based, and it seems that it takes more time than expected)
- Not done
- Morgan Richomme cleanup basic_clamp and re-run it on a daily master
- Done
- Michał Jagiełło complete basic_vm_macro
- Morgan Richomme create wiki page to reference the GUI endpoints to be tested
- Done Honolulu UIs
- all review the wiki page and add the links of the GUI you are using
- Morgan Richomme create a wiki page to improve the process on test integration in CSIT and/or CI from third parties
- all review/complete the wiki on process to integrate new external tests in CSIT or CI
- Morgan Richomme initiate an official page on simulator
- Krzysztof Kuzmicki complete the official simulator page with NS illustration
- To be Done
- Bartek Grzybowski review the official simu page (list of usable simu)
- Done
- Illia Halych review the official simu page (pythonsdk wrapper)
- To be Done
- Michał Jagiełło pnf_macro add processing for pnf_macro to wait for a clean startup of the simulator (currently time based, and it seems that it takes more time than expected)
- Admin
- Illia Halych officially new Integration commiter, welcome !
- No update on xtesting docker release
- LF IT backend , not possible to move logs, no cleanup mechanism (manual cleanup needed)
- Honolulu
- status:board review https://jira.onap.org/secure/RapidBoard.jspa?rapidView=229&selectedIssue=INT-1842
- action #all review your Jira ticket move to instanbul or close
- Daily Honolulu CI/CD chain created
- Remaining work on tests
basic_clamp: integrated in CI, several successful runs, but improvements needed to manage exceptions (Policy/DCAE/CLamp) => https://gerrit.onap.org/r/c/testsuite/pythonsdk-tests/+/120234
basic_vm_macro (replace vfw_macro): in progress still some issues on the daily honolulu
pnf_macro => add mechanism to have a better control of the pnf simu launch (timer is not enough) => # action Michal
refactoring of 5gbulkpm => #action Krzysztof => test done this afternoon => merged
tern: test declared in CI, but issue => # action morgan & Alexander
stability tests
I started the first resiliency tests: - TEST-308[Resiliency] Evaluate ONAP behavior on k8s worker node restart
- Other labs are experiencing different issues
- Nokia => pod remain in stuck
- #action Daniel, Andreas create a JIRA ticket for each resiliency test
- #action morgan create a wiki page to consolidate resiliencey tests
- Other labs are experiencing different issues
stability I planned to regive a try to Natacha's suite as soon as the other tests are OK
- to be tried asap: if enough resources => weekly Honolulu to be launched on Wednesday
consolidation of versions: I was waiting for the come back of Pawel but as far as I understood, you will move to othe r challenges soon.. :) so I need to review that
better exception catching
integration of the GUI part to generate an html reporting (for the moment I usually replay manually the tests, get the results and apply the GUI processing manually)
removal of the submodule: test in progress (seems there was an import in the simu step), once done we could review our docker build (to remove the developer mode) but we shall not forget to adapt xtesting-onap according ly (/Src/onaptests/src/onaptests => /usr/lib/python3.8/sites-available/onaptests
- #action morgan review the patch dealing with sumbmodule removal + cherry + adapt xtesting-onap
CDS / CSIT => will not have the resource to do it for Honolulu => move to Istanbul
GUI tests: wait for completion of the Wiki page and see if enough bandwidth => move to instanbul
- documentation update started
- honolulu branching done
- release testsuite 1.8.0 to be done
- status:board review https://jira.onap.org/secure/RapidBoard.jspa?rapidView=229&selectedIssue=INT-1842
- Istanbul
- requests from new project (OOF, AAI) "on how to bring new tests to integration" => How to help projects to integrate their tests in Integration?
- Discussion on the support for AAI gatling tests. By default the multitenancy support is disabled and they require a keycloak third party
- first issue is the test env => we coudl imagine to have a daily scenario daily_master_multitenancy but we need to take care of the resources. Until we cannot rely on the windriver lab, it will be hard to create new scenarios...
- installation of keycloak shall be done in a tooling namespace through a helm chart => OOM
- Mohammad Hosnidokht indicated that gatling part is almost automated (docker verifying the roles)
- William Reehil indicated that it will probably break the other tests
- it is probably more relevant to start by creating a CSIT test (fonctional test) with AAI + keycloak rather than a daily..
- #action Morgan Richomme complete wiki instanbul page to track this scenario
- run robot tests out of onap namespace: How to run the tests out of the ONAP cluster?
- requests from new project (OOF, AAI) "on how to bring new tests to integration" => How to help projects to integrate their tests in Integration?
- AoB
- Michał Jagiełło pnf_macro add processing for pnf_macro to wait for a clean startup of the simulator (currently time based, and it seems that it takes more time than expected)
- Michał Jagiełło complete basic_vm_macro
- all review the wiki page and add the links of the GUI you are using
- all review/complete the wiki on process to integrate new external tests in CSIT or CI
- Krzysztof Kuzmicki complete the official simulator page with NS illustration
- Illia Halych review the official simu page (pythonsdk wrapper)
- all review your Jira ticket move to instanbul or close
- Morgan Richomme review the patch dealing with sumbmodule removal + cherry + adapt xtesting-onap
- Morgan Richomme complete wiki instanbul page to track this scenario
- Action point follow-up
- Morgan Richomme action morgan initiate Illia Halych committer promotion procedure
- done
- Morgan RichommeMichał Jagiełło develop basic_vm_macro and basic_cbf_macro
- basic_vm ready to be tested, basic_cnf shall be very closed
- Morgan Richommecontact Lasse Kaihlavirta to see if CDS regression could be integrated in CSIT
- done, no bandwidth available, Jira created
- Morgan RichommeAndreas Geisslerlist UI and create a healthcheck test to check the availability of the UI
- done, see next section
- Morgan Richomme action morgan initiate Illia Halych committer promotion procedure
- Admin
- Committer promotion: formal vote initiated Committer Promotion Request for Integration Illia Halych
- vote in progress, please committer vote before the end of the week
- testing docker discussion (formating, unexpected removal of daily build dockers,..)
- 2 tickets created on LF
- "fresh" xtesting dockers disappear time to time on the nexus https://jira.linuxfoundation.org/plugins/servlet/theme/portal/2/IT-21812
- "impossible to release because the tag format of the snapshot docker does not respect some rules" => https://jira.linuxfoundation.org/plugins/servlet/theme/portal/2/IT-21710
- xtesting dockers are rebuilt daily based on the branches (guilin/master end of branches of all the components embedded in the docker), CI or end users cannot guess the format imposed by the LF rules (used for components e.g. 6.0-STAGING-20210315T185806Z)
- 2 tickets created on LF
- Committer promotion: formal vote initiated Committer Promotion Request for Integration Illia Halych
- Honolulu
- status: - INT-1883Reccurent various errors in smoke tests In Progress
- better results on daily master
- last patch for the release was the SDNC patch, currently in gate - shall be merged before the TSC
- dockers version (on Monday Master) - SO integrated meanwhile, wait for SDNC but otherwise it looks good: https://docs.google.com/spreadsheets/d/1t3GNRtabdkVxG4ZAxqBJ-X7OO-Zv6xRwd8KrR52V9zU/edit?usp=sharing
- weekly versions test show also some progress but we still have bad versions: https://logs.onap.org/onap-integration/weekly/onap_weekly_pod4_master/2021-03/29_08-09/security/versions/versions.html, results transfered to seccom (Amy Zwarico )
- lab
- azure staging resinstallation initiated
- done, keys pushed to the lab, ready to use: https://logs.onap.org/onap-integration/daily/onap_oom_staging_azure_1/2021-03/31_07-06/
- windriver: feedback from Intel => recontact them after Honolulu for Hardware upgrade
- azure staging resinstallation initiated
- Automated tests
- pnf_macro
- issue with the start of the simulator, wait 30s but it takes more time to start in master, a mechanism to check that everything shall be implemented #action Michał Jagiełło add mechanism in simu step to verify that everything is well started before sending a VES event.
- basic_clamp
- some regression had been introduced in master. Cleanup in progress, tests to be done after the meeting # action morgan re-run a clean basic_clamp on daily
- basic_vm_macro
- test almost ready, a config-deploy file was requested to complete the deployment, Michał Jagiełło wil include an empty file (nothing to do..) and re-test #action Michał Jagiełło complete basic_vm_macro tests
- tern integration in CI
- long duration test seen in https://logs.onap.org/onap-integration/weekly/onap_weekly_pod4_master/2021-03/29_08-09/security/tern/index.html but not sure the processing has been executed, no ternenv seen in the master-weekly lab. troubleshooting to be done (same mechanism will be reused for the stability tests.)
- Krzysztof Kuzmicki indicates that there are issues on cmpv2 and hv-ves on IPv4/Ipv6 lab due to proxy issues in Nokia lab, errors can be ignored
- pnf_macro
- stability and resiliency testing
- status: - INT-1883Reccurent various errors in smoke tests In Progress
- Istanbul
- first tests to execute the robotframework based tests in external ns
- several current robotframework, k8s job healthcheck
- url checker for GUI
- mail sent, no feedback #action Morgan Richomme create a wiki page to share the link before building the test (could be introduced in Honolulu)
- #action all review and complete this patch
- requests from new project (OOF, AAI) "on how to bring new tests to integration" => discussion on the best way..
- Q&A + official exist but are probably not enough
- #action Morgan Richomme create a wiki page to collect integration contributer view
- #action all review and complete the page
- first tests to execute the robotframework based tests in external ns
- AoB
- Do we have/should we have some Best Practices laid down for various simulators? (Lasse)
- the answer is no, we do not have, and yes we should have
- #action Morgan Richomme initiate a official doc page on simulator
- #action Krzysztof Kuzmicki use NS refactoring to complete this page
- Do we have/should we have some Best Practices laid down for various simulators? (Lasse)
- Michał Jagiełło pnf_macro add processing for pnf_macro to wait for a clean startup of the simulator (currently time based, and it seems that it takes more time than expected)
- Morgan Richomme cleanup basic_clamp and re-run it on a daily master
- Michał Jagiełło complete basic_vm_macro
- Morgan Richomme create wiki page to refeence the GUI endpoints to be tested
- all review the wiki page and add the links of the GUI you are using
- Morgan Richomme create a wiki page to improve the process on test integration in CSIT and/or CI from third patries
- all review/complete the wiki on process to integrate new external tests in CSIT or CI
- Morgan Richomme initiate an official page on simulator
- Krzysztof Kuzmicki complete the official simulator page with NS illustration
- Bartek Grzybowski review the official simu page (list of usable simu)
- Illia Halych review the official simu page (pythonsdk wrapper)
- Action point follow-up
- AP1: Morgan Richomme amend pythonsdk_tests to include the workaround
- new patch submitted yesterday (first one not efficient as no exception was raised by the SDK at this stage, just a 404 but no exception ResourceNotFound)
- https://gerrit.onap.org/r/c/testsuite/pythonsdk-tests/+/119655
- AP2: Santosh bayas identify inputs to see if parameters are missing + publish robot code in testsuite for review
- AP3: Morgan Richomme Krzysztof Kuzmicki check why we have a path difference between execution in the robot pod and in the xtesting docker
- not done
- AP4: Morgan Richomme review MR from user-98b79 for the integration of tern in ci-weekly
- first attempt done last friday but faield, new attempts to be done, see next section
- AP5: Morgan Richomme contact use case owner to collect the possible needs and remind the need to update the doc
- done see next section
- AP6: Morgan Richomme Krzysztof Kuzmicki create Epic and tasks to detail expectations on robot pod refactoring (alpine, split web/robot, python3, execution from outside the cluster, use python baseline image,....)
- AP1: Morgan Richomme amend pythonsdk_tests to include the workaround
- Admin
- update on the use cases (mail)
- Use Case for ONAP-based SON for 5G networks: "We do the testing in Winlab (Rutger’s University) within our use case team. So there is no requirement from Integration team for resources (human as well as lab infra). Some automation has been done – see https://gerrit.onap.org/r/gitweb?p=testsuite.git;a=tree;f=robot/testsuites/usecases;h=1690b9d4b7222f3748211df3a539e2a7d48e2a88;hb=HEAD."
E2E Network Slicing use case requirements for Honolulu release: "We do the testing in CMCC lab, Winlab (Rutger’s University), and at Windriver (OOF tenant) within our use case team. So there is no other requirement from Integration team for resources (human as well as lab infra). Automation – we expect to make some progress in Istanbul release."
- CCVPN - Transport Slicing use case requirements for Honolulu release Henry/Lin (some tests automated, dedicated repo created) "We are using our local lab in Ottawa to test the CCVPN use-case. And in parallel, the integration test of Network Slicing and Transport Slicing is done together with the Network Slicing use case team (i.e., using CMCC lab and Win lab)."
- doc updated
- Time to think to renew committers (some are no more very active) => suggest to promote Illia Halych
- #action morgan initiate promotion procedure
- update on the use cases (mail)
- Honolulu
- Troubleshooting campaign status: - INT-1883Reccurent various errors in smoke tests In Progress
- good progress on Master, still lots of patches in gate
- on daily Master still regular timeouts on basic_vm|cnf|network
- these 2E2 basic tests are leveraging A lacarte bpmn, it seems that most of the Service Providers priviledge the macro bpmn
- pnf_macro under integration in CI
- it would make sense to keep 1 test with a la carte workflow (basic_network) and move basic_vm and basic_cnf in macro Michał Jagiełło and Morgan Richomme started working on this task
- Daily Guilin very stable - no so many timeouts for the basic_tests...so for sure the change of DB had an impact..we are not very good in evaluating the DB performance...important to address before B&R tests
- VID issue fixed
- Certificates at large fixed (openstack, AAI,..)
- Daily master of yesterday looks fine: https://logs.onap.org/onap-integration/daily/onap_daily_pod4_master/2021-03/23_11-52/
- Majority of the issues due to timeout on SO requests
- Issues could be due to SO or any other components SO is discussing with including DB
- On Guilin it is much more stable and for a full workflow it take usually 2 times less average (1100 versus 500s)
- Code did not changes that much => could be due to the DB upgrade
- #action morgan/michal: write basic_vm_macro, basic_cnf_macro
- Lukasz Rajewski indicates that we must keep the test of a la carte bpmn because they are usefull to specify vnf order and complete macro mode
- the goal is not to exclude these tests but to create new one on macro mode which is the main used bpmn
- Krzysztof Kuzmicki indicates that some SO slow down are also affecting the pf-registrate, wich is using te macro mode
- Automated tests
- update on CDS regression test (session with Jozsef Csongva Morgan Richomme
- session done last week, it was possible to launch the mock and start running the tests
- some issues due to the fact that the test code was not fully up to date regarding master version
- #action morgan contact Lasse Kaihlavirta and see if integration in CSIT will nto make more sense
- DCAEmod integrated in CI Krzysztof Kuzmicki
- integrated in CI
- false negative (icon red but test green) due to the fact that the test was not declared in the test DB. Test added, so at teh end of xtesting the status is pushed ot the DB http://testresults.opnfv.org/onap/api/v1/results?case_name=dcaemod
- From a dashboard perspective => test in healthcheck = 1 project / Smoke = test involved several projects BUT test still executed as part of smoke test in CI (avoid race condition with DCAE healthcheck)
- tern integration in CI failed in last weekly (gitlab-ci under review)
- use of a bad env var for the weekly rules change merged to be retested
- update on CDS regression test (session with Jozsef Csongva Morgan Richomme
- Troubleshooting campaign status: - INT-1883Reccurent various errors in smoke tests In Progress
- Istambul
- Move Robot pod out of ONAP cluster (including robot)
- certificate and robot pod to be executed in onap-testing cluster
- deals only with xtesting (not the robot pod)
- the idea is to keep the onap namespace cleaned
- to be tested, only open question = cm shall be recreated in teh new onap-testing namespace? url seems OK in cm to allow x-namespace exchanges
- certificate and robot pod to be executed in onap-testing cluster
- PythonSDK
- which tests? what is poorly covered?
- vFWCL? scaleout?
- NBI
- basic loop (basic_clamp extended)
- ETSI bpmn?
- use cases?
- which tests? what is poorly covered?
- Robot
- Bell/apex (if not done in Honolulu)
- Add a retry capability in CI? other rules (if onap-helm or onap-k8s Fail "too much" => do not execute E2E tests to save CI time?)
- Open discussion on the missing tests
- Bartek Grzybowski Illia Halych Backup & Restore
- Michał Jagiełło Krzysztof Kuzmicki Service Sanity check. Healthcheck are not enough, they may be PASS when the system is not working at all..Problem reported already (lots of Healthcheck tests do not provide more information that the onap-k8s checking that the pods are up&running). We have several examples when the pods are up&running but it does not wrok. Exception shall not caught to hide the issues. We would need to formalize a little bit to discuss with the PTL.
- Andreas Geissler we are missing a simple test checking the UI endpoint
- #action andreas morgan send a mail to PTL to collect admin and user UI and create a simple test checking that the UI are available
- user-98b79 add clair scan, even if it shall be done at LF, usually showing it in weekly is the fastest way for adoption
- Open discussion on the missing tests
- CI dashboard: improved dashboard to identify regression over time (see xls graphs) - multi platforms?
- Move Robot pod out of ONAP cluster (including robot)
- AoB
- Morgan Richomme action morgan initiate Illia Halych committer promotion procedure
- Morgan Richomme Michał Jagiełło develop basic_vm_macro and basic_cbf_macro
- Morgan Richomme contact Lasse Kaihlavirta to see if CDS regression coudl be integrated in CSIT
- Morgan Richomme Andreas Geissler list UI and create a healthcheck test to check the availability of the UI
- Action point follow-up
- AP1 Illia Halych : ask OOM for the right place to host helm chart
- unformal exchange. For CDS helm charts are stored intestsuite/cds/mock-* until we coudl move the helm charts to a helm registry
- AP2 Andreas Geissler add warning in doc for backup and restore
- patch done
- AP3 Morgan Richomme contact TSC regarding B&R task force and possible scenario for Honolulu
- done, mail sent, Catherine Lefevre answered and agreed on shortmid term plans
- AP4 Morgan Richomme test basic-pnf (macro) on daily master before integrating thsi use case in CI
- done, test issues reported to Michał Jagiełło and added in - INT-1822Getting issue details... STATUS
- AP1 Illia Halych : ask OOM for the right place to host helm chart
- Admin
- congratulation to the team for the S3P..now it will be cool to fix the issues in Master/Gating
- windriver re-installation: to be planned after honolulu => answer sent to Intel team
- Azure Staging lab reinstalled
- #acrion Morgan Richomme send a mail to the community to indicate that staging azure is back (guilin maintenance release as master is broken)
- congratulation to the team for the S3P..now it will be cool to fix the issues in Master/Gating
- Honolulu
- Troubleshooting campaign status: - INT-1883Reccurent various errors in smoke tests In Progress
- several issues found
- SDC races => including offset to move to pseudo sequential seems to reduces the number of SDC issues. Still one where ceritification seems to take to much time => possibility to add a workaround in the test to retry and/or delay the creation/certification
- #action Morgan Richomme amend pythonsdk_tests to include the workaround
- VID 500 due to the presence of a "U" char in random password (which explains why we did not have always the error...à - deobuscated by VID even if not obfuscated using jetty library interprating U as unicode char breaking everything. Problem found and fixed by Krzysztof Opasiak => last gating seems reasonably better
- SO error (so-bpmn stuck) was alsready observed time to time in guilin, so it is not a new problem...it explains why sometimes an instantiation takes 3 minutes sometimes 20, a jIRA had been created on this aspect - SO-3419Getting issue details... STATUS
- rebase/remerge in gate in progress...gates will burn over the next days...but it means that master chains should be back and usable
- SDC races => including offset to move to pseudo sequential seems to reduces the number of SDC issues. Still one where ceritification seems to take to much time => possibility to add a workaround in the test to retry and/or delay the creation/certification
- several issues found
- Automated tests
- demo Apex pdp Policy Santosh bayas Prudence Au
- test done on Bell labs, adaptation needed to be intgrated in CI (config in a dedicated file not leveraging CM information)..
- #Action Santosh bayas identify inputs to see if parameters are missing + publish robot code in testsuite for review
- test done on Bell labs, adaptation needed to be intgrated in CI (config in a dedicated file not leveraging CM information)..
- CDS regression test (session with Jozsef Csongvai planned on thursday to give a try on Orange lab and see how we can integrate in CI - helm chart finally with docker file in mock repositories)
- mock in gates
- troubleshooting planned on thursday
- DCAEmod, 5Gbulkpm evolution (Krzysztof Kuzmicki ) → Morgan Richomme
- Could we move this point to next week when bulkpm over HTTPS server will be finished and DCAEMOD in daily CI ?
- #action Morgan Richomme Krzysztof Kuzmicki check why we have a path difference between execution in the robot pod and in the xtesting docker
- pythonsdk (pnf, clamp, .., removal of the submodule, patch to allow test replay)
- tern integration in CI user-98b79 Morgan Richomme - how to test/troubleshoot ansible par
- user-98b79 provides follow up in the MR https://gitlab.com/Orange-OpenSource/lfn/onap/xtesting-onap/-/merge_requests/37
- #action Morgan Richomme reriew MR for the integration of tern in ci-weekly
- #agreed moved weekly reinstallation on Friday next week
- user-98b79 provides follow up in the MR https://gitlab.com/Orange-OpenSource/lfn/onap/xtesting-onap/-/merge_requests/37
- demo Apex pdp Policy Santosh bayas Prudence Au
- S3P tests
- back on B&R mail to the community => only resilience tests to be planned for Honolulu
- stability => target?
- wait for master stabilization
- Figures to be decided: 5 // onboarding during 48, 20 // instantiation during 48?
- feed back on first test executed with litmus
- k8s chaos monkey framework
- it was possible to anote onap component and generate chaos..coudl be used for resiliency testing to check that components are recovering
- use cases
- who knows the list of Honolulu use cases?..
- Lukasz Rajewski shared the link with the Honolulu Release Requirements
- no requests received so far
- no documentation changes seen
- #action Morgan Richomme contact use case owner to collect the possible needs and remind the need to update teh doc
- who knows the list of Honolulu use cases?..
- Troubleshooting campaign status: - INT-1883Reccurent various errors in smoke tests In Progress
- Istambul => for information to be discussed next week
- Move Robot pod out of ONAP cluster (including robot)
- certificate and robot pod to be executed in onap-testing cluster
- CI dashboard: improved dashboard to identify regression over time (see xls graphs) - muti platforms?
- PythonSDK
- which tests? what is poorly covered?
- vFWCL? scaleout?
- NBI
- basic loop (basic_clamp extended)
- which tests? what is poorly covered?
- Add a retry capability in CI? other rules (if onap-helm or onap-k8s Fail "too much" => do not execute E2E tests to save CI time?)
- Robot refactoring: target (slot planned after the meeting if we are short in time)
- #action Morgan Richomme Krzysztof Kuzmicki create Epic and tasks to detail expectations on robot pod refactoring (alpine, split web/robot, python3, execution from outside the cluster, use python baseline image,....)
- Move Robot pod out of ONAP cluster (including robot)
- AoB
- AP1: Morgan Richomme amend pythonsdk_tests to include the workaround
- AP2: Santosh bayas identify inputs to see if parameters are missing + publish robot code in testsuite for review
- AP3: Morgan Richomme Krzysztof Kuzmicki check why we have a path difference between execution in the robot pod and in the xtesting docker
- AP4: Morgan Richomme review MR from user-98b79 for the integration of tern in ci-weekly
- AP5: Morgan Richomme contact use case owner to collect the possible needs and remind the need to update the doc
- AP6: Morgan Richomme Krzysztof Kuzmicki create Epic and tasks to detail expectations on robot pod refactoring (alpine, split web/robot, python3, execution from outside the cluster, use python baseline image,....)
- Action point follow-up
- AP1: Andreas Geissler check if doc of the official use cases are excplictely referencing the robot pod
- not done yet
- AP2: Morgan Richomme include Nokia pod in onap-integration web site
- AP3: all help reviewing the OOM patches to give Integration feedback
- in progress see next section
- AP1: Andreas Geissler check if doc of the official use cases are excplictely referencing the robot pod
- Admin
- Honolulu
- pythonsdk
- smoke test issues (basic_vnf, basic_cnf) 10'
last master daily results
01/03/21 03/03/21 04/03/21 05/03/21 06/03/21 08/03/21 basic_onboard OK KO OK KO OK OK basic_cds OK OK OK OK OK OK basic_vm KO KO KO KO KO KO basic_cnf OK KO KO KO KO KO basic_network OK KO KO KO KO OK Pnf-registrate OK KO OK OK KO OK 5gbulkpm OK OK OK OK OK OK Issues
- VID: - VID-960VID exception on E2E tests In Progress
- 500 on SDC (multi retry on SDC component get versions)
- gitlab.com issues (runner problems)
- today the daily guilin and the daily master are very different, daily guiling is 100% OK from E2E perspective, daily master is still bad
- however it is exactly teh same code xtesting-smoke-usecases:master and the same launch xtesting-onap:master
- the only difference is due to master changes
- review of the OOM Master patches from the 17/2 (all gates look pretty good). Daily master of th 18 and 19 were also good (not the daily from the 1st of March was not that bad as well..)
Merge Date Date of the last patchset (or rebase) used for gating ID Patch Status
At least 2 of the 3 basic_ E2E tests OK)links Comments 17/02/21 00:12:00 12/02/21 08:48:00 117758 [AAF] Give `identities.dat` to working deployments OK https://logs.onap.org/onap-integration/gating//117758-3/index.html basic_cnf and basic_network OK pb of SDC with basic_vm 17/02/21 12:13:00 16/02/21 13:34:00 114505 [MSB] Use certInitializer for MSB OK https://logs.onap.org/onap-integration/experimental-gating//114505-24/index.html 17/02/21 12:18:00 16/02/21 08:07:00 117909 [AAI] Add multi.tenancy.enabled flag to keycloak properties file OK https://logs.onap.org/onap-integration/gating//117909-2/index.html 18/02/21 16:27:00 18/02/21 08:15:00 117881 [APPC][CDT] Automatically retrieve certificates OK https://logs.onap.org/onap-integration/gating//117881-4/index.html 18/02/21 16:35:00 17/02/21 15:22:00 117807 [AAI][SPARKY] Updating the sparky config OK https://logs.onap.org/onap-integration/gating//117807-4/index.html 19/02/21 22:17:00 11/02/21 23:28:00 117743 [GENERAL] Fix failed chart list reporting in deploy plugin KO https://logs.onap.org/onap-integration/gating//117743-1/index.html 20/02/21 01:06:00 17/02/21 10:13:00 117936 [DCAEGEN2] Update inventory Api to 3.5. KO https://logs.onap.org/onap-integration/gating//117936-1/index.html 19/02/21 17:41:00 19/02/21 20:33:00 117753 [SDNC] Fix issue with certs from CMPv2 by Netconf (TLS) KO https://logs.onap.org/onap-integration/gating//117753-6/index.html note patchet 5 executed on the 15/02 os OK https://logs.onap.org/onap-integration/experimental-gating//117753-5/index.html 20/02/21 20:23:00 18/02/21 15:48:00 118049 [NBI] Fix image name OK https://logs.onap.org/onap-integration/gating//118049-1/index.html 22/02/21 15:17:00 18/02/21 23:50:00 118076 [GENERAL] Fix setting of HELM_VER variable KO https://logs.onap.org/onap-integration/experimental-gating//118076-1/index.html 22/02/21 15:22:00 20/02/21 16:54:00 117921 [AAI] Move schema service to use v23 OK https://logs.onap.org/onap-integration/gating//117921-2/index.html 22/02/21 15:31:00 18/02/21 09:07:00 117991 [DCAEGEN2] Use generic makefiles KO https://logs.onap.org/onap-integration/gating//117991-3/index.html 22/02/21 15:47:00 19/02/21 15:19:00 117889 [CLI] Retrieve certificates automatically KO https://logs.onap.org/onap-integration/gating//117889-6/index.html test from the 16/02 (patchset 2) OK (https://logs.onap.org/onap-integration/gating//117889-2/index.html)
test from the 17/02 OK (https://logs.onap.org/onap-integration/experimental-gating//117889-2/index.html)test from the 18/2 (1:29) KO (https://logs.onap.org/onap-integration/gating//117889-3/index.html)But installation is failing
19/2 KO (https://logs.onap.org/onap-integration/gating//117889-6/index.html) (lots of HC KO)22/02/21 15:40:00 18/02/21 19:11:00 117858 [GENERAL] Print helm binary used OK https://logs.onap.org/onap-integration/gating//117921-2/index.html 23/02/21 12:05:00 22/02/21 16:26:00 118171 [GENERIC][CASSANDRA] Loosen probe check time KO https://logs.onap.org/onap-integration/gating//118171-1/index.html 23/02/21 12:15:00 22/02/21 15:53:00 116836 [AAI][RESOURCES] Remove Hardcoded certificates KO https://logs.onap.org/onap-integration/gating//116386-22/index.html 12/2 (patchset 19) OK (https://logs.onap.org/onap-integration/gating//116386-19/index.html)
17/2 (patchset 20) KO – VID issue
18/2 18:00(patchset 21) basic_cnf OK
19/2 (patchset 21 oom _redeploy) OK (https://logs.onap.org/onap-integration/gating//116386-21/index.html)
22/2 (patchset 22) basic_network ok bit other basic failed23/02/21 10:09:00 22/02/21 15:51:00 117828 [AAI][TRAVERSAL] Remove Hardcoded certificates KO https://logs.onap.org/onap-integration/gating//117828-6/index.html Patchset 4 (18/02 21:00) is fully OK (https://logs.onap.org/onap-integration/gating//117828-4/index.html) something seems broken since the 19/2...only 3 patches have been merged around this date: 117743, 117936, 117753
the subsequent OK correspond to patch whose tests have been executed earlier but merge after the 20/2, revert in progress to see if one of the 3 patches coudl be a root cause of the regression on the gating chain
- smoke test issues (basic_vnf, basic_cnf) 10'
- pythonsdk
- sequential versus parallel
- tests planned to move to E2E sequential to avoid side effect due to parallelization. Parallel tests could be done on weekly...need to fist master and gating chains first.
- sequential versus parallel
- tests update (Michał Jagiełło 15' )
- pnf (macro flow)
- ready to be tested (logger fixed)
- #action morgan test basic-pnf on daily
- exceptions not caught
- patch provided
- basic_clamp
- wrapper
- OK, question on where to host the helm charts for the simulator (pnf and future one)
- #action contact OOM team => similar question than the for CDS mock objects
- it is possible to remove the submodule - just be carefull, once removed => possible to deploy onapsdk as a "standard module" no more as developer module => impact on xtesting-onap as the paces will be different
- pnf (macro flow)
- Backup/restore Morgan Richomme 15' Backup_and_restore_discussion.pdf
- discussion on a realistic use case...
- backup and restore seems too optimistic
- consensus to work on the resiliency of the core component
- #action Andreas Geissler warning shall be added in the documentation, as aarna network is not really usable for a real backup/restore (more some examples to use velero)
- Companies that tried velero has issues due to teh fact that ONAP is not really cloud native
- discussion on a realistic use case...
- AoB
- cmpv2 about to be moved from healthcheck to smoke
- Krzysztof Kuzmicki indicates that 5Gbulkpm will be updated ..may have some turbulences during the update
Actions
- AP1 Illia Halych : ask OOM for the rright place to host helm chart
- AP2 Andreas Geissler add warning in doc for backup and restore
- AP3 Morgan Richomme contact TSC regarding B&R task force and possible scenario for Honolulu
- AP4 Morgan Richomme test basic-pnf (macro) on daily master before integrating thsi use case in CI
- Action point follow-up
- no action point from last week (several docs)
- action point from the 17/02
- AP1: Morgan Richomme : contact vcpe tosca team to get their position in Honolulu (
-
TEST-239Getting issue details...
STATUS
)
- done no feedback on the JIra
- AP2: user-98b79 : get the list of projects using the baseline images
- done
- AP3: Illia Halych amend https://gerrit.onap.org/r/c/testsuite/pythonsdk-tests/+/117192 for helm3 support
- done
- AP4: Morgan Richomme test basic_pnf on Orange daily master with onaptests:master
- not done PTO..but I can reaction myself;.
- AP5: Michał Jagiełło Morgan Richomme test carefully before submitting until we got a gate to do it...
- not really an action point but the advise is wise...be carefull after removing the submodules + change the xtesting installation (using standard python module not developer way...xtesting-onap shall follow)
- AP6: Krzysztof Kuzmicki prepare a 10 minutes demo on pnf-simu to nf-simu (explaning the graph shared by mail)
- done
- AP7: Marcin Przybysz plan a demo of VID new UI, how to launch it, how it is used in PNF macromode + see with VID how we can manage tests that are using legacy VID..
- done
- AP8: Morgan Richomme contact Kenny Paul for the status of the slack chan
- done it is a free chan, no idea how LF decides to pay for a non free chan..if community thinks that we shall be able to keep the history and then take a non free chan, it is possible to report it to the TSC
- AP1: Morgan Richomme : contact vcpe tosca team to get their position in Honolulu (
-
TEST-239Getting issue details...
STATUS
)
- Admin
- A1 policy enforcement created
- nf repo created (Krzysztof Kuzmicki )
- M3 is over..JIRA review needed to know what is possible (support/test cases/) what shall be postponed...
- release cadence is new, some projects were surprised. No new code will be accepted in OOM gate. Lots of patches submitted...gating system under pressure..wait for merge to reach the RC0
- Honolulu (M3 is over..)
- CSIT issue with python2.7/Python3 (Lasse Kaihlavirta Morgan Richomme )
- fix submitted by user-98b79
- list of jenkins images known in CSIT repo but not sure it is documented. Some issues due to python 2.7/3.5/3.6/3.7/3.8/3.9...lib dependency issues NB the seccom recommended version is the 3.9
- Refactoring of testsuite
- python2.7/micro service/root
- cert
- helm chart
- Lukasz Rajewski and Andreas Geissler suggest to put the effort on migrating to pythonsdk rather than maintaining the pod
- Morgan Richomme reminds that robot tests are run through pythonsdk, the question is linked to the robot pod not the ability to run robotframework test cases, which is always possible. Who is using the e2e scripts that trigger processing on the robot pod. Lasse Kaihlavirta indicates that some usecases (under upstreaming (A1)) are in this case. For Honolulu we may ask for a waiver and remove the pod in Istambul (is nobody is using it). Morgan Richomme sent a mail to the community but no feedback so far. Scaleout and vFWCL still probably requires the pod, other usercases requires also the init offered by the bash scripts #action Andreas Geissler look in the doc which use cases explicitely make reference to the scripts in the official documentation
- status on CI/CD (delta guilin) (Morgan Richomme )
- New Nokia dualstack lab added in the pool of lab reporting results http://testresults.opnfv.org/onap/api/v1/results?last=5&case_name=full&pod=ci-dualstack-master-daily
- IPv4/IPv6 server, ONAP Master daily => results shared with the community in LF backend + Test DB
- shall we update http://testresults.opnfv.org/onap-integration/ ?
- Daniel Milaszkiewicz OK #action morgan add Nokia in http://testresults.opnfv.org/onap-integration/
- Master is currently not that bad: https://logs.onap.org/onap-integration/daily/onap_daily_pod4_master/03-01-2021_06-03/
- error due to internal cert executed in onap ns (shall be moved to an external ns)
- 1 error on enhanced policy => policy anticipated the OOM merge and already adapted the test, it shall be back OK once the new policy dockers woudl be merged
- 1 error in E2E due to SDC (exception not caught)
- lots of patch in OOM gate..if lots of errors it is not normal as the master is reasonably stable...
- #action all help reviewing the OOM patches to give Integration feedback
- Guilin is not good
- same CI errors in DT and Orange
- runnign the tests directly is usually good => issue in the way to launch the test in guilin
- "old" ansible version in the xtesting-onap branch
- tests in progress to use xtesting-onap master (assuming that the test version is provided by a env variable already) => no need of specific branch
- delta guilin/current master shared with the community
- https://docs.google.com/spreadsheets/d/1t3GNRtabdkVxG4ZAxqBJ-X7OO-Zv6xRwd8KrR52V9zU
- as usual some projects anticipate and provided features continuously (Policy, DCAE, NBI, OOF, OOM), some did not...which explains the gating bottlneck due to the M3
- New Nokia dualstack lab added in the pool of lab reporting results http://testresults.opnfv.org/onap/api/v1/results?last=5&case_name=full&pod=ci-dualstack-master-daily
- security test update
- tern (user-98b79 )
- last results: https://logs.onap.org/onap-integration/daily/onap_weekly_pod4_master/02-25-2021_02-02/security/tern/
- patch submitted to fix some projects ..but they have no gating so manual test needed
- first patch submitted
- advantage, it will be easier to chase CVE if we can rely on baseline images
- issues with Python (and lib dependencies) but few components dealing with python
- next steps work on teh integration as a weekly job in xtesting-onap to run automatically tern everyweek
- versions still some issue in weekly chain + reporting to be integrated
- patch merged for the bug
- results of last week shared => https://logs.onap.org/onap-integration/daily/onap_weekly_pod4_master/02-25-2021_02-02/security/versions/versions/index.html
- core/internal nodeports tests can be long ..any idea how to parallelize
- wait for Paweł Wieczorek come back as same issue than for versions
- tern (user-98b79 )
- Backup & restore
- postponed to next week
- simu question on the availability of the E// pnf simu availability in nexus AJAY SINGH
- answer sent by mai, changes done broke the built chain => after 40 days the snampshot images are removed, as the image was not released and not rebuilt, expiration date was reached => need to fix the build or to release an old version (pretty sure that jenkins history is also cleaned..so probably not possible)
- CSIT issue with python2.7/Python3 (Lasse Kaihlavirta Morgan Richomme )
- AoB
- Action points for next meeting
- AP1: Andreas Geissler check if doc of the official use cases are excplictely referencing the robot pod
- AP2: Morgan Richomme include Nokia pod in onap-integration web site
- AP3: all help reviewing the OOM patches to give Integration feedback
- Action point follow-up
- Admin
- Honolulu
- demo simulator Core NSSMF simulator (Zhang Min ) 20'
- demo CSIT of VNF/NFVO package management and LCM Gareth Roper , Andrew Lamb Byung-Woo Jun 10'
- nf-simulator (pnf-simu evolution) Krzysztof Kuzmicki 10'
- VID new UI Marcin Przybysz VID_MODERN_UI.pptx
- AoB
- Action point follow-up
- AP1: morgan_orange publish video and slideck on the how to
- done: Youtube video link, discussions in progress at LFN level to suggest how to publish such videos per community channel
- AP2: morgan_orange plan a new slot with praveen
- no feedback from PRAVEEN VELUGUBANTLA , need to replan a slot
- AP3: morgan_orange send mail to suggest repo, and if OK => create teh repo
- mail sent to Konrad Bańka , wait for green light before creating the new repo
- AP4: morgan_orange anser to the mail for pnf-simu evolution and plan a formal agreement from committer next week
- see next section
- AP5: natacha add the _v to manage the settings path in order to retrieve the pythonsdk logs
- done but side effect due to the fact that the settings in master are in /Src/onaptests/src/onaptests and not under /usr/lib/python3.8 in xtesting
- AP1: morgan_orange publish video and slideck on the how to
- Admin
- new ietf repo created, Bartek Grzybowski added JS linter, first upload in progress
- Guilin CI chain
- ~ OK http://testresults.opnfv.org/onap-integration/results/index-onap_daily_pod4_guilin-ONAP-oom.html
- Certificates CDS-UI expires in 3 days and VID in 29 days
- Error due to 500 from SDC
- Master chain
- pythonsdk tests broken during 2-3 days due to the introduction of onapsdk 7.4 (onapsdk custom exceptions were no more caught in onaptests)
- problem affecting all the gating chains
- Comments from Lukasz Rajewski on the slack channel => quid about the history, free account does not provide long retention period #action 8
- Honolulu
- Discussion on a recovery pipeline / resiliency tests (Marek Szwałkiewicz Morgan Richomme
- Epic created on Honolulu Resiliency and Backup and Restore test
- several tasks to perform tests
- Please do not hesitate to use the JIRA if you are performing such tests in your lab
- Epic created on Honolulu Resiliency and Backup and Restore test
- Discussion on CI/CD chain evolution (Morgan Richomme )honolulu_ci_evolution_08022021.pdf
- Krzysztof Kuzmicki indicates that dcae-mod shall be tested also through a new healthcheck test
- FWCL not possible to integrate in CI straightforward - preprovisionned resources needed, dependencies on installation (documented),
- Not sure if FW (macromode) will bring more than basic_VM + PNF and note sure Andreas Geissler will have resource to deal with it
- VCPE TOSCA (CLI) was not possible to run it on daily labs (running onwly on tester lab), improvements were planned in Guilin (seems to be done). #action 1
- #agreed No objection from the new categories and new tests for Honolulu branch
- for tern/scancode: watch out the versions of the OS, Debian9 is too old. #action 2
- Security
- versions: last update
- test created by Paweł Wieczorek
- waiver list introduced in Honolulu (patch merged)
- New GUI suggested more explicit that the json file: https://logs.onap.org/onap-integration/weekly/onap_weekly_pod4_master/02-11-2021_22-02/security/versions/versions/
- still lots of java 8 and python 2.7...
- very few projects seem to use the baseline images (that are up to date..)
- versions: last update
- Pythonsdk
- removal of the submodule?
- #agree remove the pnf submodule as soon as possible, be careful with the side effects on xtesting dockers / xtesting-onap ansible roles as we coudl move back from src/onaptests to /usr/lib/python3.8 path
- status on the wrapper (helm2/helm3)
- Illia Halych indicates that there is a client supporting helm2 and one supporting helm3 but not both
- #agree consider only helm3
- Add the statement in the README (as for the python min version 3.7 (same as pythonsdk)
- #action 3
- status on the tests basic_pnf, basic_clamp
- problem for basic_pnf seems to be fixed, patch merged on SDC
- see if the basic_pnf could be introduced in CI (test it first manually on daily weekly then add teh job xtesting-onap project that defines the CI chain) #action 4
- Improve reporting
- reporting may be misleading as it indictaes only the results of resource creation. We recently got an issue when deleting the resources => all the reporting is green even if there was an error during vfModule deletion. Michał Jagiełło submitted a patch to consider deletion in reporting #action 5
- Solution to limit issues on master gate (gating on pythonsdk?, versioning,..)
- pnf-simu > nf-simu: formal review
- #agreed
- demo to be planned next week #action6
- Discussion on a recovery pipeline / resiliency tests (Marek Szwałkiewicz Morgan Richomme
- AoB
- Marcin Przybysz indicates that a discussion is needed on VID. Vid new UI is needed for PNF macromode tests but old UI is needed for vFWCL..how do we manage that
- Michał Jagiełło tried to setup the new VID uI (VID 7.0.2) but has issue, Marcin Przybysz indicates that some flags have to be tuned #action 7
- actions for next meeting
- AP1: Morgan Richomme : contact vcpe tosca team to get their position in Honolulu ( - TEST-239Getting issue details... STATUS )
- AP2: user-98b79 : get the list of projects using the baseline images
- AP3: Illia Halych amend https://gerrit.onap.org/r/c/testsuite/pythonsdk-tests/+/117192 for helm3 support
- AP4: Morgan Richomme test basic_pnf on Orange daily master with onaptests:master
- AP5: Michał Jagiełło Morgan Richomme test carefully before submitting until we got a gate to do it...
- AP6: Krzysztof Kuzmicki prepare a 10 minutes demo on pnf-simu to nf-simu (explaning the graph shared by mail)
- AP7: Marcin Przybysz plan a demo of VID new UI, how to launch it, how it is used in PNF macromode + see with VID how we can manage tests that are using legacy VID..
- AP8: Morgan Richomme contact Kenny Paul for the status of the slack chan
- Action point follow-up
- Admin
- Honolulu
- demo 5g-core-nf simu (PRAVEEN VELUGUBANTLA ) 20'
- demo ORAN A1-based Closed Loop (Konrad Bańka ) 20' A1 Policy Enforcement PoCv1.pptx
- demo stability test launcher (Natacha Chéreau )10'
- AoB
(18:02:16) collab-meetbot: Meeting ended Wed Feb 10 17:02:15 2021 UTC. Information about MeetBot at http://wiki.debian.org/MeetBot . (v 0.1.4)
(18:02:16) collab-meetbot: Minutes: http://ircbot.wl.linuxfoundation.org/meetings/onap-int/2021/integration_weekly_meeting_10_2_2021/onap-int-integration_weekly_meeting_10_2_2021.2021-02-10-16.44.html
(18:02:16) collab-meetbot: Minutes (text): http://ircbot.wl.linuxfoundation.org/meetings/onap-int/2021/integration_weekly_meeting_10_2_2021/onap-int-integration_weekly_meeting_10_2_2021.2021-02-10-16.44.txt
(18:02:16) collab-meetbot: Log: http://ircbot.wl.linuxfoundation.org/meetings/onap-int/2021/integration_weekly_meeting_10_2_2021/onap-int-integration_weekly_meeting_10_2_2021.2021-02-10-16.44.log.html
- Action point follow-up
- Admin
- new repo follow-up
- Honolulu
- demo PNF (Michał Jagiełło )
- discussion on simulators
- simulator wrapper in pythonsdk-tests (Illia Halych)
- Preparation vEvent
- AoB
(18:57:32) collab-meetbot: Minutes: http://ircbot.wl.linuxfoundation.org/meetings/onap-int/2021/integration_weekly_meeting_27_01_2021/onap-int-integration_weekly_meeting_27_01_2021.2021-01-27-17.29.html
(18:57:32) collab-meetbot: Minutes (text): http://ircbot.wl.linuxfoundation.org/meetings/onap-int/2021/integration_weekly_meeting_27_01_2021/onap-int-integration_weekly_meeting_27_01_2021.2021-01-27-17.29.txt
(18:57:32) collab-meetbot: Log: http://ircbot.wl.linuxfoundation.org/meetings/onap-int/2021/integration_weekly_meeting_27_01_2021/onap-int-integration_weekly_meeting_27_01_2021.2021-01-27-17.29.log.html
- Action point follow-up
- Admin
- new repo follow-up
- new Real time support channel: slack (already exists), publick rocket chat, mattermost, ....
- Honolulu
- demo PNF (Michał Jagiełło )
- discussion on simulators
- simulator wrapper in pythonsdk-tests (Illia Halych)
- CSIT redesign updates (Lasse Kaihlavirta)
- stability/resiliency testing (Natacha Chéreau )
- Update on Tern tests (user-98b79 )
- Preparation vEvent
- AoB
(17:40:10) collab-meetbot: Minutes: http://ircbot.wl.linuxfoundation.org/meetings/onap-int/2021/integration_weekly_meeting_20_01_2021/onap-int-integration_weekly_meeting_20_01_2021.2021-01-20-16.06.html
(17:40:10) collab-meetbot: Minutes (text): http://ircbot.wl.linuxfoundation.org/meetings/onap-int/2021/integration_weekly_meeting_20_01_2021/onap-int-integration_weekly_meeting_20_01_2021.2021-01-20-16.06.txt
(17:40:10) collab-meetbot: Log: http://ircbot.wl.linuxfoundation.org/meetings/onap-int/2021/integration_weekly_meeting_20_01_2021/onap-int-integration_weekly_meeting_20_01_2021.2021-01-20-16.06.log.html
- Action point follow-up
- Honolulu
- demo CDS / how it is used in Bell and how to upstream (Jozsef Csongvai ) 10' +10' discussions
- demo tests DataFileCollector (Krzysztof Gajewski ) 10'
- Guilin Maintenance Release
- Preparation vEvent
- AoB
(16:26:02) collab-meetbot: Minutes: http://ircbot.wl.linuxfoundation.org/meetings/onap-int/2021/integration_weekly_meeting_13_1_2021/onap-int-integration_weekly_meeting_13_1_2021.2021-01-13-14.51.html
(16:26:03) collab-meetbot: Minutes (text): http://ircbot.wl.linuxfoundation.org/meetings/onap-int/2021/integration_weekly_meeting_13_1_2021/onap-int-integration_weekly_meeting_13_1_2021.2021-01-13-14.51.txt
(16:26:03) collab-meetbot: Log: http://ircbot.wl.linuxfoundation.org/meetings/onap-int/2021/integration_weekly_meeting_13_1_2021/onap-int-integration_weekly_meeting_13_1_2021.2021-01-13-14.51.log.html
Canceled quorum not reached (lots of public holidays)
Integration weekly meeting 2020 archives
Integration weekly meeting 2019 archives
Integration weekly meeting 2018 archives
Integration weekly meeting 2017 archives