- Action point follow-up
- Morgan Richomme review Integration simu and release versions if possible + update doc accordingly
- not done, will be done after doc, stability and resiliency testing #reaction
- Lukasz Rajewski properly tag SO bugs to reference them as blocking in Integration blicking table
- SO bugs do not appear in Honolulu Integration Blocking Points => to be crosscheck with Lukasz Rajewski
- if SDNC-1515 merged and pnf-simu OK in honoluluMorgan Richomme add pnf-macro to CI
- honolulu daily shows that the basic_vm_macro issue is not really fully fixed => #action Morgan Richomme create a JIRA for basoc_vm_macro
- Morgan Richomme grant access to Christophe Closset to check the status of SDC/DB
- done see section on SDC stability test
- Morgan Richomme retry and wait for the 5 minutes (Eviction timeout) then investigate on the networking issue
- done, Jira completed
- Morgan Richomme check ansible image availability for chained-ci
- doneAndreas Geissler gave a new try, still issue but not related to teh image. open question: shall we also reference the image in the CI page of the release?
- Morgan Richomme review Integration simu and release versions if possible + update doc accordingly
- Admin
- INFO.yaml updated needed (NS repo have no yaml + status of the project)
- Commiter Promotion Request for Bartosz Gardziejewski: please vote before end of the week. BTW if other committer proposal => feel free to suggest (to replace Pawel, Marcin and partly Thierry)
- Honolulu
- status:board review https://jira.onap.org/secure/RapidBoard.jspa?rapidView=229
- release testsuite 1.8.0 under built
- in progress https://gerrit.onap.org/r/c/testsuite/+/120960, once merged, submit a patch to OOM
- Remaining work on tests
- macro issues Lukasz Rajewski
- pnf-macro test to be done on master before integration in CI (working fine in guilin)
- basic_vm_macro good results on guilin but not so good on master/honolulu => JIRA created
Jira server ONAP JIRA serverId 425b2b0a-557c-3c0c-b515-579789cceedb key SDNC-1529
- tern: test run in CI but results not pushed yet to LF Backend => WIP to manage properly the push to lf backend. On last honolulu and master-weekly, artifacts were produced and pushed manually #action user-98b79 finalize push to LF back end
stability/resiliency tests
Resiliency test
- worker node restart (thanks Bartek Grzybowski for the full analysis) => how do we follow, shall we create Jiras?
- test done, eviction looks good. some issues detected on some pods but not always trivial to reproduce may depend on the associated evicted pods. sometimes Init error, sometimes pod Running but exception..sometimes first eviction look good, but second seems to be fail... different cases referenced in Honolulu Resiliency and Backup and Restore test
- for SDC issue, the restart may be due to teh fact that the SDC is trying to recreate tables that already exist in cassandra
- #action Morgan Richomme create resiliency chapter in doc and initiate Jira for confirmed issue (appears at least 2 times), restart of a controller still to be done to complete the test.
stability tests
- SDC tests: 72h 5 parallel on boarding
INT-1912 - SDC Stability test: success rate drops after ~ 500 onboarding Open
- Christophe Closset reports that it was useful to detect new issues, especially the continuous time increasing that my be lead to teh DB and/or a middle ware layer (explicit exception seen)
- plan to integrate such test in weekly as part of benchmark test
- instantiation tests
- 48h 5 parallel basic_vm (initial issue on SDNC (fixed by restart) then regular timeout value then internal openstack issues
- more timeout than in guilin, need to be confirmed with a 1 test (no parallelization), mariadbgalera issue observed on some replica
- 72h 1 test basic_vm => problem with mariadb galera, results will not be as good as in guilin run
- issue with mariadbgalera after 24h (but tests run after the 48h tests..) once finished (thurday morning => reinstallation of honolulu weekly to reproduce the tests with the last dockers
- mariadb galera workaround reported by Bell to OOM, seems to improve the overall quality of gate but trigger an new issue on CDS. tradefoff between redundancy and efficiency not clear
- 48h 5 parallel basic_vm (initial issue on SDNC (fixed by restart) then regular timeout value then internal openstack issues
- SDC tests: 72h 5 parallel on boarding
- macro issues Lukasz Rajewski
- documentation update started but not finished => https://gerrit.onap.org/r/c/integration/+/120236
- estimation 2-3 days still needed to compelte the doc / release note + testsuite integration
- Istanbul
- New Jira board created: https://jira.onap.org/secure/RapidBoard.jspa?rapidView=233
- robot refacroting..it eems we have a plan (Krzysztof Kuzmicki )
- discussion by mail, #action Krzysztof Kuzmicki crerate a wiki page in Istanbul page
- migration of DCAE from cloudify to helm chart..first analysis by Krzysztof Kuzmicki , impacts to be planned on several tests
- discussion by mail, #action Krzysztof Kuzmicki crerate a wiki page in Istanbul page
- New CSIT (AAI, CDS,..)
- AAI is folowwing the SDC way (Maven based), for CDS it shoudl be possible to reuse what Lasse Kaihlavirta suggested. AAI team contacted SDC
- brainstorm on new use cases for Istanbul open => https://etherpad.anuket.io/p/onap_integration_istanbul
- all review the page, it is an etherpad..so do not refrain yourself..put everything you have in mind
- AoB
- Lasse Kaihlavirta closed the CSIT refactoring EPIC
- Lasse Kaihlavirta also disabled useless SO CSIT tests (with very old images) - the Jira was closed without action by Seshu Kumar Mudiganti so we remove some jobs as they are meaningless (test with casablanca images)
- Lasse Kaihlavirta add a browser cleanup on robot healthcheck (vid) to save resource, it will be integrated in 1.8.0
- Action point follow-up
- Illia Halych review the official simu page (pythonsdk wrapper)
- done, no example for Honolulu, to be completed in Istanbul
- Morgan Richomme refine the information about the conditions of the resiliency tests of worker restart -
- See Honolulu topic
- Lasse Kaihlavirta Offline discussions with Lasse about AAI CSIT
- Done
- Krzysztof Kuzmicki Next week proposal about what we can do with robot
- Postponed to next week
- Krzysztof Kuzmicki
- used by VID test, workaround to be confirmed
- Maybe we need more information about DMaaP simulator use and to write it better in the simulator doc → Reported by Lasse Kaihlavirta
Globally we are not well structured on the simulator, on the release note of version X we should list all the versions of the simulators we used for the validation. It shall be done by the use case teams that developed a simu and:or the integration team. The difficulty ois to get the full view on the simulators, some are hidden in repositories, some are unmaintained,...# action Morgan Richomme review Integration simu and release versions if possible + update doc accordingly
- Need of consolidation of versioning of images of simulators used in CSIT → Reported by Lasse Kaihlavirta
- sure it is a bit messy...and impossible to maintain, integration can cleanup time to time..but the best way for functional tests => bring functional tests including simu in their own repo. And if the simu can be used more widely create a dedicate repo under integration/simulators..
- Illia Halych review the official simu page (pythonsdk wrapper)
- Admin
- INFO.yaml updated needed (NS repo have no yaml + status of the project)
- strange INFO.yaml is needed to create the repo => to be xchecked. Morgan Richomme indicates to Thomas Kulik that it will be done with the next INFO update (needed as at least 2 committers announced they will stop after Honolulu)
- thanks to Pawel and Marcin for their contributions and all the best for the next challenges
- INFO.yaml updated needed (NS repo have no yaml + status of the project)
- Honolulu
- status:board review https://jira.onap.org/secure/RapidBoard.jspa?rapidView=229
- Remaining work on tests
- pnf_macro => OK on daily guilin, what do we do regarding CI integration
kudos to Michał Jagiełło - first pythonsdk tests with simulator - let's wait for the merge of Dan fix for SDNC-1515, if OK #action Morgan Richomme add pnf-macro to CI
- stil SO bugs preventing Lukasz Rajewski to complete his test .. deja-vu?
- #action Lukasz Rajewski properly tag SO bugs to reference them as blocking in Integration blicking table
- tern: test run in CI
- test results available (launched manually) => https://logs.onap.org/onap-integration/weekly/onap_weekly_pod4_master/2021-04/19_18-36/security/tern/index.html
- next CI run shall be good..wait and see
stability/resiliency tests
I started the first resiliency tests:
TEST-308 - [Resiliency] Evaluate ONAP behavior on k8s worker node restart In Progress , replay done, mail sent to the community (network issue?)
- behavor different on Nokia (pod stuck for ever in Terminating state when stopping a working) and Samsung RKE2 cluster, continue discussion on the ticket Bartek Grzybowski indicates that we shall be careful especially with statefulset, it coudl explain some issues
- #action morgan retry and wait for the 5 minutes (Eviction timeout) then investigate on the networking issue
stability tests
- SDC tests: 72h 5 parallel on boarding started on the 20th of April..wait and see...
- First tests were all OK but duration ontinuously increases. No error rate is high. Wait for the end of the tests to get the graphs.
- #action Morgan Richomme grant access to Christophe Closset to check the status of SDC/DB
- instantiation tests to be planned after onboarding tests
- SDC tests: 72h 5 parallel on boarding started on the 20th of April..wait and see...
consolidation of versions
better exception catching + integration of the GUI => still action Morgan
removal of the submodule: done
- pnf_macro => OK on daily guilin, what do we do regarding CI integration
- documentation update started but not finished => https://gerrit.onap.org/r/c/integration/+/120236
- release testsuite 1.8.0 to be done?
- next week if no objection
- simulators: whuch dockers are available in nexus today? could we clarify the build chain?
- see previous topic
- Istanbul
- New Jira board created: https://jira.onap.org/secure/RapidBoard.jspa?rapidView=233
- discussion on future tests
- follow up integration AAI tests in CSIT
- CDS in CSIT?
- AoB
- Andreas Geissler mentioned that his CI chains were affected by the non availability of an ansible image used in the runner => could be a problem, need to release also such images somewhere to avoid such issues #action Morgan Richomme check ansible image availability for chained-ci (2.7.13)
- issue also on Orange chains last week (build chain of ansible image was broken, chain fixed)
- Christophe Closset raises a question on python2.7 in jenkins CI and in robot. Partly addressed by the initiative of robot refactoring. note xtesting dockers also still using python2.7 due to dependency to python-utils that has not evolved for a long time.
- Andreas Geissler mentioned that his CI chains were affected by the non availability of an ansible image used in the runner => could be a problem, need to release also such images somewhere to avoid such issues #action Morgan Richomme check ansible image availability for chained-ci (2.7.13)
...