- Created by Morgan Richomme, last modified on Mar 07, 2022
Sometimes you need to troubleshoot the errors reported during a daily chain
You must first connect to the daily lab.
The controller can be reached over SSH master.onap.eu, istanbul.onap.eu, master-weekly.onap.eu.(you SSH key must obviously be referenced see How to add keys to access integration labs?).
Once connected, it is is a "usual ONAP" lab, so you can run your test as in any ONAP platform.
All the configurations of the tests executed during the daily chains can be found under /tmp/xtesting
tests Expand source
debian@control01-onap-master:~$ cd /tmp/xtesting
debian@control01-onap-master:/tmp/xtesting$ tree -L 2
.
├── config
├── infrastructure-healthcheck
│ ├── k8s
│ └── k8s-teardown
├── security
│ ├── jdpw_ports
│ ├── kube_hunter
│ ├── nonssl_endpoints
│ ├── root_pods
│ ├── unlimitted_pods
│ └── versions
├── smoke-usecases
│ ├── basic_cds
│ ├── basic_clamp
│ ├── basic_cnf
│ ├── basic_cnf_macro
│ ├── basic_network
│ ├── basic_onboard
│ ├── basic_vm
│ ├── basic_vm_macro
│ ├── cds_resource_resolution
│ └── pnf_macro
├── xtesting-healthcheck
│ ├── core
│ ├── cps-healthcheck
│ ├── full
│ ├── healthdist
│ └── postinstall
└── xtesting-smoke-usecases-robot
├── cmpv2
├── dcaemod
├── hv-ves
├── pnf-registrate
└── ves-collector
33 directories, 1 file
The tests are run using 2 main mechanisms:
- k8s job in the onap namespace
- docker executed from the controller node
How to replay a kubernetes job?
k8s test Expand source
debian@control01-onap-master:~$ kubectl get job -n onap |grep integration
integration-onap-cmpv2 0/1 9d 9d
integration-onap-core 1/1 2m6s 9d
integration-onap-cps-healthcheck 1/1 32s 9d
integration-onap-dcaemod 0/1 9d 9d
integration-onap-full 1/1 2m50s 9d
integration-onap-healthdist 1/1 5m58s 9d
integration-onap-hv-ves 1/1 4m18s 9d
integration-onap-internal-check-certs 0/1 9d 9d
integration-onap-pnf-registrate 1/1 7m36s 9d
integration-onap-postinstall 1/1 101s 9d
integration-onap-ves-collector 1/1 11m 9d
# Delete the job of the test you want to replay
debian@control01-onap-master:~$ kubectl delete job -n onap integration-onap-cps-healthcheck
job.batch "integration-onap-cps-healthcheck" deleted
# Recreate the job (it will automatically relaunch the test pod)
debian@control01-onap-master:~$ cd oom/master/
debian@control01-onap-master:~/oom/master$ ls
deployment.yaml healthcheck-cps-healthcheck.yaml healthcheck-healthdist.yaml healthcheck-pnf-registrate.yaml onap-overrides.yaml
healthcheck-cmpv2.yaml healthcheck-dcaemod.yaml healthcheck-hv-ves.yaml healthcheck-postinstall.yaml xtesting-healthcheck
healthcheck-core.yaml healthcheck-full.yaml healthcheck-internal-check-certs.yaml healthcheck-ves-collector.yaml xtesting-smoke-usecases-robot
kubectl apply -f healthcheck-cps-healthcheck.yaml --validate=false
# Then look at the logs
kubectl logs -n onap integration-onap-cps-healthcheck--1-m7tbt -f
2022-03-07 13:27:41,030 - xtesting.ci.run_tests - INFO - Deployment description:
+-------------------------+----------------------------------------------------------+
| ENV VAR | VALUE |
+-------------------------+----------------------------------------------------------+
| CI_LOOP | daily |
| DEBUG | false |
| DEPLOY_SCENARIO | onap-nofeature-noha |
| INSTALLER_TYPE | oom |
| BUILD_TAG | gitlab_ci-functest-kubespray-baremetal-daily- |
| | master-479785435-onap |
| NODE_NAME | onap_weekly_pod4_master-ONAP-oom |
| TEST_DB_URL | http://testresults.opnfv.org/onap/api/v1/results |
| TEST_DB_EXT_URL | |
| S3_ENDPOINT_URL | |
| S3_DST_URL | |
| HTTP_DST_URL | |
+-------------------------+----------------------------------------------------------+
2022-03-07 13:27:41,061 - xtesting.ci.run_tests - INFO - Loading test case 'cps-healthcheck'...
2022-03-07 13:27:41,217 - xtesting.ci.run_tests - INFO - Running test case 'cps-healthcheck'...
2022-03-07 13:27:41,321 - xtesting.core.robotframework - INFO -
==============================================================================
Actuator :: CPS - Actuator endpoints
==============================================================================
Test CPS Enhanced Healthcheck :: Runs CPS Health Check. It will ch... | PASS |
------------------------------------------------------------------------------
Actuator :: CPS - Actuator endpoints | PASS |
1 critical test, 1 passed, 0 failed
1 test total, 1 passed, 0 failed
==============================================================================
Output: /var/lib/xtesting/results/cps-healthcheck/output.xml
2022-03-07 13:27:41,326 - xtesting.core.robotframework - INFO - Results were successfully parsed
2022-03-07 13:27:41,395 - xtesting.core.robotframework - INFO - Results were successfully generated
2022-03-07 13:27:41,395 - xtesting.ci.run_tests - INFO - Test result:
+-------------------------+---------------------+------------------+----------------+
| TEST CASE | PROJECT | DURATION | RESULT |
+-------------------------+---------------------+------------------+----------------+
| cps-healthcheck | integration | 00:00 | PASS |
+-------------------------+---------------------+------------------+----------------+
2022-03-07 13:27:41,713 - xtesting.core.testcase - INFO - The results were successfully pushed to DB:
http://testresults.opnfv.org/onap/api/v1/results/6226084d6c44f5001087e92d
2022-03-07 13:27:41,714 - xtesting.ci.run_tests - INFO - Execution exit value: Result.EX_OK
# Note by default it will push the results to the test DB..if you do not want, edit the manifest and remove the -- report option
- args:
- --test
- cps-healthcheck
- --report
How to replay a docker based test?
Several tests are executed by running a docker file on the controller. It is used for
- infrastructure tests
- security tests
- pythonsdk-test tests
docker tests Expand source
# Select the test you want to replay (e.g. basic_cds) debian@control01-onap-master:~$ cd /tmp/xtesting//smoke-usecases/basic_cds/ # run the docker debian@control01-onap-master:/tmp/xtesting/smoke-usecases/basic_cds$ cat /etc/hosts |grep onap.org && docker run -it --env-file env -v /home/debian/.kube/config:/root/.kube/config -v /home/debian/.config/openstack/clouds.yaml:/root/.config/openstack/clouds.yaml -v /tmp/xtesting/config:/usr/lib/python3.8/site-packages/onaptests/templates/artifacts/config -v /tmp/test:/var/lib/xtesting/results nexus3.onap.org:10003/onap/xtesting-smoke-usecases-pythonsdk:master bash # you shall be in the docker # for infrastructure and security tests you can directly run the test # for pythonsdk you must copy/paste the content of the /etc/hosts (you could also use parameter in the docker exec command if you want), as onap url are not publicaly routable # bash-5.1# vi /etc/hosts # copy the content displayed by the command cat /etc/hosts |grep onap.org bash-5.1# cat /etc/hosts 127.0.0.1 localhost ::1 localhost ip6-localhost ip6-loopback fe00::0 ip6-localnet ff00::0 ip6-mcastprefix ff02::1 ip6-allnodes ff02::2 ip6-allrouters 172.17.0.2 265c7c83aaf0 10.253.0.217 portal.api.simpledemo.onap.org 10.253.0.217 vid.api.simpledemo.onap.org 10.253.0.217 sdc.api.fe.simpledemo.onap.org 10.253.0.217 sdc.api.be.simpledemo.onap.org 10.253.0.217 portal-sdk.simpledemo.onap.org 10.253.0.217 policy.api.simpledemo.onap.org 10.253.0.217 aai.api.sparky.simpledemo.onap.org 10.253.0.217 cli.api.simpledemo.onap.org 10.253.0.217 msb.api.simpledemo.onap.org 10.253.0.217 so.api.simpledemo.onap.org 10.253.0.217 appc.api.simpledemo.onap.org 10.253.0.217 sdnc.api.simpledemo.onap.org 10.253.0.217 nbi.api.simpledemo.onap.org 10.253.0.217 consul.api.simpledemo.onap.org 10.253.0.217 kibana.api.simpledemo.onap.org 10.253.0.217 mr.api.simpledemo.onap.org 10.253.0.217 uui.api.simpledemo.onap.org 10.253.0.217 aaf.api.simpledemo.onap.org 10.253.0.217 clamp.api.simpledemo.onap.org 10.253.0.217 robot.api.simpledemo.onap.org 10.253.0.217 dcae.api.simpledemo.onap.org 10.253.0.217 sdc.workflow.plugin.simpledemo.onap.org 10.253.0.217 sdc.dcae.plugin.simpledemo.onap.org # execute the xtesting command bash-5.1# run_tests -t basic_cds 2022-03-07 13:46:13,968 - xtesting.ci.run_tests - INFO - Deployment description: +-------------------------+----------------------------------------------------------+ | ENV VAR | VALUE | +-------------------------+----------------------------------------------------------+ | CI_LOOP | daily | | DEBUG | False | | DEPLOY_SCENARIO | os-nosdn-nofeature-noha | | INSTALLER_TYPE | oom | | BUILD_TAG | gitlab_ci-functest-kubespray-baremetal-daily- | | | master-479785435-onap | | NODE_NAME | onap_weekly_pod4_master-ONAP-oom | | TEST_DB_URL | http://testresults.opnfv.org/onap/api/v1/results | | TEST_DB_EXT_URL | | | S3_ENDPOINT_URL | | | S3_DST_URL | | | HTTP_DST_URL | | +-------------------------+----------------------------------------------------------+ 2022-03-07 13:46:14,013 - xtesting.ci.run_tests - INFO - Loading test case 'basic_cds'... 2022-03-07 13:46:14,822 DEBUG 25:cds_blueprint_enrichment.py(13) - CDS blueprint enrichment initialization 2022-03-07 13:46:14,839 DEBUG 219:extension.py(13) - found extension EntryPoint(name='basic_clamp', value='onaptests.scenario.basic_clamp:BasicClamp', group='xtesting.testcase') [....] 2022-03-07 13:46:33,393 INFO 176:run_tests.py(13) - Test result: +-------------------+---------------------+------------------+----------------+ | TEST CASE | PROJECT | DURATION | RESULT | +-------------------+---------------------+------------------+----------------+ | basic_cds | integration | 00:18 | PASS | +-------------------+---------------------+------------------+----------------+ 2022-03-07 13:46:33,397 INFO 51:cds_blueprint_enrichment.py(13) - Generate Test report 2022-03-07 13:46:33,477 - xtesting.ci.run_tests - INFO - Execution exit value: Result.EX_OK 2022-03-07 13:46:33,477 INFO 275:run_tests.py(13) - Execution exit value: Result.EX_OK # note 1: the results are in /tmp/reporting.html and the logs in /tmp/pythonsdk.debug.log # note 2: the env var set in the env file shall corresponds to the test you run e.g for basic_cds => ONAP_PYTHON_SDK_SETTINGS=onaptests.configuration.cba_enrichment_settings # note 3 the name of the test is in the test DB and defined as the entry point of xtesting https://github.com/onap/testsuite-pythonsdk-tests/blob/master/setup.cfg # note 4, if you want to play directly with the python test or sdk code, it is respectively under /usr/lib/python3.8/site-packages/onaptests and /usr/lib/python3.8/site-packages/onapsdk bash-5.1# ls /usr/lib/python3.8/site-packages/onaptests __init__.py __pycache__ configuration scenario steps templates utils bash-5.1# ls /usr/lib/python3.8/site-packages/onapsdk __init__.py aai clamp constants.py exceptions.py nbi sdc so version.py vid __pycache__ cds configuration dmaap msb onap_service.py sdnc utils ves
- No labels