References

Attention To

Discussion

Need a better end-to-end test (that exercises model loader/babel, haproxy, resources, traversal, graphadmin)

Need to include testing for services that aren't in the critical flow (sparky, search-data-service, data-router, spike, champ, gizmo, etc)

Suggestions

8th Jan: Testing should cover this case: MSB-198 - Getting issue details... STATUS

20th Feb: Move from CSIT to Robot. See PTL 2019-02-19 presentation https://wiki.onap.org/download/attachments/53249527/Dev%20Testing_Enhancement%20for%20Dublin_V2.pptx?version=1&modificationDate=1550580552000&api=v2

21st Feb:

4th Mar:

  • Trial for CCVPN use case  AAI-2208 - Getting issue details... STATUS

4th Mar:

  • Updated suggestions from gerrit reviews https://gerrit.onap.org/r/79583:

    • Brian Freeman

      if the intent is to have a aai-backwards-compatibility-v14.robot function that would get added to each release as the version number increases then I guess you need to create aai-[something]-v14, aai-something]-v16 etc. Not sure these are end to end tests becuase we arent testing that a Dublin vFW works end to end with v14 API's when most components will be calling the v16 APIs. If its simply AAI regression as part of verify/merge CSIT jobs then we should have a nomenclature with CSIT in the filename to distribution End to End tests from CSIT tests. aai-csit-v14 , aai-csit-v11 ?
      Technically even Amsterdam had a mixture of versions supported so it seems like you may want to be testing by version not by release package name ?

    • Daniel Rose

      i looked over catherines presentation. I think if we are going to support csit type tests, i am ok. i think we need to put them in testsuites/<app> and tag those tests as csit. I still also dont think we should put release specific code in here though. I think supporting 2 versions of aai schema is ok if you ar releasing that, but calling something as casablanca is not - the branch should always work with the release it is.

    • Keong Lim
      AAI team did have an idea that our tests could be used to diagnose installation problems, e.g. pulling the wrong version and wondering why it doesn't work. So as part of the "sanity integration" testing, we would publish these tools/scripts for use on any installation (development, test labs, production, etc), not just the latest.

      I will update a new patchset.

      The naming can be updated. Wasn't sure if the release names were more familiar than the API version numbers.


Setup Procedure

Follow testsuite Readme for setup: https://gerrit.onap.org/r/gitweb?p=testsuite.git;a=blob;f=README.md;hb=HEAD

Python Installation

As of 25th Feb 2019, Python 2.7.12 was replaced by 2.7.15 so that version was installed instead. See also the note about deprecation in the python output below.

Installing pip and robotframework 
>python get-pip.py
DEPRECATION: Python 2.7 will reach the end of its life on January 1st, 2020. Ple
ase upgrade your Python as Python 2.7 won't be maintained after that date. A fut
ure version of pip will drop support for Python 2.7.
Collecting pip
  Downloading https://files.pythonhosted.org/packages/d8/f3/413bab4ff08e1fc4828d
fc59996d721917df8e8583ea85385d51125dceff/pip-19.0.3-py2.py3-none-any.whl (1.4MB)

    100% |UUUUUUUUUUUUUUUUUUUUUUUUUUUUUUUU| 1.4MB 1.5MB/s
Collecting wheel
  Downloading https://files.pythonhosted.org/packages/96/ba/a4702cbb6a3a485239fb
e9525443446203f00771af9ac000fa3ef2788201/wheel-0.33.1-py2.py3-none-any.whl
Installing collected packages: pip, wheel
  Found existing installation: pip 9.0.3
    Uninstalling pip-9.0.3:
      Successfully uninstalled pip-9.0.3
Successfully installed pip-19.0.3 wheel-0.33.1

Robotframework Installation

Installing pip and robotframework 
>pip install robotframework
DEPRECATION: Python 2.7 will reach the end of its life on January 1st, 2020. Ple
ase upgrade your Python as Python 2.7 won't be maintained after that date. A fut
ure version of pip will drop support for Python 2.7.
Collecting robotframework
  Downloading https://files.pythonhosted.org/packages/36/c6/6f89c80ac5a526a091bd
383ffdfc64c9a68d9df0c775d4b36f03d8e0ac25/robotframework-3.1.1-py2.py3-none-any.w
hl (601kB)
    100% |UUUUUUUUUUUUUUUUUUUUUUUUUUUUUUUU| 604kB 1.6MB/s
Installing collected packages: robotframework
Successfully installed robotframework-3.1.1

As of 25th Feb 2019, Nokia Robot Editor (RED) 0.7.0 was replaced by 0.8.11, so that version was installed instead: https://github.com/nokia/RED/releases/

Note that the "product" contains a stand-alone copy of Eclipse, whereas the "feature" is an add-on to a pre-existing installation of Eclipse: https://github.com/nokia/RED/blob/master/installation.md

Setup.sh

Looking into https://gerrit.onap.org/r/gitweb?p=testsuite.git;a=blob;f=setup.sh;hb=HEAD

Due to local differences in my environment, copied the "pip install" commands and ran them individually instead of the whole setup.sh script.

On the heatbridge setup, got "error: Microsoft Visual C++ 9.0 is required. Get it from http://aka.ms/vcpython27" which needs fixing.

chromedriver

As of 25th Feb 2019, chromedriver 2.43 was replaced by 2.46, so that version was installed instead: http://chromedriver.storage.googleapis.com/index.html?path=2.46/

Other

The setup for kafkacat and protobuf does not appear to support Windows, so these were not installed on this environment.

Using RED

Clone testsuite repository, see https://gerrit.onap.org/r/#/admin/projects/testsuite

In RED, use "File / Open Projects from File System" menu and choose the "testsuite/robot" sub-directory.

The file format is described in http://robotframework.org/robotframework/latest/RobotFrameworkUserGuide.html#supported-file-formats

RED screenshot

Settings section

See also http://robotframework.org/robotframework/latest/RobotFrameworkUserGuide.html#all-available-settings-in-test-data

The Settings section appears to have some references to other files as "Library" or "Resource". These files act as shared modules implementing common sub-routines. The "Documentation" appears to be a comment for the file.

Variables section

See also http://robotframework.org/robotframework/latest/RobotFrameworkUserGuide.html#variables

This section appears to define variables or macros with values that can be used in the "Keywords" section. They are potentially available to other Robot scripts as well, see also the "global_properties" script.

It appears that all variables share the global namespace, so naming conventions are used to distinguish between variables that are intended for global use (prefixed by "GLOBAL" and written in all-caps) vs variables intended for local use.

Keywords section

See also http://robotframework.org/robotframework/latest/RobotFrameworkUserGuide.html#creating-user-keywords

This section appears to define sub-routines that can be called by other "keywords". The "[Documentation]" appears to be a comment for the sub-routine. The "[Arguments]" appears to be a list of named parameters passed into the sub-routine. The lines that follow appear to be calls to other sub-routines in sequential order. Values can be returned from sub-routines and internal structures of the values can also be extracted and processed.

Worked Examples

Running Robot

  • In demo repository, "boot/robot/ete.sh" executes the "runTags.sh" script inside the openecompete_container
  • In oom repository, "kubernetes/robot/ete-k8s.sh" executes the "runTags.sh" script inside robot pod
  • In optf repository, "cmso/cmso-robot/docker/cmso-service/ete_test.sh" executes "docker-compose" to run the robot container
  • In optf repository, "cmso/cmso-robot/ete.sh" executes the ROBOT_CMD="python -m robot.run" using the older method for Python 2.7 compatibility
  • In testsuite repository, "runEteTag.sh" executes "python -m robot.run" connecting to an X-Windows display created by Xvfb and captures the command line output into a file

The robot command line needs:

  • Path to robot library modules
  • Path to robot variables file
  • Tags of the tests to run
  • A log-level setting
  • An output file
  • An input directory

See also "robot --help" or http://robotframework.org/robotframework/latest/RobotFrameworkUserGuide.html#id882

For example

Robot command line
# Using Python 2.7 run-time
python -m robot.run \
  --loglevel ${LOG_LEVEL}    # aka -L option \
  --outputdir ${OUTPUT_DIR}  # aka -d option \
  --variablefile ${VARIABLE_FILE}  # aka -V option \
  --pythonpath ${PATH_DIRECTORIES} # aka -P option \
  --include ${TAGS_INCLUDED} # aka -i option \
  --exclude ${TAGS_EXCLUDED} # aka -e option \
  ${DATA_SOURCES_DIRECTORY}

robot will create an output XML file, a log HTML file and a report HTML file by default.

robot can also create documentation using the "python -m robot.libdoc" command.

Simple example

Simple Robot command line for AAI
cd testsuite  # cloned from git/gerrit repository


# run the AAI scripts from the right directory
robot --argumentfile my-aai.txt robot/testsuites/aai/

# output looks like this
==============================================================================
Aai
==============================================================================
Aai.Aai-Regression-Test-V14 :: AAI CSIT-style regression tests for CCVPN - ...
==============================================================================
Connectivity test case                                                | PASS |
------------------------------------------------------------------------------
Aai.Aai-Regression-Test-V14 :: AAI CSIT-style regression tests for... | PASS |
1 critical test, 1 passed, 0 failed
1 test total, 1 passed, 0 failed
==============================================================================
Aai                                                                   | PASS |
1 critical test, 1 passed, 0 failed
1 test total, 1 passed, 0 failed
==============================================================================
Output:  D:\ONAP\testsuite\testsuite\output.xml
Log:     D:\ONAP\testsuite\testsuite\log.html
Report:  D:\ONAP\testsuite\testsuite\report.html

# argumentfile my-aai.txt contains
--pythonpath robot/library/:robot/library/eteutils
--variable GLOBAL_AAI_USERNAME:AAI
--variable GLOBAL_AAI_PASSWORD:AAI
--variable GLOBAL_AAI_SERVER_PROTOCOL:https
--variable GLOBAL_INJECTED_AAI1_IP_ADDR:xxx.yyy.zzz.aaa
--variable GLOBAL_AAI_SERVER_PORT:30233
--loglevel DEBUG



# for debugging of error messages, use a browser to view log.html and report.html

20th June: See also update to Jinja templating:

26th June: Robot testing for Sparky sanity tests

  • Use AAI robot scripts to create an object, e.g. vpn-binding, i.e. do not do Teardown steps that delete the objects at the end of the test
  • Login to Portal and navigate to AAI UI (Sparky)
  • Use Sparky to browse the object, thus providing minimum test of sparky-fe, sparky-be, search-data-service microservices
  • See attachments on 2019-06-20 AAI Developers Meeting


Add to hosts file:

<ONAP.IP.address> portal.api.simpledemo.onap.org
<ONAP.IP.address> vid.api.simpledemo.onap.org
<ONAP.IP.address> sdc.api.fe.simpledemo.onap.org
<ONAP.IP.address> portal-sdk.simpledemo.onap.org
<ONAP.IP.address> policy.api.simpledemo.onap.org
<ONAP.IP.address> aai.api.sparky.simpledemo.onap.org
<ONAP.IP.address> cli.api.simpledemo.onap.org
<ONAP.IP.address> aaf-onap-test.osaaf.org 
<ONAP.IP.address> aai.ui.simpledemo.onap.org

Install onap_root.crt into Windows and Firefox trusted authorities
Visit portal website: https://portal.api.simpledemo.onap.org:30225/ONAPPORTAL/login.htm