Introduction

The 72 hour Stability Test for Drools PDP has the goal of introducing a steady flow of transactions initiated from a Test Client server running JMeter. The onset messages will be submitted through REST interface of Drools PDP. The results are verified through REST interface of Drools PDP using facts, source & sink topics. The JMeter testing will exercise the following use cases:

vCPE
vFirewall
vDNS
VOLTE

The stability test will loop through these use cases sequentially in two threaded environment (for each use case) for the duration of 72 hours.

Setup details

The stability test is performed on VM's running in OpenStack cloud environment. There are 2 seperate VM's, one for running policy framework & other one for running JMeter to simulate steady flow of transactions. For this test, the components involved in the flow outside of Policy (i.e. VFC, A&AI & SO) will be simulated using "features enable controlloop-utils".

OpenStack environment details

Version: Mitaka 9.2

Policy Framework VM details

OS: Linux 4.4.0-119-generic #143-Ubuntu SMP
CPU: 4core
RAM: 8GB
HardDisk: 80GB
Docker Version: 18.03.0-ce, build 0520e24
Java: openjdk version "1.8.0_162"

JMeter VM details

OS: Linux 4.4.0-119-generic #143-Ubuntu SMP
CPU: 4core
RAM: 8GB
HardDisk: 80GB
Java: openjdk version "1.8.0_162"

Install JMeter in virtual machine

Make the etc/hosts entries

echo $(hostname -I | cut -d\  -f1) $(hostname) | sudo tee -a /etc/hosts


Make the DNS entries

echo "nameserver <PrimaryDNSIPIP>" >> /etc/resolvconf/resolv.conf.d/head
 
echo "nameserver <SecondaryDNSIP>" >> /etc/resolvconf/resolv.conf.d/head
 
resolvconf -u


Update the ubuntu software installer

apt-get update


Check & Install Java

apt-get install -y openjdk-8-jdk
 
java -version


Download & install JMeter

mkdir jMeter


cd jMeter


wget http://mirrors.whoishostingthis.com/apache//jmeter/binaries/apache-jmeter-4.0.zip


unzip apache-jmeter-4.0.zip 


Follow the instructions in this link for installing plugin manager - https://jmeter-plugins.org/wiki/PluginsManager/


Run JMeter

/home/ubuntu/jMeter/apache-jmeter-4.0/bin/jmeter

The above command will load the JMeter UI. Then navigate to File → Open → Browse and select the test plan jmx file to open.

Install ONAP Policy Framework in virtual machine

Follow the steps mentioned in the below link to install ONAP Policy Framework with controllers & policies in a virtual machine.

ONAP BEIJING Policy Framework: Standalone installation in Virtual Machine - TO BE DELETED

Install & Configure VisualVM

VisualVM needs to be installed in the virtual machine having policy framework. It will be used to monitor CPU, Memory, GC for drools PDP while stability test is running.


Install visualVM

sudo apt-get install visualvm


Login to docker container (using root)

docker exec -e -it -u 0 drools su - root


Run few commands to configure permissions

cd /usr/lib/jvm/java-8-openjdk-amd64/bin/ 

touch visualvm.policy 

vi visualvm.policy 

Add the following in visualvm.policy


grant codebase "file:/usr/lib/jvm/java-8-openjdk-amd64/lib/tools.jar" {
   permission java.security.AllPermission;
};


chmod 777 visualvm.policy


exit


Login to docker container (using policy)

docker exec -e -it -u 0 drools su - policy


Run following commands to start jstatd using port 1111

cd /usr/lib/jvm/java-8-openjdk-amd64/bin/ 


./jstatd -p 1111 -J-Djava.security.policy=visualvm.policy  &


exit


Login to VM using graphical interface in separate terminal window.

ssh -X <user>@<VM-IP-ADDRESS>


Open visualVM

visualvm &


Connect to jstatd & remote drools PDP JVM

  1. Right click on "Remote" in the left panel of the screen and select "Add Remote Host..."

  2. Enter the IP address of drools docker container. Run the following command to check the IP address of drools container in VM.

     docker inspect --format '{{ .NetworkSettings.Networks.docker_default.IPAddress}}' drools
  3. Right click on IP address, select "Add jstatd Connection..."
  4. In "jstatd Connections" tab, enter port 1111 and click OK.
  5. Right click on IP address, select "Add JMX Connection..."
  6. Enter the drools docker container IP Address (from step 2) <IP address>:9991 ( for example - 172.18.0.7:9991) and click OK.
  7. Double click on the newly added nodes under "Remote" to start monitoring CPU, Memory & GC.


Sample Screenshot of visualVM

Test Plan

The 72 hours stability test will run the following steps one by one sequentially in two threaded loop.

  • Check facts in memory - Before starting the test, we need to make sure that there are no stuck events in engine.
  • vCPE controller - The vCPE controller is responsible for trigerring the vCPE success test cases.

  • VCPE VNF ID Generator - generates a new VNF ID for every request.

  • vCPE Request ID Generator - generates a new Request ID for every request.

  • vCPE ONSET Request - sends a vCPE onset message to REST interface of drools PDP.

  • vCPE APPC Success Response - sends the APPC success response message to REST interface of drools PDP.

  • vCPE ABATEMENT Request - sends the vCPE ABATEMENT message to REST interface of drools PDP.

  • vCPE Assertion Controller - This controller is responsible for performing assertions for the request sent earlier.

  • vCPE Sink Events - fetch all the events from sink topic POLICY-CL-MGT and then verify the FINAL SUCCESS notification against the request ID of vCPE onset message sent earlier.

  • vCPE Failure Controller - The vCPE controller is responsible for trigerring the vCPE failure test cases.

  • vCPE Failure VNF ID Generator - generates a new VNF ID for every request.

  • vCPE Failure Request ID Generator - generates a new Request ID for every request.

  • vCPE Failure ONSET Request - sends a vCPE onset message to REST interface of drools PDP.

  • vCPE Failure APPC Response - sends the APPC failure response message to REST interface of drools PDP.

  • vCPE Failure Assertion Controller - This controller is responsible for performing assertions for the request sent earlier.

  • vCPE Failure Sink Events - fetch all the events from sink topic POLICY-CL-MGT and then verify the FINAL FAILURE notification against the request ID of vCPE onset message sent earlier.

  • vDNS Controller - The vDNS controller is responsible for trigerring the vDNS test flow.

  • vDNS vServer Name Generator - generates a new vServer name for every request.

  • vDNS Request ID Generator - generates a new Request ID for every request.

  • vDNS ONSET Request - sends a vDNS onset message to REST interface of drools PDP.

  • vDNS Assertion Controller - This controller is responsible for performing assertions for the request sent earlier.

  • vDNS Sink Events - fetch all the events from sink topic POLICY-CL-MGT and then verify the FINAL SUCCESS notification against the request ID of vDNS onset message sent earlier.

  • vFW controller - The vFW controller is responsible for trigerring the vFirewall test flow.

  • vFW VNF ID Generator - generate a new VNF ID for every request.

  • vFW Request ID Generator - generate a new Request ID for every request.

  • vFW ONSET Request - sends a vFirewall onset message to REST interface of drools PDP.

  • vFW APPC Legacy Success Response - sends the APPC success response message to REST interface of drools PDP.

  • vFW Assertion Controller - This controller is responsible for performing assertions for the request sent earlier.

  • vFW Sink Events - fetch all the events from sink topic POLICY-CL-MGT and then verify the FINAL SUCCESS notification against the request ID of vFirewall onset message sent earlier.

  • vFW Failure Controller - The vFW controller is responsible for trigerring the vFirewall failure test cases.

  • vFW Failure VNF ID Generator - generates a new VNF ID for every request.

  • vFW Failure Request ID Generator - generates a new Request ID for every request.

  • vFW Failure ONSET Request - sends a vFirewall onset message to REST interface of drools PDP.

  • vFW Failure APPC Response - sends the APPC failure response message to REST interface of drools PDP.

  • vFW Failure Assertion Controller - This controller is responsible for performing assertions for the request sent earlier.

  • vFW Failure Sink Events - fetch all the events from sink topic POLICY-CL-MGT and then verify the FINAL FAILURE notification against the request ID of vFirewall onset message sent earlier.

  • VOLTE Controller - The VOLTE controller is responsible for triggering the VOLTE test flow.

  • VOLTE Request ID Generator - generates a new Request ID for every request.

  • VOLTE vServer Name Generator - generates a new vServer name for every request.

  • VOLTE VNF ID Generator - generates a new VNF ID for every request.

  • VOLTE ONSET Request - sends a VOLTE onset message to REST interface of drools PDP.

  • VOLTE Assertion Controller - This controller is responsible for performing assertions for the request sent earlier.

  • VOLTE Sink Events - fetch all the events from sink topic POLICY-CL-MGT and then verify the FINAL SUCCESS notification against the request ID of VOLTE onset message sent earlier.

  • Check facts in memory - Before ending the test, we need to make sure that there are no stuck events in engine.


The following steps can be used to configure the parameters of test plan.

  • HTTP Authorization Manager - used to store user credentials which will be used for making HTTP requests.
  • HTTP Request Defaults -  used to store HTTP request details like Server Name or IP, Port, Protocol etc.

  • User Defined Variables -  used to store following parameters.

    NameDescriptionDefault Value
    sizeIndicates the maximum number of VNF instances. Used for generating VNF IDs.2000
    waitWait time after each request so that the same gets processed by engine (in milliseconds)500
    vCPEControlLoopvCPE Control loop nameControlLoop-vCPE-48f0c2c3-a172-4192-9ae3-052274181b6e
    vDNSControlLoopvDNS Control loop nameControlLoop-vDNS-6f37f56d-a87d-4b85-b6a9-cc953cf779b3
    vFWControlLoopvFirewall Controll loop nameControlLoop-vFirewall-d0a1dfc6-94f5-4fd4-a5b5-4630b438850a
    VOLTEControlLoopVOLTE Control loop nameControlLoop-VOLTE-2179b738-fd36-4843-a71a-a8c24c70c55b
    threadsNumber of threads to run test cases in parallel. It should not be more than 2 as the sink queue size is 10.2
    threadsTimeOutInMsSynchronization timer for threads running in parallel (in milliseconds).5000

                 


Screenshot of drools PDP stability test plan



Important Note: There is wait introduced after sending every ONSET, Response, Request messages to drools PDP so that it is able to process the message before getting the next one. If needed, the wait time can be suitably adjusted based on the setup and time taken by drools PDP to process the messages.

  • No labels