The APPC OOM deployment consists of a 3 node ODL Cluster plus a DB node. The details for the environment can be found at APPC WindRiver Lab. From a testing perspective a directory has been created on the Master APPC node (10.12.5.171). The directory is called "testing."
testing directory
ubuntu@k8s-master:~/testing$ ls -l total 36 drwxrwxr-x 10 ubuntu ubuntu 4096 May 5 00:11 apache-jmeter-4.0 drwxrwxr-x 3 ubuntu ubuntu 4096 May 4 15:16 ONAP-Testing -rw-rw-r-- 1 ubuntu ubuntu 1039 May 4 12:32 rebuild-VM3.json -rw-rw-r-- 1 ubuntu ubuntu 755 May 1 16:40 restart-test-dmaap.json -rw-rw-r-- 1 ubuntu ubuntu 1041 May 4 15:19 snapshot-VM3.json -rw-rw-r-- 1 ubuntu ubuntu 1031 May 4 12:11 start-VM2.json -rw-rw-r-- 1 ubuntu ubuntu 1031 May 4 11:50 stop-VM2.json -rw-rw-r-- 1 ubuntu ubuntu 1004 May 4 11:50 test.json -rw-rw-r-- 1 ubuntu ubuntu 675 Apr 30 17:56 test-restart.json ubuntu@k8s-master:~/testing/apache-jmeter-4.0/APPC-Tests$ ls -l total 167696 -rw-rw-r-- 1 ubuntu ubuntu 104327 May 5 20:34 APPC-LCM-Action-4272018.jmx -rw-rw-r-- 1 ubuntu ubuntu 48609 May 4 14:53 APPC-LCM-Action-Generic_Restart.jmx -rw-rw-r-- 1 ubuntu ubuntu 48673 May 5 20:42 APPC-LCM-Action-Rebuild.jmx -rw-rw-r-- 1 ubuntu ubuntu 56469 May 5 00:37 APPC-LCM-Action-Restart.jmx -rw-rw-r-- 1 ubuntu ubuntu 56425 May 4 14:53 APPC-LCM-Action-Start.jmx -rw-rw-r-- 1 ubuntu ubuntu 56406 May 4 18:02 APPC-LCM-Action-Stop.jmx -rw-rw-r-- 1 ubuntu ubuntu 95 May 4 23:48 appc-lcm-test.csv -rw-rw-r-- 1 ubuntu ubuntu 95 May 4 23:48 appc-lcm-test-Stability-Test-VM2.csv -rw-rw-r-- 1 ubuntu ubuntu 95 May 4 23:48 appc-lcm-test-Stability-Test-VM3.csv -rw-rw-r-- 1 ubuntu ubuntu 1451122 May 7 09:23 appc-oom-72hr.jtl -rw-rw-r-- 1 ubuntu ubuntu 2695327 May 7 09:25 appc-oom-rebuild-72hr.jtl -rw-rw-r-- 1 ubuntu ubuntu 167156886 May 7 09:30 jmeter.log drwxr-xr-x 2 root root 4096 May 4 23:48 temp -rw-r--r-- 1 root root 159 May 4 23:48 test-scott.jtl ubuntu@k8s-master:~/testing/ONAP-Testing$ ls -l total 12 drwxrwxr-x 2 ubuntu ubuntu 4096 May 4 15:17 keys -rw-rw-r-- 1 ubuntu ubuntu 26 May 4 15:17 login.txt drwxrwxr-x 5 ubuntu ubuntu 4096 May 4 15:16 Test Data ubuntu@k8s-master:~/testing/ONAP-Testing/Test Data$ ls -l total 24 drwxrwxr-x 4 ubuntu ubuntu 4096 May 4 15:17 APPC-Stability-Test -rw-rw-r-- 1 ubuntu ubuntu 999 May 4 15:16 Stop-Stability-Test-VM1-VM.json drwxrwxr-x 2 ubuntu ubuntu 4096 May 4 15:16 vCPE drwxrwxr-x 2 ubuntu ubuntu 4096 May 4 15:16 vDNS -rw-rw-r-- 1 ubuntu ubuntu 6144 May 4 15:16 vTESTVM1.xls ubuntu@k8s-master:~/testing/ONAP-Testing/Test Data$ cd APPC-Stability-Test ubuntu@k8s-master:~/testing/ONAP-Testing/Test Data/APPC-Stability-Test$ ls -l total 24 drwxrwxr-x 5 ubuntu ubuntu 4096 May 4 15:16 A&AI Data -rw-rw-r-- 1 ubuntu ubuntu 10079 May 4 15:17 APPC-AAI-Data.txt -rw-rw-r-- 1 ubuntu ubuntu 3587 May 4 15:17 APPC_Test_VM_SAMPLE.tar.gz drwxrwxr-x 2 ubuntu ubuntu 4096 May 4 15:17 vDNS ubuntu@k8s-master:~/testing/ONAP-Testing/Test Data/APPC-Stability-Test$ cd "A&AI Data" ubuntu@k8s-master:~/testing/ONAP-Testing/Test Data/APPC-Stability-Test/A&AI Data$ ls -l total 12 drwxrwxr-x 2 ubuntu ubuntu 4096 May 4 15:17 Stability-Test-VM1 drwxrwxr-x 2 ubuntu ubuntu 4096 May 4 15:17 Stability-Test-VM2 drwxrwxr-x 2 ubuntu ubuntu 4096 May 4 15:17 Stability-Test-VM3 ubuntu@k8s-master:~/testing/ONAP-Testing/Test Data/APPC-Stability-Test/A&AI Data/Stability-Test-VM1$ ls -l total 76 -rw-rw-r-- 1 ubuntu ubuntu 3325 May 4 15:17 cloud-region.json -rw-rw-r-- 1 ubuntu ubuntu 300 May 4 15:17 generic-vnf.json -rw-rw-r-- 1 ubuntu ubuntu 289 May 4 15:17 model.json -rw-rw-r-- 1 ubuntu ubuntu 314 May 4 15:17 named-query.json -rw-rw-r-- 1 ubuntu ubuntu 3381 May 4 15:17 put_closed_loop.sh -rw-rw-r-- 1 ubuntu ubuntu 1103 May 4 15:17 put_named_query.sh -rw-rw-r-- 1 ubuntu ubuntu 794 May 4 15:17 update-esr.sh -rw-rw-r-- 1 ubuntu ubuntu 518 May 4 15:17 update-esr-status.json -rw-rw-r-- 1 ubuntu ubuntu 125 May 4 15:17 update-identity-url.json -rw-rw-r-- 1 ubuntu ubuntu 492 May 4 15:17 update.sh -rw-rw-r-- 1 ubuntu ubuntu 337 May 4 15:17 update-vnf-status.json -rw-rw-r-- 1 ubuntu ubuntu 597 May 4 15:17 update-vserver-selflink.json -rw-rw-r-- 1 ubuntu ubuntu 799 May 4 15:17 update-vserver.sh -rw-rw-r-- 1 ubuntu ubuntu 450 May 4 15:17 verify.sh -rw-rw-r-- 1 ubuntu ubuntu 950 May 4 15:17 VNFC_APPC_Test_VM.json -rw-rw-r-- 1 ubuntu ubuntu 10901 May 4 15:17 vnf-to-esr-info-1.0.json -rw-rw-r-- 1 ubuntu ubuntu 115 May 4 15:17 vserver-generic-vnf-relationship.json ubuntu@k8s-master:~/testing/ONAP-Testing/Test Data/APPC-Stability-Test/A&AI Data/Stability-Test-VM1$
Within the testing directory there are sample test json requests along with jmeter and A&AI data load scripts. The breakdown for these directories is below:
apache-jmeter-4.0
This directory contains APACHE Jmeter along with APPC-Tests. The APPC-Tests subfolder contains jmx files that can be utilized to run specific APPC LCM Actions. The tests can be executed either command line or via the UI.
- Command Line (The example below is executing the test from the testing->apache-jmeter-4.0->APPC-Tests folder):
- ../bin/jmeter.sh -n -L DEBUG -t APPC-LCM-Action-Rebuild.jmx l APPCLCM-Action-Rebuild.jtl
- The jmx file actually contains the test "script" and the jtl file is the formatted test results file.
- ../bin/jmeter.sh -n -L DEBUG -t APPC-LCM-Action-Rebuild.jmx l APPCLCM-Action-Rebuild.jtl
- Executing JMeter from the UI:
- /home/ubuntu/testing/apache-jmeter-4.0/jmeter
- This will pull up the UI. However, you will need to have X11 setup on your desktop.
ONAP-Testing
ONAP Testing directory contains the data and scripts that will allow the user to update a particular A&AI instance.
put_closed_loop_sh is the script that actually "puts" news data into A&AI.
./put_closed_loop.sh aaiporhost |python -m json.tool
The below files need to be configured for the specific VNF being loaded:
cloud-region.json
model.json
generic-vnf-.json
vserver-generic-vnf-relationship.json
Additionally, for the Stability Testing 3 VMs were created. Each VM has a separate directory named accordingly. These files for the specific VNFs do not require modification if there is need to load the data for that VNF in a new or cleaned out A&AI>
verify.sh is the script that can be utilized to verify the VNF is loaded into A&AI. In order to query for the specific VNF simply modify the named-query.json file to reflect that VNF.
Pre-Defined data has been created for:
Stability-Test-VM1
Stability-Test-VM2
Stability-Test-VM3
vDNS
Common commands utilzied in the OOM env:
- kubectl get pods --all-namespaces -o wide -w
Displays all the running pods and related information.
POD InformationCubuntu@k8s-master:~$ kubectl get pods --all-namespaces -o wide -w NAMESPACE NAME READY STATUS RESTARTS AGE IP NODE kube-system etcd-k8s-master 1/1 Running 1 6d 10.12.5.171 k8s-master kube-system kube-apiserver-k8s-master 1/1 Running 1 6d 10.12.5.171 k8s-master kube-system kube-controller-manager-k8s-master 1/1 Running 1 6d 10.12.5.171 k8s-master kube-system kube-dns-86f4d74b45-px44s 3/3 Running 12 19d 10.32.0.3 k8s-master kube-system kube-proxy-25tm5 1/1 Running 4 19d 10.12.5.171 k8s-master kube-system kube-proxy-6dt4z 1/1 Running 3 19d 10.12.5.174 k8s-appc1 kube-system kube-proxy-jmv67 1/1 Running 3 19d 10.12.5.193 k8s-appc2 kube-system kube-proxy-l8fks 1/1 Running 4 19d 10.12.5.194 k8s-appc3 kube-system kube-scheduler-k8s-master 1/1 Running 1 6d 10.12.5.171 k8s-master kube-system tiller-deploy-84f4c8bb78-p8b6j 1/1 Running 0 6d 10.36.0.4 k8s-appc3 kube-system weave-net-bz7wr 2/2 Running 13 19d 10.12.5.194 k8s-appc3 kube-system weave-net-c2pxd 2/2 Running 10 19d 10.12.5.174 k8s-appc1 kube-system weave-net-jw29c 2/2 Running 10 19d 10.12.5.171 k8s-master kube-system weave-net-kxxpl 2/2 Running 10 19d 10.12.5.193 k8s-appc2 onap onap-appc-0 2/2 Running 0 3d 10.44.0.5 k8s-appc1 onap onap-appc-1 2/2 Running 0 1h 10.47.0.1 k8s-appc2 onap onap-appc-2 2/2 Running 0 1h 10.36.0.5 k8s-appc3 onap onap-appc-cdt-5474c7cb88-8sfqp 1/1 Running 0 3d 10.36.0.10 k8s-appc3 onap onap-appc-db-0 2/2 Running 0 3d 10.47.0.2 k8s-appc2 onap onap-appc-dgbuilder-5dccdf57b5-qntzg 1/1 Running 0 3d 10.47.0.6 k8s-appc2 onap onap-dbcl-db-0 1/1 Running 0 3d 10.47.0.4 k8s-appc2 onap onap-dbcl-db-1 1/1 Running 0 3d 10.44.0.8 k8s-appc1 onap onap-dmaap-7dfc8f84d6-qmqmx 1/1 Running 1 3d 10.36.0.8 k8s-appc3 onap onap-dmaap-bus-controller-6c4d4d55cf-jrf4w 1/1 Running 0 3d 10.47.0.8 k8s-appc2 onap onap-global-kafka-6bb8cf75c8-dr5rm 1/1 Running 2 3d 10.36.0.11 k8s-appc3 onap onap-log-elasticsearch-855f94ccc4-jsc8t 1/1 Running 0 3d 10.47.0.7 k8s-appc2 onap onap-log-kibana-764bd47f-wxhhh 1/1 Running 0 3d 10.47.0.9 k8s-appc2 onap onap-log-logstash-776fd9fbd6-7hzhg 1/1 Running 0 3d 10.44.0.2 k8s-appc1 onap onap-robot-d7c758c5f-fz68f 1/1 Running 0 3d 10.44.0.4 k8s-appc1 onap onap-zookeeper-7774c66fb9-5nwmd 1/1 Running 0 3d 10.44.0.9 k8s-appc1
- kubectl exec -ti onap-appc-0 -c appc -n onap bash
- This command is executed from the Master node and opens a bash shell into the particular pod.
New or Clean APPC DB
Data for the different actions must be loaded into the PROTOCOL_REFERENCE table whenever a new instance of the APPC DB is instantiated or cleaned. In order to load the Stability Tests VNFs use the following:
1. kubectl exec -ti onap-appc-db-0 -c appc-db -n onap bash 2. mysql -u sdnctl -p 3. Enter the following SQL command to insert the data: INSERT INTO `PROTOCOL_REFERENCE` VALUES (1,'Stop','vTESTVM','OS',now(),'NO','vm'); INSERT INTO `PROTOCOL_REFERENCE` VALUES (2,'Start','vTESTVM','OS',now(),'NO','vm'); INSERT INTO `PROTOCOL_REFERENCE` VALUES (4,'Rebuild','vTESTVM','OS',now(),'NO','vm'); INSERT INTO `PROTOCOL_REFERENCE` VALUES (3,'Restart','vTESTVM','OS',now(),'NO','vnf'); INSERT INTO `PROTOCOL_REFERENCE` VALUES (5,'Restart','vTESTVM','OS',now(),'NO','vm'); INSERT INTO `PROTOCOL_REFERENCE` VALUES (18,'Snapshot','vTESTVM','OS',now(),'NO','vm');
KIbana Queries
- Querying for a specific string in the Message filed:
- message:"REBUILD_STATUS -- SUCCESS"
- Querying for all transactions for a RequestId:
- RequestId:"RequestId you are searching for"
- View for Restart and Rebuilds can be found here.