Versions Compared

Key

  • This line was added.
  • This line was removed.
  • Formatting was changed.

...

Test #CommentMessage

k8 utilization

Wed May 20 18:45:15 UTC 2020

Memory:
root@long-nfs:~/oom/kubernetes/robot# kubectl -n onap top pods | sort -rn -k 3 | head -25
dev-appc-0 7m 2901Mi
dev-portal-cassandra-59f5cb4cf5-9phmg 159m 2777Mi
dev-appc-2 10m 2705Mi
dev-appc-1 19m 2681Mi
dev-cassandra-0 73m 2417Mi
dev-cassandra-2 48m 2394Mi
dev-cassandra-1 70m 2391Mi
dev-sdnc-2 71m 1868Mi
dev-policy-59f48bd84b-q2fp8 7m 1820Mi
dev-sdnc-0 139m 1627Mi
dev-sdnc-1 26m 1574Mi
dev-vid-5b7558dcdc-rx2d7 9m 1510Mi
dev-clamp-dash-es-6cb85979b5-cvrcs 32m 1480Mi
dev-awx-0 244m 1434Mi
dev-aai-elasticsearch-55b56f855c-f5pp5 2m 1422Mi
dev-sdc-be-77d55774f5-zkfrt 6m 1381Mi
dev-dcae-cloudify-manager-6f854859f9-ctdcv 90m 1312Mi
dep-dcae-tca-analytics-55dbd5cd9d-fsm89 511m 1262Mi
dev-aaf-cass-7d55bfc874-sqcdq 6m 1244Mi
dev-aai-traversal-847c4c6994-qbpst 3m 956Mi
dev-so-bpmn-infra-7b58b75b76-n59sf 5m 953Mi
dev-message-router-zookeeper-2 2m 946Mi
dev-aai-resources-74dd6994d4-nh24m 5m 869Mi
dev-aai-graphadmin-65db8cfc67-svvkd 2m 836Mi
dev-music-cassandra-2 147m 801Mi

#1

Status
colourGreen
titleTOOLING

Startup issues - modified customer uuid to shorten the string in the tooling since it looked like robot selenium was having trouble "seeing" the string in the drop down.

vDNS: NoSuchElementException: Message: Could not locate element with visible text: ETE_Customer_aaaf3926-d765-4c47-93b9-857e674d2d01

vvG: NoSuchElementException: Message: Could not locate element with visible text: ETE_Customer_08f8a099-3e2b-480f-8153-5b4173d9394a

vFW: Succeeded


#4

Status
colourGreen
titleENV

${vnf} = vFWCLvPKG

Robot heat bridge run after the deployment failed trying to find the stack in openstack usually means that openstack was slow in deploying the VNF. Heatbridge had succeeded for the vFWCLvSNK inside the same service instantiate.

Keyword 'Get Deployed Stack' failed after retrying for 10 minutes. The last error was: KeyError: 'stack'
#13

Status
colourGreen
titleENV

${vnf} = vFWCLvPKG

Robot heat bridge run after the deployment failed trying to find the stack in openstack usually means that openstack was slow in deploying the VNF. Heatbridge had succeeded for the vFWCLvSNK inside the same service instantiate.

Keyword 'Get Deployed Stack' failed after retrying for 10 minutes. The last error was: KeyError: 'stack'
#14

Status
colourGreen
titleTOOLING
or 
Status
colourGreen
titleENV

vDNS and vVG robot script couldnt find elements on the GUI drop downs. Likely transient networking issues. vFW succeeded and all three are in the test run (vDNS, vVG, vFW in that order).

vDNS : Keyword 'Wait For Model' failed after retrying for 3 minutes. The last error was: Element 'xpath=//tr[td/span/text() = 'vLB 2020-05-20 13-06-03']/td/button[contains(text(),'Deploy')]' not visible after 1 minute.


vVG: NoSuchElementException: Message: Could not locate element with visible text: ETE_Customer_9f739343-cbc7-4ee4-8697-ea52f06e7796


vFW Succeeded

#15

Status
colourGreen
titleTOOLING

Virtual Volume Group - Failure in robot selenium to find customer in search window. Timing issue.

NoSuchElementException: Message: Could not locate element with visible text: ETE_Customer_26e85655-1f44-4e7e-8cd2-e9fab290af01
#17

Status
colourGreen
titleENV
   or
Status
colourGreen
titleTOOLING

Failure in robot selenium at second VNF in service package. Likely tuning of robot needed waiting for the module name to appear in the drop down under transient conditions.

Element 'xpath=//div[contains(.,'Ete_vFWCLvPKG_f716b1bd_1')]/div/button[contains(.,'Add VF-Module')]' did not appear in 1 minute.
#18

Status
colourRed
titleENV

K8 worker node problem . kubectl top nodes listed k8s-04 as unkown.

k8s-04 is on 10.12.6.0 which could be contributing factor - .0 and .32 addresses in windriver have suspect behavoir.

Worker down caused a set of containers to be restarted which is the right behavoir from a k8 standpoint. Test could not run while robot container was down.

12:00:25 Instantiate Virtual DNS GRA command terminated with exit code 137
12:22:22 + retval=137
12:22:22 ++ echo 'kubectl exec -n onap dev-robot-56c5b65dd-dkks4 -- ls -1t /share/logs | grep stability72hr | head -1'
12:22:22 ++ ssh -i /var/lib/jenkins/.ssh/onap_key ubuntu@10.12.5.205 sudo su
12:22:25 error: unable to upgrade connection: container not found ("robot")

#19

#20

Status
colourGreen
titleTOOLING

k8 restarted robot pod. Manual fixes to vnf_orchestration_test_template to fix heat3 parsing issues were removed.

reapplied manual fixes so parsing sdc artifacts to find the base_vlb resource succeeded again.

Unable to find catalog resource for vLB base_vlb'


#32

Status
colourGreen
titleTOOLING
 

Robot script did not find subscriber name in search results

Likely timing issue that robot is too fast in looking for json data in the drop down before it is fully loaded.

Create Service Instance → vid_interface . Click On Element When Visible //select[@prompt='Select Subscriber Name'
#35

Status
colourGreen
titleENV

vDNS instantiate failed at openstack stage. Potentially slowed openstack caused SO to resubmit a request that subsequently became a duplicate from openstack perspective.

Looks like functional bug with SO to Openstack issue triggered by the environment not stability related.

CREATE failed: Conflict: resources.vlb_0_onap_private_port_0: IP address 10.0.211.24 already allocated in subnet be057760-1ffa-4827-a6df-75d355c4d45a\nNeutron server returns request_ids: ['req-ca6e5f39-7462-47c6-aaa8-9653783828cb']

View file
name10.0.211.24.debug.log
height250

#37

Status
colourGreen
titleENV

vG and vFW failed on VID screen errors looking for data items. Investigation shows that aai-traversal pod restarted. Looks like slow networking caused the pod to be redeployed but not conclusive. Initially so, vid failed healtch check until aai traversal was up then both passed healthcheck.



Thu May 21 12:33:45 UTC 2020

Memory:

root@long-nfs:/home/ubuntu# kubectl -n onap top pod | sort -rn -k3 | head -20
dev-appc-0 7m 2834Mi
dev-portal-cassandra-59f5cb4cf5-9phmg 152m 2780Mi
dev-appc-1 19m 2700Mi
dev-appc-2 10m 2694Mi
dev-cassandra-2 15m 2449Mi
dev-cassandra-1 21m 2434Mi
dev-vid-5b7558dcdc-rx2d7 16m 1786Mi
dev-sdnc-2 64m 1664Mi
dev-sdnc-0 131m 1631Mi
dev-sdc-be-77d55774f5-zkfrt 9m 1578Mi
dev-sdnc-1 29m 1566Mi
dev-awx-0 291m 1524Mi
dev-clamp-dash-es-6cb85979b5-cvrcs 37m 1496Mi
dep-dcae-tca-analytics-55dbd5cd9d-fsm89 664m 1318Mi
dev-dcae-cloudify-manager-6f854859f9-ctdcv 76m 1302Mi
dev-aaf-cass-7d55bfc874-sqcdq 5m 1250Mi
dev-cds-blueprints-processor-7fd988d584-mvdkz 40m 1228Mi
dev-message-router-zookeeper-1 5m 1127Mi
dev-message-router-zookeeper-0 6m 1023Mi
dev-so-bpmn-infra-7b58b75b76-n59sf 8m 941Mi




#38

Status
colourGreen
titleENV

vDNS - Timeout waiting for model to be visible via Deploy button in VID

vVG and vFW Succeeded

Transient Slowness since the 2nd and 3rd VNF succeeded.

Keyword 'Wait For Model' failed after retrying for 3 minutes. The last error was: TypeError: object of type 'NoneType' has no len()





Closed Loop Tests

This test uses the onap-ci job "Project windriver-longevity-vfwclosedloop".

...