PASS
Monitoring shift, EDT (11am -12am, integration team)
5/14/2018: 1:00am-5:00AM (Ran), 5:00am -8:00AM, 8:00AM -12:00 (Brian), 12:--16 (Marco), 16—20 (Gary), 20-24(Helen)
5/15/2018: 1:00am-2:00am (Helen), 3:00am -8:00AM (Ran), 8:00AM -12:00 (Marco), 12:--16 (Yang), 16—20 (Helen), 20-24(Gary)
5/16/2018: 1:00am-2:00AM , 3:00am -8:00AM (Ran), 8:00AM -12:00 (Brian), 12:--16 (Yang), 16—20 (Marco), 20-24(Gary/Helen)
Stability Testing Starts at 5/13/2018,11:34PM EDT
| Time (EDT) | Report By | Health Check | Robot Test (Every 30 minutes) | Closed Loop | Manual Check | Notes |
|---|---|---|---|---|---|---|
| 5/16 20 - 24 EDT | Gary | Pass (39/40) | Pass | Pass | Portal is live | ete.sh stability72hrCurrent status: 100% Success! vFWCL continued soak testCurrent status: Success Health Checks39/40 Passed OOM-Homing failed |
5/16 4PM-8PM EDT | Marco | Pass (39/40) | Pass | Pass | Automated testing IssuesDate: 05-15-2018 @4PM-8PM EDT Affected component: OOF Test: Health check Error: 404 != 200 Comment: Non-blocking issue. A fix is available in Gerrit: https://gerrit.onap.org/r/#/c/47509/ Manual testingTest: vFirewall instantiation and closed loop Status: Success Test: vFirewall teardown Status: Success Ongoing testingTest: vFirewall instantiation and closed loop - 72-hour soaking test Running since: 05-13-2018 @ 10PM EDT Current status: Success (I had to reboot the vPacketGen VM. It seems that after many hour of work the vPacketGen gets stuck on 5 flows, which doesn't activate closed loop) | |
| 5/16 12:00-16:00PM | Yang | Pass (39/40) | Pass | Pass | No issue was observed, except the health check for OOF-Homing failure (known issue) | |
| 5/16 8:00 AM - 12:00 PM | Brian | Pass | Pass | Pass | Cleaned up a vVG from yesterday that we had left in place. Control Loop running normal Stability72hr tests normal | |
5/16 3:00am -8:00AM | Ran | Pass (39/40) | Pass (12/13) | Pass | NA (Created VLM & VSP on the onboarding page) Didn't observe any issue. | ete.sh stability72hr Current status: Success; no ONAP errors observed during this period. Health Test Fail: OOF-Homing(Known issue). Knowen issue "authentication" observe during the run only 1 time: error.message=HEAT-E99001 Service nova is not available for resource type OS::Nova::KeyPair, reason: The request you have made requires authentication. (HTTP 401). Stacks - I was able to observe Vfmodule on Stacks that should have been deleted(only one during my Shift). |
| 5/15 20 - 24 EDT | Gary | Pass (39/40) | Pass (22/24) | Pass | ete.sh stability72hrCurrent status: Success; no ONAP errors observed during this period. Continued intermittent failures of "The request you have made requires authentication" from OpenStack. Failed in 2 test cases out of 24 = 8.3% error rate from OpenStack. Had to manually delete one vFWSNK stack orphaned by the OpenStack errors. vFWCL continued soak testCurrent status: Success Health Checks39/40 Passed OOM-Homing failed | |
| 5/15 16 - 20 EDT | Helen | Pass (39/40) | Pass (22/24) | Pass | Health Test Fail: OOF-Homing Test Case: Instantiate Virtual Volume Group Status: FAIL (critical) Message: '{"request":{"requestId":"f5afa2df-40ba-4492-9b42-9c05d4215240","startTime":"Tue, 15 May 2018 22:16:42 GMT","requestScope":"vfModule","requestType":"createInstance","instanceReferences":{"serviceInstanceId":"c75edfa0-49a1-4503-b673-bba740a50cd5","vnfInstanceId":"ea3c464a-8cf1-4071-9927-9995de2fa616","vfModuleInstanceName":"Vfmodule_Ete_vVG_73c32a00-f705-47ca-baa3-b032d116679e","requestorId":"demo"},"requestStatus":{"requestState":"FAILED","statusMessage":"Received vfModuleException from VnfAdapter: category='INTERNAL' message='0 null: null, error.type=null, error.message=The request you have made requires authentication.' rolledBack='true'","percentProgress":100,"finishTime":"Tue, 15 May 2018 22:16:51 GMT"}}}' contains 'FAILED' | |
| 5/15 12-16 | Yang | Pass (39/40) | Pass (6/8) | Pass | Pass | Health Check Test OOF-Homing failed with 404 != 200 error, same as reported earlier Robot Test 2 out 8 failed with the authentication error in Robot, the time are 13:33:36 and 15:03:37. The issue is the same as reported repeatedly. Closed Loop Test worked fine during the period Manual Test Tested VNF onboarding, service creation and distribution with VoLTE use case, it worked fine |
| 5/15 8 AM - 12 PM | Brian/Marco | 39 out of 40 Pass | Pass 8/9 | Pass* VNF issue on Packet Generator not ONAP related | Automated testing issuesDate: 05-15-2018 @12PM-4PM EDT Affected component: OOF Test: Health check Error: 404 != 200 Comment: Non-blocking issue. Team identified the issue. A fix has been submitted and available in Gerrit: https://gerrit.onap.org/r/#/c/47509/ Date: 05-15-2018 @8:30AM EDT Affected component: vLB Test: Instantiation Error:'{"request":{"requestId":"bc9dee7f-b9e1-45b8-91ad-5289e62f2cfa","startTime":"Tue, 15 May 2018 12:37:31 GMT","requestScope":"vfModule","requestType":"createInstance","instanceReferences":{"serviceInstanceId":"f7b4da8f-cd86-4be0-a822-a1b3026686e0","vnfInstanceId":"7936c473-959c-400c-bb40-3bd6707f1766","vfModuleInstanceName":"Vfmodule_Ete_vLB_a63e39c5-89f8-4a01-9ebf-9f1182a8d3b7","requestorId":"demo"},"requestStatus":{"requestState":"FAILED","statusMessage":"Received vfModuleException from VnfAdapter: category='INTERNAL' message='Exception during create VF 400 Bad Request: The server could not comply with the request since it is either malformed or otherwise incorrect., error.type=ResourceTypeUnavailable, error.message=HEAT-E99001 Service nova is not available for resource type OS::Nova::KeyPair, reason: The request you have made requires authentication. (HTTP 401) (Request-ID: req-07c0b84a-0cdb-4abc-9afa-af8c90655cb5)' rolledBack='true'","percentProgress":100,"finishTime":"Tue, 15 May 2018 12:37:39 GMT"}}}' contains 'FAILED' Comment: Intermittent failure Manual testingTest: vFirewall instantiation and closed loop Status: Success Test: vFirewall teardown Status: Success Ongoing testingTest: vFirewall instantiation and closed loop - 72-hour soaking test Running since: 05-13-2018 @ 10PM EDT Current status: Success (I had to reboot the vPacketGen VM due to some input from outside that may have confused the vPacketGen) Brian's notes:Deleted 1 vFW and 1 vVG from failed to delete activity Closed loop is not looking correct as of 8:29 AM - back to normal ~9:30 after restarting packet generator VNF.
| |
| Ran | ||||||
| to 5/15/2018 2:00 EDT | Helen | Pass (39/40) | Pass (14/15) | Pass | Test Case: Instantiate Virtual DNS Status: FAIL (critical) Message: '{"request":{"requestId":"fb3322e5-ef59-417c-8891-814f3dde76b9","startTime":"Tue, 15 May 2018 05:07:31 GMT","requestScope":"vfModule","requestType":"createInstance","instanceReferences":{"serviceInstanceId":"c8038ce6-695d-4f96-9079-07dffcbb7f01","vnfInstanceId":"2449ea65-7044-4ccc-ae1b-28bd8e47c80e","vfModuleInstanceName":"Vfmodule_Ete_vLB_ca5da6e1-b2e8-42e0-99b7-15283803b3ea","requestorId":"demo"},"requestStatus":{"requestState":"FAILED","statusMessage":"Received vfModuleException from VnfAdapter: category='INTERNAL' message='Exception during create VF 400 Bad Request: The server could not comply with the request since it is either malformed or otherwise incorrect., error.type=StackValidationFailed, error.message=Property error: : resources.vpg_0.properties.flavor: : The request you have made requires authentication. (HTTP 401)' rolledBack='true'","percentProgress":100,"finishTime":"Tue, 15 May 2018 05:07:41 GMT"}}}' contains 'FAILED' | |
| to 5/14/2018 24:00 EDT | Helen | Pass (39/40) | Pass (22/24) | Pass | 5/14/2018 23:30 EDT Test Case: Instantiate Virtual FirewallCL Status: '{"request":{"requestId":"348d71db-7846-4a01-9496-9d058e993387","startTime":"Tue, 15 May 2018 03:51:36 GMT","requestScope":"vfModule","requestType":"createInstance","instanceReferences":{"serviceInstanceId":"b3a63233-8321-4535-af44-a6599ea40877","vnfInstanceId":"9ac48053-7e1d-43e3-8734-c09a15dfabca","vfModuleInstanceName":"Vfmodule_Ete_vFWSNK_151b24fd-d8ba-4079-a4e4-68b0dae8cc8e","requestorId":"demo"},"requestStatus":{"requestState":"FAILED","statusMessage":"Received vfModuleException from VnfAdapter: category='INTERNAL' message='Exception during create VF 400 Bad Request: The server could not comply with the request since it is either malformed or otherwise incorrect., error.type=ResourceTypeUnavailable, error.message=HEAT-E99001 Service nova is not available for resource type OS::Nova::KeyPair, reason: The request you have made requires authentication. (HTTP 401) (Request-ID: req-bd852abb-421b-4a77-8bcf-04c6d433b8b6)' rolledBack='true'","percentProgress":100,"finishTime":"Tue, 15 May 2018 03:51:44 GMT"}}}' contains 'FAILED' ====================================================== 5/14/2018 21:30 EDT: Test Case: Instantiate Virtual Volume Group Status: FAIL (critical) Message: '{"request":{"requestId":"b0d0e687-14c2-4002-a359-92341e6cc465","startTime":"Tue, 15 May 2018 01:46:29 GMT","requestScope":"vfModule","requestType":"createInstance","instanceReferences":{"serviceInstanceId":"5bd36e34-8923-4068-9d63-c62e0dd53ff3","vnfInstanceId":"992422ac-5652-4a46-a513-4bde82347ccd","vfModuleInstanceName":"Vfmodule_Ete_vVG_51f740af-8319-4c9f-8c27-a9a7979e2333","requestorId":"demo"},"requestStatus":{"requestState":"FAILED","statusMessage":"Received vfModuleException from VnfAdapter: category='INTERNAL' message='0 null: null, error.type=null, error.message=The request you have made requires authentication.' rolledBack='true'","percentProgress":100,"finishTime":"Tue, 15 May 2018 01:46:37 GMT"}}}' contains 'FAILED' | |
| 5/14 4 PM to 8 PM EDT | Gary | Pass (39/40) | Pass (22/24) | Pass | Pass (Incomplete) | Automated Testingete.sh stability72hrCurrent status: Success; no ONAP errors observed during this period. Continued intermittent failures of "The request you have made requires authentication" from OpenStack. Failed in 2 test cases out of 24 = 8.3% error rate from OpenStack. Had to manually delete one set of vFWSNK and vPKG stacks orphaned by the OpenStack errors. vFWCL continued soak testCurrent status: Success Manual TestingOnly got as far as service creation in VID for vFWCL. Did not get to complete SDNC Preload. Everything attempted in the UI so far worked. |
| 5/14 12PM-4PM EDT | Marco | 38 out of 40 Passed | 21 out of 24 test Passed | Pass | Pass/Fail | Automated testing issuesDate: 05-14-2018 @12PM-4PM EDT Affected component: AAF Test: Health check Error: ConnectionError: HTTPSConnectionPool(host='aaf.api.simpledemo.onap.org', port=8100): Max retries exceeded with url: /authz/perms/user/demo@people.osaaf.org (Caused by NewConnectionError('<urllib3.connection.VerifiedHTTPSConnection object at 0x7ff9bf31b590>: Failed to establish a new connection: [Errno 111] Connection refused',)) Comment: Non-blocking issue, a fix has been submitted to Gerrit Date: 05-14-2018 @12PM-4PM EDT Affected component: OOF Test: Health check Error: 404 != 200 Comment: Non-blocking issue Date: 05-14-2018 @1PM EDT Affected component: vVG Test: Instantiation Error: '{"request":{"requestId":"cd6440cc-4a8a-499a-995a-9cd1cedb6be5","startTime":"Mon, 14 May 2018 17:16:37 GMT","requestScope":"vfModule","requestType":"createInstance","instanceReferences":{"serviceInstanceId":"cc267011-9954-4a65-8f8b-c65b981092de","vnfInstanceId":"e9491247-cdb4-41a3-9fd7-3033215b45c6","vfModuleInstanceName":"Vfmodule_Ete_vVG_a591c63a-77ca-40e1-93b5-b22af2b4cdfe","requestorId":"demo"},"requestStatus":{"requestState":"FAILED","statusMessage":"Received vfModuleException from VnfAdapter: category='INTERNAL' message='Exception during create VF 0 null: null, error.type=null, error.message=The request you have made requires authentication.' rolledBack='true'","percentProgress":100,"finishTime":"Mon, 14 May 2018 17:17:00 GMT"}}}' contains 'FAILED' Comment: Intermittent failure Manual testingTest: vFirewall instantiation and closed loop Status: Success Test: vFirewall teardown Status: Success Test: vLoadBalancer/vDNS closed loop Status: Failed Affected component: SO Error: Exception caught mapping requestBody to RequestDetails com.fasterxml.jackson.databind.exc.UnrecognizedPropertyException: Unrecognized field "requestDetails" Jira ticket: https://jira.onap.org/browse/SO-621 A fix has been submitted to Gerrit Test: vLB/vDNS (for manual scaling use case) design, distribution, and instantiation Status: Success
Ongoing testingTest: vFirewall instantiation and closed loop - 72-hour soaking test Running since: 05-13-2018 @ 10PM EDT Current status: Success |
| Brian | Just an FYI: Here are the messages coming out of TCA to Policy "{\"closedLoopEventClient\":\"DCAE_INSTANCE_ID.dcae-tca\",\"policyVersion\":\"v0.0.1\",\"policyName\":\"DCAE.Config_tca-hi-lo\",\"policyScope\":\"DCAE\",\"target_type\":\"VM\",\"AAI\":{\"vserver.prov-status\":\"ACTIVE\",\"vserver.resource-version\":\"1526307045632\",\"vserver.is-closed-loop-disabled\":false,\"vserver.vserver-name2\":\"zdfw1lb01lb01-0514-1\",\"vserver.vserver-id\":\"cb2a02d3-88f0-494b-89f3-e8c3366669e8\",\"vserver.vserver-selflink\":\"http://10.12.25.2:8774/v2.1/ebb0ea7144004bacac1e39ff23105fa7/servers/cb2a02d3-88f0-494b-89f3-e8c3366669e8\",\"vserver.in-maint\":false,\"vserver.vserver-name\":\"zdfw1lb01lb01-0514-1\"},\"closedLoopAlarmStart\":1526308051843950,\"closedLoopEventStatus\":\"ONSET\",\"closedLoopControlName\":\"ControlLoop-vDNS-6f37f56d-a87d-4b85-b6a9-cc953cf779b3\",\"version\":\"1.0.2\",\"target\":\"vserver.vserver-name\",\"requestID\":\"26a32b02-c874-4af8-af94-57573ffa343e\",\"from\":\"DCAE\"}"
another vVG stack not deleted - wonder if that is correlated to the Openstack Authentication Error ? Vfmodule_Ete_vVG_a591c63a-77ca-40e1-93b5-b22af2b4cdfe @ 1:30 PM EDT test (or so) Yes it is http://10.12.5.33:88/logs/ETE_0473_stability72hr/log.html is the same VNF | |||||
| 5/14 8:00 AM | Brian | Pass 2 OS fails in last 10 test. 3X8 + 2X2 = | Pass | Each cycle is 3 tests (vFW, vVG, vLB) - the Openstack Error throws out one of those when it happens. 3X8 + 2x2 = 28 tests PASS out of 30 total tests 28/30 ~93% success if we count the Openstack errors. 100% if we ignore the Openstack errors. | ||
| 5/14/2018 1:00AM-8:00AM | Ran | Pass | 2/11 had issues of Open-Stack NET Please Observe the Notes => | Pass | N/A | (1) Issues while running the VFW(Occured after passing 3 Tests Flows): http://10.12.5.33:88/logs/ETE_0401_stability72hr/log.html Problem : 401 Issue found. Assumption - Open stack Issue of NET Causing Authorization failed: error.message=The request you have made requires authentication.' rolledBack='true'","percentProgress":100,"finishTime":"Mon, 14 May 2018 05:21:35 GMT"}}}' contains 'FAILED' The next flow of VFW Passed:
(2) Another Failure has occurred after 2 Tests Flow Passed: Instantiate Virtual DNS - error.message=The request you have made requires authentication http://10.12.5.33:88/logs/ETE_0410_stability72hr/log.html Looks like the same NET Issue we observe... Agin after this Fail mention above the next Test Passed, PSB!
Note : Another Issue regarding not deleting the VM, PSB:
|
| 5/13: 23:34 – 1:00AM | Team | Pass | Pass | Pass | N/A |
