Monitoring shift, EDT (11am -12am, integration team)
5/14/2018: 1:00am-5:00AM (Ran), 5:00am -8:00AM, 8:00AM -12:00 (Brian), 12:--16 (Marco), 16—20 (Gary), 20-24(Helen)
5/15/2018: 1:00am-2:00am (Helen), 3:00am -8:00AM (Ran), 8:00AM -12:00 (Marco), 12:--16 (Yang), 16—20 (??), 20-24(Gary)
5/16/2018: 1:00am-2:00AM , 3:00am -8:00AM (Ran), 8:00AM -12:00 (Brian), 12:--16 (Yang), 16—20 (Marco), 20-24(Gary/Helen)
Stability Testing Starts at 5/13/2018,11:34PM EDT
| Time (EDT) | Report By | Health Check | Robot Test (Every 30 minutes) | Closed Loop | Manual Check | Notes |
|---|---|---|---|---|---|---|
| Ran | ||||||
| to 5/15/2018 2:00 EDT | Helen | Pass (39/40) | Pass (14/15) | Pass | Test Case: Instantiate Virtual DNS Status: FAIL (critical) Message: '{"request":{"requestId":"fb3322e5-ef59-417c-8891-814f3dde76b9","startTime":"Tue, 15 May 2018 05:07:31 GMT","requestScope":"vfModule","requestType":"createInstance","instanceReferences":{"serviceInstanceId":"c8038ce6-695d-4f96-9079-07dffcbb7f01","vnfInstanceId":"2449ea65-7044-4ccc-ae1b-28bd8e47c80e","vfModuleInstanceName":"Vfmodule_Ete_vLB_ca5da6e1-b2e8-42e0-99b7-15283803b3ea","requestorId":"demo"},"requestStatus":{"requestState":"FAILED","statusMessage":"Received vfModuleException from VnfAdapter: category='INTERNAL' message='Exception during create VF 400 Bad Request: The server could not comply with the request since it is either malformed or otherwise incorrect., error.type=StackValidationFailed, error.message=Property error: : resources.vpg_0.properties.flavor: : The request you have made requires authentication. (HTTP 401)' rolledBack='true'","percentProgress":100,"finishTime":"Tue, 15 May 2018 05:07:41 GMT"}}}' contains 'FAILED' | |
| to 5/14/2018 24:00 EDT | Helen | Pass (39/40) | Pass (22/24) | Pass | 5/14/2018 23:30 EDT Test Case: Instantiate Virtual FirewallCL Status: '{"request":{"requestId":"348d71db-7846-4a01-9496-9d058e993387","startTime":"Tue, 15 May 2018 03:51:36 GMT","requestScope":"vfModule","requestType":"createInstance","instanceReferences":{"serviceInstanceId":"b3a63233-8321-4535-af44-a6599ea40877","vnfInstanceId":"9ac48053-7e1d-43e3-8734-c09a15dfabca","vfModuleInstanceName":"Vfmodule_Ete_vFWSNK_151b24fd-d8ba-4079-a4e4-68b0dae8cc8e","requestorId":"demo"},"requestStatus":{"requestState":"FAILED","statusMessage":"Received vfModuleException from VnfAdapter: category='INTERNAL' message='Exception during create VF 400 Bad Request: The server could not comply with the request since it is either malformed or otherwise incorrect., error.type=ResourceTypeUnavailable, error.message=HEAT-E99001 Service nova is not available for resource type OS::Nova::KeyPair, reason: The request you have made requires authentication. (HTTP 401) (Request-ID: req-bd852abb-421b-4a77-8bcf-04c6d433b8b6)' rolledBack='true'","percentProgress":100,"finishTime":"Tue, 15 May 2018 03:51:44 GMT"}}}' contains 'FAILED' ====================================================== 5/14/2018 21:30 EDT: Test Case: Instantiate Virtual Volume Group Status: FAIL (critical) Message: '{"request":{"requestId":"b0d0e687-14c2-4002-a359-92341e6cc465","startTime":"Tue, 15 May 2018 01:46:29 GMT","requestScope":"vfModule","requestType":"createInstance","instanceReferences":{"serviceInstanceId":"5bd36e34-8923-4068-9d63-c62e0dd53ff3","vnfInstanceId":"992422ac-5652-4a46-a513-4bde82347ccd","vfModuleInstanceName":"Vfmodule_Ete_vVG_51f740af-8319-4c9f-8c27-a9a7979e2333","requestorId":"demo"},"requestStatus":{"requestState":"FAILED","statusMessage":"Received vfModuleException from VnfAdapter: category='INTERNAL' message='0 null: null, error.type=null, error.message=The request you have made requires authentication.' rolledBack='true'","percentProgress":100,"finishTime":"Tue, 15 May 2018 01:46:37 GMT"}}}' contains 'FAILED' | |
| 5/14 4 PM to 8 PM EDT | Gary | Pass (39/40) | Pass (22/24) | Pass | Pass (Incomplete) | Automated Testingete.sh stability72hrCurrent status: Success; no ONAP errors observed during this period. Continued intermittent failures of "The request you have made requires authentication" from OpenStack. Failed in 2 test cases out of 24 = 8.3% error rate from OpenStack. Had to manually delete one set of vFWSNK and vPKG stacks orphaned by the OpenStack errors. vFWCL continued soak testCurrent status: Success Manual TestingOnly got as far as service creation in VID for vFWCL. Did not get to complete SDNC Preload. Everything attempted in the UI so far worked. |
| 5/4 12PM-4PM EDT | Marco | 38 out of 40 Passed | 21 out of 24 test Passed | Pass | Pass/Fail | Automated testing issuesDate: 05-14-2018 @12PM-4PM EDT Affected component: AAF Test: Health check Error: ConnectionError: HTTPSConnectionPool(host='aaf.api.simpledemo.onap.org', port=8100): Max retries exceeded with url: /authz/perms/user/demo@people.osaaf.org (Caused by NewConnectionError('<urllib3.connection.VerifiedHTTPSConnection object at 0x7ff9bf31b590>: Failed to establish a new connection: [Errno 111] Connection refused',)) Comment: Non-blocking issue, a fix has been submitted to Gerrit Date: 05-14-2018 @12PM-4PM EDT Affected component: OOF Test: Health check Error: 404 != 200 Comment: Non-blocking issue Date: 05-14-2018 @1PM EDT Affected component: vVG Test: Instantiation Error: '{"request":{"requestId":"cd6440cc-4a8a-499a-995a-9cd1cedb6be5","startTime":"Mon, 14 May 2018 17:16:37 GMT","requestScope":"vfModule","requestType":"createInstance","instanceReferences":{"serviceInstanceId":"cc267011-9954-4a65-8f8b-c65b981092de","vnfInstanceId":"e9491247-cdb4-41a3-9fd7-3033215b45c6","vfModuleInstanceName":"Vfmodule_Ete_vVG_a591c63a-77ca-40e1-93b5-b22af2b4cdfe","requestorId":"demo"},"requestStatus":{"requestState":"FAILED","statusMessage":"Received vfModuleException from VnfAdapter: category='INTERNAL' message='Exception during create VF 0 null: null, error.type=null, error.message=The request you have made requires authentication.' rolledBack='true'","percentProgress":100,"finishTime":"Mon, 14 May 2018 17:17:00 GMT"}}}' contains 'FAILED' Comment: Intermittent failure Manual testingTest: vFirewall instantiation and closed loop Status: Success Test: vFirewall teardown Status: Success Test: vLoadBalancer/vDNS closed loop Status: Failed Affected component: SO Error: Exception caught mapping requestBody to RequestDetails com.fasterxml.jackson.databind.exc.UnrecognizedPropertyException: Unrecognized field "requestDetails" Jira ticket: https://jira.onap.org/browse/SO-621 A fix has been submitted to Gerrit Test: vLB/vDNS (for manual scaling use case) design, distribution, and instantiation Status: Success
Ongoing testingTest: vFirewall instantiation and closed loop - 72-hour soaking test Running since: 05-13-2018 @ 10PM EDT Current status: Success |
| Brian | Just an FYI: Here are the messages coming out of TCA to Policy "{\"closedLoopEventClient\":\"DCAE_INSTANCE_ID.dcae-tca\",\"policyVersion\":\"v0.0.1\",\"policyName\":\"DCAE.Config_tca-hi-lo\",\"policyScope\":\"DCAE\",\"target_type\":\"VM\",\"AAI\":{\"vserver.prov-status\":\"ACTIVE\",\"vserver.resource-version\":\"1526307045632\",\"vserver.is-closed-loop-disabled\":false,\"vserver.vserver-name2\":\"zdfw1lb01lb01-0514-1\",\"vserver.vserver-id\":\"cb2a02d3-88f0-494b-89f3-e8c3366669e8\",\"vserver.vserver-selflink\":\"http://10.12.25.2:8774/v2.1/ebb0ea7144004bacac1e39ff23105fa7/servers/cb2a02d3-88f0-494b-89f3-e8c3366669e8\",\"vserver.in-maint\":false,\"vserver.vserver-name\":\"zdfw1lb01lb01-0514-1\"},\"closedLoopAlarmStart\":1526308051843950,\"closedLoopEventStatus\":\"ONSET\",\"closedLoopControlName\":\"ControlLoop-vDNS-6f37f56d-a87d-4b85-b6a9-cc953cf779b3\",\"version\":\"1.0.2\",\"target\":\"vserver.vserver-name\",\"requestID\":\"26a32b02-c874-4af8-af94-57573ffa343e\",\"from\":\"DCAE\"}"
another vVG stack not deleted - wonder if that is correlated to the Openstack Authentication Error ? Vfmodule_Ete_vVG_a591c63a-77ca-40e1-93b5-b22af2b4cdfe @ 1:30 PM EDT test (or so) Yes it is http://10.12.5.33:88/logs/ETE_0473_stability72hr/log.html is the same VNF | |||||
| 5/14 8:00 AM | Brian | Pass 2 OS fails in last 10 test. 3X8 + 2X2 = | Pass | Each cycle is 3 tests (vFW, vVG, vLB) - the Openstack Error throws out one of those when it happens. 3X8 + 2x2 = 28 tests PASS out of 30 total tests 28/30 ~93% success if we count the Openstack errors. 100% if we ignore the Openstack errors. | ||
| 5/14/2018 1:00AM-8:00AM | Ran | Pass | 2/11 had issues of Open-Stack NET Please Observe the Notes => | Pass | N/A | (1) Issues while running the VFW(Occured after passing 3 Tests Flows): http://10.12.5.33:88/logs/ETE_0401_stability72hr/log.html Problem : 401 Issue found. Assumption - Open stack Issue of NET Causing Authorization failed: error.message=The request you have made requires authentication.' rolledBack='true'","percentProgress":100,"finishTime":"Mon, 14 May 2018 05:21:35 GMT"}}}' contains 'FAILED' The next flow of VFW Passed:
(2) Another Failure has occurred after 2 Tests Flow Passed: Instantiate Virtual DNS - error.message=The request you have made requires authentication http://10.12.5.33:88/logs/ETE_0410_stability72hr/log.html Looks like the same NET Issue we observe... Agin after this Fail mention above the next Test Passed, PSB!
Note : Another Issue regarding not deleting the VM, PSB:
|
| 5/13: 23:34 – 1:00AM | Team | Pass | Pass | Pass | N/A |
