Monitoring shift, EDT (11am -12am, integration team)
5/14/2018: 1:00am-5:00AM (Ran), 5:00am -8:00AM, 8:00AM -12:00 (Brian), 12:--16 (Marco), 16—20 (Gary), 20-24(Helen/??)
5/15/2018: 12am-4:00AM , 4:00am -8:00AM, 8:00AM -12:00 (Marco), 12:--16 (Yang), 16—20 (??), 20-24(Gary)
5/16/2018: 12am-4:00AM , 4:00am -8:00AM, 8:00AM -12:00 (Brian), 12:--16 (Yang), 16—20 (Marco), 20-24(Gary/Helen)
Stability Testing Starts at 5/13/2018,11:34PM EDT
Time (EDT) | Report By | Health Check | Robot Test (Every 30 minutes) | Closed Loop | Manual Check | Notes |
---|---|---|---|---|---|---|
5/13: 23:34 – 1:00AM | Team | Pass | Pass | Pass | N/A | |
5/14/2018 1:00AM-8:00AM | Ran | Pass | 2/11 had issues of Open-Stack NET Please Observe the Notes => | Pass | N/A | (1) Issues while running the VFW(Occured after passing 3 Tests Flows): http://10.12.5.33:88/logs/ETE_0401_stability72hr/log.html Problem : 401 Issue found. Assumption - Open stack Issue of NET Causing Authorization failed: error.message=The request you have made requires authentication.' rolledBack='true'","percentProgress":100,"finishTime":"Mon, 14 May 2018 05:21:35 GMT"}}}' contains 'FAILED' The next flow of VFW Passed: (2) Another Failure has occurred after 2 Tests Flow Passed: Instantiate Virtual DNS - error.message=The request you have made requires authentication http://10.12.5.33:88/logs/ETE_0410_stability72hr/log.html Looks like the same NET Issue we observe... Agin after this Fail mention above the next Test Passed, PSB! Note : Another Issue regarding not deleting the VM, PSB: |
5/14 8:00 AM | Brian | Pass 2 OS fails in last 10 test. 3X8 + 2X2 = | Pass | Each cycle is 3 tests (vFW, vVG, vLB) - the Openstack Error throws out one of those when it happens. 3X8 + 2x2 = 28 tests PASS out of 30 total tests 28/30 ~93% success if we count the Openstack errors. 100% if we ignore the Openstack errors. | ||
Just an FYI: Here are the messages coming out of TCA to Policy "{\"closedLoopEventClient\":\"DCAE_INSTANCE_ID.dcae-tca\",\"policyVersion\":\"v0.0.1\",\"policyName\":\"DCAE.Config_tca-hi-lo\",\"policyScope\":\"DCAE\",\"target_type\":\"VM\",\"AAI\":{\"vserver.prov-status\":\"ACTIVE\",\"vserver.resource-version\":\"1526307045632\",\"vserver.is-closed-loop-disabled\":false,\"vserver.vserver-name2\":\"zdfw1lb01lb01-0514-1\",\"vserver.vserver-id\":\"cb2a02d3-88f0-494b-89f3-e8c3366669e8\",\"vserver.vserver-selflink\":\"http://10.12.25.2:8774/v2.1/ebb0ea7144004bacac1e39ff23105fa7/servers/cb2a02d3-88f0-494b-89f3-e8c3366669e8\",\"vserver.in-maint\":false,\"vserver.vserver-name\":\"zdfw1lb01lb01-0514-1\"},\"closedLoopAlarmStart\":1526308051843950,\"closedLoopEventStatus\":\"ONSET\",\"closedLoopControlName\":\"ControlLoop-vDNS-6f37f56d-a87d-4b85-b6a9-cc953cf779b3\",\"version\":\"1.0.2\",\"target\":\"vserver.vserver-name\",\"requestID\":\"26a32b02-c874-4af8-af94-57573ffa343e\",\"from\":\"DCAE\"}" another vVG stack not deleted - wonder if that is correlated to the Openstack Authentication Error ? Vfmodule_Ete_vVG_a591c63a-77ca-40e1-93b5-b22af2b4cdfe @ 1:30 PM EDT test (or so) Yes it is http://10.12.5.33:88/logs/ETE_0473_stability72hr/log.html is the same VNF | ||||||
5/4 12PM-4PM EDT | Marco | 38 out of 40 Passed | 21 out of 24 test Passed | Pass | Pass/Fail | Automated testing issuesDate: 05-14-2018 @12PM-4PM EDT Affected component: AAF Test: Health check Error: ConnectionError: HTTPSConnectionPool(host='aaf.api.simpledemo.onap.org', port=8100): Max retries exceeded with url: /authz/perms/user/demo@people.osaaf.org (Caused by NewConnectionError('<urllib3.connection.VerifiedHTTPSConnection object at 0x7ff9bf31b590>: Failed to establish a new connection: [Errno 111] Connection refused',)) Comment: Non-blocking issue, a fix has been submitted to Gerrit
Date: 05-14-2018 @12PM-4PM EDT Affected component: AAF-SMS Test: Health check Error: 404 != 200 Comment: Non-blocking issue
Date: 05-14-2018 @1PM EDT Affected component: vVG Test: Instantiation Error: '{"request":{"requestId":"cd6440cc-4a8a-499a-995a-9cd1cedb6be5","startTime":"Mon, 14 May 2018 17:16:37 GMT","requestScope":"vfModule","requestType":"createInstance","instanceReferences":{"serviceInstanceId":"cc267011-9954-4a65-8f8b-c65b981092de","vnfInstanceId":"e9491247-cdb4-41a3-9fd7-3033215b45c6","vfModuleInstanceName":"Vfmodule_Ete_vVG_a591c63a-77ca-40e1-93b5-b22af2b4cdfe","requestorId":"demo"},"requestStatus":{"requestState":"FAILED","statusMessage":"Received vfModuleException from VnfAdapter: category='INTERNAL' message='Exception during create VF 0 null: null, error.type=null, error.message=The request you have made requires authentication.' rolledBack='true'","percentProgress":100,"finishTime":"Mon, 14 May 2018 17:17:00 GMT"}}}' contains 'FAILED' Comment: Intermittent failure Manual testingTest: vFirewall instantiation and closed loop Status: Success
Test: vFirewall teardown Status: Success
Test: vLoadBalancer/vDNS closed loop Status: Failed Affected component: SO Error: Exception caught mapping requestBody to RequestDetails com.fasterxml.jackson.databind.exc.UnrecognizedPropertyException: Unrecognized field "requestDetails" Jira ticket: https://jira.onap.org/browse/SO-621 A fix has been submitted to Gerrit
Test: vLB/vDNS (for manual scaling use case) design, distribution, and instantiation Status: Success
Ongoing testingTest: vFirewall instantiation and closed loop - 72-hour soaking test Running since: 05-13-2018 @ 10PM EDT Current status: Success |