- Created by Brian Freeman, last modified on Jul 13, 2017
Install and run clearwater automated test suite
- Install the clearwater live testsuite on a VM
- could use the robot framework since its lightly loaded unless you want a VM that will be around if you want to re-install ONAP
- Get the public IPs for :
- bono-o (PROXY):
- ellis-0 (ELLIS):
- sprout-0 (GEMINI) :
- MOMENTO_HTTP: <same as sprout>
- MOMENTO_SID: <same as sprout>
- Get the domain from the preload data
- The default Clearwater SIGNUP_CODE/password is "secret"
- Run the clearwater-live-tests
ubuntu@vm1-robot:~/clearwater-live-test$ rake test[vimstest.onap.org] SIGNUP_CODE=secret ELLIS=<ellis-0 IP> PROXY=<bono-0 IP> GEMINI=<sprout-0 IP> MEMENTO_SIP=<sprout-0 IP> MOMENTO_HTTP=<sprout-0 IP> ELLIS_API_KEY=secret
6. the automated test report is displayed in real time with Passed/Failed
7. The default configuration should have 11 failed out of 62 run tests (88 total tests but 26 skipped)
Test Results
ubuntu@vm1-robot:~/clearwater-live-test$ rake test[vimstest.onap.org] SIGNUP_CODE=secret ELLIS=<ellis-0 IP> PROXY=<bono-0 IP> GEMINI=<sprout-0 IP> MOMENTO_SIP=<sprout-0 IP> MOMENTO_HTTP=<sprout-0 IP> ELLIS_API_KEY=secret
/usr/lib/ruby/vendor_ruby/bundler/shared_helpers.rb:78: warning: Insecure world writable dir /home in PATH, mode 040777
Basic Call - Mainline (TCP) - (5555551002, 5555551008) Passed
Basic Call - SDP (TCP) - (5555551011, 5555551009) Passed
Basic Call - Tel URIs (TCP) - (5555551002, 5555551008) Passed
Basic Call - Unknown number (TCP) - (5555551008, 5555551002) Passed
Basic Call - Rejected by remote endpoint (TCP) - (5555551001, 5555551008) Passed
Basic Call - Messages - Pager model (TCP) - (5555551019, 5555551005) Passed
Basic Call - Pracks (TCP) - (5555551018, 5555551002) Passed
Basic Registration (TCP) - (5555551001) Passed
Multiple Identities (TCP) - (5555551006, 5555551004) Passed
Call Barring - Outbound Rejection (TCP) - (5555551015, 5555551005) Passed
Call Barring - Allow non-international call (TCP) - Skipped (No PSTN support)
- Call with PSTN=true to run test
Call Barring - Reject international call (TCP) - Skipped (No PSTN support)
- Call with PSTN=true to run test
Call Barring - Inbound Rejection (TCP) - (5555551010, 5555551002) Passed
Call Diversion AS - Unconditional (TCP) - Skipped (No Call Diversion AS hostname provided)
- Call with CDIV_AS=<hostname>
Call Diversion AS - Busy (TCP) - Skipped (No Call Diversion AS hostname provided)
- Call with CDIV_AS=<hostname>
Call Diversion AS - No answer (TCP) - Skipped (No Call Diversion AS hostname provided)
- Call with CDIV_AS=<hostname>
Call Diversion - Not registered (TCP) - (5555551012, 5555551016, 5555551017) Passed
Call Diversion - Not reachable (not registered) (TCP) - (5555551019, 5555551006, 5555551000) Passed
Call Diversion - Not reachable (408) (TCP) - (5555551000, 5555551006, 5555551019) Passed
Call Diversion - Not reachable (503) (TCP) - (5555551007, 5555551016, 5555551019) Passed
Call Diversion - Not reachable (500) (TCP) - (5555551001, 5555551009, 5555551012) Passed
Call Diversion - Busy (TCP) - (5555551006, 5555551011, 5555551015) Passed
Call Diversion - Unconditional (TCP) - (5555551011, 5555551009, 5555551013) Passed
Call Diversion - No answer (TCP) - (5555551009, 5555551005, 5555551013) Passed
Call Diversion - Bad target URI (TCP) - (5555551004, 5555551019) Passed
Call Diversion - Audio-only call (TCP) - (5555551006, 5555551018, 5555551014, 5555551002) Passed
Call Diversion - Audio-video call (TCP) - (5555551006, 5555551005, 5555551007, 5555551002) Passed
Call Waiting - Accepted (TCP) - (5555551009, 5555551008, 5555551017) Passed
Call Waiting - Cancelled (TCP) - (5555551011, 5555551012, 5555551007) Passed
CANCEL - Mainline (TCP) - (5555551011, 5555551017) Passed
Filtering - Accept-Contact (TCP) - (5555551000, 5555551016) Passed
Filtering - Accept-Contact no match (TCP) - (5555551005, 5555551015) Passed
Filtering - Accept-Contact negated match (TCP) - (5555551016, 5555551019) Passed
Filtering - RFC3841 example (TCP) - (5555551000, 5555551008) Passed
Filtering - Reject-Contact no match (TCP) - (5555551012, 5555551011) Passed
Filtering - Reject-Contact match (TCP) - (5555551005, 5555551019) Passed
Gemini - INVITE - VoIP device answers (TCP) - (5555551019, 5555551000, 1235555551000@vimstest.onap.org) Failed
Endpoint threw exception:
- undefined method `include?' for nil:NilClass
- /home/ubuntu/clearwater-live-test/lib/tests/gemini.rb:72:in `block (2 levels) in <top (required)>'
Terminating other threads after failure
Gemini - INVITE - Mobile device answers (TCP) - (5555551016, 5555551011, 1235555551011@vimstest.onap.org) Failed
Endpoint threw exception:
- undefined method `include?' for nil:NilClass
- /home/ubuntu/clearwater-live-test/lib/tests/gemini.rb:165:in `block (2 levels) in <top (required)>'
Terminating other threads after failure
Gemini - INVITE - VoIP device rejects (TCP) - (5555551015, 5555551003, 1235555551003@vimstest.onap.org) Failed
Endpoint threw exception:
- undefined method `include?' for nil:NilClass
- /home/ubuntu/clearwater-live-test/lib/tests/gemini.rb:259:in `block (2 levels) in <top (required)>'
Terminating other threads after failure
Gemini - INVITE - Mobile device rejects (TCP) - (5555551014, 5555551009, 1235555551009@vimstest.onap.org) Failed
Endpoint threw exception:
- undefined method `include?' for nil:NilClass
- /home/ubuntu/clearwater-live-test/lib/tests/gemini.rb:347:in `block (2 levels) in <top (required)>'
Terminating other threads after failure
Gemini - INVITE - Mobile device rejects with a 480 (TCP) - (5555551013, 5555551001, 1235555551001@vimstest.onap.org) Failed
Endpoint threw exception:
- undefined method `include?' for nil:NilClass
- /home/ubuntu/clearwater-live-test/lib/tests/gemini.rb:443:in `block (2 levels) in <top (required)>'
Terminating other threads after failure
Endpoint threw exception:
- undefined method `include?' for nil:NilClass
- /home/ubuntu/clearwater-live-test/lib/tests/gemini.rb:458:in `block (2 levels) in <top (required)>'
Gemini - INVITE - Both reject, choose mobile response (TCP) - (5555551008, 5555551001, 1235555551001@vimstest.onap.org) Failed
Endpoint threw exception:
- undefined method `include?' for nil:NilClass
- /home/ubuntu/clearwater-live-test/lib/tests/gemini.rb:535:in `block (2 levels) in <top (required)>'
Terminating other threads after failure
Gemini - INVITE - Both reject, choose VoIP response (TCP) - (5555551011, 5555551013, 1235555551013@vimstest.onap.org) Failed
Endpoint threw exception:
- undefined method `include?' for nil:NilClass
- /home/ubuntu/clearwater-live-test/lib/tests/gemini.rb:610:in `block (2 levels) in <top (required)>'
Terminating other threads after failure
Gemini - INVITE - Successful call with GR (TCP) - (5555551015, 5555551002, 1235555551002@vimstest.onap.org) Passed
Gemini - INVITE - Failed call with GR (TCP) - (5555551003, 5555551002, 1235555551002@vimstest.onap.org) Passed
Gemini - INVITE - Successful call with Accept-Contact (TCP) - (5555551015, 5555551017, 1235555551017@vimstest.onap.org) Failed
Endpoint threw exception:
- sip:1235555551017@vimstest.onap.org timed out waiting for new incoming call
- /home/ubuntu/.bundle/ruby/2.3.0/bundler/gems/quaff-f92372cac3e1/lib/endpoint.rb:65:in `rescue in incoming_call'
- /home/ubuntu/.bundle/ruby/2.3.0/bundler/gems/quaff-f92372cac3e1/lib/endpoint.rb:62:in `incoming_call'
- /home/ubuntu/clearwater-live-test/lib/tests/gemini.rb:808:in `block (2 levels) in <top (required)>'
Terminating other threads after failure
Gemini - INVITE - Failed call with Accept-Contact (TCP) - (5555551005, 5555551003, 1235555551003@vimstest.onap.org) Failed
Endpoint threw exception:
- sip:1235555551003@vimstest.onap.org timed out waiting for new incoming call
- /home/ubuntu/.bundle/ruby/2.3.0/bundler/gems/quaff-f92372cac3e1/lib/endpoint.rb:65:in `rescue in incoming_call'
- /home/ubuntu/.bundle/ruby/2.3.0/bundler/gems/quaff-f92372cac3e1/lib/endpoint.rb:62:in `incoming_call'
- /home/ubuntu/clearwater-live-test/lib/tests/gemini.rb:864:in `block (2 levels) in <top (required)>'
Terminating other threads after failure
Gemini - SUBSCRIBE - Mobile Notifies (TCP) - (5555551015, 5555551000, 1235555551000@vimstest.onap.org) Failed
Endpoint threw exception:
- undefined method `include?' for nil:NilClass
- /home/ubuntu/clearwater-live-test/lib/tests/gemini.rb:926:in `block (2 levels) in <top (required)>'
Terminating other threads after failure
Gemini - SUBSCRIBE - Joint 408 (TCP) - (5555551003, 5555551004, 1235555551004@vimstest.onap.org) Failed
Endpoint threw exception:
- undefined method `include?' for nil:NilClass
- /home/ubuntu/clearwater-live-test/lib/tests/gemini.rb:988:in `block (2 levels) in <top (required)>'
Terminating other threads after failure
GRUU - REGISTER - two bindings with and without GRUU (TCP) - (5555551010) Passed
GRUU - REGISTER - binding suggested GRUU (TCP) - (5555551014) Passed
GRUU - REGISTER - instance ID requires escaping (TCP) - (5555551003) Passed
GRUU - REGISTER - three bindings with GRUUs (TCP) - (5555551014) Passed
GRUU - Call - first endpoint GRUU as target (TCP) - (5555551000, 5555551017) Passed
GRUU - Call - second endpoint GRUU as target (TCP) - (5555551013, 5555551006) Passed
GRUU - Call - only GRUU as target (TCP) - (5555551013, 5555551019) Passed
GRUU - Call - AoR as target (TCP) - (5555551015, 5555551014) Passed
GRUU - Call - unknown GRUU as target (TCP) - (5555551003, 5555551008) Passed
GRUU - Call - unknown GRUU as target - no GRUUs assigned (TCP) - (5555551003, 5555551005) Passed
GRUU - Call - Reject-Contact interop (TCP) - (5555551013, 5555551010) Passed
GRUU - Call - Accept-Contact interop (TCP) - (5555551003, 5555551012) Passed
GRUU - Call - AoR with other param as target (TCP) - (5555551013, 5555551003) Passed
GRUU - Call - GRUU with other param as target (TCP) - (5555551013, 5555551006) Passed
ISC Interface - Terminating (TCP) - Skipped (No hostname given)
- Call with HOSTNAME=<publicly accessible hostname/IP of this machine>
ISC Interface - Terminating (UDP AS) (TCP) - Skipped (No hostname given)
- Call with HOSTNAME=<publicly accessible hostname/IP of this machine>
ISC Interface - Terminating Failed (TCP) - Skipped (No hostname given)
- Call with HOSTNAME=<publicly accessible hostname/IP of this machine>
ISC Interface - Third-party Registration (TCP) - Skipped (No hostname given)
- Call with HOSTNAME=<publicly accessible hostname/IP of this machine>
ISC Interface - Third-party Registration - implicit registration (TCP) - Skipped (No hostname given)
- Call with HOSTNAME=<publicly accessible hostname/IP of this machine>
ISC Interface - Redirect (TCP) - Skipped (No hostname given)
- Call with HOSTNAME=<publicly accessible hostname/IP of this machine>
ISC Interface - B2BUA (TCP) - Skipped (No hostname given)
- Call with HOSTNAME=<publicly accessible hostname/IP of this machine>
Live Call - Dial out to a real number (TCP) - Skipped (No live number given)
- Call with LIVENUMBER=<number>
Memento - Incorrect Password (TCP) - Skipped (No memento hostnames provided)
- Call with MEMENTO_SIP=<SIP hostname> and MEMENTO_HTTP=<HTTP hostname>
Memento - Wrong Call List (TCP) - Skipped (No memento hostnames provided)
- Call with MEMENTO_SIP=<SIP hostname> and MEMENTO_HTTP=<HTTP hostname>
Memento - Basic Call (TCP) - Skipped (No memento hostnames provided)
- Call with MEMENTO_SIP=<SIP hostname> and MEMENTO_HTTP=<HTTP hostname>
Memento - Unknown Number (TCP) - Skipped (No memento hostnames provided)
- Call with MEMENTO_SIP=<SIP hostname> and MEMENTO_HTTP=<HTTP hostname>
Memento - Rejected Call (TCP) - Skipped (No memento hostnames provided)
- Call with MEMENTO_SIP=<SIP hostname> and MEMENTO_HTTP=<HTTP hostname>
Memento - Cancelled Call (TCP) - Skipped (No memento hostnames provided)
- Call with MEMENTO_SIP=<SIP hostname> and MEMENTO_HTTP=<HTTP hostname>
Memento - Privacy Call (TCP) - Skipped (No memento hostnames provided)
- Call with MEMENTO_SIP=<SIP hostname> and MEMENTO_HTTP=<HTTP hostname>
Memento - Barred Call (TCP) - Skipped (No memento hostnames provided)
- Call with MEMENTO_SIP=<SIP hostname> and MEMENTO_HTTP=<HTTP hostname>
Memento - Busy Call Forwarding (TCP) - Skipped (No memento hostnames provided)
- Call with MEMENTO_SIP=<SIP hostname> and MEMENTO_HTTP=<HTTP hostname>
Nonce-Count - Mainline (TCP) - Skipped (No nonce-count support)
- Call with NONCE_COUNT=true to run test
Nonce-Count - Reject Re-Use (TCP) - Skipped (No nonce-count support)
- Call with NONCE_COUNT=true to run test
Off-net calls - tel: URI (TCP) - Skipped (No off-net number given)
- Call with OFF_NET_TEL=<a number set up in ENUM/BGCF to route to port 5072 on this machine>
Off-net calls - sip: URI (TCP) - Skipped (No off-net number given)
- Call with OFF_NET_TEL=<a number set up in ENUM/BGCF to route to port 5072 on this machine>
SUBSCRIBE - reg-event (TCP) - (5555551015) Passed
SUBSCRIBE - reg-event with a GRUU (TCP) - (5555551005) Passed
SUBSCRIBE - Subscription timeout (TCP) - (5555551002) Passed
SUBSCRIBE - Registration timeout (TCP) - (5555551010, 5555551014) Passed
11 failures out of 88 tests run
Gemini - INVITE - VoIP device answers (TCP) at 2017-07-13 19:41:11 +0000
Gemini - INVITE - Mobile device answers (TCP) at 2017-07-13 19:41:17 +0000
Gemini - INVITE - VoIP device rejects (TCP) at 2017-07-13 19:41:23 +0000
Gemini - INVITE - Mobile device rejects (TCP) at 2017-07-13 19:41:29 +0000
Gemini - INVITE - Mobile device rejects with a 480 (TCP) at 2017-07-13 19:41:37 +0000
Gemini - INVITE - Both reject, choose mobile response (TCP) at 2017-07-13 19:41:43 +0000
Gemini - INVITE - Both reject, choose VoIP response (TCP) at 2017-07-13 19:41:49 +0000
Gemini - INVITE - Successful call with Accept-Contact (TCP) at 2017-07-13 19:42:35 +0000
Gemini - INVITE - Failed call with Accept-Contact (TCP) at 2017-07-13 19:43:08 +0000
Gemini - SUBSCRIBE - Mobile Notifies (TCP) at 2017-07-13 19:43:14 +0000
Gemini - SUBSCRIBE - Joint 408 (TCP) at 2017-07-13 19:43:20 +0000
26 tests skipped
Error logs, including Call-IDs of failed calls, are in the 'logfiles' directory