...
TODO: update/link Running the ONAP Demos
TODO :20171127 20171207
- using MultiCloud inside SO and changing - To Conclude Cloud-Config.json .json for using MultiCloud for creating the Vf Module .
MultiCloud Support in OOM
- Fixing the External IP Address issue with vFW VM.
TODO :20171128
| Jira | ||||||
|---|---|---|---|---|---|---|
|
- To Conclude Cloud-Config.json for using MultiCloud for creating the Vf Module .
TODO :20171207
- Fixing the External IP Address issue with vFWCL VM's ..OOM Challenges
- Is there similar splitting applicable to vLB/vDNS similar to vFWCL package . TO have a list of small list of kubectl commands needed for OOM .
- OOM Kubernetes Pods gets rebooted and every information is wiped off in OOM .This requires doing the complete service distribution and AAI Steps and VNF/VF Creation all over again .
This page aims to capture all the information , challenges and troubleshooting tricks to run vFWCL Demo successfully . This page assumes that you should have
...
a.4 try check in the browser < private IP address of Sink VM :667 > and check if the Graph are displayed .
a.5 for vLB/vDNS - There is no splitting done as in vFWCL .
vFW Oboarding is step4 to step12 . Post vFW/Service Distribution the workFlow of vFW instantiation gets executed. a.6 vFWCL/vDNS/vLB requires some additional steps to make them pingable .
KubeCtl Commands (OOM Use)
1. Deploy the ONAP using the latest heat template .
1.a location of the onap heat environment files ( environment and yaml file ) is
1.b Modify the environment file as per your environment Openstack/VIO Deployment .
1.c our Environment File with filled values .
1.d Heat Command to be executed on controller Node .
Edit the mso-docker.json file
I see two ways of doing this:
- from the pods itself
kubectl --namespace=onap-mso exec -it mso-3784963895-brdxx bash
vi /shared/mso-docker.json
- from the host running the containers
vi /dockerdata-nfs/onap/mso/mso/mso-docker.json
2. Bounce the mso pod
kubectl --namespace=onap-mso delete pod mso-3784963895-brdxx
openstack stack create -t heat_ONAP_onap_openstack.yaml -e heat_ONAP_onap_openstack.env ONAP
1.e ONAP Stack Creation + ONAP VM's Deployment Challenges
vFW Oboarding is step4 to step12 . Post vFW/Service Distribution the workFlow of vFW instantiation gets executed.
1. Deploy the ONAP using the latest heat template .
1.a location of the onap heat environment files ( environment and yaml file ) is - ONAP Stack gets created within 15-30 mins
- Docker Pull as a part of Each ONAP VM Creation is generally not clean https://gerrit.onap.org/r/gitweb?p=demo.git;a=tree;f=heat/ONAP;h=f948399f5fcbe2300f0d7b8417792aea4682425d;hb=refs/heads/master
1.b Modify the environment file as per your environment Openstack/VIO Deployment .
1.c our Environment File with filled values .
Docker Pull Errors 1.d Heat Command to be executed on controller Node .
a. - Image Not Found Error on Nexus Repo for multiple different Docker Images .openstack stack create -t heat_ONAP_onap_openstack.yaml -e heat_ONAP_onap_openstack.env ONAP
1.e ONAP Stack Creation + ONAP VM's Deployment Challenges
Error: image aaionap/hbase:1.0.0 not found
- ONAP Stack gets created within 15-30 mins
- Docker Pull as a part of Each ONAP VM Creation is generally not clean .
Docker Pull Errors
b- TLS Handshake Error for few docker images .
a. - Image Not Found Error on Nexus Repo for multiple different Docker Images .
Error: image aaionap/hbase:1.0.0 not found
b- TLS Handshake Error for few docker images .
Error response Error response from daemon: Get https://nexus3.onap.org:10001/v2/openecomp/data-router/manifests/1.1-STAGING-latest: net/http: TLS handshake timeout
Error response from daemon: Get https://nexus3.onap.org:10001/v2/openecomp/aai-traversal/manifests/1.1-STAGING-latest: net/http: TLS handshake timeout
c- I/O Timeout
Error response from daemon: Get https://nexus3.onap.org:10001/v1/_ping: dial tcp 199.204.45.137:10001: i/o timeout
...
e- directory creation in VID , SDNC and SDC failes . t
d and e requires manual creation of directory and relaunch Init scripts manually
f f- SDC/SDNC does a GITClone for roughly 800 MB of Size .This hangs intermittently .
workaround -
Post Docker Download errors . VID - remove vid directory and placed docker-compose file inside /opt and start /vid_install.sh
SDC
- create mkdir -p /opt/sdc and trigger reinstall.sh
Post Docker Download errors .
- AAI2 and AAI1 - has Init issues if certain order is not followed .
...
10. Create the Customer inside A&AI where the Region is to be same as given in AAI_ZONE in /var/opt/OpenECOMP_ETE/robot/resources/global_properties.robot (Note that this file is under the openecompete_container Docker)
10.1 create the customer
...
{
"cloud_config":
{
"identity_services":
{
"DEFAULT_KEYSTONE":
{
"identity_url": "http://10.0.14.1/api/multicloud/v0/vmware_vio/identity/v2.0",
"mso_id": "onap",
"mso_pass": "f8cf78bd37b4e258e85076eabb161977",
"admin_tenant": "service",
"member_role": "admin",
"tenant_metadata": true,
"identity_server_type": "KEYSTONE",
"identity_authentication_type": "USERNAME_PASSWORD"
}
},
"cloud_sites":
{
"nova":
{
"region_id": "nova",
"clli": "nova",
"aic_version": "2.5",
"identity_service_id": "DEFAULT_KEYSTONE"
}
}
}
}
19. vFW Network Topology
toConfirm 20171203-
1- for each of the network - i create Router in the horizon .
2 . I update /etc/resolve.conf in each VM to point to external DNS So that VM Can reach out to Open Internet .
3. Sink VM and PktGen VM are generally able to ping on Tenant and ONAP OOM network .
4 Sink VM can ping the protected network gateway .
5. PktGen can not ping unprotected network gateway .
6. firewall VM Can not ping Gateway of ONAP OOM , Protected and unprotected ..
19 .TODO – Issues faced from step13 to Step18 and workaround used .
19.1 Challenges faced
19.1.1 SDC Sanity Docker keep exiting – Needs to raise a JIRA Ticket for the same .This result into every operation from Portal VID GUI resulting in 500 error or 400 error and no operation succeed .
WorkAround
18.2 registering MultiCloud to AAI-ESR
There are two ways to register a VIM to A&AI
You can register VIM from esr gui http://MSB_SERVER_IP:80/iui/aai-esr-gui/extsys/vim/vimView.html . For the ESR usage detail you can refer to http://onap.readthedocs.io/en/latest/submodules/aai/esr-gui.git/docs/platform/installation.html.
2. Register VIM with the API from A&AI, here is an example
PUT https://A&AI_SERVER_IP:8443/aai/v11/cloud-infrastructure/cloud-regions/cloud-region/ZTE/region-one
Authorization:
header:
body:
{
"cloud-owner": "ZTE",
"cloud-region-id": "region-one",
"cloud-type": "openstack",
"owner-defined-type": "owner-defined-type",
"cloud-region-version": "ocata",
"cloud-zone": "cloud zone",
"complex-name": "complex name",
"sriov-automation": false,
"cloud-extra-info": "cloud-extra-info",
"esr-system-info-list": {
"esr-system-info": [
{ This is works as designed .Non Issue .
19.1.2 SDNC VM "root" becomes 100% full this results into container being unstable and keep exiting - JIRA Ticket raised "esr-system-info-id": "432ac032-e996-41f2-84ed-9c7a1766eb29",
19.1.3 Once the SDC is into Issue of 500 or 400 error - The sevices gets into inconsistent state and that requires creating , distributiing the service all over again including customer creation into AAI ."service-url": "http://10.74.151.22:5000/v2.0",
19.1.4 SDNC Login page throwing Error :- "Unable to get database connection :Error :connect ECONNREFUSED 127.0.0.1:3306" . Pls see the screen below
WorkAround
"user-name": "admin",
-- remove the SDNC Docker – restart the SDNC Docker .
TODO - raise a JIRA Ticket for the SDNC Issue .
19.1.5 SO Complaining about Certificate Error while calling createVfModule . createVfModule fails
017-11-22T13:31:17.150Z|19374e93-7461-4303-8de9-13d105ab519b|keystoneUrl=http://10.110.208.162:5000/v2.0
2017-11-22T13:31:17.478Z|19374e93-7461-4303-8de9-13d105ab519b|heatUrl=https://10.110.209.230:8004/v1/74c7fa9e54f246f5878c902c346e590d, region=nova
2017-11-22T13:31:17.478Z|19374e93-7461-4303-8de9-13d105ab519b|Caching HEAT Client for nova:74c7fa9e54f246f5878c902c346e590d
2017-11-22T13:31:17.478Z|19374e93-7461-4303-8de9-13d105ab519b|Found: com.woorea.openstack.heat.Heat@3ca4e80c
2017-11-22T13:31:17.487Z|19374e93-7461-4303-8de9-13d105ab519b|OpenstackConnectException at:org.openecomp.mso.openstack.utils.MsoHeatUtils.queryHeatStack request:StackResource.GetStack Retry indicated. Attempts remaining:2
2017-11-22T13:31:22.496Z|19374e93-7461-4303-8de9-13d105ab519b|OpenstackConnectException at:org.openecomp.mso.openstack.utils.MsoHeatUtils.queryHeatStack request:StackResource.GetStack Retry indicated. Attempts remaining:1
2017-11-22T13:31:27.508Z|19374e93-7461-4303-8de9-13d105ab519b|OpenstackConnectException at:org.openecomp.mso.openstack.utils.MsoHeatUtils.queryHeatStack request:StackResource.GetStack Retry indicated. Attempts remaining:0
2017-11-22T13:31:32.518Z|19374e93-7461-4303-8de9-13d105ab519b|MSO-RA-9202E Exception communicating with OpenStack: Openstack Heat connection error on QueryAllStack: com.woorea.openstack.base.client.OpenStackConnectException: sun.security.validator.ValidatorException: PKIX path building failed: sun.security.provider.certpath.SunCertPathBuilderException: unable to find valid certification path to requested target
2017-11-22T13:31:32.518Z|19374e93-7461-4303-8de9-13d105ab519b|
Solution
picked up the VIO Certifcate from the loadBalance VM
: /usr/local/share/ca-certificates and copied to : /usr/local/share/ca-certificates
inside MSO_TestLab Container .
update-ca-certificates with root inside the mso_testlab docker
19.1.6 ADD VNF Failes From VID GUI with the error – No Valid Catalogue Entry Specified
WorkAround
--- Create a new user via AAI Rest Command and new service using demo.sh init .
"password": "admin",
"system-type": "VIM",
"ssl-insecure": true,
"cloud-domain": "cloud-domain"
}
]
}
}
19. vFW Network Topology
20 Additional Step For vFWCL on VIO 20171207-
1- for each of the network - create Router in the horizon .
2 . update /etc/resolve.conf in firewall VM to point to external DNS (10.112.64.1 ) So that VM Can reach out to Open Internet .
3. Check on Horizon under the networks → <Network Name> → Ports to see if the Gateway IP address of the subnet/network is showen as "router:interface"
4 Since we are updating the network given in the sdnc preload - please make sure you login to each VM (Firewall VM , Sink VM and PktGen VM )
4.1 Login using tenant network and remove every other network .
4.2 once logged into VM update the ip address and cidr files present in /opt/config in each VM
4.3 also update the /etc/network/interfaces files for eth1 , eth2 for Sink and PktGen VM and eth1 ,eth2 , eth3 for Firewall VM .
4.4 attach the Network via Horizon to each of these VM for Firewall VM , try disabling the v_firewall_install.sh and v_firewall_init.sh after running these script onces .
5. Sink VM and PktGen VM are generally able to ping on Tenant and ONAP OOM network including Robot VM .
6 Sink VM can ping the protected network gateway .
7. PktGen can ping unprotected network gateway .
8. firewall VM Can not ping Gateway of ONAP OOM .-- Why this is piece of investigation . .
tocompare with 20171205
1. Network Toplogy Example from a successful vFWCL Lab on Openstack /OOM
2. Nothing was done as in doing additional steps .
3. 667 Port was not open on Sink VM in successful vFW Demo case .
21 Additional Step For vLB/ vDNS on VIO 20171207-
1- for each of the network - create Router in the horizon .
2 . update /etc/resolve.conf in firewall VM to point to external DNS So that VMs can reach out to Open Internet .
3. Check on Horizon under the networks → <Network Name> → Ports to see if the Gateway IP address of the subnet/network is showen as "router:interface"
4 Since we are updating the network given in the sdnc preload - please make sure you login to each VM (Load Balancer VM, DNS VM and PktGen VM )
4.1 Login using tenant network and remove every other network
4.2 once logged into VM update the ip address and cidr files present in /opt/config in each VM
4.3 also update the /etc/network/interfaces files for eth1 , eth2 for DNS and PktGen VM and eth1 ,eth2 , eth3 for Load Balancer VM .
4.4 attach the Network via Horizon to each of these VM for Load Balancer VM , try disabling the v_firewall_install.sh and v_firewall_init.sh after running these script onces .
5. DNS VM and PktGen VM are generally able to ping on Tenant and ONAP OOM network including Robot VM .
6 DNS VM can ping the protected network gateway .
7. PktGen can ping unprotected network gateway .
8. Load Balancer VM Can not ping Gateway of ONAP OOM .-- Why this, is piece of investigation..
22 .TODO – Issues faced from step13 to Step18 and workaround used .
22.1 Challenges faced
22.1.1 SDC Sanity Docker keep exiting – Needs to raise a JIRA Ticket for the same .This result into every operation from Portal VID GUI resulting in 500 error or 400 error and no operation succeed .
WorkAround
This is works as designed .Non Issue .
22.1.2 SDNC VM "root" becomes 100% full this results into container being unstable and keep exiting - JIRA Ticket raised
22.1.3 Once the SDC is into Issue of 500 or 400 error - The sevices gets into inconsistent state and that requires creating , distributiing the service all over again including customer creation into AAI .
22.1.4 SDNC Login page throwing Error :- "Unable to get database connection :Error :connect ECONNREFUSED 127.0.0.1:3306" . Pls see the screen below
WorkAround
-- remove the SDNC Docker
– restart the SDNC Docker .
TODO - raise a JIRA Ticket for the SDNC Issue .
22.1.5 SO Complaining about Certificate Error while calling createVfModule . createVfModule fails
017-11-22T13:31:17.150Z|19374e93-7461-4303-8de9-13d105ab519b|keystoneUrl=http://10.110.208.162:5000/v2.0
2017-11-22T13:31:17.478Z|19374e93-7461-4303-8de9-13d105ab519b|heatUrl=https://10.110.209.230:8004/v1/74c7fa9e54f246f5878c902c346e590d, region=nova
2017-11-22T13:31:17.478Z|19374e93-7461-4303-8de9-13d105ab519b|Caching HEAT Client for nova:74c7fa9e54f246f5878c902c346e590d
2017-11-22T13:31:17.478Z|19374e93-7461-4303-8de9-13d105ab519b|Found: com.woorea.openstack.heat.Heat@3ca4e80c
2017-11-22T13:31:17.487Z|19374e93-7461-4303-8de9-13d105ab519b|OpenstackConnectException at:org.openecomp.mso.openstack.utils.MsoHeatUtils.queryHeatStack request:StackResource.GetStack Retry indicated. Attempts remaining:2
2017-11-22T13:31:22.496Z|19374e93-7461-4303-8de9-13d105ab519b|OpenstackConnectException at:org.openecomp.mso.openstack.utils.MsoHeatUtils.queryHeatStack request:StackResource.GetStack Retry indicated. Attempts remaining:1
2017-11-22T13:31:27.508Z|19374e93-7461-4303-8de9-13d105ab519b|OpenstackConnectException at:org.openecomp.mso.openstack.utils.MsoHeatUtils.queryHeatStack request:StackResource.GetStack Retry indicated. Attempts remaining:0
2017-11-22T13:31:32.518Z|19374e93-7461-4303-8de9-13d105ab519b|MSO-RA-9202E Exception communicating with OpenStack: Openstack Heat connection error on QueryAllStack: com.woorea.openstack.base.client.OpenStackConnectException: sun.security.validator.ValidatorException: PKIX path building failed: sun.security.provider.certpath.SunCertPathBuilderException: unable to find valid certification path to requested target
2017-11-22T13:31:32.518Z|19374e93-7461-4303-8de9-13d105ab519b|
Solution
picked up the VIO Certifcate from the loadBalance VM
: /usr/local/share/ca-certificates and copied to : /usr/local/share/ca-certificates
inside MSO_TestLab Container .
update-ca-certificates with root inside the mso_testlab docker
22.1.6 ADD VNF Failes From VID GUI with the error – No Valid Catalogue Entry Specified
WorkAround
--- Create a new user via AAI Rest Command and new service using demo.sh init .
- Once the service is successfully distributed Try creating the VNF from VID Gui , It shall succeed .
23. Working with Multi-cloud
- Multi-cloud project provides interfaces to work with a variety of VIMs. When using Multi-cloud its service interfaces are used by the applications instead of VIM interfaces.
- To register your VIM and use it through Multi-cloud interfaces, following need to be done:
- Register AAI services with MSB. Following are the REST Requests to do it:
curl -X POST -H “Content-Type: application/json” -d ‘{“serviceName”: “aai-cloudInfrastructure”, “version”: “v11”, “url”: “/aai/v11/cloud-infrastructure”,”protocol”: “REST”, “enable_ssl”:”true”, “visualRange”:”1”, “nodes”: [ {“ip”: “A&AI_SERVER_IP”,”port”: “8443”}]}’ “http://MSB_SERVER_IP:10081/api/microservices/v1/services“
curl -X POST -H “Content-Type: application/json” -d ‘{“serviceName”: “aai-externalSystem”, “version”: “v11”, “url”: “/aai/v11/external-system”,”protocol”: “REST”, “enable_ssl”:”true”, “visualRange”:”1”, “nodes”: [ {“ip”: “A&AI_SERVER_IP”,”port”: “8443”}]}’ “http://MSB_SERVER_IP:10081/api/microservices/v1/services“ - Register ESR services with MSB. Following are the REST Requests to do it:
curl -X POST -H “Content-Type: application/json” -d ‘{“serviceName”: “aai-esr-server”, “version”: “v1”, “url”: “/api/aai-esr-server/v1”,”protocol”: “REST”, “visualRange”:”1”, “nodes”: [ {“ip”: “ESR_SERVER_IP”,”port”: “9518”}]}’ “http://MSB_SERVER_IP:10081/api/microservices/v1/services“
curl -X POST -H “Content-Type: application/json” -d ‘{“serviceName”: “aai-esr-gui”, “url”: “/esr-gui”,”protocol”: “UI”, “visualRange”:”1”, “path”:”/iui/aai-esr-gui”, “nodes”: [ {“ip”: “ESR_SERVER_IP”,”port”: “9519”}]}’ “http://MSB_SERVER_IP:10081/api/microservices/v1/services“ - Register Multicloud framework services and your VIM specific services with MSB. Following are the REST Requests to do it:
curl -X POST -H “Content-Type: application/json” -d ‘{“serviceName”: “multicloud”, “version”: “v0”, “url”: “/api/multicloud/v0”,”protocol”: “REST”, “nodes”: [ {“ip”: “’$MultiCloud_IP’”,”port”: “9001”}]}’ “http://$MSB_SERVER_IP:10081/api/microservices/v1/services“
curl -X POST -H “Content-Type: application/json” -d ‘{“serviceName”: “multicloud-vio”, “version”: “v0”, “url”: “/api/multicloud-vio/v0”,”protocol”: “REST”, “nodes”: [ {“ip”: “’$MultiCloud_IP’”,”port”: “9004”}]}’ “http://$MSB_SERVER_IP:10081/api/microservices/v1/services“ - Register VIM Information in AAI with region name “vmware” and region id “vio”. Following is the REST Request to do it:
curl -X PUT -H "Authorization: Basic QUFJOkFBSQ==" -H "Content-Type: application/json" -H "X-TransactionId:get_aai_subcr" \
https://aai_resource_docker_host_ip:30233/aai/v01/cloud-infrastructure/cloud-regions/cloud-region/vmware/vio \
- Register AAI services with MSB. Following are the REST Requests to do it:
-d "{
"cloud-owner": "vmware",
"cloud-type": "vmware",
"cloud-region-version": "4.0",
"esr-system-info-list": {
"esr-system-info": [
{
"esr-system-info-id": "123-456",
"system-name": "vim-vio",
"system-type": "vim",
"type": "vim",
"user-name": "admin",
"password": "vmware",
"service-url": "<keystone auth url>",
"cloud-domain": "default",
"default-tenant": "admin",
"ssl-insecure": false
}
]
}
}"
Please note: The IP and port numbers may vary in HEAT vs OOM based ONAP setup. This need to be considered while formulating the curl requests - Once the service is successfully distributed Try creating the VNF from VID Gui , It shall succeed .