This page details deployment of ONAP on any environment that supports Kubernetes based containers.
Quickstart
In review
- OOM-710Getting issue details... STATUS , - OOM-716Getting issue details... STATUS , - OOM-715Getting issue details... STATUS ,
- OOM-714Getting issue details... STATUS
- OOM-711Getting issue details... STATUS
- OOM-713Getting issue details... STATUS
Two choices, run the single oom_deployment.sh ARM, CloudFormation, Heat template wrapper as a oneclick - or use it to bring up an empty single VM and run oom_entrypoint.sh manually. Once the VM comes up the oom_entrypoint.sh script will run - which will download the oom_rancher_setup.sh script to setup docker, rancher, kubernetes and helm - the entrypoint script will then run the cd.sh script to bring up onap based on your values.yaml config by running helm install on it. Recommend running a cluster - I am splitting the script and will post a 2nd set shortly
# if you wish to run the oom_entrypoint script yourself - edit/break the cloud init section at the end of the arm/cloudformation/heat template and do it yourself below # download and edit values.yaml with your onap preferences and openstack tenant config wget https://jira.onap.org/secure/attachment/11414/values.yaml # download and run the bootstrap and onap install script, the -s server name can be an IP, FQDN or hostname - this is on OOM-710 wget https://jira.onap.org/secure/attachment/11518/oom_entrypoint.sh chmod 777 oom_entrypoint.sh sudo ./oom_entrypoint.sh -b master -s devops.onap.info -e onap # wait 15 min for rancher to finish, then 30-90 min for onap to come up
entrypoint aws/azure/openstack | Ubuntu 16 rancher install | oom deployment CD script | |
---|---|---|---|
Requirements
Hardware Requirements
Release | RAM | HD | vCores | Ports | Network |
---|---|---|---|---|---|
any | 63-70G at startup Recommend 3 x 64G class VM | 40G per host min (30G for dockers) 100G after a week 5G min per NFS | 8 min 60 peak at startup recommended 16-64 vCores | see list on PortProfile Recommend 0.0.0.0/0 (all open) inside VPC Block 10249-10255 outside | 170 MB/sec |
Amazon AWS
Account Provider: (2) Robin of Amazon and Michael O'Brien of Amdocs
Amazon has donated an allocation enough for 256G of VM space in order to run CD systems since Dec 2017 - thank you |
Amazon AWS is currently hosting our RI for ONAP Continuous Deployment - this is a joint Proof Of Concept between Amazon and ONAP.
Auto Continuous Deployment via Jenkins and Kibana
AWS CLI Installation
Install the AWS CLI on the bastion VM
https://docs.aws.amazon.com/cli/latest/userguide/cli-install-macos.html
OSX
obrien:obrienlabs amdocs$ pip --version pip 9.0.1 from /Library/Python/2.7/site-packages/pip-9.0.1-py2.7.egg (python 2.7) obrien:obrienlabs amdocs$ curl -O https://bootstrap.pypa.io/get-pip.py obrien:obrienlabs amdocs$ python3 get-pip.py --user Requirement already up-to-date: pip in /Library/Frameworks/Python.framework/Versions/3.6/lib/python3.6/site-packages obrien:obrienlabs amdocs$ pip3 install awscli --upgrade --user Successfully installed awscli-1.14.41 botocore-1.8.45 pyasn1-0.4.2 s3transfer-0.1.13
Ubuntu
obrien:obrienlabs amdocs$ ssh ubuntu@dev.onap.info $ sudo apt install python-pip $ pip install awscli --upgrade --user $ aws --version aws-cli/1.14.41 Python/2.7.12 Linux/4.4.0-1041-aws botocore/1.8.45
Windows Powershell
Configure Access Keys for your Account
$aws configure AWS Access Key ID [None]: AK....Q AWS Secret Access Key [None]: Dl....l Default region name [None]: us-east-1 Default output format [None]: json $aws ec2 describe-regions --output table || ec2.ca-central-1.amazonaws.com | ca-central-1 || ....
Option 0: Deploy OOM Kubernetes to a spot VM
Peak Performance Metrics
We hit a peak of 44 cores during startup, with an external network peak of 1.2Gbps (throttled nexus servers at ONAP), a peak SSD write rate of 4Gbps and 55G ram on a 64 vCore/256G VM on AWS Spot.
Kubernetes Installation via CLI
Allocate an EIP static public IP (one-time)
https://docs.aws.amazon.com/cli/latest/reference/ec2/allocate-address.html
$aws ec2 allocate-address { "PublicIp": "35.172..", "Domain": "vpc", "AllocationId": "eipalloc-2f743..."}
Create a Route53 Record Set - Type A (one-time)
$ cat route53-a-record-change-set.json {"Comment": "comment","Changes": [ { "Action": "CREATE", "ResourceRecordSet": { "Name": "amazon.onap.cloud", "Type": "A", "TTL": 300, "ResourceRecords": [ { "Value": "35.172.36.." }]}}]} $ aws route53 change-resource-record-sets --hosted-zone-id Z...7 --change-batch file://route53-a-record-change-set.json { "ChangeInfo": { "Status": "PENDING", "Comment": "comment", "SubmittedAt": "2018-02-17T15:02:46.512Z", "Id": "/change/C2QUNYTDVF453x" }} $ dig amazon.onap.cloud ; <<>> DiG 9.9.7-P3 <<>> amazon.onap.cloud amazon.onap.cloud. 300 IN A 35.172.36.. onap.cloud. 172800 IN NS ns-1392.awsdns-46.org.
Request a spot EC2 Instance
# request the usually cheapest $0.13 spot 64G EBS instance at AWS aws ec2 request-spot-instances --spot-price "0.25" --instance-count 1 --type "one-time" --launch-specification file://aws_ec2_spot_cli.json # don't pass in the the following - it will be generated for the EBS volume "SnapshotId": "snap-0cfc17b071e696816" launch specification json { "ImageId": "ami-c0c964ba", "InstanceType": "r4.2xlarge", "KeyName": "obrien_systems_aws_2015", "BlockDeviceMappings": [ {"DeviceName": "/dev/sda1", "Ebs": { "DeleteOnTermination": true, "VolumeType": "gp2", "VolumeSize": 120 }}], "SecurityGroupIds": [ "sg-322c4842" ]} # results { "SpotInstanceRequests": [{ "Status": { "Message": "Your Spot request has been submitted for review, and is pending evaluation.", "Code": "pending-evaluation",
Get EC2 instanceId after creation
aws ec2 describe-spot-instance-requests --spot-instance-request-id sir-1tyr5etg "InstanceId": "i-02a653592cb748e2x",
Associate EIP with EC2 Instance
Can be done separately as long as it is in the first 30 sec during initialization and before rancher starts on the instance.
$aws ec2 associate-address --instance-id i-02a653592cb748e2x --allocation-id eipalloc-375c1d0x { "AssociationId": "eipassoc-a4b5a29x"}
Reboot EC2 Instance to apply DNS change to Rancher in AMI
$aws ec2 reboot-instances --instance-ids i-02a653592cb748e2x
Clustered Deployment
EC2 Cluster Creation
EFS share for shared NFS
"From the NFS wizard"
Setting up your EC2 instance
- Using the Amazon EC2 console, associate your EC2 instance with a VPC security group that enables access to your mount target. For example, if you assigned the "default" security group to your mount target, you should assign the "default" security group to your EC2 instance. Learn more
- Open an SSH client and connect to your EC2 instance. (Find out how to connect)
If you're not using the EFS mount helper, install the NFS client on your EC2 instance:- On an Ubuntu instance:
sudo apt-get install nfs-common
- On an Ubuntu instance:
Mounting your file system
- Open an SSH client and connect to your EC2 instance. (Find out how to connect)
- Create a new directory on your EC2 instance, such as "efs".
- sudo mkdir efs
- Mount your file system. If you require encryption of data in transit, use the EFS mount helper and the TLS mount option. Mounting considerations
- Using the EFS mount helper:
sudo mount -t efs fs-43b2763a:/ efs - Using the EFS mount helper and encryption of data in transit:
sudo mount -t efs -o tls fs-43b2763a:/ efs - Using the NFS client:
sudo mount -t nfs4 -o nfsvers=4.1,rsize=1048576,wsize=1048576,hard,timeo=600,retrans=2 fs-43b2763a.efs.us-east-2.amazonaws.com:/ efs
- Using the EFS mount helper:
If you are unable to connect, see our troubleshooting documentation.
https://docs.aws.amazon.com/efs/latest/ug/mounting-fs.html
# client install (per node) sudo curl https://releases.rancher.com/install-docker/17.03.sh | sh sudo usermod -aG docker ubuntu sudo apt-get install nfs-common -y sudo mkdir /dockerdata-nfs # fs-xxxxx - replace with your AWS EFS sudo mount -t nfs4 -o nfsvers=4.1,rsize=1048576,wsize=1048576,hard,timeo=600,retrans=2 fs-43b2763a.efs.us-east-2.amazonaws.com:/ /dockerdata-nfs # generated by oom_rancher_install.sh sudo docker run --rm --privileged -v /var/run/docker.sock:/var/run/docker.sock -v /var/lib/rancher:/var/lib/rancher rancher/agent:v1.2.9 http://master.onap.info:8880/v1/scripts/D23B53158B5E1D3DED5D:1514678400000:TGu0bANFdJf9TX6AEe4qV5Fj8
4 Node Kubernetes Cluster on AWS
Notice that we are vCore bound - the 32 vCores across four 64G VMs are saturated even 1 hour in to deployment when 95% of the system is up. Ideally we need 64 vCores for a minimal production system
Client Install
obrienbiometrics:onap_oom-714_heat michaelobrien$ ssh ubuntu@18.0.0.0 ubuntu@ip-10-0-0-70:~$ sudo curl https://releases.rancher.com/install-docker/17.03.sh | sh % Total % Received % Xferd Average Speed Time Time Time Current sudo usermod -aG docker ubuntu ubuntu@ip-10-0-0-70:~$ sudo usermod -aG docker ubuntu ubuntu@ip-10-0-0-70:~$ exit logout obrienbiometrics:onap_oom-714_heat michaelobrien$ ssh ubuntu@18.0.0.0 ubuntu@ip-10-0-0-70:~$ sudo apt-get install nfs-common -y Get:11 http://us-east-2.ec2.archive.ubuntu.com/ubuntu xenial/main amd64 rpcbind amd64 0.2.3-0.2 [40.3 kB] ubuntu@ip-10-0-0-70:~$ sudo mkdir /dockerdata-nfs ubuntu@ip-10-0-0-70:~$ sudo mount -t nfs4 -o nfsvers=4.1,rsize=1048576,wsize=1048576,hard,timeo=600,retrans=2 fs-43b2763a.efs.us-east-2.amazonaws.com:/ /dockerdata-nfs ubuntu@ip-10-0-0-70:~$ ls /dockerdata-nfs/ onap test.sh ubuntu@ip-10-0-0-70:~$ sudo docker run --rm --privileged -v /var/run/docker.sock:/var/run/docker.sock -v /var/lib/rancher:/var/lib/rancher rancher/agent:v1.2.9 http://10.0.0.19:8880/v1/scripts/D9715793EEE25FBDCD1F:1514678400000:ARuGZI6AKln5WOmIPOO1mGcII INFO: Running Agent Registration Process, CATTLE_URL=http://10.0.0.19:8880/v1 INFO: Attempting to connect to: http://10.0.0.19:8880/v1 INFO: http://10.0.0.19:8880/v1 is accessible INFO: ENV: CATTLE_URL=http://10.0.0.19:8880/v1 INFO: ENV: DETECTED_CATTLE_AGENT_IP=10.0.0.70 INFO: ENV: RANCHER_AGENT_IMAGE=rancher/agent:v1.2.9 INFO: Launched Rancher Agent: 1dee580c2281186ae0b06fd194991b2d4cd83375b08388470059ac43928ff16b
Kubernetes Installation via CloudFormation
ONAP Installation
SSH and upload OOM
oom_rancher_install.sh is in - OOM-715Getting issue details... STATUS under https://gerrit.onap.org/r/#/c/32019/
Run OOM
see - OOM-710Getting issue details... STATUS
cd.sh in - OOM-716Getting issue details... STATUS under https://gerrit.onap.org/r/#/c/32653/
Scenario: installing Rancher on clean Ubuntu 16.04 64g VM (single collocated server/host) and the master branch of onap via OOM deployment (2 scripts)
1 hour video of automated installation on an AWS EC2 spot instance
Run Healthcheck
Run Automated Robot parts of vFirewall VNF
Report Results
Stop Spot Instance
$ aws ec2 terminate-instances --instance-ids i-0040425ac8c0d8f6x { "TerminatingInstances": [ { "InstanceId": "i-0040425ac8c0d8f63", "CurrentState": { "Code": 32, "Name": "shutting-down" }, "PreviousState": { "Code": 16, "Name": "running" } } ]}
Verify Instance stopped
Video on Installing and Running the ONAP Demos#ONAPDeploymentVideos
WE can run ONAP on an AWS EC2 instance for $0.17/hour as opposed to Rackspace at $1.12/hour for a 64G Ubuntu host VM.
I have created an AMI on Amazon AWS under the following ID that has a reference 20170825 tag of ONAP 1.0 running on top of Rancher
ami-b8f3f3c3 : onap-oom-k8s-10
EIP 34.233.240.214 maps to http://dev.onap.info:8880/env/1a7/infra/hosts
A D2.2xlarge with 61G ram on the spot market https://console.aws.amazon.com/ec2sp/v1/spot/launch-wizard?region=us-east-1 at $0.16/hour for all of ONAP
It may take up to 3-8 min for kubernetes pods to initialize as long as you preload the docker images - OOM-328Getting issue details... STATUS
Workaround for the disk space error - even though we are running with a 1.9 TB NVMe SSD
https://github.com/kubernetes/kubernetes/issues/48703
Use a flavor that uses EBS like M4.4xLarge which is OK
Use a flavor that uses EBS like M4.4xLarge which is OK - except for AAI right now
Expected Monthly Billing
r4.2xlarge is the smallest and most cost effective 64g min instance to use for full ONAP deployment - it requires EBS stores. This is assuming 1 instance up at all times and a couple ad-hoc instances up a couple hours for testing/experimentation.
Option 1: Migrating Heat to CloudFormation
Resource Correspondence
ID | Type | Parent | AWS | Openstack |
---|---|---|---|---|
Using the CloudFormationDesigner
https://console.aws.amazon.com/cloudformation/designer/home?region=us-east-1#
Decoupling and Abstracting Southbound Orchestration via Plugins
Part of getting another infrastructure provider like AWS to work with ONAP will be in identifying and decoupling southbound logic from any particular cloud provider using an extensible plugin architecture on the SBI interface.
see Multi VIM/Cloud (5/11/17), VID project (5/17/17), Service Orchestrator (5/14/17), ONAP Operations Manager (5/10/17), ONAP Operations Manager / ONAP on Containers
Design Issues
DI 1: Refactor nested orchestration in DCAE
Replace the DCAE Controller
DI 2: Elastic IP allocation
DI 3: Investigate Cloudify plugin for AWS
Cloudify is Tosca based - https://github.com/cloudify-cosmo/cloudify-aws-plugin
OOM Automated Installation Videos
Latest 20171206 AWS install from clean Ubuntu 16.04 VM using rancher setup script below and the cd.sh script to bring up OOM - after the 20 min prepull of dockers - OOM comes up fully with only the known aaf issue 84 of 85 containers - all healthcheck passes except DCAE at 29/30, portal tested and an AAI cloud-region put
Links
Waiting for the EC2 C5 instance types under the C620 chipset to arrive at AWS so we can experiment under EC2 Spot - http://technewshunter.com/cpus/intel-launches-xeon-w-cpus-for-workstations-skylake-sp-ecc-for-lga2066-41771/ https://aws.amazon.com/about-aws/whats-new/2016/11/coming-soon-amazon-ec2-c5-instances-the-next-generation-of-compute-optimized-instances/
http://docs.aws.amazon.com/cli/latest/userguide/cli-install-macos.html
use
curl "https://s3.amazonaws.com/aws-cli/awscli-bundle.zip" -o "awscli-bundle.zip" unzip awscli-bundle.zip sudo ./awscli-bundle/install -i /usr/local/aws -b /usr/local/bin/aws aws --version aws-cli/1.11.170 Python/2.7.13 Darwin/16.7.0 botocore/1.7.28
EC2 VMs
AWS Clustered Deployment
AWS EC2 Cluster Creation
AWS EFS share for shared NFS
You need an NFS share between the VM's in your Kubernetes cluster - an Elastic File System share will wrap NFS
"From the NFS wizard"
Setting up your EC2 instance
- Using the Amazon EC2 console, associate your EC2 instance with a VPC security group that enables access to your mount target. For example, if you assigned the "default" security group to your mount target, you should assign the "default" security group to your EC2 instance. Learn more
- Open an SSH client and connect to your EC2 instance. (Find out how to connect)
If you're not using the EFS mount helper, install the NFS client on your EC2 instance:- On an Ubuntu instance:
sudo apt-get install nfs-common
- On an Ubuntu instance:
Mounting your file system
- Open an SSH client and connect to your EC2 instance. (Find out how to connect)
- Create a new directory on your EC2 instance, such as "efs".
- sudo mkdir efs
- Mount your file system. If you require encryption of data in transit, use the EFS mount helper and the TLS mount option. Mounting considerations
- Using the EFS mount helper:
sudo mount -t efs fs-43b2763a:/ efs - Using the EFS mount helper and encryption of data in transit:
sudo mount -t efs -o tls fs-43b2763a:/ efs - Using the NFS client:
sudo mount -t nfs4 -o nfsvers=4.1,rsize=1048576,wsize=1048576,hard,timeo=600,retrans=2 fs-43b2763a.efs.us-east-2.amazonaws.com:/ efs
- Using the EFS mount helper:
If you are unable to connect, see our troubleshooting documentation.
https://docs.aws.amazon.com/efs/latest/ug/mounting-fs.html
ubuntu@ip-10-0-0-66:~$ sudo apt-get install nfs-common ubuntu@ip-10-0-0-66:~$ cd / ubuntu@ip-10-0-0-66:~$ sudo mkdir /dockerdata-nfs root@ip-10-0-0-19:/# sudo mount -t nfs4 -o nfsvers=4.1,rsize=1048576,wsize=1048576,hard,timeo=600,retrans=2 fs-43b2763a.efs.us-east-2.amazonaws.com:/ /dockerdata-nfs # write something on one vm - and verify it shows on another ubuntu@ip-10-0-0-8:~$ ls /dockerdata-nfs/ test.sh
Microsoft Azure
Subscription Sponsor: (1) Microsoft
VMs
Deliverables are deployment scripts, arm/cli templates for various deployment scenarios (single, multiple, federated servers)
In review - OOM-711Getting issue details... STATUS
Quickstart
Single collocated VM
Automation is currently only written for single VM that hosts both the rancher server and the deployed onap pods. Use the ARM template below to deploy your VM and provision it (adjust your config parameters)
Two choices, run the single oom_deployment.sh ARM wrapper - or use it to bring up an empty vm and run oom_entrypoint.sh manually. Once the VM comes up the oom_entrypoint.sh script will run - which will download the oom_rancher_setup.sh script to setup docker, rancher, kubernetes and helm - the entrypoint script will then run the cd.sh script to bring up onap based on your values.yaml config by running helm install on it.
# login to az cli, wget the deployment script, arm template and parameters file - edit the parameters file (dns, ssh key ...) and run the arm template wget https://jira.onap.org/secure/attachment/11459/oom_deployment.sh wget https://jira.onap.org/secure/attachment/11419/arm_deploy_ons_sut_parameters.json wget https://jira.onap.org/secure/attachment/11420/arm_deploy_ons_sut.json # either run the entrypoint which creates a resource template and runs the stack - or do those two commands manually ./oom_deployment.sh -b master -s azure.onap.cloud -e onap -r a_auto-youruserid_20180421 -t arm_deploy_ons_sut.json -p arm_deploy_ons_sut_parameters.json # wait for the VM to finish in about 75 min or watch progress by ssh'ing into the vm and doing root@ons-auto-201803181110z: sudo tail -f /var/lib/waagent/custom-script/download/0/stdout # if you wish to run the oom_entrypoint script yourself - edit/break the cloud init section at the end of the arm template and do it yourself below # download and edit values.yaml with your onap preferences and openstack tenant config wget https://jira.onap.org/secure/attachment/11414/values.yaml # download and run the bootstrap and onap install script, the -s server name can be an IP, FQDN or hostname wget https://jira.onap.org/secure/attachment/11518/oom_entrypoint.sh chmod 777 oom_entrypoint.sh sudo ./oom_entrypoint.sh -b master -s devops.onap.info -e onap # wait 15 min for rancher to finish, then 30-90 min for onap to come up
- OOM-714Getting issue details... STATUS see https://jira.onap.org/secure/attachment/11455/oom_openstack.yaml and https://jira.onap.org/secure/attachment/11454/oom_openstack_oom.env
- OOM-710Getting issue details... STATUS see https://jira.onap.org/secure/attachment/11439/oom_entrypoint.sh
customize your template (true/false for any components, docker overrides etc...)
https://jira.onap.org/secure/attachment/11414/values.yaml
Run oom_entrypoint.sh after you verified values.yaml - it will run both scripts below for you - a single node kubernetes setup running what you configured in values.yaml will be up in 50-90 min. If you want to just configure your vm without bringing up ONAP - comment out the cd.sh line and run that separately.
- OOM-715Getting issue details... STATUS see https://jira.onap.org/secure/attachment/11438/oom_rancher_setup.sh
- OOM-716Getting issue details... STATUS see https://jira.onap.org/secure/attachment/11413/cd.sh
Verify your system is up by doing a kubectl get pods --all-namespaces and checking the 8880 port to bring up the rancher or kubernetes gui.
20180319: Amsterdam fully automated ARM template to ONAP up
20180312: Master 2 step ARM template - then run entrypoint.sh to ONAP up
Login to Azure CLI
https://portal.azure.com/#blade/HubsExtension/Resources/resourceType/Microsoft.Resources%2Fresources
Download Azure ONAP ARM template
see
- OOM-711Getting issue details... STATUS
Edit Azure ARM template environment parameters
Create Resource Group
az group create --name onap_eastus --location eastus
Run ARM template
az group deployment create --resource-group onap_eastus --template-file oom_azure_arm_deploy.json --parameters @oom_azure_arm_deploy_parameters.json
Wait for Rancher/Kubernetes install
The oom_entrypoint.sh script will be run as a cloud-init script on the VM - see
- OOM-710Getting issue details... STATUS
which runs
- OOM-715Getting issue details... STATUS
Wait for OOM ONAP install
see
- OOM-716Getting issue details... STATUS
Verify ONAP installation
kubectl get pods --all-namespaces # raise/lower onap components from the installed directory if using the oneclick arm template # amsterdam only root@ons-auto-master-201803191429z:/var/lib/waagent/custom-script/download/0/oom/kubernetes/oneclick# ./createAll.bash -n onap
Azure CLI Installation
Requirements
Azure subscription
OSX
https://docs.microsoft.com/en-us/cli/azure/install-azure-cli?view=azure-cli-latest
Install homebrew first (reinstall if you are on the latest OSX 10.13.2 https://github.com/Homebrew/install because of 3718)
Will install Python 3.6
$brew update $brew install azure-cli
https://docs.microsoft.com/en-us/cli/azure/get-started-with-azure-cli?view=azure-cli-latest
$ az login To sign in, use a web browser to open the page https://aka.ms/devicelogin and enter the code E..D to authenticate. [ { "cloudName": "AzureCloud", "id": "f4...b", "isDefault": true, "name": "Pay-As-You-Go", "state": "Enabled", "tenantId": "bcb.....f", "user": { "name": "michael@....org", "type": "user" }}]
Bastion/Jumphost VM in Azure
https://docs.microsoft.com/en-us/cli/azure/install-azure-cli-apt?view=azure-cli-latest
# in root AZ_REPO=$(lsb_release -cs) echo "deb [arch=amd64] https://packages.microsoft.com/repos/azure-cli/ $AZ_REPO main" | sudo tee /etc/apt/sources.list.d/azure-cli.list apt-key adv --keyserver packages.microsoft.com --recv-keys 52E16F86FEE04B979B07E28DB02C46DF417A0893 apt-get install apt-transport-https apt-get update && sudo apt-get install azure-cli az login # verify root@ons-dmz:~# ps -ef | grep az root 1427 1 0 Mar17 ? 00:00:00 /usr/lib/linux-tools/4.13.0-1011-azure/hv_vss_daemon -n
Windows Powershell
https://docs.microsoft.com/en-us/cli/azure/install-azure-cli-windows?view=azure-cli-latest
ARM Template
Follow https://docs.microsoft.com/en-us/azure/azure-resource-manager/resource-manager-create-first-template
Create a Storage Account
$ az login To sign in, use a web browser to open the page https://aka.ms/devicelogin and enter the code E...Z to authenticate. $ az group create --name examplegroup --location "South Central US" { "id": "/subscriptions/f4b...e8b/resourceGroups/examplegroup", "location": "southcentralus", "managedBy": null, "name": "examplegroup", "properties": { "provisioningState": "Succeeded" }, "tags": null } obrien:obrienlabs amdocs$ vi azuredeploy_storageaccount.json obrien:obrienlabs amdocs$ az group deployment create --resource-group examplegroup --template-file azuredeploy_storageaccount.json { "id": "/subscriptions/f4...e8b/resourceGroups/examplegroup/providers/Microsoft.Resources/deployments/azuredeploy_storageaccount", "name": "azuredeploy_storageaccount", "properties": { "additionalProperties": { "duration": "PT32.9822642S", "outputResources": [ { "id": "/subscriptions/f4..e8b/resourceGroups/examplegroup/providers/Microsoft.Storage/storageAccounts/storagekj6....kk2w", "resourceGroup": "examplegroup" }], "templateHash": "11440483235727994285"}, "correlationId": "41a0f79..90c291", "debugSetting": null, "dependencies": [], "mode": "Incremental", "outputs": {}, "parameters": {}, "parametersLink": null, "providers": [ { "id": null, "namespace": "Microsoft.Storage", "registrationState": null, "resourceTypes": [ { "aliases": null, "apiVersions": null, "locations": [ "southcentralus" ], "properties": null, "resourceType": "storageAccounts" }]}], "provisioningState": "Succeeded", "template": null, "templateLink": null, "timestamp": "2018-02-17T16:15:11.562170+00:00" }, "resourceGroup": "examplegroup"}
Pick a region
az account list-locations northcentralus for example
Create a resource group
# create a resource group if not already there az group create --name obrien_jenkins_b_westus2 --location westus2
Create a VM
We need a 128G VM with at least 8vCores (peak is 60) and a 100+GB drive. The sizes are detailed on https://docs.microsoft.com/en-ca/azure/virtual-machines/windows/sizes-memory - we will use the Standard_D32s_v3 type
We need an "all open 0.0.0.0/0" security group and a reassociated data drive as boot drive - see the arm template in OOM-711
Get the ARM template
see open review in - OOM-711Getting issue details... STATUS
"ubuntuOSVersion": "16.04.0-LTS" "imagePublisher": "Canonical", "imageOffer": "UbuntuServer", "vmSize": "Standard_E8s_v3" "osDisk": {"createOption": "FromImage"},"dataDisks": [{"diskSizeGB": 511,"lun": 0, "createOption": "Empty" }]
Follow
https://github.com/Azure/azure-quickstart-templates/tree/master/101-acs-kubernetes
https://docs.microsoft.com/en-us/azure/azure-resource-manager/resource-group-template-deploy
https://github.com/Azure/azure-quickstart-templates/tree/master/101-vm-simple-linux
It needs a security group https://docs.microsoft.com/en-us/azure/virtual-network/virtual-networks-create-nsg-arm-template
{ "apiVersion": "2017-03-01", "type": "Microsoft.Network/networkSecurityGroups", "name": "[variables('networkSecurityGroupName')]", "location": "[resourceGroup().location]", "tags": { "displayName": "NSG - Front End" }, "properties": { "securityRules": [ { "name": "in-rule", "properties": { "description": "All in", "protocol": "Tcp", "sourcePortRange": "*", "destinationPortRange": "*", "sourceAddressPrefix": "Internet", "destinationAddressPrefix": "*", "access": "Allow", "priority": 100, "direction": "Inbound" } }, { "name": "out-rule", "properties": { "description": "All out", "protocol": "Tcp", "sourcePortRange": "*", "destinationPortRange": "*", "sourceAddressPrefix": "Internet", "destinationAddressPrefix": "*", "access": "Allow", "priority": 101, "direction": "Outbound" } } ] } } , { "apiVersion": "2017-04-01", "type": "Microsoft.Network/virtualNetworks", "name": "[variables('virtualNetworkName')]", "location": "[resourceGroup().location]", "dependson": [ "[concat('Microsoft.Network/networkSecurityGroups/', variables('networkSecurityGroupName'))]" ], "properties": { "addressSpace": { "addressPrefixes": [ "[variables('addressPrefix')]" ] }, "subnets": [ { "name": "[variables('subnetName')]", "properties": { "addressPrefix": "[variables('subnetPrefix')]", "networkSecurityGroup": { "id": "[resourceId('Microsoft.Network/networkSecurityGroups', variables('networkSecurityGroupName'))]" } } } ] } },
# validate first (validate instead of create) az group deployment create --resource-group obrien_jenkins_b_westus2 --template-file oom_azure_arm_deploy.json --parameters @oom_azure_arm_cd_amsterdam_deploy_parameters.json
SSH into your VM and run the Kubernetes and OOM installation scripts
Use the entrypoint script in - OOM-710Getting issue details... STATUS
# clone the oom repo to get the install directory git clone https://gerrit.onap.org/r/oom # run the Rancher RI installation (to install kubernetes) oom/install/rancher/oom_rancher_install.sh -b master -s 192.168.240.32 -e onap # run the oom deployment script # get a copy of onap-parametes.yaml and place in this folder oom/install/deployment/cd.sh -b master -s 192.168.240.32 -e onap
oom_rancher_install.sh is in - OOM-715Getting issue details... STATUS under https://gerrit.onap.org/r/#/c/32019/
cd.sh in - OOM-716Getting issue details... STATUS under https://gerrit.onap.org/r/#/c/32653/
Delete the VM and resource group
# delete the vm and resources az group deployment delete --resource-group ONAPAMDOCS --name oom_azure_arm_deploy # the above deletion will not delete the actual resources - only a delete of the group or each individual resource works # optionally delete the resource group az group delete --name ONAPAMDOCS -y
Azure devops
create static IP
az network public-ip create --name onap-argon --resource-group a_ONAP_argon_prod_donotdelete --location eastus --allocation-method Static
ONAP on Azure Container Service
AKS Installation
Follow https://docs.microsoft.com/en-us/azure/aks/tutorial-kubernetes-deploy-cluster
Register for AKS preview via az cli
obrienbiometrics:obrienlabs michaelobrien$ az provider register -n Microsoft.ContainerService Registering is still on-going. You can monitor using 'az provider show -n Microsoft.ContainerService'
Create an AKS resource group
Raise your AKS vCPU quota - optional
http://aka.ms/corequotaincrease
https://portal.azure.com/#blade/Microsoft_Azure_Support/HelpAndSupportBlade/newsupportrequest
Deployment failed. Correlation ID: 4b4707a7-2244-4557-855e-11bcced556de. Provisioning of resource(s) for container service onapAKSCluster in resource group onapAKS failed. Message: Operation results in exceeding quota limits of Core. Maximum allowed: 10, Current in use: 10, Additional requested: 1. Please read more about quota increase at http://aka.ms/corequotaincrease.. Details:
Create AKS cluster
obrienbiometrics:obrienlabs michaelobrien$ az aks create --resource-group onapAKS --name onapAKSCluster --node-count 1 --generate-ssh-keys - Running .. "fqdn": "onapaksclu-onapaks-f4....3.hcp.eastus.azmk8s.io",
AKS cluster VM granularity
The cluster will start with a 3.5G VM before scaling
Resources for your AKS cluster
Bring up AAI only for now
Design Issues
Resource Group
A resource group makes it easier to package and remove everything for a deployment - essentially making the deployment stateless
Network Security Group
Global or local to the resource group?
Static public IP
Register a CNAME for an existing domain and use the same IP address everytime the deployment comes up
Entrypoint cloud init script
How to attach the cloud init script to provision the VM
ARM template chaining
passing derived varialbles into the next arm template - for example when bringing up an entire federated set in one or more DCs
see script attached to
Troubleshooting
DNS propagation and caching
It takes about 2 min for DNS entries to propagate out from A record DNS changes. For example the following IP/DNS association took 2 min to appear in dig.
obrienbiometrics:onap_oom_711_azure michaelobrien$ dig azure.onap.info ; <<>> DiG 9.9.7-P3 <<>> azure.onap.info ;; global options: +cmd ;; Got answer: ;; ->>HEADER<<- opcode: QUERY, status: NOERROR, id: 10599 ;; flags: qr rd ra; QUERY: 1, ANSWER: 1, AUTHORITY: 0, ADDITIONAL: 1 ;; OPT PSEUDOSECTION: ; EDNS: version: 0, flags:; udp: 512 ;; QUESTION SECTION: ;azure.onap.info. IN A ;; ANSWER SECTION: azure.onap.info. 251 IN A 52.224.233.230 ;; Query time: 68 msec ;; SERVER: 8.8.8.8#53(8.8.8.8) ;; WHEN: Tue Feb 20 10:26:59 EST 2018 ;; MSG SIZE rcvd: 60 obrienbiometrics:onap_oom_711_azure michaelobrien$ dig azure.onap.info ; <<>> DiG 9.9.7-P3 <<>> azure.onap.info ;; global options: +cmd ;; Got answer: ;; ->>HEADER<<- opcode: QUERY, status: NOERROR, id: 30447 ;; flags: qr rd ra; QUERY: 1, ANSWER: 1, AUTHORITY: 0, ADDITIONAL: 1 ;; OPT PSEUDOSECTION: ; EDNS: version: 0, flags:; udp: 512 ;; QUESTION SECTION: ;azure.onap.info. IN A ;; ANSWER SECTION: azure.onap.info. 299 IN A 13.92.225.167 ;; Query time: 84 msec ;; SERVER: 8.8.8.8#53(8.8.8.8) ;; WHEN: Tue Feb 20 10:27:04 EST 2018
Corporate Firewall Access
Inside the corporate firewall - avoid it PS C:\> az login Please ensure you have network connection. Error detail: HTTPSConnectionPool(host='login.microsoftonline.com', port=443) : Max retries exceeded with url: /common/oauth2/devicecode?api-version=1.0 (Caused by NewConnectionError('<urllib3.conne ction.VerifiedHTTPSConnection object at 0x04D18730>: Failed to establish a new connection: [Errno 11001] getaddrinfo fai led',)) at home or cell hotspot PS C:\> az login To sign in, use a web browser to open the page https://aka.ms/devicelogin and enter the code E...2W to authenticate. [ { "cloudName": "AzureCloud", "id": "4...da1", "isDefault": true, "name": "Microsoft Azure Internal Consumption", "state": "Enabled", "tenantId": "72f98....47", "user": { "name": "fran...ocs.com", "type": "user" }] On corporate account (need permissions bump to be able to create a resource group prior to running an arm template https://wiki.onap.org/display/DW/ONAP+on+Kubernetes+on+Microsoft+Azure#ONAPonKubernetesonMicrosoftAzure-ARMTemplate PS C:\> az group create --name onapKubernetes --location eastus The client 'fra...s.com' with object id '08f98c7e-...ed' does not have authorization to per form action 'Microsoft.Resources/subscriptions/resourcegroups/write' over scope '/subscriptions/42e...8 7da1/resourcegroups/onapKubernetes'. try my personal = OK PS C:\> az login To sign in, use a web browser to open the page https://aka.ms/devicelogin and enter the code EE...ULR to authenticate. Terminate batch job (Y/N)? y # hangs when first time login in a new pc PS C:\> az login To sign in, use a web browser to open the page https://aka.ms/devicelogin and enter the code E.PBKS to authenticate. [ { "cloudName": "AzureCloud", "id": "f4b...b", "isDefault": true "name": "Pay-As-You-Go", "state": "Enabled", "tenantId": "bcb...f4f", "user": "name": "michael@obrien...org", "type": "user" } }] PS C:\> az group create --name onapKubernetes2 --location eastus { "id": "/subscriptions/f4b....b/resourceGroups/onapKubernetes2", "location": "eastus", "managedBy": null, "name": "onapKubernetes2", "properties": { "provisioningState": "Succeeded" }, "tags": null}
Design Issues
20180228: Deployment delete does not delete resources without a resourceGroup delete
I find that a delete deployment deletes the deployment but not the actual resources. The workaround is to delete the resource group - but in some constrained subscriptions the cli user may not have the ability to create a resource group - and hence delete it.
see
https://github.com/Azure/azure-sdk-for-java/issues/1167
deleting the resources manually for now - is a workaround if you cannot create/delete resource groups
# delete the vm and resources az group deployment delete --resource-group ONAPAMDOCS --name oom_azure_arm_deploy
# the above deletion will not delete the actual resources - only a delete of the group or each individual resource works
# optionally delete the resource group az group delete --name ONAPAMDOCS -y
However modifying the template to add resources works well. For example adding a reference to a network security group
20180228: Resize the OS disk
ONAP requires at least 75g - the issue is than in most VM templates on Azure - the OS disk is 30g - we need to either switch to the data disk or resize the os disk.
# add diskSizeGB to the template "osDisk": { "diskSizeGB": 255, "createOption": "FromImage" }, ubuntu@oom-auto-deploy:~$ df Filesystem 1K-blocks Used Available Use% Mounted on udev 65989400 0 65989400 0% /dev tmpfs 13201856 8848 13193008 1% /run /dev/sda1 259142960 1339056 257787520 1% / tmpfs 66009280 0 66009280 0% /dev/shm tmpfs 5120 0 5120 0% /run/lock tmpfs 66009280 0 66009280 0% /sys/fs/cgroup none 64 0 64 0% /etc/network/interfaces.dynamic.d /dev/sdb1 264091588 60508 250592980 1% /mnt tmpfs 13201856 0 13201856 0% /run/user/1000 ubuntu@oom-auto-deploy:~$ free total used free shared buff/cache available Mem: 132018560 392336 131242164 8876 384060 131012328
20180301: Add oom_entrypoint.sh bootstrap script to install rancher and onap
in review under OOM-715
https://jira.onap.org/secure/attachment/11206/oom_entrypoint.sh
If using amsterdam - swap out the onap-parameters.yaml (the curl is hardcoded to a master branch version)
20180303: cloudstorage access on OSX via Azure Storage Manager
use this method instead of installing az cli directly - for certain corporate oauth configurations
https://azure.microsoft.com/en-us/features/storage-explorer/
Install AZM using the name and access key of a storage account created manually or by enabling the az cli on the browser
20180318: add oom_entrypoint.sh to cloud-init on the arm template
See https://docs.microsoft.com/en-us/azure/templates/microsoft.compute/virtualmachines/extensions it looks like Azure has a similar setup to AWS ebextentions
Targetting
type | string | No | Specifies the type of the extension; an example is "CustomScriptExtension". |
https://docs.microsoft.com/en-us/azure/virtual-machines/linux/extensions-customscript
deprecated { "apiVersion": "2015-06-15", "type": "Microsoft.Compute/virtualMachines/extensions", "name": "[concat(parameters('vmName'),'/onap')]", "location": "[resourceGroup().location]", "dependsOn": ["[concat('Microsoft.Compute/virtualMachines/', parameters('vmName'))]"], "properties": { "publisher": "Microsoft.Azure.Extensions", "type": "CustomScript", "typeHandlerVersion": "1.9", "autoUpgradeMinorVersion": true, "settings": { "fileUris": [ "https://jira.onap.org/secure/attachment/11263/oom_entrypoint.sh" ], "commandToExecute": "[concat('./' , parameters('scriptName'), ' -b master -s dns/pub/pri-ip -e onap' )]" } } } use { "apiVersion": "2017-12-01", "type": "Microsoft.Compute/virtualMachines/extensions", "name": "[concat(parameters('vmName'),'/onap')]", "location": "[resourceGroup().location]", "dependsOn": ["[concat('Microsoft.Compute/virtualMachines/', parameters('vmName'))]"], "properties": { "publisher": "Microsoft.Azure.Extensions", "type": "CustomScript", "typeHandlerVersion": "2.0", "autoUpgradeMinorVersion": true, "settings": { "fileUris": [ "https://jira.onap.org/secure/attachment/11281/oom_entrypoint.sh" ], "commandToExecute": "[concat('./' , parameters('scriptName'), ' -b master ', ' -s ', 'ons-auto-201803181110z', ' -e onap' )]" } } }
ubuntu@ons-dmz:~$ ./oom_deployment.sh
Deployment template validation failed: 'The template resource 'entrypoint' for type 'Microsoft.Compute/virtualMachines/extensions' at line '1' and column '6182' has incorrect segment lengths. A nested resource type must have identical number of segments as its resource name. A root resource type must have segment length one greater than its resource name. Please see https://aka.ms/arm-template/#resources for usage details.'.
ubuntu@ons-dmz:~$ ./oom_deployment.sh
Deployment failed. Correlation ID: 532b9a9b-e0e8-4184-9e46-6c2e7c15e7c7. {
"error": {
"code": "ParentResourceNotFound",
"message": "Can not perform requested operation on nested resource. Parent resource '[concat(parameters('vmName'),'' not found."
}
}
fixed 20180318:1600
Install runs - but I need visibility - checking /var/lib/waagent/custom-script/download/0/
progress
./oom_deployment.sh # 7 min to delete old deployment ubuntu@ons-dmz:~$ az vm extension list -g a_ONAP_auto_201803181110z --vm-name ons-auto-201803181110z .. "provisioningState": "Creating", "settings": { "commandToExecute": "./oom_entrypoint.sh -b master -s ons-auto-201803181110zons-auto-201803181110z.eastus.cloudapp.azure.com -e onap", "fileUris": [ "https://jira.onap.org/secure/attachment/11263/oom_entrypoint.sh" ubuntu@ons-auto-201803181110z:~$ sudo su - root@ons-auto-201803181110z:~# docker ps CONTAINER ID IMAGE COMMAND CREATED STATUS PORTS NAMES 83458596d7a6 rancher/server:v1.6.14 "/usr/bin/entry /u..." 3 minutes ago Up 3 minutes 3306/tcp, 0.0.0.0:8880->8080/tcp rancher_server root@ons-auto-201803181110z:~# tail -f /var/log/azure/custom-script/handler.log time=2018-03-18T22:51:59Z version=v2.0.6/git@1008306-clean operation=enable seq=0 file=0 event="download complete" output=/var/lib/waagent/custom-script/download/0 time=2018-03-18T22:51:59Z version=v2.0.6/git@1008306-clean operation=enable seq=0 event="executing command" output=/var/lib/waagent/custom-script/download/0 time=2018-03-18T22:51:59Z version=v2.0.6/git@1008306-clean operation=enable seq=0 event="executing public commandToExecute" output=/var/lib/waagent/custom-script/download/0 root@ons-auto-201803181110z:~# docker ps CONTAINER ID IMAGE COMMAND CREATED STATUS PORTS NAMES 539733f24c01 rancher/agent:v1.2.9 "/run.sh run" 13 seconds ago Up 13 seconds rancher-agent 83458596d7a6 rancher/server:v1.6.14 "/usr/bin/entry /u..." 5 minutes ago Up 5 minutes 3306/tcp, 0.0.0.0:8880->8080/tcp rancher_server root@ons-auto-201803181110z:~# ls -la /var/lib/waagent/custom-script/download/0/ total 31616 -rw-r--r-- 1 root root 16325186 Aug 31 2017 helm-v2.6.1-linux-amd64.tar.gz -rw-r--r-- 1 root root 4 Mar 18 22:55 kube_env_id.json drwxrwxr-x 2 ubuntu ubuntu 4096 Mar 18 22:53 linux-amd64 -r-x------ 1 root root 2822 Mar 18 22:51 oom_entrypoint.sh -rwxrwxrwx 1 root root 7288 Mar 18 22:52 oom_rancher_setup.sh -rwxr-xr-x 1 root root 12213376 Mar 18 22:53 rancher -rw-r--r-- 1 root root 3736787 Dec 20 19:41 rancher-linux-amd64-v0.6.7.tar.gz drwxr-xr-x 2 root root 4096 Dec 20 19:39 rancher-v0.6.7
testing via http://jenkins.onap.cloud/job/oom_azure_deployment/
Need the ip address and not the domain name - via linked template
or
https://docs.microsoft.com/en-us/azure/templates/microsoft.network/publicipaddresses
https://github.com/Azure/azure-quickstart-templates/issues/583
Arm templates cannot specify a static ip - without a private subnet
reference(variables('publicIPAddressName')).ipAddress
for
reference(variables('nicName')).ipConfigurations[0].properties.privateIPAddress
Using the hostname instead of the private/public ip works (verify /etc/hosts though)
obrienbiometrics:oom michaelobrien$ ssh ubuntu@13.90.207.60 ubuntu@ons-auto-201803181110z:~$ sudo su - root@ons-auto-201803181110z:/var/lib/waagent/custom-script/download/0# cat stdout INFO: Running Agent Registration Process, CATTLE_URL=http://ons-auto-201803181110z:8880/v1 INFO: Attempting to connect to: http://ons-auto-201803181110z:8880/v1 INFO: http://ons-auto-201803181110z:8880/v1 is accessible INFO: Inspecting host capabilities INFO: Boot2Docker: false INFO: Host writable: true INFO: Token: xxxxxxxx INFO: Running registration INFO: Printing Environment INFO: ENV: CATTLE_ACCESS_KEY=9B0FA1695A3E3CFD07DB INFO: ENV: CATTLE_HOME=/var/lib/cattle INFO: ENV: CATTLE_REGISTRATION_ACCESS_KEY=registrationToken INFO: ENV: CATTLE_REGISTRATION_SECRET_KEY=xxxxxxx INFO: ENV: CATTLE_SECRET_KEY=xxxxxxx INFO: ENV: CATTLE_URL=http://ons-auto-201803181110z:8880/v1 INFO: ENV: DETECTED_CATTLE_AGENT_IP=172.17.0.1 INFO: ENV: RANCHER_AGENT_IMAGE=rancher/agent:v1.2.9 INFO: Launched Rancher Agent: b44bd62fd21c961f32f642f7c3b24438fc4129eabbd1f91e1cf58b0ed30b5876 waiting 7 min for host registration to finish 1 more min KUBECTL_TOKEN base64 encoded: QmFzaWMgUWpBNE5EWkdRVE5HTmpKRVFVRkZOa1k1UlRNNlMzZzBWRTVUZVc5VGEycFhaVE5HTm5rNWNGWlhVRWg0TVVkV1ZEbDVTRmQxUkc1d2MwWTJRZz09 run the following if you installed a higher kubectl version than the server helm init --upgrade Verify all pods up on the kubernetes system - will return localhost:8080 until a host is added kubectl get pods --all-namespaces NAMESPACE NAME READY STATUS RESTARTS AGE kube-system heapster-76b8cd7b5-v5jrd 1/1 Running 0 5m kube-system kube-dns-5d7b4487c9-9bwk5 3/3 Running 0 5m kube-system kubernetes-dashboard-f9577fffd-cpwv7 1/1 Running 0 5m kube-system monitoring-grafana-997796fcf-s4sjm 1/1 Running 0 5m kube-system monitoring-influxdb-56fdcd96b-2mn6r 1/1 Running 0 5m kube-system tiller-deploy-cc96d4f6b-fll4t 1/1 Running 0 5m
20180318: Create VM image without destroying running VM
In AWS we can select the "no reboot" option and create an image from a running VM as-is with no effect on the running system.
Having issues with the Azure image creator - it is looking for the ubuntu pw even though I only use key based access
20180319: New Relic Monitoring
20180319: document devops flow
aka: travellers guide
20180319: Document Virtual Network Topology
20180429: Helm repo n/a after reboot - rerun helm serve
If you run into issues doing a make all - your helm server is not running
# rerun helm serve & helm repo add local http://127.0.0.1:8879
Training
(from Microsoft - thank you)
General Azure Documentation
Azure Site http://azure.microsoft.com
Azure Documentation Site https://docs.microsoft.com/en-us/azure/
Azure Training Courses https://azure.microsoft.com/en-us/training/free-online-courses/
Azure Portal http://portal.azure.com
Developer Documentation
Azure AD Authentication Libraries https://docs.microsoft.com/en-us/azure/active-directory/develop/active-directory-authentication-libraries
Java Overview on Azure https://azure.microsoft.com/en-us/develop/java/
Java Docs for Azure https://docs.microsoft.com/en-us/java/azure/
Java SDK on GitHub https://github.com/Azure/azure-sdk-for-java
Python Overview on Azure https://azure.microsoft.com/en-us/develop/python/
Python Docs for Azure https://docs.microsoft.com/en-us/python/azure/
Python SDK on GitHub https://github.com/Azure/azure-sdk-for-python
REST Api and CLI Documentation
REST API Documentation https://docs.microsoft.com/en-us/rest/api/
CLI Documentation https://docs.microsoft.com/en-us/cli/azure/index
Other Documentation
Using Automation for VM shutdown & startup https://docs.microsoft.com/en-us/azure/automation/automation-solution-vm-management
Azure Resource Manager (ARM) QuickStart Templates https://github.com/Azure/azure-quickstart-templates
Known Forks
The code in this github repo has 2 month old copies of cd.sh and oom_rancher_install.sh
https://github.com/taranki/onap-azure
Use the official ONAP code in review under
https://jira.onap.org/browse/OOM-710
https://jira.onap.org/browse/OOM-715
https://gerrit.onap.org/r/32019
https://jira.onap.org/browse/OOM-716
https://gerrit.onap.org/r/32653
The original seed source from 2017 below is deprecated - use onap links above
https://github.com/obrienlabs/onap-root
Links
https://azure.microsoft.com/en-us/services/container-service/
https://docs.microsoft.com/en-us/azure/templates/microsoft.compute/virtualmachines
https://kubernetes.io/docs/concepts/containers/images/#using-azure-container-registry-acr
https://azure.microsoft.com/en-us/features/storage-explorer/
https://docs.microsoft.com/en-ca/azure/virtual-machines/linux/capture-image
AKS
Google GCE
Account Provider: Michael O'Brien of Amdocs
OOM Installation on a GCE VM
The purpose of this page is to detail getting ONAP on Kubernetes (OOM) setup on a GCE VM.
I recommend using the ONAP on Kubernetes on Amazon EC2 Amazon EC2 Spot API - as it runs around $0.12-0.25/hr at 75% off instead of the $0.60 below (33% off for reserved instances) - this page is here so we can support GCE and also work with the kubernetes open source project in a space it was originally designed in at Google.
Login to your google account and start creating a 128g Ubuntu 16.04 VM
Install google command line tools
?????????????????????????????????????????????????????????????????????????????????????????????????????????????? ? Components ? ??????????????????????????????????????????????????????????????????????????????????????????????????????????????? ? Status ? Name ? ID ? Size ? ??????????????????????????????????????????????????????????????????????????????????????????????????????????????? ? Not Installed ? App Engine Go Extensions ? app-engine-go ? 97.7 MiB ? ? Not Installed ? Cloud Bigtable Command Line Tool ? cbt ? 4.0 MiB ? ? Not Installed ? Cloud Bigtable Emulator ? bigtable ? 3.5 MiB ? ? Not Installed ? Cloud Datalab Command Line Tool ? datalab ? < 1 MiB ? ? Not Installed ? Cloud Datastore Emulator ? cloud-datastore-emulator ? 17.7 MiB ? ? Not Installed ? Cloud Datastore Emulator (Legacy) ? gcd-emulator ? 38.1 MiB ? ? Not Installed ? Cloud Pub/Sub Emulator ? pubsub-emulator ? 33.2 MiB ? ? Not Installed ? Emulator Reverse Proxy ? emulator-reverse-proxy ? 14.5 MiB ? ? Not Installed ? Google Container Local Builder ? container-builder-local ? 3.7 MiB ? ? Not Installed ? Google Container Registry's Docker credential helper ? docker-credential-gcr ? 2.2 MiB ? ? Not Installed ? gcloud Alpha Commands ? alpha ? < 1 MiB ? ? Not Installed ? gcloud Beta Commands ? beta ? < 1 MiB ? ? Not Installed ? gcloud app Java Extensions ? app-engine-java ? 116.0 MiB ? ? Not Installed ? gcloud app PHP Extensions ? app-engine-php ? 21.9 MiB ? ? Not Installed ? gcloud app Python Extensions ? app-engine-python ? 6.2 MiB ? ? Not Installed ? kubectl ? kubectl ? 15.9 MiB ? ? Installed ? BigQuery Command Line Tool ? bq ? < 1 MiB ? ? Installed ? Cloud SDK Core Libraries ? core ? 5.9 MiB ? ? Installed ? Cloud Storage Command Line Tool ? gsutil ? 3.3 MiB ? ??????????????????????????????????????????????????????????????????????????????????????????????????????????????? ==> Source [/Users/michaelobrien/gce/google-cloud-sdk/completion.bash.inc] in your profile to enable shell command completion for gcloud. ==> Source [/Users/michaelobrien/gce/google-cloud-sdk/path.bash.inc] in your profile to add the Google Cloud SDK command line tools to your $PATH. gcloud init obrienbiometrics:google-cloud-sdk michaelobrien$ source ~/.bash_profile obrienbiometrics:google-cloud-sdk michaelobrien$ gcloud components update All components are up to date.
Connect to your VM by getting a dynamic SSH key
obrienbiometrics:google-cloud-sdk michaelobrien$ gcloud compute ssh instance-1 WARNING: The public SSH key file for gcloud does not exist. WARNING: The private SSH key file for gcloud does not exist. WARNING: You do not have an SSH key for gcloud. WARNING: SSH keygen will be executed to generate a key. Generating public/private rsa key pair. Enter passphrase (empty for no passphrase): Enter same passphrase again: Your identification has been saved in /Users/michaelobrien/.ssh/google_compute_engine. Your public key has been saved in /Users/michaelobrien/.ssh/google_compute_engine.pub. The key fingerprint is: SHA256:kvS8ZIE1egbY+bEpY1RGN45ruICBo1WH8fLWqO435+Y michaelobrien@obrienbiometrics.local The key's randomart image is: +---[RSA 2048]----+ | o=o+* o | | . .oo+*.= . | |o o ..=.=+. | |.o o ++X+o | |. . ..BoS | | + * . | | . . . | | . o o | | .o. *E | +----[SHA256]-----+ Updating project ssh metadata.../Updated [https://www.googleapis.com/compute/v1/projects/onap-184300]. Updating project ssh metadata...done. Waiting for SSH key to propagate. Warning: Permanently added 'compute.2865548946042680113' (ECDSA) to the list of known hosts. Welcome to Ubuntu 16.04.3 LTS (GNU/Linux 4.10.0-37-generic x86_64) * Documentation: https://help.ubuntu.com * Management: https://landscape.canonical.com * Support: https://ubuntu.com/advantage Get cloud support with Ubuntu Advantage Cloud Guest: http://www.ubuntu.com/business/services/cloud 0 packages can be updated. 0 updates are security updates. michaelobrien@instance-1:~$
Open up firewall rules or the entire VM
We need at least port 8880 for rancher
obrienbiometrics:20171027_log_doc michaelobrien$ gcloud compute firewall-rules create open8880 --allow tcp:8880 --source-tags=instance-1 --source-ranges=0.0.0.0/0 --description="8880" Creating firewall...|Created [https://www.googleapis.com/compute/v1/projects/onap-184300/global/firewalls/open8880]. Creating firewall...done. NAME NETWORK DIRECTION PRIORITY ALLOW DENY open8880 default INGRESS 1000 tcp:8880
Better to edit the existing internal firewall rule to the CIDR 0.0.0.0/0
Continue with ONAP on Kubernetes
ONAP on Kubernetes#QuickstartInstallation
Kubernetes
Kubernetes DevOps
downgrade docker if required
sudo apt-get autoremove -y docker-engine
Operations
Get failed/pending containers
kubectl get pods --all-namespaces | grep -E "0/|1/2" | wc -l
kubectl cluster-info # get pods/containers kubectl get pods --all-namespaces # get port mappings kubectl get services --all-namespaces -o wide NAMESPACE NAME READY STATUS RESTARTS AGE default nginx-1389790254-lgkz3 1/1 Running 1 5d kube-system heapster-4285517626-x080g 1/1 Running 1 6d kube-system kube-dns-638003847-tst97 3/3 Running 3 6d kube-system kubernetes-dashboard-716739405-fnn3g 1/1 Running 2 6d kube-system monitoring-grafana-2360823841-hr824 1/1 Running 1 6d kube-system monitoring-influxdb-2323019309-k7h1t 1/1 Running 1 6d kube-system tiller-deploy-737598192-x9wh5 1/1 Running 1 6d # ssh into a pod kubectl -n default exec -it nginx-1389790254-lgkz3 /bin/bash # get logs kubectl -n default logs -f nginx-1389790254-lgkz3
Exec
kubectl -n onap-aai exec -it aai-resources-1039856271-d9bvq bash
Bounce/Fix a failed container
Periodically one of the higher containers in a dependency tree will not get restarted in time to pick up running child containers - usually this is the kibana container
Fix this or "any" container by deleting the container in question and kubernetes will bring another one up.
root@a-onap-auto-20180412-ref:~# kubectl get services --all-namespaces | grep log onap dev-vfc-catalog ClusterIP 10.43.210.8 <none> 8806/TCP 5d onap log-es NodePort 10.43.77.87 <none> 9200:30254/TCP 5d onap log-es-tcp ClusterIP 10.43.159.93 <none> 9300/TCP 5d onap log-kibana NodePort 10.43.41.102 <none> 5601:30253/TCP 5d onap log-ls NodePort 10.43.180.165 <none> 5044:30255/TCP 5d onap log-ls-http ClusterIP 10.43.13.180 <none> 9600/TCP 5d root@a-onap-auto-20180412-ref:~# kubectl get pods --all-namespaces | grep log onap dev-log-elasticsearch-66cdc4f855-wmpkz 1/1 Running 0 5d onap dev-log-kibana-5b6f86bcb4-drpzq 0/1 Running 1076 5d onap dev-log-logstash-6d9fdccdb6-ngq2f 1/1 Running 0 5d onap dev-vfc-catalog-7d89bc8b9d-vxk74 2/2 Running 0 5d root@a-onap-auto-20180412-ref:~# kubectl delete pod dev-log-kibana-5b6f86bcb4-drpzq -n onap pod "dev-log-kibana-5b6f86bcb4-drpzq" deleted root@a-onap-auto-20180412-ref:~# kubectl get pods --all-namespaces | grep log onap dev-log-elasticsearch-66cdc4f855-wmpkz 1/1 Running 0 5d onap dev-log-kibana-5b6f86bcb4-drpzq 0/1 Terminating 1076 5d onap dev-log-kibana-5b6f86bcb4-gpn2m 0/1 Pending 0 12s onap dev-log-logstash-6d9fdccdb6-ngq2f 1/1 Running 0 5d onap dev-vfc-catalog-7d89bc8b9d-vxk74 2/2 Running 0 5d
Remove containers stuck in terminating
a helm namespace delete or a kubectl delete or a helm purge may not remove everything based on hanging PVs - use
melliott [12:11 PM] kubectl delete pods <pod> --grace-period=0 --force
Persistent Volumes
Several applications in ONAP require persistent configuration or storage outside of the stateless docker containers managed by Kubernetes. In this case Kubernetes can act as a direct wrapper of native docker volumes or provide its own extended dynamic persistence for use cases where we are running scaled pods on multiple hosts.
https://kubernetes.io/docs/concepts/storage/persistent-volumes/
The SDNC clustering poc - https://gerrit.onap.org/r/#/c/25467/23
For example the following has a patch that exposes a dir into the container just like a docker volume or a volume in docker-compose - the issue here is mixing emptyDir (exposing dirs between containers) and exposing dirs outside to the FS/NFS
https://jira.onap.org/browse/LOG-52
This is only one way to do a static PV in K8S
https://jira.onap.org/secure/attachment/10436/LOG-50-expose_mso_logs.patch
Token
Thanks Joey
root@ip-172-31-27-86:~# kubectl describe secret $(kubectl get secrets | grep default | cut -f1 -d ' ') Name: default-token-w1jq0 Namespace: default Labels: <none> Annotations: kubernetes.io/service-account.name=default kubernetes.io/service-account.uid=478eae11-f0f4-11e7-b936-022346869a82 Type: kubernetes.io/service-account-token Data ==== ca.crt: 1025 bytes namespace: 7 bytes token: eyJhbGciOiJSUzI1NiIsInR5cCI6IkpXVCJ9.eyJpc3MiOiJrdWJlcm5ldGVzL3NlcnZpY2VhY2NvdW50Iiwia3ViZXJuZXRlcy5pby9zZXJ2aWNlYWNjb3VudC9uYW1lc3BhY2UiOiJkZWZhdWx0Iiwia3ViZXJuZXRlcy5pby9zZXJ2aWNlYWNjb3VudC9zZWNyZXQubmFtZSI6ImRlZmF1bHQtdG9rZW4tdzFqcTAiLCJrdWJlcm5ldGVzLmlvL3NlcnZpY2VhY2NvdW50L3NlcnZpY2UtYWNjb3VudC5uYW1lIjoiZGVmYXVsdCIsImt1YmVybmV0ZXMuaW8vc2VydmljZWFjY291bnQvc2VydmljZS1hY2NvdW50LnVpZCI6IjQ3OGVhZTExLWYwZjQtMTFlNy1iOTM2LTAyMjM0Njg2OWE4MiIsInN1YiI6InN5c3RlbTpzZXJ2aWNlYWNjb3VudDpkZWZhdWx0OmRlZmF1bHQifQ.Fjv6hA1Kzurr-Cie5EZmxMOoxm-3Uh3zMGvoA4Xu6h2U1-NBp_fw_YW7nSECnI7ttGz67mxAjknsgfze-1JtgbIUtyPP31Hp1iscaieu5r4gAc_booBdkV8Eb8gia6sF84Ye10lsS4nkmmjKA30BdqH9qjWspChLPdGdG3_RmjApIHEOjCqQSEHGBOMvY98_uO3jiJ_XlJBwLL4uydjhpoANrS0xlS_Evn0evLdits7_piklbc-uqKJBdZ6rWyaRbkaIbwNYYhg7O-CLlUVuExynAAp1J7Mo3qITNV_F7f4l4OIzmEf3XLho4a1KIGb76P1AOvSrXgTzBq0Uvh5fUw |
Auto Scaling
Using the example on page 122 of Kubernetes Up & Running.
kubectl run nginx --image=nginx:1.7.12 kubectl get deployments nginx kubectl scale deployments nginx --replicas=3 kubectl get deployments nginx kubectl get replicasets --selector=run=nginx kubectl get pods --all-namespaces kubectl scale deployments nginx --replicas=64
Openstack
Windriver Intel Lab
see - OOM-714Getting issue details... STATUS
Openlab VNC and CLI
The following is missing some sections and a bit out of date (v2 deprecated in favor of v3)
Get an openlab account - Integration / Developer Lab Access | |
Install openVPN - Using Lab POD-ONAP-01 Environment For OSX both Viscosity and TunnelBlick work fine | |
Login to Openstack | |
Install openstack command line tools | Tutorial: Configuring and Starting Up the Base ONAP Stack#InstallPythonvirtualenvTools(optional,butrecommended) |
get your v3 rc file | |
verify your openstack cli access (or just use the jumpbox) | obrienbiometrics:aws michaelobrien$ source logging-openrc.sh obrienbiometrics:aws michaelobrien$ openstack server list +--------------------------------------+---------+--------+-------------------------------+------------+ | ID | Name | Status | Networks | Image Name | +--------------------------------------+---------+--------+-------------------------------+------------+ | 1ed28213-62dd-4ef6-bdde-6307e0b42c8c | jenkins | ACTIVE | admin-private-mgmt=10.10.2.34 | | +--------------------------------------+---------+--------+-------------------------------+------------+ |
get some elastic IP's | You may need to release unused IPs from other tenants - as we have 4 pools of 50 |
fill in your stack env parameters | |
Run the HEAT stack | obrienbiometrics:onap_oom-714_heat michaelobrien$ openstack stack list +--------------------------------------+----------------------------+-----------------+----------------------+----------------------+ | ID | Stack Name | Stack Status | Creation Time | Updated Time | +--------------------------------------+----------------------------+-----------------+----------------------+----------------------+ | 52379aea-d0a9-48db-a13e-35ca00876768 | dcae | CREATE_COMPLETE | 2018-03-04T22:02:12Z | 2018-03-04T22:02:12Z | | 15203d3e-b603-4ccd-8be8-57c93f90da80 | oom20181214_obrien_rancher | CREATE_COMPLETE | 2018-02-15T01:26:27Z | 2018-02-15T01:26:27Z | +--------------------------------------+----------------------------+-----------------+----------------------+----------------------+ obrienbiometrics:onap_oom-714_heat michaelobrien$ openstack stack create -t oom_openstack.yaml -e logging_openstack_oom.env OOM20180507a +---------------------+-----------------------------------------+ | Field | Value | +---------------------+-----------------------------------------+ | id | f40a14a1-8114-463c-b1ba-c4df28215385 | | stack_name | OOM20180507a | | description | Heat template to install OOM components | | creation_time | 2018-05-07T16:47:53Z | | updated_time | 2018-05-07T16:47:53Z | | stack_status | CREATE_IN_PROGRESS | | stack_status_reason | Stack CREATE started | +---------------------+-----------------------------------------+ |
ssh in | obrienbiometrics:onap_oom-714_heat michaelobrien$ ssh ubuntu@10.12.6.151 ubuntu@onap-oom-obrien-rancher:~$ docker version Client: Version: 17.03.2-ce API version: 1.27 |
ONAP Deployment Specification
Security Profile
ONAP will require certain ports open by CIDR to several static domain names in order to deploy defined in a security group. At runtime the list is reduced.
Ideally these are all inside a private network.
It looks like we will need a standard public/private network locked down behind a combined ACL/SG for AWS VPC or a NSG for Azure where we only expose what we need outside the private network.
Still working on a list of ports but we should not need any of these exposed if we use a bastion/jumpbox + nat combo inside the network.
Known Security Vulnerabilities
https://medium.com/handy-tech/analysis-of-a-kubernetes-hack-backdooring-through-kubelet-823be5c3d67c
https://github.com/kubernetes/kubernetes/pull/59666 fixed in Kubernetes 1.10
ONAP Port Profile
ONAP on deployment will require the following incoming and outgoing ports. Note: within ONAP rest calls between components will be handled inside the Kubernetes namespace by the DNS server running as part of K8S.
port | protocol | incoming/outgoing | application | source | destination | Notes |
---|---|---|---|---|---|---|
22 | ssh | ssh | developer vm | host | ||
443 | tiller | client | host | |||
8880 | http | rancher | client | host | ||
9090 | http | kubernetes | host | |||
10001 | https | nexus3 | nexus3.onap.org | |||
10003 | https | nexus3 | nexus3.onap.org | |||
https | nexus | nexus.onap.org | ||||
https ssh | git | git.onap.org | ||||
30200-30399 | http/https | REST api | developer vm | host | ||
5005 | tcp | java debug port | developer vm | host | ||
Lockdown ports | ||||||
8080 | outgoing | |||||
10250-10255 | in/out | Lock these down via VPC or a source CIDR that equals only the server/client IP list https://medium.com/handy-tech/analysis-of-a-kubernetes-hack-backdooring-through-kubelet-823be5c3d67c |
Azure Security Group
AWS VPC + Security Group
Links
https://kubernetes.io/docs/user-guide/kubectl-cheatsheet/
ONAP on Kubernetes#QuickstartInstallation
https://kubernetes.io/docs/tasks/run-application/run-replicated-stateful-application/
https://kubernetes.io/docs/tasks/job/fine-parallel-processing-work-queue/