This page is currently a WIP for the next 2 weeks starting 20180214. Deliverables will deployment scripts, arm/cli templates for various deployment scenarios (single, multiple, federated servers)
under - OOM-524Getting issue details... STATUS , - OOM-635Getting issue details... STATUS , - OOM-607Getting issue details... STATUS , - OOM-710Getting issue details... STATUS , - OOM-711Getting issue details... STATUS
Azure CLI Installation
Requirements
Azure subscription
OSX
https://docs.microsoft.com/en-us/cli/azure/install-azure-cli?view=azure-cli-latest
Install homebrew first (reinstall if you are on the latest OSX 10.13.2 https://github.com/Homebrew/install because of 3718)
Will install Python 3.6
$brew update $brew install azure-cli
https://docs.microsoft.com/en-us/cli/azure/get-started-with-azure-cli?view=azure-cli-latest
$ az login To sign in, use a web browser to open the page https://aka.ms/devicelogin and enter the code E..D to authenticate. [ { "cloudName": "AzureCloud", "id": "f4...b", "isDefault": true, "name": "Pay-As-You-Go", "state": "Enabled", "tenantId": "bcb.....f", "user": { "name": "michael@....org", "type": "user" }}]
Windows Powershell
https://docs.microsoft.com/en-us/cli/azure/install-azure-cli-windows?view=azure-cli-latest
ONAP on Azure VMs
Pre-requisite
The supported versions are as follows:
ONAP Release | Rancher | Kubernetes | Helm | Kubectl | Docker |
---|---|---|---|---|---|
Amsterdam | 1.6.10 | 1.7.7 | 2.3.0 | 1.7.7 | 1.12.x |
Beijing | 1.6.14 | 1.8.6 | 2.6.1+ | 1.8.6 | 17.03-ce |
Reference Server
http://azure.onap.info:8880/r/projects/1a7/kubernetes-dashboard:9090/#!/pod?namespace=_all
Cluster Configuration
Single Server
Azure Artifacts
Static IP
Secondary Drive
VM
ARM Template
Follow https://docs.microsoft.com/en-us/azure/azure-resource-manager/resource-manager-create-first-template
Create a Storage Account
$ az login To sign in, use a web browser to open the page https://aka.ms/devicelogin and enter the code E...Z to authenticate. $ az group create --name examplegroup --location "South Central US" { "id": "/subscriptions/f4b...e8b/resourceGroups/examplegroup", "location": "southcentralus", "managedBy": null, "name": "examplegroup", "properties": { "provisioningState": "Succeeded" }, "tags": null } obrien:obrienlabs amdocs$ vi azuredeploy_storageaccount.json obrien:obrienlabs amdocs$ az group deployment create --resource-group examplegroup --template-file azuredeploy_storageaccount.json { "id": "/subscriptions/f4...e8b/resourceGroups/examplegroup/providers/Microsoft.Resources/deployments/azuredeploy_storageaccount", "name": "azuredeploy_storageaccount", "properties": { "additionalProperties": { "duration": "PT32.9822642S", "outputResources": [ { "id": "/subscriptions/f4..e8b/resourceGroups/examplegroup/providers/Microsoft.Storage/storageAccounts/storagekj6....kk2w", "resourceGroup": "examplegroup" }], "templateHash": "11440483235727994285"}, "correlationId": "41a0f79..90c291", "debugSetting": null, "dependencies": [], "mode": "Incremental", "outputs": {}, "parameters": {}, "parametersLink": null, "providers": [ { "id": null, "namespace": "Microsoft.Storage", "registrationState": null, "resourceTypes": [ { "aliases": null, "apiVersions": null, "locations": [ "southcentralus" ], "properties": null, "resourceType": "storageAccounts" }]}], "provisioningState": "Succeeded", "template": null, "templateLink": null, "timestamp": "2018-02-17T16:15:11.562170+00:00" }, "resourceGroup": "examplegroup"}
Pick a region
az account list-locations northcentralus for example
Create a resource group
# create a resource group if not already there az group create --name obrien_devops_northcentralus --location northcentralus "id": "/subscriptions/f.8b/resourceGroups/brien_devops_northcentralus"
Create a VM
We need a 128G VM with at least 8vCores (peak is 60) and a 100+GB drive. The sizes are detailed on https://docs.microsoft.com/en-ca/azure/virtual-machines/windows/sizes-memory - we will use the Standard_D32s_v3 type
We need an "all open 0.0.0.0/0" security group and
a reassociated data drive as boot drive
Get the ARM template
see open review in - OOM-711Getting issue details... STATUS
"ubuntuOSVersion": "16.04.0-LTS" "imagePublisher": "Canonical", "imageOffer": "UbuntuServer", "vmSize": "Standard_E8s_v3" "osDisk": {"createOption": "FromImage"},"dataDisks": [{"diskSizeGB": 511,"lun": 0, "createOption": "Empty" }]
Follow
https://github.com/Azure/azure-quickstart-templates/tree/master/101-acs-kubernetes
https://docs.microsoft.com/en-us/azure/azure-resource-manager/resource-group-template-deploy
https://github.com/Azure/azure-quickstart-templates/tree/master/101-vm-simple-linux
It needs a security group https://docs.microsoft.com/en-us/azure/virtual-network/virtual-networks-create-nsg-arm-template
"networkSecurityGroup": { "id": "[resourceId('Microsoft.Network/networkSecurityGroups', parameters('frontEndNSGName'))]" }
# validate first az group deployment validate --resource-group ONAPAMDOCS --template-file oom_azure_arm_deploy.json --parameters @oom_azure_arm_deploy_parameters.json # run deployment az group deployment create --resource-group ONAPAMDOCS --template-file oom_azure_arm_deploy.json --parameters @oom_azure_arm_deploy_parameters.json
SSH into your VM and run the Kubernetes and OOM installation scripts
# clone the oom repo to get the install directory git clone https://gerrit.onap.org/r/oom # run the Rancher RI installation (to install kubernetes) oom/install/rancher/oom_rancher_install.sh -b master -s 192.168.240.32 -e onap # run the oom deployment script # get a copy of onap-parametes.yaml and place in this folder oom/install/deployment/cd.sh -b master -s 192.168.240.32 -e onap
oom_rancher_install.sh is in - OOM-715Getting issue details... STATUS under https://gerrit.onap.org/r/#/c/32019/
cd.sh in - OOM-716Getting issue details... STATUS under https://gerrit.onap.org/r/#/c/32653/
Delete the VM and resource group
# delete the vm and resources az group deployment delete --resource-group ONAPAMDOCS --name oom_azure_arm_deploy # the above deletion will not delete the actual resources - only a delete of the group or each individual resource works # optionally delete the resource group az group delete --name ONAPAMDOCS -y
ONAP on Azure Container Service
AKS Installation
Follow https://docs.microsoft.com/en-us/azure/aks/tutorial-kubernetes-deploy-cluster
Register for AKS preview via az cli
obrienbiometrics:obrienlabs michaelobrien$ az provider register -n Microsoft.ContainerService Registering is still on-going. You can monitor using 'az provider show -n Microsoft.ContainerService'
Create an AKS resource group
Raise your AKS vCPU quota - optional
http://aka.ms/corequotaincrease
https://portal.azure.com/#blade/Microsoft_Azure_Support/HelpAndSupportBlade/newsupportrequest
Deployment failed. Correlation ID: 4b4707a7-2244-4557-855e-11bcced556de. Provisioning of resource(s) for container service onapAKSCluster in resource group onapAKS failed. Message: Operation results in exceeding quota limits of Core. Maximum allowed: 10, Current in use: 10, Additional requested: 1. Please read more about quota increase at http://aka.ms/corequotaincrease.. Details:
Create AKS cluster
obrienbiometrics:obrienlabs michaelobrien$ az aks create --resource-group onapAKS --name onapAKSCluster --node-count 1 --generate-ssh-keys - Running .. "fqdn": "onapaksclu-onapaks-f4....3.hcp.eastus.azmk8s.io",
AKS cluster VM granularity
The cluster will start with a 3.5G VM before scaling
Resources for your AKS cluster
Bring up AAI only for now
Design Issues
Resource Group
A resource group makes it easier to package and remove everything for a deployment - essentially making the deployment stateless
Network Security Group
Global or local to the resource group?
Static public IP
Register a CNAME for an existing domain and use the same IP address everytime the deployment comes up
Entrypoint cloud init script
How to attach the cloud init script to provision the VM
ARM template chaining
passing derived varialbles into the next arm template - for example when bringing up an entire federated set in one or more DCs
Troubleshooting
DNS propagation and caching
It takes about 2 min for DNS entries to propagate out from A record DNS changes. For example the following IP/DNS association took 2 min to appear in dig.
obrienbiometrics:onap_oom_711_azure michaelobrien$ dig azure.onap.info ; <<>> DiG 9.9.7-P3 <<>> azure.onap.info ;; global options: +cmd ;; Got answer: ;; ->>HEADER<<- opcode: QUERY, status: NOERROR, id: 10599 ;; flags: qr rd ra; QUERY: 1, ANSWER: 1, AUTHORITY: 0, ADDITIONAL: 1 ;; OPT PSEUDOSECTION: ; EDNS: version: 0, flags:; udp: 512 ;; QUESTION SECTION: ;azure.onap.info. IN A ;; ANSWER SECTION: azure.onap.info. 251 IN A 52.224.233.230 ;; Query time: 68 msec ;; SERVER: 8.8.8.8#53(8.8.8.8) ;; WHEN: Tue Feb 20 10:26:59 EST 2018 ;; MSG SIZE rcvd: 60 obrienbiometrics:onap_oom_711_azure michaelobrien$ dig azure.onap.info ; <<>> DiG 9.9.7-P3 <<>> azure.onap.info ;; global options: +cmd ;; Got answer: ;; ->>HEADER<<- opcode: QUERY, status: NOERROR, id: 30447 ;; flags: qr rd ra; QUERY: 1, ANSWER: 1, AUTHORITY: 0, ADDITIONAL: 1 ;; OPT PSEUDOSECTION: ; EDNS: version: 0, flags:; udp: 512 ;; QUESTION SECTION: ;azure.onap.info. IN A ;; ANSWER SECTION: azure.onap.info. 299 IN A 13.92.225.167 ;; Query time: 84 msec ;; SERVER: 8.8.8.8#53(8.8.8.8) ;; WHEN: Tue Feb 20 10:27:04 EST 2018
Corporate Firewall Access
Inside the corporate firewall - avoid it PS C:\> az login Please ensure you have network connection. Error detail: HTTPSConnectionPool(host='login.microsoftonline.com', port=443) : Max retries exceeded with url: /common/oauth2/devicecode?api-version=1.0 (Caused by NewConnectionError('<urllib3.conne ction.VerifiedHTTPSConnection object at 0x04D18730>: Failed to establish a new connection: [Errno 11001] getaddrinfo fai led',)) at home or cell hotspot PS C:\> az login To sign in, use a web browser to open the page https://aka.ms/devicelogin and enter the code E...2W to authenticate. [ { "cloudName": "AzureCloud", "id": "4...da1", "isDefault": true, "name": "Microsoft Azure Internal Consumption", "state": "Enabled", "tenantId": "72f98....47", "user": { "name": "fran...ocs.com", "type": "user" }] On corporate account (need permissions bump to be able to create a resource group prior to running an arm template https://wiki.onap.org/display/DW/ONAP+on+Kubernetes+on+Microsoft+Azure#ONAPonKubernetesonMicrosoftAzure-ARMTemplate PS C:\> az group create --name onapKubernetes --location eastus The client 'fra...s.com' with object id '08f98c7e-...ed' does not have authorization to per form action 'Microsoft.Resources/subscriptions/resourcegroups/write' over scope '/subscriptions/42e...8 7da1/resourcegroups/onapKubernetes'. try my personal = OK PS C:\> az login To sign in, use a web browser to open the page https://aka.ms/devicelogin and enter the code EE...ULR to authenticate. Terminate batch job (Y/N)? y # hangs when first time login in a new pc PS C:\> az login To sign in, use a web browser to open the page https://aka.ms/devicelogin and enter the code E.PBKS to authenticate. [ { "cloudName": "AzureCloud", "id": "f4b...b", "isDefault": true "name": "Pay-As-You-Go", "state": "Enabled", "tenantId": "bcb...f4f", "user": "name": "michael@obrien...org", "type": "user" } }] PS C:\> az group create --name onapKubernetes2 --location eastus { "id": "/subscriptions/f4b....b/resourceGroups/onapKubernetes2", "location": "eastus", "managedBy": null, "name": "onapKubernetes2", "properties": { "provisioningState": "Succeeded" }, "tags": null}
20180228: Deployment delete does not delete resources without a resourceGroup delete
I find that a delete deployment deletes the deployment but not the actual resources. The workaround is to delete the resource group - but in some constrained subscriptions the cli user may not have the ability to create a resource group - and hence delete it.
see
https://github.com/Azure/azure-sdk-for-java/issues/1167
deleting the resources manually for now - is a workaround if you cannot create/delete resource groups
# delete the vm and resources az group deployment delete --resource-group ONAPAMDOCS --name oom_azure_arm_deploy
# the above deletion will not delete the actual resources - only a delete of the group or each individual resource works
# optionally delete the resource group az group delete --name ONAPAMDOCS -y
However modifying the template to add resources works well. For example adding a reference to a network security group
20180228: Resize the OS disk
ONAP requires at least 75g - the issue is than in most VM templates on Azure - the OS disk is 30g - we need to either switch to the data disk or resize the os disk.
# add diskSizeGB to the template "osDisk": { "diskSizeGB": 255, "createOption": "FromImage" }, ubuntu@oom-auto-deploy:~$ df Filesystem 1K-blocks Used Available Use% Mounted on udev 65989400 0 65989400 0% /dev tmpfs 13201856 8848 13193008 1% /run /dev/sda1 259142960 1339056 257787520 1% / tmpfs 66009280 0 66009280 0% /dev/shm tmpfs 5120 0 5120 0% /run/lock tmpfs 66009280 0 66009280 0% /sys/fs/cgroup none 64 0 64 0% /etc/network/interfaces.dynamic.d /dev/sdb1 264091588 60508 250592980 1% /mnt tmpfs 13201856 0 13201856 0% /run/user/1000 ubuntu@oom-auto-deploy:~$ free total used free shared buff/cache available Mem: 132018560 392336 131242164 8876 384060 131012328
20180301: Add oom_entrypoint.sh bootstrap script to install rancher and onap
in review under OOM-715
https://jira.onap.org/secure/attachment/11206/oom_entrypoint.sh
If using amsterdam - swap out the onap-parameters.yaml (the curl is hardcoded to a master branch version)
20180303: cloudstorage access on OSX via Azure Storage Manager
use this method instead of installing az cli directly - for certain corporate oauth configurations
https://azure.microsoft.com/en-us/features/storage-explorer/
Install AZM using the name and access key of a storage account created manually or by enabling the az cli on the browser
Training
(from Microsoft - thank you)
General Azure Documentation
Azure Site http://azure.microsoft.com
Azure Documentation Site https://docs.microsoft.com/en-us/azure/
Azure Training Courses https://azure.microsoft.com/en-us/training/free-online-courses/
Azure Portal http://portal.azure.com
Developer Documentation
Azure AD Authentication Libraries https://docs.microsoft.com/en-us/azure/active-directory/develop/active-directory-authentication-libraries
Java Overview on Azure https://azure.microsoft.com/en-us/develop/java/
Java Docs for Azure https://docs.microsoft.com/en-us/java/azure/
Java SDK on GitHub https://github.com/Azure/azure-sdk-for-java
Python Overview on Azure https://azure.microsoft.com/en-us/develop/python/
Python Docs for Azure https://docs.microsoft.com/en-us/python/azure/
Python SDK on GitHub https://github.com/Azure/azure-sdk-for-python
REST Api and CLI Documentation
REST API Documentation https://docs.microsoft.com/en-us/rest/api/
CLI Documentation https://docs.microsoft.com/en-us/cli/azure/index
Other Documentation
Using Automation for VM shutdown & startup https://docs.microsoft.com/en-us/azure/automation/automation-solution-vm-management
Azure Resource Manager (ARM) QuickStart Templates https://github.com/Azure/azure-quickstart-templates
Links
https://azure.microsoft.com/en-us/services/container-service/
https://docs.microsoft.com/en-us/azure/templates/microsoft.compute/virtualmachines
https://kubernetes.io/docs/concepts/containers/images/#using-azure-container-registry-acr
https://azure.microsoft.com/en-us/features/storage-explorer/