You are viewing an old version of this page. View the current version.

Compare with Current View Page History

« Previous Version 8 Next »



Requirements

Docker images = 30G

Shared NFS = 5G

RAM = 70G+

vCores = 12+ (max 55)

Cluster = 3 + 1 (hosts + rancher master)

Network = 170 MB/sec (2700 packets/sec at 64k/packet max) = NFS + docker pulls


Amazon AWS

Account Provider: (2) Robin of Amazon and Michael O'Brien of Amdocs


EC2 VMs

AWS Clustered Deployment

AWS EC2 Cluster Creation

AWS EFS share for shared NFS

You need an NFS share between the VM's in your Kubernetes cluster - an Elastic File System share will wrap NFS


"From the NFS wizard"

Setting up your EC2 instance

  1. Using the Amazon EC2 console, associate your EC2 instance with a VPC security group that enables access to your mount target. For example, if you assigned the "default" security group to your mount target, you should assign the "default" security group to your EC2 instance. Learn more
  2. Open an SSH client and connect to your EC2 instance. (Find out how to connect)

  3. If you're not using the EFS mount helper, install the NFS client on your EC2 instance:
    • On an Ubuntu instance:
      sudo apt-get install nfs-common

Mounting your file system

  1. Open an SSH client and connect to your EC2 instance. (Find out how to connect)
  2. Create a new directory on your EC2 instance, such as "efs".
    • sudo mkdir efs
  3. Mount your file system. If you require encryption of data in transit, use the EFS mount helper and the TLS mount option. Mounting considerations
    • Using the EFS mount helper:
      sudo mount -t efs fs-43b2763a:/ efs
    • Using the EFS mount helper and encryption of data in transit:
      sudo mount -t efs -o tls fs-43b2763a:/ efs
    • Using the NFS client:
      sudo mount -t nfs4 -o nfsvers=4.1,rsize=1048576,wsize=1048576,hard,timeo=600,retrans=2 fs-43b2763a.efs.us-east-2.amazonaws.com:/ efs

If you are unable to connect, see our troubleshooting documentation.

https://docs.aws.amazon.com/efs/latest/ug/mounting-fs.html

ubuntu@ip-10-0-0-66:~$ sudo apt-get install nfs-common
ubuntu@ip-10-0-0-66:~$ cd /
ubuntu@ip-10-0-0-66:~$ sudo mkdir /dockerdata-nfs
root@ip-10-0-0-19:/# sudo mount -t nfs4 -o nfsvers=4.1,rsize=1048576,wsize=1048576,hard,timeo=600,retrans=2 fs-43b2763a.efs.us-east-2.amazonaws.com:/ /dockerdata-nfs
# write something on one vm - and verify it shows on another
ubuntu@ip-10-0-0-8:~$ ls /dockerdata-nfs/
test.sh


Microsoft Azure

Subscription Sponsor: (1) Microsoft

VMs

Deliverables are deployment scripts, arm/cli templates for various deployment scenarios (single, multiple, federated servers)

In review  OOM-710 - Getting issue details... STATUS ,  OOM-711 - Getting issue details... STATUS ,  OOM-716 - Getting issue details... STATUS ,  OOM-715 - Getting issue details... STATUS

Quickstart

Single collocated VM

Automation is currently only written for single VM that hosts both the rancher server and the deployed onap pods. Use the ARM template below to deploy your VM and provision it (adjust your config parameters)

Two choices, run the single oom_deployment.sh ARM wrapper - or use it to bring up an empty vm and run oom_entrypoint.sh manually. Once the VM comes up the oom_entrypoint.sh script will run - which will download the oom_rancher_setup.sh script to setup docker, rancher, kubernetes and helm - the entrypoint script will then run the cd.sh script to bring up onap based on your values.yaml config by running helm install on it.

# login to az cli, wget the deployment script, arm template and parameters file - edit the parameters file (dns, ssh key ...) and run the arm template
wget https://jira.onap.org/secure/attachment/11459/oom_deployment.sh
wget https://jira.onap.org/secure/attachment/11419/arm_deploy_ons_sut_parameters.json
wget https://jira.onap.org/secure/attachment/11420/arm_deploy_ons_sut.json
# either run the entrypoint which creates a resource template and runs the stack - or do those two commands manually
./oom_deployment.sh -b master -s azure.onap.cloud -e onap -r a_auto-youruserid_20180421 -t arm_deploy_ons_sut.json -p arm_deploy_ons_sut_parameters.json
# wait for the VM to finish in about 75 min or watch progress by ssh'ing into the vm and doing
root@ons-auto-201803181110z: sudo tail -f /var/lib/waagent/custom-script/download/0/stdout

# if you wish to run the oom_entrypoint script yourself - edit/break the cloud init section at the end of the arm template and do it yourself below
# download and edit values.yaml with your onap preferences and openstack tenant config
wget https://jira.onap.org/secure/attachment/11414/values.yaml
# download and run the bootstrap and onap install script, the -s server name can be an IP, FQDN or hostname
wget https://jira.onap.org/secure/attachment/11518/oom_entrypoint.sh
chmod 777 oom_entrypoint.sh
sudo ./oom_entrypoint.sh -b master -s devops.onap.info -e onap
# wait 15 min for rancher to finish, then 30-90 min for onap to come up


OOM-714 - Getting issue details... STATUS  see https://jira.onap.org/secure/attachment/11455/oom_openstack.yaml and https://jira.onap.org/secure/attachment/11454/oom_openstack_oom.env

OOM-710 - Getting issue details... STATUS  see https://jira.onap.org/secure/attachment/11439/oom_entrypoint.sh

customize your template (true/false for any components, docker overrides etc...)

https://jira.onap.org/secure/attachment/11414/values.yaml

Run oom_entrypoint.sh after you verified values.yaml - it will run both scripts below for you - a single node kubernetes setup running what you configured in values.yaml will be up in 50-90 min.  If you want to just configure your vm without bringing up ONAP - comment out the cd.sh line and run that separately.

OOM-715 - Getting issue details... STATUS  see https://jira.onap.org/secure/attachment/11438/oom_rancher_setup.sh

OOM-716 - Getting issue details... STATUS  see https://jira.onap.org/secure/attachment/11413/cd.sh

Verify your system is up by doing a kubectl get pods --all-namespaces and checking the 8880 port to bring up the rancher or kubernetes gui.

20180319: Amsterdam fully automated ARM template to ONAP up

20180312: Master 2 step ARM template - then run entrypoint.sh to ONAP up

Login to Azure CLI

https://portal.azure.com/#blade/HubsExtension/Resources/resourceType/Microsoft.Resources%2Fresources

Download Azure ONAP ARM template

see 

OOM-711 - Getting issue details... STATUS

Edit Azure ARM template environment parameters
Create Resource Group
az group create --name onap_eastus --location eastus

Run ARM template
az group deployment create --resource-group onap_eastus --template-file oom_azure_arm_deploy.json --parameters @oom_azure_arm_deploy_parameters.json
Wait for Rancher/Kubernetes install

The oom_entrypoint.sh script will be run as a cloud-init script on the VM - see 

OOM-710 - Getting issue details... STATUS

which runs

OOM-715 - Getting issue details... STATUS

Wait for OOM ONAP install

see 

OOM-716 - Getting issue details... STATUS

Verify ONAP installation 
kubectl get pods --all-namespaces
# raise/lower onap components from the installed directory if using the oneclick arm template
# amsterdam only
root@ons-auto-master-201803191429z:/var/lib/waagent/custom-script/download/0/oom/kubernetes/oneclick# ./createAll.bash -n onap


Azure CLI Installation

Requirements

Azure subscription

OSX

https://docs.microsoft.com/en-us/cli/azure/install-azure-cli?view=azure-cli-latest

Install homebrew first (reinstall if you are on the latest OSX 10.13.2 https://github.com/Homebrew/install because of 3718)

Will install Python 3.6

$brew update
$brew install azure-cli

https://docs.microsoft.com/en-us/cli/azure/get-started-with-azure-cli?view=azure-cli-latest

$ az login
To sign in, use a web browser to open the page https://aka.ms/devicelogin and enter the code E..D to authenticate.
[ {
    "cloudName": "AzureCloud",
    "id": "f4...b",
    "isDefault": true,
    "name": "Pay-As-You-Go",
    "state": "Enabled",
    "tenantId": "bcb.....f",
    "user": {
      "name": "michael@....org",
      "type": "user"
    }}]
Bastion/Jumphost VM in Azure

https://docs.microsoft.com/en-us/cli/azure/install-azure-cli-apt?view=azure-cli-latest

# in root
AZ_REPO=$(lsb_release -cs)
echo "deb [arch=amd64] https://packages.microsoft.com/repos/azure-cli/ $AZ_REPO main" |      sudo tee /etc/apt/sources.list.d/azure-cli.list
apt-key adv --keyserver packages.microsoft.com --recv-keys 52E16F86FEE04B979B07E28DB02C46DF417A0893
apt-get install apt-transport-https
apt-get update && sudo apt-get install azure-cli
az login


# verify
root@ons-dmz:~# ps -ef | grep az
root       1427      1  0 Mar17 ?        00:00:00 /usr/lib/linux-tools/4.13.0-1011-azure/hv_vss_daemon -n

Windows Powershell

https://docs.microsoft.com/en-us/cli/azure/install-azure-cli-windows?view=azure-cli-latest

ARM Template

Follow https://docs.microsoft.com/en-us/azure/azure-resource-manager/resource-manager-create-first-template

Create a Storage Account
$ az login
To sign in, use a web browser to open the page https://aka.ms/devicelogin and enter the code E...Z to authenticate.
$ az group create --name examplegroup --location "South Central US"
{
  "id": "/subscriptions/f4b...e8b/resourceGroups/examplegroup",
  "location": "southcentralus",
  "managedBy": null,
  "name": "examplegroup",
  "properties": {
    "provisioningState": "Succeeded"
  },
  "tags": null
}
obrien:obrienlabs amdocs$ vi azuredeploy_storageaccount.json
obrien:obrienlabs amdocs$ az group deployment create --resource-group examplegroup --template-file azuredeploy_storageaccount.json
{
  "id": "/subscriptions/f4...e8b/resourceGroups/examplegroup/providers/Microsoft.Resources/deployments/azuredeploy_storageaccount",
  "name": "azuredeploy_storageaccount",
  "properties": {
    "additionalProperties": {
      "duration": "PT32.9822642S",
      "outputResources": [
        {
          "id": "/subscriptions/f4..e8b/resourceGroups/examplegroup/providers/Microsoft.Storage/storageAccounts/storagekj6....kk2w",
          "resourceGroup": "examplegroup"
        }],
      "templateHash": "11440483235727994285"},
    "correlationId": "41a0f79..90c291",
    "debugSetting": null,
    "dependencies": [],
    "mode": "Incremental",
    "outputs": {},
    "parameters": {},
    "parametersLink": null,
    "providers": [
      {
        "id": null,
        "namespace": "Microsoft.Storage",
        "registrationState": null,
        "resourceTypes": [
          {
            "aliases": null,
            "apiVersions": null,
            "locations": [
              "southcentralus"
            ],
            "properties": null,
            "resourceType": "storageAccounts"
          }]}],
    "provisioningState": "Succeeded",
    "template": null,
    "templateLink": null,
    "timestamp": "2018-02-17T16:15:11.562170+00:00"
  },
  "resourceGroup": "examplegroup"}
Pick a region
az account list-locations
northcentralus
for example
Create a resource group
# create a resource group if not already there
az group create --name obrien_jenkins_b_westus2 --location westus2

Create a VM

We need a 128G VM with at least 8vCores (peak is 60) and a 100+GB drive. The sizes are detailed on https://docs.microsoft.com/en-ca/azure/virtual-machines/windows/sizes-memory - we will use the Standard_D32s_v3 type

We need an "all open 0.0.0.0/0" security group and a reassociated data drive as boot drive - see the arm template in OOM-711

Get the ARM template

see open review in  OOM-711 - Getting issue details... STATUS 


"ubuntuOSVersion": "16.04.0-LTS"
"imagePublisher": "Canonical",
"imageOffer": "UbuntuServer",
"vmSize": "Standard_E8s_v3"
"osDisk": {"createOption": "FromImage"},"dataDisks": [{"diskSizeGB": 511,"lun": 0, "createOption": "Empty" }]

Follow

https://github.com/Azure/azure-quickstart-templates/tree/master/101-acs-kubernetes

https://docs.microsoft.com/en-us/azure/azure-resource-manager/resource-group-template-deploy

https://docs.microsoft.com/en-us/cli/azure/group/deployment?view=azure-cli-latest#az_group_deployment_create

https://github.com/Azure/azure-quickstart-templates/tree/master/101-vm-simple-linux

It needs a security group https://docs.microsoft.com/en-us/azure/virtual-network/virtual-networks-create-nsg-arm-template

{
"apiVersion": "2017-03-01",
"type": "Microsoft.Network/networkSecurityGroups",
"name": "[variables('networkSecurityGroupName')]",
"location": "[resourceGroup().location]",
"tags": { "displayName": "NSG - Front End" },
"properties": {
  "securityRules": [
    {
      "name": "in-rule",
      "properties": {
        "description": "All in",
        "protocol": "Tcp",
        "sourcePortRange": "*",
        "destinationPortRange": "*",
        "sourceAddressPrefix": "Internet",
        "destinationAddressPrefix": "*",
        "access": "Allow",
        "priority": 100,
        "direction": "Inbound"
      }
    },
    {
      "name": "out-rule",
      "properties": {
        "description": "All out",
        "protocol": "Tcp",
        "sourcePortRange": "*",
        "destinationPortRange": "*",
        "sourceAddressPrefix": "Internet",
        "destinationAddressPrefix": "*",
        "access": "Allow",
        "priority": 101,
        "direction": "Outbound"
      }
    }
  ]
}
}
, 
{
      "apiVersion": "2017-04-01",
      "type": "Microsoft.Network/virtualNetworks",
      "name": "[variables('virtualNetworkName')]",
      "location": "[resourceGroup().location]",
      "dependson": [
        "[concat('Microsoft.Network/networkSecurityGroups/', variables('networkSecurityGroupName'))]"
      ],
      "properties": {
        "addressSpace": {
          "addressPrefixes": [
            "[variables('addressPrefix')]"
          ]
        },
        "subnets": [
          {
            "name": "[variables('subnetName')]",
            "properties": {
              "addressPrefix": "[variables('subnetPrefix')]",
              "networkSecurityGroup": {
                 "id": "[resourceId('Microsoft.Network/networkSecurityGroups', variables('networkSecurityGroupName'))]"
              }
            }
          }
        ]
      }
    },
# validate first (validate instead of create)
az group deployment create --resource-group obrien_jenkins_b_westus2 --template-file oom_azure_arm_deploy.json --parameters @oom_azure_arm_cd_amsterdam_deploy_parameters.json

SSH into your VM and run the Kubernetes and OOM installation scripts

Use the entrypoint script in  OOM-710 - Getting issue details... STATUS 

# clone the oom repo to get the install directory
git clone https://gerrit.onap.org/r/oom
# run the Rancher RI installation (to install kubernetes)
oom/install/rancher/oom_rancher_install.sh -b master -s 192.168.240.32 -e onap
# run the oom deployment script
# get a copy of onap-parametes.yaml and place in this folder
oom/install/deployment/cd.sh -b master -s 192.168.240.32 -e onap

oom_rancher_install.sh is in  OOM-715 - Getting issue details... STATUS  under https://gerrit.onap.org/r/#/c/32019/

cd.sh in  OOM-716 - Getting issue details... STATUS  under https://gerrit.onap.org/r/#/c/32653/

Delete the VM and resource group
# delete the vm and resources
az group deployment delete --resource-group ONAPAMDOCS --name oom_azure_arm_deploy
# the above deletion will not delete the actual resources - only a delete of the group or each individual resource works
# optionally delete the resource group
az group delete --name ONAPAMDOCS -y

Azure devops
create static IP

az network public-ip create --name onap-argon --resource-group a_ONAP_argon_prod_donotdelete --location eastus --allocation-method Static

ONAP on Azure Container Service

AKS Installation

Follow https://docs.microsoft.com/en-us/azure/aks/tutorial-kubernetes-deploy-cluster

Register for AKS preview via az cli
obrienbiometrics:obrienlabs michaelobrien$ az provider register -n Microsoft.ContainerService
Registering is still on-going. You can monitor using 'az provider show -n Microsoft.ContainerService'
Create an AKS resource group

Raise your AKS vCPU quota - optional

https://docs.microsoft.com/en-us/azure/azure-subscription-service-limits#container-service-aks-limits

http://aka.ms/corequotaincrease

https://portal.azure.com/#blade/Microsoft_Azure_Support/HelpAndSupportBlade/newsupportrequest

Deployment failed. Correlation ID: 4b4707a7-2244-4557-855e-11bcced556de. Provisioning of resource(s) for container service onapAKSCluster in resource group onapAKS failed. Message: Operation results in exceeding quota limits of Core. Maximum allowed: 10, Current in use: 10, Additional requested: 1. Please read more about quota increase at http://aka.ms/corequotaincrease.. Details:

Create AKS cluster
obrienbiometrics:obrienlabs michaelobrien$ az aks create --resource-group onapAKS --name onapAKSCluster --node-count 1 --generate-ssh-keys
 - Running ..
 "fqdn": "onapaksclu-onapaks-f4....3.hcp.eastus.azmk8s.io",
AKS cluster VM granularity

The cluster will start with a 3.5G VM before scaling

Resources for your AKS cluster



Bring up AAI only for now


Design Issues

Resource Group

A resource group makes it easier to package and remove everything for a deployment - essentially making the deployment stateless

Network Security Group

Global or local to the resource group?

Static public IP

Register a CNAME for an existing domain and use the same IP address everytime the deployment comes up

Entrypoint cloud init script

How to attach the cloud init script to provision the VM

ARM template chaining

passing derived varialbles into the next arm template - for example when bringing up an entire federated set in one or more DCs

see script attached to 

Troubleshooting

DNS propagation and caching

It takes about 2 min for DNS entries to propagate out from A record DNS changes.  For example the following IP/DNS association took 2 min to appear in dig.

obrienbiometrics:onap_oom_711_azure michaelobrien$ dig azure.onap.info
; <<>> DiG 9.9.7-P3 <<>> azure.onap.info
;; global options: +cmd
;; Got answer:
;; ->>HEADER<<- opcode: QUERY, status: NOERROR, id: 10599
;; flags: qr rd ra; QUERY: 1, ANSWER: 1, AUTHORITY: 0, ADDITIONAL: 1
;; OPT PSEUDOSECTION:
; EDNS: version: 0, flags:; udp: 512
;; QUESTION SECTION:
;azure.onap.info.		IN	A
;; ANSWER SECTION:
azure.onap.info.	251	IN	A	52.224.233.230
;; Query time: 68 msec
;; SERVER: 8.8.8.8#53(8.8.8.8)
;; WHEN: Tue Feb 20 10:26:59 EST 2018
;; MSG SIZE  rcvd: 60

obrienbiometrics:onap_oom_711_azure michaelobrien$ dig azure.onap.info
; <<>> DiG 9.9.7-P3 <<>> azure.onap.info
;; global options: +cmd
;; Got answer:
;; ->>HEADER<<- opcode: QUERY, status: NOERROR, id: 30447
;; flags: qr rd ra; QUERY: 1, ANSWER: 1, AUTHORITY: 0, ADDITIONAL: 1
;; OPT PSEUDOSECTION:
; EDNS: version: 0, flags:; udp: 512
;; QUESTION SECTION:
;azure.onap.info.		IN	A
;; ANSWER SECTION:
azure.onap.info.	299	IN	A	13.92.225.167
;; Query time: 84 msec
;; SERVER: 8.8.8.8#53(8.8.8.8)
;; WHEN: Tue Feb 20 10:27:04 EST 2018
Corporate Firewall Access
Inside the corporate firewall - avoid it

PS C:\> az login
Please ensure you have network connection. Error detail: HTTPSConnectionPool(host='login.microsoftonline.com', port=443)
: Max retries exceeded with url: /common/oauth2/devicecode?api-version=1.0 (Caused by NewConnectionError('<urllib3.conne
ction.VerifiedHTTPSConnection object at 0x04D18730>: Failed to establish a new connection: [Errno 11001] getaddrinfo fai
led',))

at home or cell hotspot

PS C:\> az login
To sign in, use a web browser to open the page https://aka.ms/devicelogin and enter the code E...2W to authenticate.
[  {    "cloudName": "AzureCloud",    "id": "4...da1",    "isDefault": true,    "name": "Microsoft Azure Internal Consumption",    "state": "Enabled",    "tenantId": "72f98....47",    "user": {      "name": "fran...ocs.com",      "type": "user"    }]

On corporate account (need permissions bump to be able to create a resource group prior to running an arm template
https://wiki.onap.org/display/DW/ONAP+on+Kubernetes+on+Microsoft+Azure#ONAPonKubernetesonMicrosoftAzure-ARMTemplate
PS C:\> az group create --name onapKubernetes --location eastus
The client 'fra...s.com' with object id '08f98c7e-...ed' does not have authorization to per
form action 'Microsoft.Resources/subscriptions/resourcegroups/write' over scope '/subscriptions/42e...8
7da1/resourcegroups/onapKubernetes'.

try my personal = OK
PS C:\> az login
To sign in, use a web browser to open the page https://aka.ms/devicelogin and enter the code EE...ULR to authenticate.
Terminate batch job (Y/N)? y
# hangs when first time login in a new pc
PS C:\> az login
To sign in, use a web browser to open the page https://aka.ms/devicelogin and enter the code E.PBKS to authenticate.
[  {    "cloudName": "AzureCloud",    "id": "f4b...b",    "isDefault": true    "name": "Pay-As-You-Go",    "state": "Enabled",   "tenantId": "bcb...f4f",   "user":       "name": "michael@obrien...org",    "type": "user"    }  }]
PS C:\> az group create --name onapKubernetes2 --location eastus
{  "id": "/subscriptions/f4b....b/resourceGroups/onapKubernetes2",  "location": "eastus",  "managedBy": null,  "name": "onapKubernetes2",  "properties": {    "provisioningState": "Succeeded"  },  "tags": null}

Design Issues

20180228: Deployment delete does not delete resources without a resourceGroup delete

I find that a delete deployment deletes the deployment but not the actual resources.  The workaround is to delete the resource group - but in some constrained subscriptions the cli user may not have the ability to create a resource group - and hence delete it.

see

https://github.com/Azure/azure-sdk-for-java/issues/1167

deleting the resources manually for now - is a workaround if you cannot create/delete resource groups

# delete the vm and resources
az group deployment delete --resource-group ONAPAMDOCS --name oom_azure_arm_deploy
# the above deletion will not delete the actual resources - only a delete of the group or each individual resource works
# optionally delete the resource group
az group delete --name ONAPAMDOCS -y

However modifying the template to add resources works well.  For example adding a reference to a network security group

20180228: Resize the OS disk

ONAP requires at least 75g - the issue is than in most VM templates on Azure - the OS disk is 30g - we need to either switch to the data disk or resize the os disk.

# add diskSizeGB to the template
          "osDisk": {
                "diskSizeGB": 255,
                "createOption": "FromImage"
            },
ubuntu@oom-auto-deploy:~$ df
Filesystem     1K-blocks    Used Available Use% Mounted on
udev            65989400       0  65989400   0% /dev
tmpfs           13201856    8848  13193008   1% /run
/dev/sda1      259142960 1339056 257787520   1% /
tmpfs           66009280       0  66009280   0% /dev/shm
tmpfs               5120       0      5120   0% /run/lock
tmpfs           66009280       0  66009280   0% /sys/fs/cgroup
none                  64       0        64   0% /etc/network/interfaces.dynamic.d
/dev/sdb1      264091588   60508 250592980   1% /mnt
tmpfs           13201856       0  13201856   0% /run/user/1000
ubuntu@oom-auto-deploy:~$ free
              total        used        free      shared  buff/cache   available
Mem:      132018560      392336   131242164        8876      384060   131012328
20180301: Add oom_entrypoint.sh bootstrap script to install rancher and onap

in review under OOM-715 

https://jira.onap.org/secure/attachment/11206/oom_entrypoint.sh

If using amsterdam - swap out the onap-parameters.yaml  (the curl is hardcoded to a master branch version)

20180303: cloudstorage access on OSX via Azure Storage Manager

use this method instead of installing az cli directly - for certain corporate oauth configurations

https://azure.microsoft.com/en-us/features/storage-explorer/

Install AZM using the name and access key of a storage account created manually or by enabling the az cli on the browser

20180318: add oom_entrypoint.sh to cloud-init on the arm template

See https://docs.microsoft.com/en-us/azure/templates/microsoft.compute/virtualmachines/extensions it looks like Azure has a similar setup to AWS ebextentions

Targetting 

typestringNoSpecifies the type of the extension; an example is "CustomScriptExtension".


https://docs.microsoft.com/en-us/azure/virtual-machines/linux/extensions-customscript

deprecated
 {
    "apiVersion": "2015-06-15",
    "type": "Microsoft.Compute/virtualMachines/extensions",
    "name": "[concat(parameters('vmName'),'/onap')]",
    "location": "[resourceGroup().location]",
    "dependsOn": ["[concat('Microsoft.Compute/virtualMachines/', parameters('vmName'))]"],
    "properties": {
        "publisher": "Microsoft.Azure.Extensions",
        "type": "CustomScript",
        "typeHandlerVersion": "1.9",
        "autoUpgradeMinorVersion": true,
        "settings": {
            "fileUris": [ "https://jira.onap.org/secure/attachment/11263/oom_entrypoint.sh" ],
"commandToExecute": "[concat('./' , parameters('scriptName'), ' -b master -s dns/pub/pri-ip -e onap' )]" }
        }
     }


use
    {
    "apiVersion": "2017-12-01",
    "type": "Microsoft.Compute/virtualMachines/extensions",
    "name": "[concat(parameters('vmName'),'/onap')]",
    "location": "[resourceGroup().location]",
    "dependsOn": ["[concat('Microsoft.Compute/virtualMachines/', parameters('vmName'))]"],
    "properties": {
        "publisher": "Microsoft.Azure.Extensions",
        "type": "CustomScript",
        "typeHandlerVersion": "2.0",
        "autoUpgradeMinorVersion": true,
        "settings": {
            "fileUris": [ "https://jira.onap.org/secure/attachment/11281/oom_entrypoint.sh" ],
            "commandToExecute": "[concat('./' , parameters('scriptName'), ' -b master ', ' -s ', 'ons-auto-201803181110z', ' -e onap' )]"
           }
        }
     }


ubuntu@ons-dmz:~$ ./oom_deployment.sh 

Deployment template validation failed: 'The template resource 'entrypoint' for type 'Microsoft.Compute/virtualMachines/extensions' at line '1' and column '6182' has incorrect segment lengths. A nested resource type must have identical number of segments as its resource name. A root resource type must have segment length one greater than its resource name. Please see https://aka.ms/arm-template/#resources for usage details.'.

ubuntu@ons-dmz:~$ ./oom_deployment.sh 

Deployment failed. Correlation ID: 532b9a9b-e0e8-4184-9e46-6c2e7c15e7c7. {

  "error": {

    "code": "ParentResourceNotFound",

    "message": "Can not perform requested operation on nested resource. Parent resource '[concat(parameters('vmName'),'' not found."

  }

}

fixed 20180318:1600

Install runs - but I need visibility - checking /var/lib/waagent/custom-script/download/0/

progress

./oom_deployment.sh


# 7 min to delete old deployment
ubuntu@ons-dmz:~$ az vm extension list -g a_ONAP_auto_201803181110z --vm-name ons-auto-201803181110z
..
    "provisioningState": "Creating",
 "settings": {
      "commandToExecute": "./oom_entrypoint.sh -b master  -s ons-auto-201803181110zons-auto-201803181110z.eastus.cloudapp.azure.com -e onap",
      "fileUris": [
        "https://jira.onap.org/secure/attachment/11263/oom_entrypoint.sh"  


ubuntu@ons-auto-201803181110z:~$ sudo su -
root@ons-auto-201803181110z:~# docker ps
CONTAINER ID        IMAGE                    COMMAND                  CREATED             STATUS              PORTS                              NAMES
83458596d7a6        rancher/server:v1.6.14   "/usr/bin/entry /u..."   3 minutes ago       Up 3 minutes        3306/tcp, 0.0.0.0:8880->8080/tcp   rancher_server

root@ons-auto-201803181110z:~# tail -f /var/log/azure/custom-script/handler.log
time=2018-03-18T22:51:59Z version=v2.0.6/git@1008306-clean operation=enable seq=0 file=0 event="download complete" output=/var/lib/waagent/custom-script/download/0
time=2018-03-18T22:51:59Z version=v2.0.6/git@1008306-clean operation=enable seq=0 event="executing command" output=/var/lib/waagent/custom-script/download/0
time=2018-03-18T22:51:59Z version=v2.0.6/git@1008306-clean operation=enable seq=0 event="executing public commandToExecute" output=/var/lib/waagent/custom-script/download/0
root@ons-auto-201803181110z:~# docker ps
CONTAINER ID        IMAGE                    COMMAND                  CREATED             STATUS              PORTS                              NAMES
539733f24c01        rancher/agent:v1.2.9     "/run.sh run"            13 seconds ago      Up 13 seconds                                          rancher-agent
83458596d7a6        rancher/server:v1.6.14   "/usr/bin/entry /u..."   5 minutes ago       Up 5 minutes        3306/tcp, 0.0.0.0:8880->8080/tcp   rancher_server
root@ons-auto-201803181110z:~# ls -la /var/lib/waagent/custom-script/download/0/
total 31616
-rw-r--r-- 1 root   root   16325186 Aug 31  2017 helm-v2.6.1-linux-amd64.tar.gz
-rw-r--r-- 1 root   root          4 Mar 18 22:55 kube_env_id.json
drwxrwxr-x 2 ubuntu ubuntu     4096 Mar 18 22:53 linux-amd64
-r-x------ 1 root   root       2822 Mar 18 22:51 oom_entrypoint.sh
-rwxrwxrwx 1 root   root       7288 Mar 18 22:52 oom_rancher_setup.sh
-rwxr-xr-x 1 root   root   12213376 Mar 18 22:53 rancher
-rw-r--r-- 1 root   root    3736787 Dec 20 19:41 rancher-linux-amd64-v0.6.7.tar.gz
drwxr-xr-x 2 root   root       4096 Dec 20 19:39 rancher-v0.6.7

testing via http://jenkins.onap.cloud/job/oom_azure_deployment/

Need the ip address and not the domain name - via linked template

https://docs.microsoft.com/en-ca/azure/azure-resource-manager/resource-group-linked-templates#return-state-from-a-template

or

https://docs.microsoft.com/en-us/azure/templates/microsoft.network/publicipaddresses

https://github.com/Azure/azure-quickstart-templates/issues/583

Arm templates cannot specify a static ip - without a private subnet

reference(variables('publicIPAddressName')).ipAddress

for

reference(variables('nicName')).ipConfigurations[0].properties.privateIPAddress

Using the hostname instead of the private/public ip works (verify /etc/hosts though)

obrienbiometrics:oom michaelobrien$ ssh ubuntu@13.90.207.60
ubuntu@ons-auto-201803181110z:~$ sudo su -
root@ons-auto-201803181110z:/var/lib/waagent/custom-script/download/0# cat stdout
INFO: Running Agent Registration Process, CATTLE_URL=http://ons-auto-201803181110z:8880/v1
INFO: Attempting to connect to: http://ons-auto-201803181110z:8880/v1
INFO: http://ons-auto-201803181110z:8880/v1 is accessible
INFO: Inspecting host capabilities
INFO: Boot2Docker: false
INFO: Host writable: true
INFO: Token: xxxxxxxx
INFO: Running registration
INFO: Printing Environment
INFO: ENV: CATTLE_ACCESS_KEY=9B0FA1695A3E3CFD07DB
INFO: ENV: CATTLE_HOME=/var/lib/cattle
INFO: ENV: CATTLE_REGISTRATION_ACCESS_KEY=registrationToken
INFO: ENV: CATTLE_REGISTRATION_SECRET_KEY=xxxxxxx
INFO: ENV: CATTLE_SECRET_KEY=xxxxxxx
INFO: ENV: CATTLE_URL=http://ons-auto-201803181110z:8880/v1
INFO: ENV: DETECTED_CATTLE_AGENT_IP=172.17.0.1
INFO: ENV: RANCHER_AGENT_IMAGE=rancher/agent:v1.2.9
INFO: Launched Rancher Agent: b44bd62fd21c961f32f642f7c3b24438fc4129eabbd1f91e1cf58b0ed30b5876
waiting 7 min for host registration to finish
1 more min
KUBECTL_TOKEN base64 encoded: QmFzaWMgUWpBNE5EWkdRVE5HTmpKRVFVRkZOa1k1UlRNNlMzZzBWRTVUZVc5VGEycFhaVE5HTm5rNWNGWlhVRWg0TVVkV1ZEbDVTRmQxUkc1d2MwWTJRZz09
run the following if you installed a higher kubectl version than the server
helm init --upgrade
Verify all pods up on the kubernetes system - will return localhost:8080 until a host is added
kubectl get pods --all-namespaces
NAMESPACE     NAME                                   READY     STATUS    RESTARTS   AGE
kube-system   heapster-76b8cd7b5-v5jrd               1/1       Running   0          5m
kube-system   kube-dns-5d7b4487c9-9bwk5              3/3       Running   0          5m
kube-system   kubernetes-dashboard-f9577fffd-cpwv7   1/1       Running   0          5m
kube-system   monitoring-grafana-997796fcf-s4sjm     1/1       Running   0          5m
kube-system   monitoring-influxdb-56fdcd96b-2mn6r    1/1       Running   0          5m
kube-system   tiller-deploy-cc96d4f6b-fll4t          1/1       Running   0          5m

20180318: Create VM image without destroying running VM

In AWS we can select the "no reboot" option and create an image from a running VM as-is with no effect on the running system.

Having issues with the Azure image creator - it is looking for the ubuntu pw even though I only use key based access

20180319: New Relic Monitoring
20180319: document devops flow

aka: travellers guide

20180319: Document Virtual Network Topology
20180429: Helm repo n/a after reboot - rerun helm serve

If you run into issues doing a make all - your helm server is not running

# rerun
helm serve &
helm repo add local http://127.0.0.1:8879


Training

(from Microsoft - thank you)

General Azure Documentation

Azure Site http://azure.microsoft.com

Azure Documentation Site https://docs.microsoft.com/en-us/azure/

Azure Training Courses https://azure.microsoft.com/en-us/training/free-online-courses/

Azure Portal http://portal.azure.com

Developer Documentation

Azure AD Authentication Libraries https://docs.microsoft.com/en-us/azure/active-directory/develop/active-directory-authentication-libraries

Java Overview on Azure https://azure.microsoft.com/en-us/develop/java/

Java Docs for Azure https://docs.microsoft.com/en-us/java/azure/

Java SDK on GitHub https://github.com/Azure/azure-sdk-for-java

Python Overview on Azure https://azure.microsoft.com/en-us/develop/python/

Python Docs for Azure https://docs.microsoft.com/en-us/python/azure/

Python SDK on GitHub https://github.com/Azure/azure-sdk-for-python

REST Api and CLI Documentation

REST API Documentation https://docs.microsoft.com/en-us/rest/api/

CLI Documentation https://docs.microsoft.com/en-us/cli/azure/index

Other Documentation

Using Automation for VM shutdown & startup https://docs.microsoft.com/en-us/azure/automation/automation-solution-vm-management

Azure Resource Manager (ARM) QuickStart Templates https://github.com/Azure/azure-quickstart-templates

Known Forks

The code in this github repo has 2 month old copies of cd.sh and oom_rancher_install.sh 

https://github.com/taranki/onap-azure

Use the official ONAP code in review under 

https://jira.onap.org/browse/OOM-710

https://jira.onap.org/browse/OOM-715

https://gerrit.onap.org/r/32019

https://jira.onap.org/browse/OOM-716

https://gerrit.onap.org/r/32653

The original seed source from 2017 below is deprecated -  use onap links above

https://github.com/obrienlabs/onap-root

https://azure.microsoft.com/en-us/services/container-service/

https://docs.microsoft.com/en-us/azure/templates/microsoft.compute/virtualmachines

https://docs.microsoft.com/en-us/azure/container-service/kubernetes/container-service-kubernetes-helm

https://kubernetes.io/docs/concepts/containers/images/#using-azure-container-registry-acr

https://azure.microsoft.com/en-us/features/storage-explorer/

https://docs.microsoft.com/en-ca/azure/virtual-machines/linux/capture-image







AKS

Google GCE

Account Provider: Michael O'Brien of Amdocs

GCE

  • No labels