VID-17 - Getting issue details... STATUS

Amazon AWS is currently hosting our RI for ONAP Continuous Deployment - this is a joint Proof Of Concept between Amazon and ONAP.

Auto Continuous Deployment via Jenkins and Kibana

Requirements

AWS CLI Installation

Install the AWS CLI on the bastion VM

https://docs.aws.amazon.com/cli/latest/userguide/cli-install-macos.html

OSX

obrien:obrienlabs amdocs$ pip --version
pip 9.0.1 from /Library/Python/2.7/site-packages/pip-9.0.1-py2.7.egg (python 2.7)
obrien:obrienlabs amdocs$ curl -O https://bootstrap.pypa.io/get-pip.py
obrien:obrienlabs amdocs$ python3 get-pip.py --user
Requirement already up-to-date: pip in /Library/Frameworks/Python.framework/Versions/3.6/lib/python3.6/site-packages
obrien:obrienlabs amdocs$ pip3 install awscli --upgrade --user
Successfully installed awscli-1.14.41 botocore-1.8.45 pyasn1-0.4.2 s3transfer-0.1.13

Ubuntu

obrien:obrienlabs amdocs$ ssh ubuntu@dev.onap.info
$ sudo apt install python-pip
$ pip install awscli --upgrade --user
$ aws --version
aws-cli/1.14.41 Python/2.7.12 Linux/4.4.0-1041-aws botocore/1.8.45

Windows Powershell

Configure Access Keys for your Account

$aws configure
AWS Access Key ID [None]: AK....Q
AWS Secret Access Key [None]: Dl....l
Default region name [None]: us-east-1
Default output format [None]: json
$aws ec2 describe-regions --output table
||  ec2.ca-central-1.amazonaws.com   |  ca-central-1    ||
....

Option 0: Deploy OOM Kubernetes to a spot VM

Peak Performance Metrics

We hit a peak of 44 cores during startup, with an external network peak of 1.2Gbps (throttled nexus servers at ONAP), a peak SSD write rate of 4Gbps and 55G ram on a 64 vCore/256G VM on AWS Spot.

Kubernetes Installation via CLI

Allocate an EIP static public IP (one-time)

https://docs.aws.amazon.com/cli/latest/reference/ec2/allocate-address.html

$aws ec2 allocate-address
{    "PublicIp": "35.172..",     "Domain": "vpc",     "AllocationId": "eipalloc-2f743..."}

Create a Route53 Record Set - Type A (one-time)

$ cat route53-a-record-change-set.json 
{"Comment": "comment","Changes": [
    { "Action": "CREATE",
      "ResourceRecordSet": {
        "Name": "amazon.onap.cloud",
        "Type": "A", "TTL": 300,
        "ResourceRecords": [
          { "Value": "35.172.36.." }]}}]}
$ aws route53 change-resource-record-sets --hosted-zone-id Z...7 --change-batch file://route53-a-record-change-set.json
{    "ChangeInfo": {        "Status": "PENDING",         "Comment": "comment", 
       "SubmittedAt": "2018-02-17T15:02:46.512Z",         "Id": "/change/C2QUNYTDVF453x"    }}


$ dig amazon.onap.cloud
; <<>> DiG 9.9.7-P3 <<>> amazon.onap.cloud
amazon.onap.cloud.	300	IN	A	35.172.36..
onap.cloud.		172800	IN	NS	ns-1392.awsdns-46.org.

Request a spot EC2 Instance

# request the usually cheapest $0.13 spot 64G EBS instance at AWS
aws ec2 request-spot-instances --spot-price "0.25" --instance-count 1 --type "one-time" --launch-specification file://aws_ec2_spot_cli.json

# don't pass in the the following - it will be generated for the EBS volume
            "SnapshotId": "snap-0cfc17b071e696816"
launch specification json
{      "ImageId": "ami-c0c964ba",
      "InstanceType": "r4.2xlarge",
      "KeyName": "obrien_systems_aws_2015",
      "BlockDeviceMappings": [
        {"DeviceName": "/dev/sda1",
          "Ebs": {
            "DeleteOnTermination": true,
            "VolumeType": "gp2",
            "VolumeSize": 120
          }}],
      "SecurityGroupIds": [ "sg-322c4842" ]}
# results
{    "SpotInstanceRequests": [{   "Status": {
                "Message": "Your Spot request has been submitted for review, and is pending evaluation.", 
                "Code": "pending-evaluation", 

Get EC2 instanceId after creation

aws ec2 describe-spot-instance-requests  --spot-instance-request-id sir-1tyr5etg
            "InstanceId": "i-02a653592cb748e2x",

Associate EIP with EC2 Instance

Can be done separately as long as it is in the first 30 sec during initialization and before rancher starts on the instance.

$aws ec2 associate-address --instance-id i-02a653592cb748e2x --allocation-id eipalloc-375c1d0x
{    "AssociationId": "eipassoc-a4b5a29x"}

Reboot EC2 Instance to apply DNS change to Rancher in AMI

$aws ec2 reboot-instances --instance-ids i-02a653592cb748e2x

Clustered Deployment

EC2 Cluster Creation

EFS share for shared NFS

"From the NFS wizard"

Setting up your EC2 instance

1. Using the Amazon EC2 console, associate your EC2 instance with a VPC security group that enables access to your mount target. For example, if you assigned the "default" security group to your mount target, you should assign the "default" security group to your EC2 instance. Learn more
2. Open an SSH client and connect to your EC2 instance. (Find out how to connect)
3. 
   If you're not using the EFS mount helper, install the NFS client on your EC2 instance:
   
   • On an Ubuntu instance:
     sudo apt-get install nfs-common

Mounting your file system

1. Open an SSH client and connect to your EC2 instance. (Find out how to connect)
2. Create a new directory on your EC2 instance, such as "efs".
   • sudo mkdir efs
3. Mount your file system. If you require encryption of data in transit, use the EFS mount helper and the TLS mount option. Mounting considerations
   • Using the EFS mount helper:
     sudo mount -t efs fs-43b2763a:/ efs
   • Using the EFS mount helper and encryption of data in transit:
     sudo mount -t efs -o tls fs-43b2763a:/ efs
   • Using the NFS client:
     sudo mount -t nfs4 -o nfsvers=4.1,rsize=1048576,wsize=1048576,hard,timeo=600,retrans=2 fs-43b2763a.efs.us-east-2.amazonaws.com:/ efs

If you are unable to connect, see our troubleshooting documentation.

https://docs.aws.amazon.com/efs/latest/ug/mounting-fs.html

ubuntu@ip-10-0-0-66:~$ sudo apt-get install nfs-common
ubuntu@ip-10-0-0-66:~$ cd /
ubuntu@ip-10-0-0-66:~$ sudo mkdir /dockerdata-nfs
root@ip-10-0-0-19:/# sudo mount -t nfs4 -o nfsvers=4.1,rsize=1048576,wsize=1048576,hard,timeo=600,retrans=2 fs-43b2763a.efs.us-east-2.amazonaws.com:/ /dockerdata-nfs
# write something on one vm - and verify it shows on another
ubuntu@ip-10-0-0-8:~$ ls /dockerdata-nfs/
test.sh

Kubernetes Installation via CloudFormation

ONAP Installation

SSH and upload OOM

oom_rancher_install.sh is in  OOM-715 - Getting issue details... STATUS  under https://gerrit.onap.org/r/#/c/32019/

Run OOM

see  OOM-710 - Getting issue details... STATUS

cd.sh in  OOM-716 - Getting issue details... STATUS  under https://gerrit.onap.org/r/#/c/32653/

Scenario: installing Rancher on clean Ubuntu 16.04 64g VM (single collocated server/host) and the master branch of onap via OOM deployment (2 scripts)

1 hour video of automated installation on an AWS EC2 spot instance

oom_rancher_cd_full_auto_install_on_aws_ec2_20180227.mp4

Run Healthcheck

Run Automated Robot parts of vFirewall VNF

Report Results

Stop Spot Instance

$ aws ec2 terminate-instances --instance-ids i-0040425ac8c0d8f6x
{    "TerminatingInstances": [        {
            "InstanceId": "i-0040425ac8c0d8f63", 
            "CurrentState": {
                "Code": 32, 
                "Name": "shutting-down"           }, 
            "PreviousState": {
                "Code": 16, 
                "Name": "running"
            }        }    ]}

Verify Instance stopped

Video on Installing and Running the ONAP Demos#ONAPDeploymentVideos

WE can run ONAP on an AWS EC2 instance for $0.17/hour as opposed to Rackspace at $1.12/hour for a 64G Ubuntu host VM.

I have created an AMI on Amazon AWS under the following ID that has a reference 20170825 tag of ONAP 1.0 running on top of Rancher

ami-b8f3f3c3 : onap-oom-k8s-10

https://console.aws.amazon.com/ec2/v2/home?region=us-east-1#Images:visibility=public-images;search=ami-b8f3f3c3;sort=name

EIP 34.233.240.214 maps to http://dev.onap.info:8880/env/1a7/infra/hosts

A D2.2xlarge with 61G ram on the spot market https://console.aws.amazon.com/ec2sp/v1/spot/launch-wizard?region=us-east-1 at $0.16/hour for all of ONAP

It may take up to 3-8 min for kubernetes pods to initialize as long as you preload the docker images  OOM-328 - Getting issue details... STATUS

Workaround for the disk space error - even though we are running with a 1.9 TB NVMe SSD

https://github.com/kubernetes/kubernetes/issues/48703

Use a flavor that uses EBS like M4.4xLarge which is OK

Use a flavor that uses EBS like M4.4xLarge which is OK - except for AAI right now

Expected Monthly Billing

r4.2xlarge is the smallest and most cost effective 64g min instance to use for full ONAP deployment - it requires EBS stores.  This is assuming 1 instance up at all times and a couple ad-hoc instances up a couple hours for testing/experimentation.

Option 1: Migrating Heat to CloudFormation

Resource Correspondence

Using the CloudFormationDesigner

https://console.aws.amazon.com/cloudformation/designer/home?region=us-east-1#

Decoupling and Abstracting Southbound Orchestration via Plugins

Part of getting another infrastructure provider like AWS to work with ONAP will be in identifying and decoupling southbound logic from any particular cloud provider using an extensible plugin architecture on the SBI interface.

see Multi VIM/Cloud (5/11/17), VID project (5/17/17), Service Orchestrator (5/14/17), ONAP Operations Manager (5/10/17), ONAP Operations Manager / ONAP on Containers

Design Issues

DI 1: Refactor nested orchestration in DCAE

Replace the DCAE Controller

DI 2: Elastic IP allocation

DI 3: Investigate Cloudify plugin for AWS

Cloudify is Tosca based - https://github.com/cloudify-cosmo/cloudify-aws-plugin

OOM Automated Installation Videos

Latest 20171206 AWS install from clean Ubuntu 16.04 VM using rancher setup script below and the cd.sh script to bring up OOM - after the 20 min prepull of dockers - OOM comes up fully with only the known aaf issue 84 of 85 containers - all healthcheck passes except DCAE at 29/30, portal tested and an AAI cloud-region put

Links

Waiting for the EC2 C5 instance types under the C620 chipset to arrive at AWS so we can experiment under EC2 Spot - http://technewshunter.com/cpus/intel-launches-xeon-w-cpus-for-workstations-skylake-sp-ecc-for-lga2066-41771/ https://aws.amazon.com/about-aws/whats-new/2016/11/coming-soon-amazon-ec2-c5-instances-the-next-generation-of-compute-optimized-instances/

http://docs.aws.amazon.com/cli/latest/userguide/cli-install-macos.html

use

curl "https://s3.amazonaws.com/aws-cli/awscli-bundle.zip" -o "awscli-bundle.zip"
unzip awscli-bundle.zip
sudo ./awscli-bundle/install -i /usr/local/aws -b /usr/local/bin/aws
aws --version
aws-cli/1.11.170 Python/2.7.13 Darwin/16.7.0 botocore/1.7.28