Versions Compared

Key

  • This line was added.
  • This line was removed.
  • Formatting was changed.
Comment: AWS costing for CD + dev

...

The audience is approving I.T. and Finance personnel.

Up to date content is on Cloud Native Deployment

Public Cloud Deployment

Executive Summary

...

Marketplace/services Utilization - no use of marketplace services is required in either AWS or Azure.  The ONAP deployment brings its own open source software stack to a bare Ubuntu VM.  At this time Kubernetes as a Service is also not required and all resiliency/scaling/replication/load-balancing/federation behaviour for HA and Geo-Redundancy is handled natively by the Kubernetes framework.a Service is also not required and all resiliency/scaling/replication/load-balancing/federation behaviour for HA and Geo-Redundancy is handled natively by the Kubernetes framework.

Beijing release

Corporate Allocation

The following table details an example of what is required to run a continuous delivery system around ONAP beijing and some developer profiles when working directly on a cloud provider.

Beijing requires the following to run: Note there is a 110 pod limit per VM - we currently deploy 175+ pods - hence 2+ VMs for full deployment

  • Full production/staging
    • Minimum 2 VMs, 12vCores total, 96G total ram, 100G HD per VM
    • Recommended 3-9 VMs, 24-64 vCores, 128+ total ram, 160G HD per VM + 1 8G/100G kubernetes master VM
  • Developer:
    • Minimum 1 VM at 4+ vCores, 16-64G ram, 100G HD - collocated kubernetes master, host and jumpbox (deploys a subset of ONAP)
    • Recommended 3 VMs 1x8G kubernetes master and 2 x 64G hosts

Amazon AWS

Total US $/month

(0*)

EBS

cost (3*)

spot

vm/hr

Artifact#vCoresRamHDFlavorUseDescription
$66$0.06$0.032
1215
R4.largeDevOpsJump box - cloudformation
$66$0.06$0.032
1215
R4.largeDevOpsJenkins server
$66$0.06$0.032
1215
R4.largeDevOpsKibana (ELK) server
$238 x 4 = $950$0.06$0.27
416122
R4.4xlargeDevOps

production CD cluster

1*

$66$0.06$0.032
1215
R4.largeDevOpsproduction kubernetes master
$138 x 4 = $552$0.06$0.13
4861
R4.2xlargeDevOps

staging cluster

2*

$66$0.06$0.032
1215
R4.large
staging kubernetes master
$89 x 4 = $356$0.06$0.063
4431
R4.xlargeDevOpslong duration cluster
$66$0.06$0.032
1215
R4.large
long duration cluster k8s master
$138 x 2 = $276$0.06$0.13
2861
R4.2xlargeDevdeveloper cluster
$66$0.06$0.032
1215
R4.large
developer cluster kubernetes master
$138$0.06$0.13
1861
R4.2xlargeDevdeveloper onap subset collocated VM
$0$0$0






Route53 DNS/EIP

$2/month per unused EIP

$0

$0$0






VPC, NG, SG, IAM,

(network costs)

Total cost / month - SPOT (max prod cluster - max dev cluster x 1)

$66 + $66 + $66 + $950 + $66 + $356 + $276 + $66 = $1912/month = $23k/year

(4*)

Total cost / month - SPOT (medium prod cluster - min dev cluster x 1)

$66 + $66 + $66 + $552 + $66 + $356 + $138 = $1310/month = $16k/year

(4*)

Total cost / month - reserved

Notes:

0 - assumes us-east-1 region (us-east-2 region is cleaper (ohio) but the spot market there is more unstable) - if you use ohio - cut costs by about 40% for spot - ie: r4.2xlarg is 0.13 but 0.07 in ohio)

1 - ONAP is CPU bound - it will peak at over 55 vCores during startup - vCPUs over 8 are required for a stable deployment - we could use C4/C5 compute optimized images - more expensive and get timed out of spot more often - it is the same price and more stable to run an R4.2x/4x instance with twice the ram but the same vCores.

2 - a cluster running the 8 core R4.2xlarge vms will be OK but will be CPU throttled during startup and any during any container under test or rogue container episode.

3 - EBS cost is usually around 45% of the ec2 cost for an R4.large for the average 100G HD

4 - some of these costs are reduced if we use AMI's and cloudformation/cli templates to raise/lower systems (ie: on the weekend off, for CD systems raise for 2 hours test and terminate for 2 hours on 4 hour cycles)

Amsterdam release

Deployment Use Cases

There are several deployment scenarios that include VMs and containers both for ONAP itself and the VNFs that are managed.  Container deployments are further segregated by managed kubernetes and Kubernetes as a Service types.  We assume that all ONAP components run as Docker containers whether they are managed per VM (HEAT) or managed in a Kubernetes cluster namespace (KaaS or managed).

...

TypeONAP(VMs or Containers)VNF (VMs or Containers)

Kubernetes containers on VMsVM Rackspace/Openstack









Deployment Example: Full Kubernetes on a VM cluster

This is the RI (Reference implementation) of ONAP Beijing release - it consists of all the ONAP containers 90+ deployed to a particular (dev/stg/prod) namespace ecosystem running on Kubernetes.  The Kubernetes implementation is running under any management layer - here Rancher and not on a KaaS.  The Kubernetes cluster undercloud can run on 1 or more VMs - in this example we colocate the server and single host on a single VM which currently fits in 55G.

Note: DCAEGEN2 is currently being fully containerized and should arrive as a native Kubernetes set of containers by Beijing R2 release. Currently DCAE runs in a 64G VM specifically on a configured Openstack system.  There is a reverse proxy mechanism that joins DCAE to the rest of ONAP running in Kubernetes already.  DCAE is required for VNF closed loop operations - but not for VNF orchestration.  When DCAE is fully refactored for Kubernetes then the memory requirement will jump over the 64G baseline and push it to 96 to 128G depending on the size of the CDAP Hadoop cluster running which is 3-7 containers.

Security Profile

ONAP will require certain ports open by CIDR to several static domain names in order to deploy defined in a security group.  At runtime the list is reduced.

...

Still working on a list of ports but we should not need any of these exposed if we use a bastion/jumpbox + nat combo inside the network.

Known Security Vulnerabilities

https://medium.com/handy-tech/analysis-of-a-kubernetes-hack-backdooring-through-kubelet-823be5c3d67c

https://github.com/kubernetes/kubernetes/pull/59666 fixed in Kubernetes 1.10

ONAP Port Profile

ONAP on deployment will require the following incoming and outgoing ports.  Note: within ONAP rest calls between components will be handled inside the Kubernetes namespace by the DNS server running as part of K8S.

portprotocolincoming/outgoingapplicationsourcedestinationNotes
22ssh
sshdeveloper vmhost
443

tillerclienthost
8880http
rancherclienthost
9090http
kubernetes
host
10001https
nexus3
nexus3.onap.org
10003https
nexus3
nexus3.onap.org

https
nexus
nexus.onap.org

https

ssh


git
git.onap.org
30200-30399http/https
REST apideveloper vmhost
5005tcp
java debug portdeveloper vmhost


Lockdown ports



8080

outgoing





1025010249-10255
in/out


Lock these down via VPC or a source CIDR that equals only the server/client IP list

https://medium.com/handy-tech/analysis-of-a-kubernetes-hack-backdooring-through-kubelet-823be5c3d67c


Azure

AWS




Software Profile

Rancher 1.6.14

Helm 2.8.0

Kubernetes 1.8.6

...

The rest of the software versions are specific to the 90+ docker containers running for example MariaDB, Jetty... etc.  All of the software is open source and encapsulated in the containers themselves.  The containers implement a REST based microservices architecture.

Hardware Profile

ONAP on Kubernetes#HardwareRequirements

Microsoft Azure

Monthly Cost

Cost $USArtifactDetails

$365/m at $0.65 (CAN)/h

reduce by 42% for 1 year reserved instances = $212/month

VM running Ubuntu 16.04

E8s V3

(64g/8vCores)

128G SSD
30G SSD

in CA central DC


$0 TBDno extra volume (only 16G VMs have 30G disks)


IP

$0image snapshot

$0Cloud Services


Total

$212/m


on-demand

Reserved


Amazon AWS

VM: Minimum EC2 instance of size 61G (ideally 128G) with a 120+GB EBS volume (ideally 1024GB) and at least 8 vCores (ideally 64 vCores), network 1Gbps (ideally 10Gbps).

...

Example of 2 or 3 network VPC peering setup http://files.meetup.com/18216364/aws_vpc_beanstalk_20150224_post.pdf


Cost

There is an R4.2xlarge instance type that has been the lowest cost instance that can run all of ONAP (except DCAE) and has been demonstrated since Amsterdam on the CD system.  The cost on the spot market is between 72 to 89% off the reserved cost at around $0.14/hour on us-east(N. Virgina DC) and 0.07/hour in us-east(Ohio DC)

...

Cost $USArtifactDetails
$52/m at $0.07/h

EC2 spot VM running Ubuntu 16.04

R4.2xlarge

(64g/8vCores)

in the ohio DC


$12.0/m at $0.1/m/GbEBS volume120Gb
$2.0/mElastic EIP


AMI image snapshot

$0Cloud Services


Total

$64/m


Artifacts

Amazon

Spot template

security group

...

EBS Volume

Auto scaling group

Amazon Cloudformation Template
Code Block
SPOT only
{
  "IamFleetRole": "arn:aws:iam::453279094200:role/aws-ec2-spot-fleet-tagging-role",
  "AllocationStrategy": "lowestPrice",
  "TargetCapacity": 1,
  "SpotPrice": "0.532",
  "ValidFrom": "2018-01-31T20:31:16Z",
  "ValidUntil": "2019-01-31T20:31:16Z",
  "TerminateInstancesWithExpiration": true,
  "LaunchSpecifications": [
    {
      "ImageId": "ami-aa2ea6d0",
      "InstanceType": "r4.2xlarge",
      "KeyName": "obrien_systems_aws_20141115",
      "SpotPrice": "0.532",
      "BlockDeviceMappings": [
        {
          "DeviceName": "/dev/sda1",
          "Ebs": {
            "DeleteOnTermination": true,
            "VolumeType": "gp2",
            "VolumeSize": 120,
            "SnapshotId": "snap-0dcc947e7c10bed94"
          }
        }
      ],
      "SecurityGroups": [
        {
          "GroupId": "sg-de2185a9"
        }
      ]
    }
  ],
  "Type": "request"
}


Microsoft

Azure Resource Manager Template

...