Placement Controllers


HPA Placement Controller

HPA Placement Controller is a stand-alone microservice that returns available cluster sites for placing applications that require specific hardware acceleration features.

HPA Placement Controller Design

HPA Placement Controller (HPC) is a gRPC Microservice written in Go Lang that stores data in mongo DB and etcd.


-- Store cluster features in unique sets of whats available in the cluster


Feature Retrieval Design 

HPA Placement Controller on prompt from scheduler or northbound API goes to a k8s cluster and retrieves feature labels

  1. requires auth/kubectl file for cluster access - some security implications
  2. allows use of placement controller independently of scheduler


HPC Registration

K8s Cluster Registration


HPA Instantiation Flow

HPA Placement Controller Northbound APIs

OpenAPI File: openapi.yaml

HPA Intent API

HPA Placement Intent

https://NODE_IP:30280/api/multicloud-k8s/v2/project/{project-name}/rb/{rb-name}/{rb-version}/hpa-placement-intent


Use HPA API expressed here Policy Specification and Retrieval for OOF as a base for hardware features.

POST
POST
URL: /v2/projects/{project-name}/composite-apps/{composite-app-name}/{version}/hpa-placement-intents
{
  "metadata": {
    "name": "Edge-1-intent",
    "description": "Edge Intent for Sec. and Load Balancing",
    "status": "active",
    "userdata1": {
      "external-proxy-ip": "10.7.100.4"
    },
    "userdata2": ""
  }
}

URL: /v2/projects/{project-name}/composite-apps/{composite-app-name}/{version}/hpa-placement-intents/{hpa-intent-name}/workload-placements/
{
  "spec": [
    {
      "application-name": "Edge-Security",
      "workload-resource": "Firewall",
      "platform-features": [
        {
          "platform-feature-name": "pci-0300_102b.present",
          "mandatory": "True",
          "score": 5,
          "platform-feature-attribute": [
            {
              "feature-attribute-key": "qatPresence",
              "feature-attribute-value": "True",
              "operator": "ALL",
              "unit": ""
            }
          ]
        },
        {
          "platform-feature-name": "memory-numa",
          "mandatory": "True",
          "score": 2,
          "platform-feature-attribute": [
            {
              "feature-attribute-key": "numaNodes",
              "feature-attribute-value": "2",
              "operator": "=",
              "unit": ""
            },
            {
              "feature-attribute-key": "numaNodes",
              "feature-attribute-value": "4",
              "operator": "=",
              "unit": ""
            }
          ]
        }
      ]
    }
  ]
}

RETURN STATUS: 201
GET, DELETE
POST
GET
URL: /v2/project/{project-name}/rb/{rb-name}/{rb-version}/hpa-placement-intent/{intent-name}

RETURN STATUS: 200
RETURN BODY: 
{
  "metadata": {
    "name": "Edge-1-intent",
    "description": "Edge Intent for Sec. and Load Balancing",
    "status": "active",
    "userdata1": {
      "external-proxy-ip": "10.7.100.4"
    },
    "userdata2": ""
  },
  "spec": [
    {
      "application-name": "Edge-Security",
      "workload-resource": "Firewall",
      "platform-features": [
        {
          "platform-feature-name": "pci-0300_102b.present",
          "mandatory": "True",
          "score": 5,
          "platform-feature-attribute": [
            {
              "feature-attribute-key": "qatPresence",
              "feature-attribute-value": "True",
              "operator": "ALL",
              "unit": ""
            }
          ]
        },
        {
          "platform-feature-name": "memory-numa",
          "mandatory": "True",
          "score": 2,
          "platform-feature-attribute": [
            {
              "feature-attribute-key": "numaNodes",
              "feature-attribute-value": "2",
              "operator": "=",
              "unit": ""
            },
            {
              "feature-attribute-key": "numaNodes",
              "feature-attribute-value": "4",
              "operator": "=",
              "unit": ""
            }
          ]
        }
      ]
    }
  ]
}


DELETE
URL: /v2/project/{project-name}/rb/{rb-name}/{rb-version}/hpa-placement-intent/{intent-name}

RETURN STATUS: 204

Sites API

https://NODE_IP:30280/api/multicloud-k8s/v2/placement/hpa/update-sites


Note: if this were to be uncoupled with scheduler it would need to have body for POST containing the following info and it would need to return app-context-name in return body:

  • candidate sites, apps, sub-apps
  • hpa-intent-name

     Could we create another api endpoint "update-sites-standalone" that contains above to uncouple?


POST
POST
URL: /v2/placement/hpa/update-sites
POST BODY:
{
   "intent-group":"intent-group-1",
   "app-context":"app-context-1"
}

RETURN STATUS: 201
RETURN BODY: 
{ 
  "app-context-updated": "true"
}

Feature API

https://NODE_IP:30280/api/multicloud-k8s/v2/placement/hpa/features

POST
POST
URL: /v2/placement/hpa/features
POST BODY:
{
  "cluster-name": "cluster1"
}
RETURN STATUS: 201
GET, DELETE
POST
GET
URL: /v2/placement/hpa/features/{cluster-name}

RETURN STATUS: 200
RETURN BODY: 
{
  "cluster-name": "cluster1",
  "hardware-features": [
    {
     "hardware-feature-name": "pci-0300_102b.present"
    },
    {
     "hardware-feature-name": "memory-numa"
    },
    {
     "hardware-feature-name": "network-sriov.configured"
    }      
 ]
}

DELETE
URL: /v2/placement/hpa/features/{cluster-name}

RETURN STATUS: 204

HPA Placement Controller gRPC APIs

see gRPC messages above.


HPA Placement Controller Targeted Hardware Features


Controllers gRPC Interface


Each controller will communicate with the Multi Cluster Application Scheduler through gRPC.


gRPC Code

gRPC patch: https://gerrit.onap.org/r/c/multicloud/k8s/+/100148

gRPC Messages

HPA Placement Controller

syntax = "proto3";

service placmentController {
	// Placement Controllers
    rpc UpdateSites(SitesRequest) returns (SitesResponse) {
    }

	// HPA Placement Controller
	rpc UpdateFeatures(UpdateFeaturesRequest) returns (UpdateFeaturesResponse) {
    }
}

message SitesRequest {
    string intent_group = 1;
	string app_context = 2;
}

message SitesResponse {
    bool app_context_updated = 1;
}


message UpdateFeaturesRequest {
	string cluster_name = 1;
}

message UpdateFeaturesResponse {
	bool features_stored = 1;
}

Traffic Controller

syntax = "proto3";

service trafficController {

	// Traffic Controller
	//rpc rpcMessage(TrafficRequest) returns (TrafficResponse) {
}

message TrafficRequest {
    string intent_group = 1;
	string app_context = 2;
}

message TrafficResponse {
    bool app_context_updated = 1;
}

Register Controller

syntax = "proto3";

service registerController {

	// Traffic Controller
	//rpc registerController(RegisterRequest) returns (RegisterResponse) {
}

message RegisterRequest {
    string controller_type = 1;
	string controller = 2;
}

message RegisterResponse {
    bool app_context_updated = 1;
}

gRPC Server


gRPC Client


gRPC Controller Registration

?

ETCD Client


Generic ETCD Client module should be created in the scheduler/orchestrator and imported and used in scheduler and controllers.


Backup

gRPC getSites using profile

service controller {
    rpc GetSites(SitesRequest) returns (SitesResponse) {
    }
}

message SitesRequest {
    string intent_name = 1;
	string rb_name = 2;
    string rb_version = 3;
    string profile_name = 4;
	repeated CandidateSites sites = 5;
}

message CandidateSite {
    string cluster_name = 1;
 
}

message SitesResponse {
    repeated Resource resources = 1;
}

message Resource {
	repeated Site available_sites = 1;

}

message Site {    
    string cluster_name = 1;
}

old 

syntax = "proto3";

service controller {
	// Placement Controllers
    rpc GetSites(SitesRequest) returns (SitesResponse) {
    }

	rpc StoreIntent(StoreIntentRequest) returns (StoreIntentResponse) {
	}

	rpc DeleteIntent(DeleteIntentRequest) returns (DeleteIntentResponse) {
	}

	rpc ModifyIntent(ModifyIntentRequest) returns (ModifyIntentResponse) {
	}

	// HPA Placement Controller
	rpc StoreFeatures(StoreFeaturesRequest) returns (StoreFeaturesResponse) {
    }

	// Traffic Controller
	// TBD
}

message SitesRequest {
    repeated Intent intents = 1;
	string rb_name = 2;
    string rb_version = 3;
    string profile_name = 4;
	repeated CandidateSites sites = 3;
}

message Intent {
    string intent_name = 1;
}

message CandidateSite {
    string cluster_name = 1;
}


message SitesResponse {
    repeated ResourceSite resource_sites = 1;
}

message ResourceSite {
	repeated Site available_sites = 1;

}

message Site {    
    string cluster_name = 1;
}


message StoreFeaturesRequest {
	string cluster_name = 1;
	repeated Feature features = 2;
}

message Feature {
	string feature = 1;
}

message StoreFeaturesResponse {
	int reply = 1;
}

message StoreIntentRequest {
	string intent_name = 1;
	repeated Intent intents = 2;
}

message Intent {
	string intent_body = 1;
}

message StoreIntentResponse {
	int reply = 1;
}

message DeleteIntentRequest {
	string intent_name = 1;
}

message DeleteIntentResponse {
	int reply = 1;
}

message ModifyIntentRequest {
	string intent_name = 1;
	repeated Intent intents = 2;
}

message ModifyIntentResponse {
	int reply = 1;
}

feature backup 

Feature API

https://NODE_IP:30280/api/multicloud-k8s/v2/placement/hpa/features

POST

POST
URL: /v2/placement/hpa/features
POST BODY:
{
  "cluster-name": "cluster1",
    "nodes": [
    {
      "node-name": "node1",
      "node-hardware-feature-set": [
        {
          "hardware-feature-name": "pci-0300_102b.present"
        },
        {
          "hardware-feature-name": "memory-numa"
        },
        {
          "hardware-feature-name": "network-sriov.configured"
        }      
      ]
    },
    {
      "node-name": "node2",
      "node-hardware-feature-set": [
        {
          "hardware-feature-name": "network-sriov.configured"
        }
      ]
    }
  ]
}
RETURN STATUS: 201

GET, DELETE

POST
GET
URL: /v2/placement/hpa/features/{cluster-name}

RETURN STATUS: 200
RETURN BODY: 
{
  "cluster-name": "cluster1",
    "nodes": [
    {
      "node-name": "node1",
      "node-hardware-feature-set": [
        {
          "hardware-feature-name": "pci-0300_102b.present"
        },
        {
          "hardware-feature-name": "memory-numa"
        },
        {
          "hardware-feature-name": "network-sriov.configured"
        }      
      ]
    },
    {
      "node-name": "node2",
      "node-hardware-feature-set": [
        {
          "hardware-feature-name": "network-sriov.configured"
        }
      ]
    }
  ]
}

DELETE
URL: /v2/placement/hpa/features/{cluster-name}

RETURN STATUS: 204
  • No labels