...
draw.io Diagram | ||||||||||||||||||
---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
|
Multi-cluster Orchestrator micro-service components:
Project Module:
Definition Module:
Profile Module:
Day 2 Config Module:
Intent Group Module:
Generic Placement Module:
Instantiation Module:
Projects allow multi-tenancy in the application from a user perspective.
Composite App Module: K8splugin artifacts start in the form of a composite application representing an application. A composite application is a collection of Helm Charts. Each Helm chart represents an application inside the composite application. The orchestrator decides the placement of each application and their associated Kubernetes resources after considering inputs from the placement controllers like the HPA controller, action controllers like traffic management controller and other controllers in future.
Profile Module: Once a composite application is uploaded, then some profiles can be created to customize that Helm charts and instantiate it in Kubernetes.
Deployment Intent Group Module: Intent Group API is used to create Intent Groups that can then will be used at instantiation time. Some intents that can be part of an Intent Group are Generic Placement Intent, HPA Intent, and traffic controller Intent.
Generic Placement Module: Generic Placement Intent API is used to create generic placement intents for applications within a composite application. Intent created using this API will be added to an Intent Group to use during instantiation.
Instantiation Module: Instantiates an application based on the Intent group. During instantiation, placement controllers and traffic controllers will be called to help with placement decisions and creating configurations for applications.
Cluster Registration Module: Registering a cluster with cluster-specific information to be used by the Resource sync module.
Day 2 Config Module: Day 2 Configurations for applications are applied using K8S kinds (typically CRDs) implemented by application-specific operators. Cluster Registration Module
Resource sync micro-service components:
Resource Sync Module: Responsible for installed Kubernetes resources on the identified clusters
Status Update Module: Maintains the state of all the resources deployed for a composite application and its applications.
Internal Design details
Guidelines that need to
...
keep in mind
- Support for metrics that can be retrieved by Prometheus
- Support for Jaeger distributed tracing by including opentracing operating libraries around HTTP calls.
- Support for logging that is understood by fluentd
- Mutual exclusion of database operations (keeping internal modules accessing database records simultaneously and also by replication entities of the scheduler micro-service).
- Resilience - ensure that the information returned by controllers is not lost as the synchronization of resources to remote edge clouds can take hours or even days when the edge is not up and running and possibility of restart of scheduler micro service in the meantime.
- Concurrency - Support multiple operations at a time and even synchronizing resources in various edge clouds in parallel.
- Performance - Avoiding file system operations as much as possible.
RESTful North API (with examples)
Onboarding
Profiles
Deployment Intent
Interface API to Controllers (with examples)
Replication controller interface
Placement controller interface for internal controllers
Placement controller interface for external controllers
Resource modifier controller interface for internal controllers
Resource modifier controller interface for external controllers
Action controller interface for internal controllers
Action controller interface for external controllers
Restful South API (to Resource installer service, with examples)
Modules (Description, internal structures etc..)
....
Sequence flows
Composite Application Structure
Composite application resource represents a collection of applications and each application is represented by helm charts. Each Helm chart is wrapped in tar.gz format.
Instantiation Creation Flow
- Get the clusters that form the logical cloud to schedule the application. Validate the application, intent group, and generic placement intent.
- Create etcd Context for the application
- Process the Generic placement context and populate the etcd context with the edge list.
- Generate Kubernetes resources from the Helm charts by using Helm templating and from macros. Copy resources in etcd store based on the context for the application.
- Placement stage: Call all the placement controllers including the HPA controller in order. Wait for one controller to complete before going to the next controller.
- HPA controller: Kubernetes resource manipulation. Add additional information to the deployments. In the case where the generic intent calls for anyof clusters, the HPA controller can make a selection for one or more clusters meeting the HPA requirements for the application.
- Action Controller Stage: Call all the action controllers including Traffic Controller in order. Wait for one controller to complete before going to the next controller.
- Traffic Controller: Creates more resources for traffic routing and management between clusters.
- After all the controllers are done processing if there are still multiple clusters that meet the requirements, the orchestrator will pick up the first cluster to schedule the resource.
- Send notification to the resource sync to schedule the resources on the selected clusters. Once the resources are scheduled to be created the resource scheduler sends back notification to the orchestrator which in turn returns HTTP response as a success for the instantiation API.
- Resource Sync will periodically check the status of the resources and update the database with the status of the resource.
draw.io Diagram | ||||||||||||||||||
---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
|
Instantiation Deletion Flow
- Get etcd Context for the application
- Send notification to the resource sync to delete the resources on the selected clusters. Once the resources are scheduled to be deleted the resource scheduler sends back notification to the orchestrator which in turn returns http response as a success for the instantiation delete API.
- Resource Sync will periodically check the status of the resources and update the database with the status of the resource.
Instantiation Update Flow
This flow is for the case no changes are made to the Helm charts of the application. Changes to the intents are realized here.
- Get the clusters that form the logical cloud to schedule the composite application. Validate the application, intent group, and generic placement intent.
- Create etcd Context for the application
- Process the Generic placement context and populate the etcd context with the edge list.
- Generate Kubernetes resources from the Helm charts by using Helm templating and from macros. Copy resources in etcd store based on the context for the application.
- Placement stage: Call all the placement controllers including the HPA controller in order. Wait for one controller to complete before going to the next controller.
- HPA controller: Kubernetes resource manipulation. Add additional information to the deployments. In the case where the generic intent calls for anyof clusters, the HPA controller can make a selection for one or more clusters meeting the HPA requirements for the application.
- Action Controller Stage: Call all the action controllers including Traffic Controller in order. Wait for one controller to complete before going to the next controller.
- Traffic Controller: Creates more resources for traffic routing and management between clusters.
- After all the controllers are done processing if there are still multiple clusters that meet the requirements, the orchestrator will pick up the first cluster to schedule the resource.
- Diff the original application resources and the edges that they are created on with this application context. Generate a list of resources to be changed.
- Send notification to the resource sync to create/update/delete the resources on the selected clusters. Once the resources are scheduled to be updated the resource scheduler sends back notification to the orchestrator which in turn returns HTTP response as a success for the instantiation API.
- Resource Sync will periodically check the status of the resources and update the database with the status of the resource.
Orchestrator High-Level Design
draw.io Diagram | ||||||||||||||||||||
---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
|
Infrastructure Library
- MongoDB - Create, Read, Write
- Etcd - Create, Read, Write
Module Access Library
Project API- CreateProject, DeleteProject, GetProject
Composite App API - CreateCompositeApp, GetCompositeApp, DeleteCompositeApp, AddApptoCompositeApp, DeleteAppFromCompositeApp
Composite Profile API - CreateCompositeProfile, GetCompositeProfile, DeleteCompositeProfile, AddProfiletoCompositeProfile, DeleteProfileFromCompositeProfile
Generic Placement API - CreateGenericPlacement, DeleteGenericPlacement, GetGenericPlacement, AddGenericIntent, DeleteGenericIntent, GetIntent
Deployment Intent API - CreateDeploymentIntent, DeleteDeploymentIntent, GetDeploymentIntent, AddIntent, DeleteIntent,
Status API - GetStatus(deploymet)
Runtime Resource Context Library
CreateContext(project, name) returns contextId
AddApp(contextId, name) return error
AddCluster(contextId, app-name, cluster-name)
AddResource(contextId, app-name, cluster-name, resourceName)
UpdateResourceField(contextId, app-name, cluster-name, resourceName, fieldName, Value)
Test cases
Composite App with EdgeX, Firewall with Intents
...