...
draw.io Diagram | ||||||||||||||||||
---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
|
Multi-cluster Orchestrator micro-service components:
Project Module: Projects allow multi-tenancy in the application from a user perspective.
Definition Composite App Module: K8splugin artifacts start in the form of Definitions a composite application representing an application. These are nothing but Helm Charts wrapped with some metadata about the application itself. The Definition is a tar file. If an application is made up of multiple applications then Definition will be made up multiple Helm charts representing each sub-A composite application is a collection of Helm Charts. Each Helm chart represents an application inside the composite application. The orchestrator decides the placement of each sub- application and their associated Kubernetes resources after considering inputs from the placement controllers like the HPA controller, action controllers like traffic management controller and other controllers in future.
Profile Module: Once the Definitions are created, we are ready to create some profiles so that we can customize that definition a composite application is uploaded, then some profiles can be created to customize that Helm charts and instantiate it in Kubernetes. Profile names are unique within a definition and version.
Day 2 Config Module: Day 2 Configurations for applications are applied using K8S kinds (typically CRDs) implemented by application-specific operators.
Deployment Intent Group Module: Intent Group API is used to create Intent Groups that can then will be used at instantiation time. Some intents that can be part of an Intent Group are Generic Placement Intent, HPA Intent, and traffic controller Intent.
Generic Placement Module: Generic Placement Intent API is used to create generic placement intents for sub-applications in an applications within a composite application. Intent created using this API will be added to an Intent Group to use during instantiation.
Instantiation Module: Instantiates an application based on the Intent group. During instantiation, placement controllers and traffic controllers will be called to help with placement decisions and creating configurations for sub- applications.
Cluster Registration Module: Registering a cluster with cluster-specific information to be used by the Resource sync module.
Day 2 Config Module: Day 2 Configurations for applications are applied using K8S kinds (typically CRDs) implemented by application-specific operators.
Resource sync micro-service components:
...
Status Update Module: Maintains the state of all the resources deployed for an a composite application and its sub- applications.
Internal Design details
...
- Support for metrics that can be retrieved by Prometheus
- Support for Jaeger distributed tracing by including opentracing operating libraries around HTTP calls.
- Support for logging that is understood by fluentd
- Mutual exclusion of database operations (keeping internal modules accessing database records simultaneously and also by replication entities of the scheduler micro-service).
- Resilience - ensure that the information returned by controllers is not lost as the synchronization of resources to remote edge clouds can take hours or even days when the edge is not up and running and possibility of restart of scheduler micro service in the meantime.
- Concurrency - Support multiple operations at a time and even synchronizing resources in various edge clouds in parallel.
- Performance - Avoiding file system operations as much as possible.
...
- Get the clusters that form the logical cloud to schedule the application. Validate the application, intent group, and generic placement intent.
- Create etcd Context for the application
- Process the Generic placement context and populate the etcd context with the edge list.
- Generate Kubernetes resources from the Helm charts by using Helm templating and from macros. Copy resources in etcd store based on the context for the application.
- Placement stage: Call all the placement controllers including the HPA controller in order. Wait for one controller to complete before going to the next controller.
- HPA controller: Kubernetes resource manipulation. Add additional information to the deployments. In the case where the generic intent calls for anyof clusters, the HPA controller can make a selection for one or more clusters meeting the HPA requirements for the sub- application.
- Action Controller Stage: Call all the action controllers including Traffic Controller in order. Wait for one controller to complete before going to the next controller.
- Traffic Controller: Creates more resources for traffic routing and management between clusters.
- After all the controllers are done processing if there are still multiple clusters that meet the requirements, the orchestrator will pick up the first cluster to schedule the resource.
- Send notification to the resource sync to schedule the resources on the selected clusters. Once the resources are scheduled to be created the resource scheduler sends back notification to the orchestrator which in turn returns HTTP response as a success for the instantiation API.
- Resource Sync will periodically check the status of the resources and update the database with the status of the resource.
...
draw.io Diagram | ||||||||||||||||||
---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
|
Instantiation Deletion Flow
...
- Get the clusters that form the logical cloud to schedule the composite application. Validate the application, intent group, and generic placement intent.
- Create etcd Context for the application
- Process the Generic placement context and populate the etcd context with the edge list.
- Generate Kubernetes resources from the Helm charts by using Helm templating and from macros. Copy resources in etcd store based on the context for the application.
- Placement stage: Call all the placement controllers including the HPA controller in order. Wait for one controller to complete before going to the next controller.
- HPA controller: Kubernetes resource manipulation. Add additional information to the deployments. In the case where the generic intent calls for anyof clusters, the HPA controller can make a selection for one or more clusters meeting the HPA requirements for the sub- application.
- Action Controller Stage: Call all the action controllers including Traffic Controller in order. Wait for one controller to complete before going to the next controller.
- Traffic Controller: Creates more resources for traffic routing and management between clusters.
- After all the controllers are done processing if there are still multiple clusters that meet the requirements, the orchestrator will pick up the first cluster to schedule the resource.
- Diff the original application resources and the edges that they are created on with this application context. Generate a list of resources to be changed.
- Send notification to the resource sync to create/update/delete the resources on the selected clusters. Once the resources are scheduled to be updated the resource scheduler sends back notification to the orchestrator which in turn returns HTTP response as a success for the instantiation API.
- Resource Sync will periodically check the status of the resources and update the database with the status of the resource.
RESTful North API (with examples)
Onboarding
Profiles
Deployment Intent
Interface API to Controllers (with examples)
Replication controller interface
Placement controller interface for internal controllers
Placement controller interface for external controllers
Resource modifier controller interface for internal controllers
Resource modifier controller interface for external controllers
Action controller interface for internal controllers
Action controller interface for external controllers
Restful South API (to Resource installer service, with examples)
Modules (Description, internal structures etc..)
....
Sequence flows
...
Orchestrator High-Level Design
draw.io Diagram | ||||||||||||||||||||
---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
|
Infrastructure Library
- MongoDB - Create, Read, Write
- Etcd - Create, Read, Write
Module Access Library
Project API- CreateProject, DeleteProject, GetProject
Composite App API - CreateCompositeApp, GetCompositeApp, DeleteCompositeApp, AddApptoCompositeApp, DeleteAppFromCompositeApp
Composite Profile API - CreateCompositeProfile, GetCompositeProfile, DeleteCompositeProfile, AddProfiletoCompositeProfile, DeleteProfileFromCompositeProfile
Generic Placement API - CreateGenericPlacement, DeleteGenericPlacement, GetGenericPlacement, AddGenericIntent, DeleteGenericIntent, GetIntent
Deployment Intent API - CreateDeploymentIntent, DeleteDeploymentIntent, GetDeploymentIntent, AddIntent, DeleteIntent,
Status API - GetStatus(deploymet)
Runtime Resource Context Library
CreateContext(project, name) returns contextId
AddApp(contextId, name) return error
AddCluster(contextId, app-name, cluster-name)
AddResource(contextId, app-name, cluster-name, resourceName)
UpdateResourceField(contextId, app-name, cluster-name, resourceName, fieldName, Value)
Test cases
Composite App with EdgeX, Firewall with Intents