Background
L7 Proxy Service Mesh Controller intends to provide connectivity, shape the traffic, apply policies, RBAC and provide
mutual TLS for applications/microservices running across clusters (with service mesh), within the cluster
and with external applications. The functionalities are subjected to the usage of underlying service mesh technology.
Design Overview
Traffic Controller Design Internals
NOTE - Current implementation will support the ISTIO service mesh technology and SD-WAN load balancer. The plugin architecture of the controller makes it extensible to work with any Service mesh technology and any external load balancer as well. It is also designed to configure and communicate with external DNS servers.
JIRA
| Component | JIRA Items |
|---|---|
| MULTICLOUD-913 - Getting issue details... STATUS |
| 2. Controller Interface, Backend Process | MULTICLOUD-914 - Getting issue details... STATUS |
| 3. Developing backend code with interfaces | MULTICLOUD-915 - Getting issue details... STATUS |
| 4. Loadbalancer configuration (Firewall, IPVS, NAT, and other L3 connectivity) | MULTICLOUD-924 - Getting issue details... STATUS |
| 5. External DNS Configuration | MULTICLOUD-909 - Getting issue details... STATUS |
| 6. Testing | MULTICLOUD-918 - Getting issue details... STATUS |
| 7.Documentation | MULTICLOUD-923 - Getting issue details... STATUS |
API
RESTful North API (with examples)
Inter Micro-service communication intents
- Considering microservice replication across multiple locations with replication within each cluster.
Intent Example for an HTTP application and TCP application
POST
URL: /v2/project/{project-name}/rb/{rb-name}/{version}/intent/{intent-name}/connectivity/servicerecord/{servicerecord-name}/
POST BODY:
{
"intent-name": "test-connectivity",
"servicerecord-name": "test-servicerecord",
"description": "connectivity intent for microservice replication across multiple locations and clusters"
"connectivityResource": [
{ "microservice01":
"egressgateway": "true", // All the outbound traffic from this service will flow through a dedicated egress gateway
"externalPrefix": "httpservice", // This is the prefix used to expose this service outside the cluster.
"mutualTLS": "true"
[{
"targetServiceName": "httpbin",
"protocol": "HTTP",
"type": "kubernetes",
"headless": "false",
"serviceMesh": "istio",
"port" : "80",
"loadbalancing": "true",
"traffic percentage" : "33",
"egress" : "true", // Setting this to true will create a dedicated egrees gateway for the service httpbin on whichever cluster it is running on
},
{
"targetServiceName": "httpbin", // Note that this is a replication of the "sleep" serivce which may be running is another cluster. The details of this service will be known only after scheduling.
"protocol": "HTTP",
"type": "kubernetes",
"headless": "false",
"serviceMesh": "istio",
"port" : "80",
"loadbalancing": "true",
"traffic percentage" : "33",
"egress" : "true", // Setting this to true will create a dedicated egrees gateway for the service httpbin on whichever cluster it is running on
},
{
"targetServiceName": "sleep",
"protocol": "HTTP",
"type": "kubernetes"
"headless": "false",
"serviceMesh": "istio"
"port" : "80",
"loadbalancing": "true",
"traffic percentage" : "34",
"egress" : "false", // Setting this to true will create a dedicated egrees gateway for the service sleep on whichever cluster it is running on
}]
},
{ "microservice02": [
{
"targetServiceName": "edition.cnn.com",
"protocol": "HTTPS",
"type": "externalService",
"headless": "false",
"serviceMesh": "",
"port" : "443",
"loadbalancing": "false",
"traffic percentage" : "",
"egress" : "false", // Setting this to true will create a dedicated egrees gateway for the service httpbin on whichever cluster it is running on
},
{
"targetServiceName": "sleep",
"protocol": "HTTP",
"type": "kubernetes",
"headless": "false",
"serviceMesh": "istio",
"port" : "80",
"loadbalancing": "false",
"traffic percentage" : "",
"egress" : "false", // Setting this to true will create a dedicated egrees gateway for the service sleep on whichever cluster it is running on
},
{
"targetServiceName": "sleep", // Note that this is a replication of the "sleep" serivce which may be running is another cluster. The details of this service will be known only after scheduling.
"protocol": "HTTP",
"type": "kubernetes",
"headless": "false",
"serviceMesh": "istio",
"port" : "80",
"loadbalancing": "false",
"traffic percentage" : "",
"egress" : "false", // Setting this to true will create a dedicated egrees gateway for the service sleep on whichever cluster it is running on
},
{ "microservice03": [
{
"targetServiceName": "", // As this a tcp service, we need the IP address of external service instead of a FQDN
"protocol": "TCP",
"targetServiceIP": "172.25.55.99", // Note that we need IP address instead of a FQDN for a TCP service
"type": "externalService",
"headless": "false",
"serviceMesh": "",
"port" : "22",
"loadbalancing": "false",
"traffic percentage" : "",
"egress" : "true", // Setting this to true will create a dedicated egrees gateway for the service httpbin on whichever cluster it is running on
}]
}
RETURN STATUS: 201
RETURN BODY:
{
"intent-name": "test-connectivity",
"description": "connectivity intent for microservice replication across multiple locations and clusters"
}
2. Considering instantiation of the same application multiple times in multiple logical clouds that span across the same edge locations. - The investigation is in Progress
POST
URL: /v2/project/{project-name}/rb/{rb-name}/{version}/intent/{intent-name}/connectivity/intercluster/servicerecord/{servicerecord-name}/
POST BODY:
{
"intent-name": "test-connectivity",
"servicerecord-name": "test-servicerecord",
"description": "instantiation of the same application multiple times in multiple logical clouds that span across the same edge locations"
"connectivityResource": [
{ "application01":
"egressgateway": "true", // All the outbound traffic from this service will flow through a dedicated egress gateway
"externalPrefix": "", // This field is not used since an application has multiple microservices
"mutualTLS": "true", // Establishes mtls between all the microservices under the application
"microservices": ["microservice01", "microservice02", "microservice03"] // list of all microservice under the application
[{
"sourceService": "microservice01",
"targetServiceName": "microservice02", // If "microservice01" and "microservice02" are placed on the same cluster and same logical cloud, then no configuration will be generated.
"protocol": "HTTP",
"type": "kubernetes",
"headless": "false",
"serviceMesh": "istio",
"port" : "80",
"loadbalancing": "true",
"traffic percentage" : "",
"egress" : "true", // Setting this to true will create a dedicated egrees gateway for the service httpbin on whichever cluster it is running on
},
{
"targetServiceName": "httpbin", // Note that this is a replication of the "sleep" serivce which may be running is another cluster. The details of this service will be known only after scheduling.
"protocol": "HTTP",
"type": "kubernetes",
"headless": "false",
"serviceMesh": "istio",
"port" : "80",
"loadbalancing": "true",
"traffic percentage" : "33",
"egress" : "true", // Setting this to true will create a dedicated egrees gateway for the service httpbin on whichever cluster it is running on
},
{
"targetServiceName": "sleep",
"protocol": "HTTP",
"type": "kubernetes"
"headless": "false",
"serviceMesh": "istio"
"port" : "80",
"loadbalancing": "true",
"traffic percentage" : "34",
"egress" : "false", // Setting this to true will create a dedicated egrees gateway for the service sleep on whichever cluster it is running on
}]
},
{ "microservice02": [
{
"targetServiceName": "edition.cnn.com",
"protocol": "HTTPS",
"type": "externalService",
"headless": "false",
"serviceMesh": "",
"port" : "443",
"loadbalancing": "false",
"traffic percentage" : "",
"egress" : "false", // Setting this to true will create a dedicated egrees gateway for the service httpbin on whichever cluster it is running on
},
{
"targetServiceName": "sleep",
"protocol": "HTTP",
"type": "kubernetes",
"headless": "false",
"serviceMesh": "istio",
"port" : "80",
"loadbalancing": "false",
"traffic percentage" : "",
"egress" : "false", // Setting this to true will create a dedicated egrees gateway for the service sleep on whichever cluster it is running on
},
{
"targetServiceName": "sleep", // Note that this is a replication of the "sleep" serivce which may be running is another cluster. The details of this service will be known only after scheduling.
"protocol": "HTTP",
"type": "kubernetes",
"headless": "false",
"serviceMesh": "istio",
"port" : "80",
"loadbalancing": "false",
"traffic percentage" : "",
"egress" : "false", // Setting this to true will create a dedicated egrees gateway for the service sleep on whichever cluster it is running on
},
{ "microservice03": [
{
"targetServiceName": "times.news.com",
"protocol": "HTTP",
"targetServiceIP": "172.25.55.99", // Note that we need IP address instead of a FQDN for a TCP service
"type": "externalService",
"headless": "false",
"serviceMesh": "",
"port" : "22",
"loadbalancing": "false",
"traffic percentage" : "",
"egress" : "true", // Setting this to true will create a dedicated egrees gateway for the service httpbin on whichever cluster it is running on
}]
}
RETURN STATUS: 201
RETURN BODY:
{
"intent-name": "test-connectivity",
"description": "instantiation of the same application multiple times in multiple logical clouds that span across the same edge locations"
}
3. Considering instantiation of the same application multiple times in the same logical cloud
POST
URL: /v2/project/{project-name}/rb/{rb-name}/{version}/intent/{intent-name}/connectivity/intercluster/servicerecord/{servicerecord-name}/
POST BODY:
{
"intent-name": "test-connectivity",
"servicerecord-name": "test-servicerecord",
"description": "instantiation of the same application multiple times in the same logical cloud"
"connectivityResource": [
{ "application01":
"egressgateway": "true", // All the outbound traffic from this service will flow through a dedicated egress gateway
"externalPrefix": "httpservice", // This is the prefix used to expose this service outside the cluster.
"mutualTLS": "true"
"microservices": ["microservice01", "microservice02", "microservice03"]
[{
"targetServiceName": "httpbin",
"protocol": "HTTP",
"type": "kubernetes",
"headless": "false",
"serviceMesh": "istio",
"port" : "80",
"loadbalancing": "true",
"traffic percentage" : "33",
"egress" : "true", // Setting this to true will create a dedicated egrees gateway for the service httpbin on whichever cluster it is running on
},
{
"targetServiceName": "httpbin", // Note that this is a replication of the "sleep" serivce which may be running is another cluster. The details of this service will be known only after scheduling.
"protocol": "HTTP",
"type": "kubernetes",
"headless": "false",
"serviceMesh": "istio",
"port" : "80",
"loadbalancing": "true",
"traffic percentage" : "33",
"egress" : "true", // Setting this to true will create a dedicated egrees gateway for the service httpbin on whichever cluster it is running on
},
{
"targetServiceName": "sleep",
"protocol": "HTTP",
"type": "kubernetes"
"headless": "false",
"serviceMesh": "istio"
"port" : "80",
"loadbalancing": "true",
"traffic percentage" : "34",
"egress" : "false", // Setting this to true will create a dedicated egrees gateway for the service sleep on whichever cluster it is running on
}]
},
{ "microservice02": [
{
"targetServiceName": "edition.cnn.com",
"protocol": "HTTPS",
"type": "externalService",
"headless": "false",
"serviceMesh": "",
"port" : "443",
"loadbalancing": "false",
"traffic percentage" : "",
"egress" : "false", // Setting this to true will create a dedicated egrees gateway for the service httpbin on whichever cluster it is running on
},
{
"targetServiceName": "sleep",
"protocol": "HTTP",
"type": "kubernetes",
"headless": "false",
"serviceMesh": "istio",
"port" : "80",
"loadbalancing": "false",
"traffic percentage" : "",
"egress" : "false", // Setting this to true will create a dedicated egrees gateway for the service sleep on whichever cluster it is running on
},
{
"targetServiceName": "sleep", // Note that this is a replication of the "sleep" serivce which may be running is another cluster. The details of this service will be known only after scheduling.
"protocol": "HTTP",
"type": "kubernetes",
"headless": "false",
"serviceMesh": "istio",
"port" : "80",
"loadbalancing": "false",
"traffic percentage" : "",
"egress" : "false", // Setting this to true will create a dedicated egrees gateway for the service sleep on whichever cluster it is running on
},
{ "microservice03": [
{
"targetServiceName": "", // As this a tcp service, we need the IP address of external service instead of a FQDN
"protocol": "TCP",
"targetServiceIP": "172.25.55.99", // Note that we need IP address instead of a FQDN for a TCP service
"type": "externalService",
"headless": "false",
"serviceMesh": "",
"port" : "22",
"loadbalancing": "false",
"traffic percentage" : "",
"egress" : "true", // Setting this to true will create a dedicated egrees gateway for the service httpbin on whichever cluster it is running on
}]
}
RETURN STATUS: 201
RETURN BODY:
{
"intent-name": "test-connectivity",
"description": "instantiation of the same application multiple times in the same logical cloud"
}
4. Considering RBAC/ABAC - TBD
External DNS Update Overview
This section covers the design for how external DNS are updated.
The following sequence diagram illustrates the approach:
Elements of the DNS update design
external-dns-agent
This is deployed as part of the Edge Cluster. Essentially, it is an instance of external-dns configures with istio-gateway as the source and a variant of 'coredns' as the provider. The variant is that instead of updating a CoreDNS instance, it will generate a custom resource (DNS CR) that is stored on the central controller cluster.
DNS CR
This DNS CR will have the following information:
- Cluster ID
- Project ID
- DNS record information
DNS Provider Intent
The intents for the distributed application provide DNS Provider information.
- Cluster ID
- Project ID
- DNS Provider Type (e.g. coredns, aws route 53, ...)
- DNS Provider locatoin (e.g. IP, hostname)
- DNS Provider credentials (e.g. username, pw, TLS credentials)
- DNS Provider other parameters (maybe provider specific)
Question: Is the DNS Provider information really an Intent ? which is associated with resource bundle, or is it a set of information associated with the edge cluster / project that is common across resource bundles for that project?
external-dns-controller
This component operates at the centralized controller. When changes are made to the DNS CR's, this component will retrieve the DNS Provider information for the associated Cluster/Project and ensure that the external DNS records are updated to match the DNS records in the DNS CR.
Example DNS Update diagram
External application communication intents
Considering DNS resolution, No DNS resolution (IP addresses), Egress proxies of the Service Mesh, Third-party egress proxy
User facing communication intents
Considering Multiple DNS Servers
Considering multiple user-facing entities
Considering RBAC/ABAC
Internal Design details
Guidelines that need to kept in mind
- Support for metrics that can be retrieved by Prometheus
- Support for Jaeger distributed tracing by including opentracing libraries around HTTP calls.
- Support for logging that is understood by fluentd
- Mutual exclusion of database operations (keeping internal modules accessing database records simultaneously and also by replication entities of the scheduler micro-service).
- Resilience - ensure that the information returned by controllers is not lost as the synchronization of resources to remote edge clouds can take hours or even days when the edge is not up and running and possibility of restart of scheduler micro service in the meantime.
- Concurrency - Support multiple operations at a time and even synchronizing resources in various edge clouds in parallel.
- Performance - Avoiding file system operations as much as possible.
Modules (Description, internal structures etc..)
....