...
PlantUML Macro |
---|
@startuml hide empty description title Deployment Intent Group state transitions\nwith AppContext state transitions illustrated state Created #LightGreen state Approved #LightGreen state Instantiated #LightGreen state Terminated #LightGreen [*] -down-> Created : Create Deployment\nIntent Group Created --> Created : //Modify intents// Created -down-> Approved : POST approve note left of Created //Modify intents// encompasses adding/deleting/updating the intents assigned to a Deployment Intent Group as well as updating the Deployment Intent Group itself end note Approved -down-> Instantiated : POST instantiate Approved -> Created : //Modify intents// Instantiated -down-> Terminated: POST terminate Terminated -up-> Instantiated: POST instantiate state Instantiated #LightGreen { state "Instantiated" as acInstantiated #LightBlue state Instantiating #LightBlue state InstantiateFailed #LightBlue Instantiating -> InstantiateFailed Instantiating -> acInstantiated } state Terminated #LightGreen { state "Terminated" as acTerminated #LightBlue state "Terminating" #LightBlue state "TerminateFailed" as termFailed #LightBlue Terminating -> termFailed Terminating -> acTerminated } Terminated --> Created: //Modify intents// Created --> [*]: DELETE Deployment Intent Group @enduml |
AppContext State
The rsync process will maintain a top level state for the AppContext. The states are:
- Instantiating: Once rsync is invoked to instantiate an AppContext, the state will be set to Instantiating.
- Instantiated: Rsyinc will set the .AppContext state to Instantiated after all Resources in the AppContext have been Applied.
- InstantiateFailed: This indicates that one or more Resources Failed to be applied.
- PreTerminate: If rsync is invoked to terminate an AppContext which is still in the Instantiating state, rsync needs to shutdown any threads in the process of trying to instantiate resources before beginning the terminate sequence.
- Terminating: When terminate is invoked, this state will be entered directly if the AppContext is in Instantiated or InstantiateFailed state. Otherwise, it will enter this state from PreTerminate after the initialize sequence has been shutdown.
- Terminated: This indicates that rsync has successfully Deleted all resources.
- TerminateFailed: This indicates that rsync has received a failure response from one or more Resources when attempting to delete them.
PlantUML Macro |
---|
@startuml
hide empty description
title AppContext State Transition
state Instantiated #LightBlue
state Instantiating #LightBlue
state InstantiateFailed #LightBlue
state PreTerminate #LightBlue
state Terminating #LightBlue
state Terminated #LightBlue
state TerminateFailed #LightBlue
[*] -> Instantiating : **gRPC instantiate**
Instantiating -> InstantiateFailed : //any resource//\n//Apply Failed//
Instantiating -> Instantiated : //all resources//\n//Applied//
Instantiating --> Terminating : **gRPC terminate**\n//old instantiate//\n//go routines//\n//cleaned up//
Instantiated --> Terminating : **gRPC terminate**
InstantiateFailed --> Terminating : **gRPC terminate**
note left of Terminating
Transition to Terminating
is allowed to occur when the controlling
EMCO resource (e.g. DeploymentIntentGroup
or Cluster) is in the Instantiated state
end note
Terminating -> Terminated : //all resources//\n//Deleted//
Terminating --> TerminateFailed : //some resources//\n//Delete Failed//
@enduml |
Rsync resource state values
The state of the rsync resource is maintained in the EMCO AppContext by rsync. The status values are defined as follows.
- Pending: Upon initial creation by ncm (for cluster network intents) or orchestrator (for DIGs), the rsync resources will be initialized to a Pending status to indicate they have not been handled by rsync yet.
- Applied: This indicates that rsync has successfully applied the rsync resource to a destination cluster. This does not indicate anything about the actual status of the corresponding cluster resource(s) - other than it is expected that the cluster resource does exist.
- Failed: This indicates that rsync has received a failure response when either attempting to apply or delete the rsync resource from the destination cluster. rsync is taking no further action with this resource.
- Retrying: This indicates that rsync is continuing to attempt to apply or delete the rsync resource from the destination cluster. This may occur because connectivity to the destination cluster is currently unavailable, but may resume at a later time. This will continue until a different lifecyle state is invoked on the controlling EMCO resource.
- Terminated: This indicates that rsync has successfully deleted the rsync resource from the destination cluster. This does not indicate anything about the actual status of the corresponding cluster resource(s) - other than it is expected to that the cluster resource does exist or is in the process of terminating.
PlantUML Macro |
---|
@startuml
hide empty description
title Rsync Resource State Transition (with corresponding AppContext states)
[*] -left-> Instantiating #LightBlue : **gRPC**\n**instantiate**
[*] -right-> Pending : **AppContext**\n**changes to**\n**Instantiating**
Pending -> Applied : Successful\napply
Pending --> Failed : Failed apply\nor delete
Pending --> Retrying
Failed --> Deleted : **AppContext**\n**changes to**\n**Terminating**
Pending --> Deleted : **AppContext**\n**changes to**\n**Terminating**
Retrying --> Deleted : **AppContext**\n**changes to**\n**Terminating**
Applied --> Pending : **AppContext**\n**changes to**\n**Terminating**
Pending --> Deleted : successful\ndelete
Retrying -> Applied
Retrying --> Retrying
Retrying --> Deleted : successful\ndelete
Retrying --> Failed : Exceed\nretry time
Instantiating -left-> Terminating #LightBlue : **//gRPC//**\n**//terminate//**\n//old instantiate threads//\n//etc. cleaned up//
X #LightBlue -up-> Terminating : **//gRPC//**\n**//terminate//**\n//from InstantiateFailed//\n//or Instantiated//
Failed --> InstantiateFailed #LightBlue : //any resource//
Failed --> TerminateFailed #LightBlue : //any resource//
Applied --> Instantiated #LightBlue : //all resources//
Deleted --> Terminated #LightBlue : //all resources//
@enduml |
Cluster resource status
The status of resources deployed by rsync to clusters is detected as follows.
- When rsync instantiates rsync resources, it will also instantiate a ResourceBundleState CR to the cluster. For a given Composite Application, a ResourceBundleState CR will be deployed for each App (in the Composite App) per cluster. A label will be applied by rsync to all cluster resources of a given App and will be matched by label to the corresponding ResourceBundleState CR in the cluster. The label format is: "emco/deployment-id: <AppContext identifier>-<app-name>"
- A 'monitor' pod is present in each cluster is monitors all resources with the "emco/deployement-id" label. When it detects changes to those resources, it will update the matching ResourceBundleState CR with the details of the resource. In the example ResourceBundleState CR below, for example, all pod resources that are labeled with emco/deployment-id: 171887448792644816-sink will be included in the 'podStatuses' array.
A Watcher thread is started per cluster by rsync to watch for changes to ResourceBundleState CRs in the cluster. When an updated CR is detected, the Watcher retrieves it and saves it into the corresponding AppContext per App/Cluster so it is available to provide information for cluster resource queries.
Code Block | ||
---|---|---|
| ||
apiVersion: k8splugin.io/v1alpha1
kind: ResourceBundleState
metadata:
labels:
emco/deployment-id: 171887448792644816-sink
name: sink-171887448792644816
namespace: default
spec:
selector:
matchLabels:
emco/deployment-id: 171887448792644816-sink
status:
ready: false
resourceCount: 0
configMapStatuses: []
daemonSetStatuses: []
deploymentStatuses: []
ingressStatuses: []
jobStatuses: []
podStatuses: []
secretStatuses: []
serviceStatuses: []
statefulSetStatuses: [] |
The cluster resource status is provided in two forms.
- The actual status{} portion of the cluster resource (if present) is made available in the information returned via the ResourceBundleState CR.
- Summarized in a value as follows.
- Unknown: The unknown status represents the case where a ResourceBundleState CR has not been received yet, or that the ResourceBundleState CR does not support that resource type.
- NotPresent: For resource types (Kinds) that are supported by the ResourceBundleState CR, if an rsync resource does not have a corresponding cluster resource in the CR, then the cluster status of the resource is NotPresent.
- Present: For rsync resources with a corresponding cluster resource in the ResourceBundleState CR, the clustesr status is present.
- TBD: Further work can be done to summarize the Status{} portion of cluster resources to identify the status more precisely than Present - such as: Active, Ready, Not Ready, etc.
Instantiate Sequence
This illustrates the Deployment Intent Group instantiate sequence
PlantUML Macro |
---|
@startuml
title Deploymenet Intent Group Instantiation Sequence
actor Admin
box "Orchestrator"
participant scheduler
participant generic_placement_controller
end box
box "EMCO DB"
database mongo
end box
box "AppContext"
participant etcd
end box
box "resource synchronizer"
participant rsync
participant appcontext_watcher
collections worker_thread
collections cluster_watcher
end box
box "Edge Cluster"
participant API_Server
participant cluster_etcd
participant monitor
end box
Admin -> scheduler : POST instantiate\nDeployment Intent Group
activate scheduler
activate scheduler #Red
scheduler -> mongo : retrieve Deployment Intent Group
scheduler -> Admin : ERROR if wrong state
deactivate scheduler #Red
scheduler -> generic_placement_controller : create AppContext
activate generic_placement_controller
generic_placement_controller -> mongo : get generic placement intent
generic_placement_controller -> etcd : create AppContext and metadata
generic_placement_controller ->scheduler : done
deactivate generic_placement_controller
scheduler -> scheduler : invoke all other\nplacement and action\ncontrollers (not shown)
scheduler -> rsync : gRPC call to instantiate AppContext
activate rsync
activate rsync #Red
rsync -> etcd : Initialize Appcontext state=Instantiating\nSet Resources state=Pending
rsync -> etcd : Initialize StopFlag=False
rsync -> scheduler : Return Error if on occurs
deactivate rsync #Red
scheduler --> Admin : Return ERROR
create appcontext_watcher
rsync -> appcontext_watcher: invoke thread to instantiate AppContext
activate appcontext_watcher
appcontext_watcher -> etcd : start watching StopFlag
rsync -> scheduler : return OK
deactivate rsync
scheduler -> Admin : Return OK
deactivate scheduler
create worker_thread
loop for each app
appcontext_watcher -> worker_thread : start thread to handle app
loop for each cluster
worker_thread -> worker_thread : start thread to handle cluster
create cluster_watcher
worker_thread -> cluster_watcher : invoke Watcher for\nResourceBundleState CRs\n(may already exist)
loop until all resources are Applied or a Failure occurs
worker_thread -> API_Server : Send Resource Apply to Cluster
worker_thread -> etcd : If success Resource State = Applied\nElse If Connectivity Error Resource State = Retrying\nElse Resource State = Failed
end
worker_thread -> API_Server : Apply ResourceBundleState CR
end
end
worker_thread -> appcontext_watcher : all worker threads complete
appcontext_watcher -> etcd : if any Resource = Failed, AppContext State = Failed\nElse All Resources=Applied, AppContext State = Instantiated
deactivate appcontext_watcher
@enduml |
Terminate Sequence
This illustrates the Deployment Intent Group terminate sequence
PlantUML Macro |
---|
@startuml
title Deployment Intent Group Termination Sequence
actor Admin
box "Orchestrator"
participant scheduler
participant generic_placement_controller
end box
box "EMCO DB"
database mongo
end box
box "AppContext"
participant etcd
end box
box "resource synchronizer"
participant rsync
participant old_appcontext_watcher
participant appcontext_watcher
collections old_worker_thread
collections worker_thread
collections cluster_watcher
end box
box "Edge Cluster"
participant API_Server
participant cluster_etcd
participant monitor
end box
activate old_appcontext_watcher
note right of old_appcontext_watcher : may be a thread still\nstill trying to instantiate\nthe appcontext
activate old_worker_thread
note left of old_worker_thread : may be thread(s) still trying\nto instantiate resources
old_appcontext_watcher <- etcd : watching StopFlag
Admin -> scheduler : POST terminate\nDeployment Intent Group
activate scheduler
activate scheduler #Red
scheduler -> mongo : retrieve Deployment Intent Group
scheduler -> Admin : ERROR if wrong state
deactivate scheduler #Red
scheduler -> rsync : gRPC call to terminate AppContext
activate rsync
create appcontext_watcher
rsync -> appcontext_watcher : invoke thread to terminate AppContext
appcontext_watcher -> etcd : Set StopFlag = True
etcd -> appcontext_watcher ++ #Red : start watching until AppContext==Instantiated OR StopFlag==False
etcd -> old_appcontext_watcher ++ #Red : detects StopFlag=True
old_appcontext_watcher -> old_worker_thread : terminate any running Instantiate threads
rsync -> scheduler : Return Error if one occurs at this stage
scheduler --> Admin : Return ERROR
rsync -> scheduler : return OK
deactivate rsync
scheduler -> Admin : Return OK
deactivate scheduler
old_worker_thread -> old_appcontext_watcher : all old threads return
deactivate old_worker_thread
etcd <- old_appcontext_watcher -- #Red : Set StopFlag=False
rsync <- old_appcontext_watcher : old instantiate thread returns
deactivate old_appcontext_watcher
etcd <- appcontext_watcher -- : Detects AppContext==Instantiated OR StopFlag==False
etcd <- appcontext_watcher : Set AppContext=Terminating
activate appcontext_watcher #LightGreen
loop initialize all resources states for terminate handling
appcontext_watcher -> etcd : If Resource = Applied, Resource= Pending\nelseif Resource = (Pending, Failed, Retrying), Resource=Deleted
end
loop for each app
create worker_thread
appcontext_watcher -> worker_thread : start thread to handle app
loop for each cluster
worker_thread -> worker_thread : start thread to handle cluster
loop until all resources are Deleted or Failed
worker_thread -> API_Server : Send Resource Delete to Cluster
worker_thread -> etcd : If success Resource State = Deleted\nElse If Connectivity Error Resource State = Retrying\nElse Resource State = Failed
end
loop waiting until deleted resources disappear from ResourceBundleState CR
monitor -> cluster_etcd : Update ResourceBundleState CR\nas resources get deleted
cluster_watcher <- cluster_etcd : Notifications of changes in\nResourcebundleState CR
cluster_watcher -> etcd : Update Status at app-cluster\nin AppContext
worker_thread -> etcd : wait until ResourceBundleState CR is empty
end
worker_thread -> API_Server : Delete ResourceBundleState CR
end
end
worker_thread -> appcontext_watcher : all worker threads complete
appcontext_watcher -> etcd : if any Resource = Failed, AppContext State = Failed\nElse All Resources=Deleted, AppContext State = Terminated
deactivate appcontext_watcher
@enduml |
Status Query Sequence
This illustrates a status query sequence
PlantUML Macro |
---|
@startuml
title Status / State Update Sequence
actor Admin
box "Orchestrator"
participant status_api
end box
box "EMCO DB"
database mongo
end box
box "AppContext"
participant etcd
end box
box "resource synchronizer"
participant rsync
participant appcontext_watcher
collections worker_thread
collections cluster_watcher
end box
box "Edge Cluster"
participant API_Server
participant cluster_etcd
participant monitor
end box
worker_thread -> API_Server : App Resources Applied
worker_thread -> API_Server : Create ResourceBundleState CR
loop monitoring for lifetime of ResourceBundleStateCR
monitor <- cluster_etcd : watches for changes on labeled resources
monitor -> cluster_etcd : updates associated ResourceBundleStateCR
cluster_watcher <- cluster_etcd : watches for changes on ResourceBundleStateCR
cluster_watcher -> etcd : Update app/cluster Status object in AppContext
end
Admin -> status_api : GET Deployment Intent\nGroup Status
status_api -> mongo : get Deployment Intent Group
status_api -> etcd : access AppContext State\n(from Deployment Intent Group state
loop for all requested apps/clusters/resources
status_api -> etcd : access AppContext Cluster Status
status_api -> etcd : access AppContext Resource State
end
status_api -> status_api : prepare response
status_api -> Admin : return Status query results
@enduml |
Status Query
The status query, and variations with query parameters, on an EMCO resource will present the information described previously to the caller. The basic status query for the two EMCO resources discussed above will look like the following:
URL: GET /v2/cluster-providers/{cluster-provider-name}/clusters/{cluster-name}/status
URL: GET /v2/projects/{project-name}/composite-apps/{composite-app-name}/{version}/deployment-intent-groups/{deployment-intent-group-name}/status
The following query parameters will be available:
output=<summary | all | detail | rsync>
default output value is: 'all'
summary: will just show the top level EMC resource state and aggregated resource statuses but no resource detail information
all: will include a list of resources showing basic resource identification and resource statuses
detail: includes in the resource list the metadata, spec, and status of the resource as received in the ResourceBundleState CR
rsync: similar to detail, but only includes in the resource list the rsync resources - so the metadata and spec information will be taken from the AppContext. No cluster status information will be included.
app=<appname>
default is all apps
This will filter the results of the query to show results only for the resources of the specified App(s).
cluster=<cluster>
default is all clusters
This will filter the results of the query to show results only for the specified cluster(s)
resource=<resource name>
default is all resources
This will filter the results of the query to show results only for the specified resource(s)
Output of Status Query
The following table shows the essential structure of the status query response with a description of which elements are present based on the 'output' parameter.
...
Format of the status query response
...
Description
...
'summary' query
...
'all' query
...
'detail' query
...
'rsync'
query
...
{
...
"name": "<name>",
...
The name of the Deployment Intent Group or Cluster
...
X
...
X
...
X
...
X
...
"project": "<project name>",
...
Present for the Deployment Intent Group
...
X
...
X
...
X
...
X
...
"composite-app-name": "<composite-app-name>",
...
Present for the Deployment Intent Group
...
X
...
X
...
X
...
X
...
"composite-app-version": "<composite-app-version>",
...
Present for the Deployment Intent Group
...
X
...
X
...
X
...
X
...
"composite-profile-name": "<composite-profile-name>",
...
Present for the Deployment Intent Group
...
X
...
X
...
X
...
X
...
"state": "[ Created, Approved, Instantiated, Terminated ]",
...
'state' is the action made by the user
...
X
...
X
...
X
...
X
...
"rsync-status": {
...
'rsync-status' is the aggregated rsync-status of the resources - subject to query parameter filters
...
X
...
X
...
X
...
X
...
"Pending": 0,
...
elements with zero can be dropped
...
X
...
X
...
X
...
X
...
"Applied": 5,
...
X
...
X
...
X
...
X
...
"Failed": 2,
...
X
...
X
...
X
...
X
...
"Retrying": 3,
...
X
...
X
...
X
...
X
...
"Terminated": 0
...
X
...
X
...
X
...
X
...
},
...
X
...
X
...
X
...
X
...
"cluster-status" : {
...
'cluster-status' is the aggregated cluster-status of the resources - subject to query parameter filters
...
X
...
X
...
X
...
"NotPresent": 0,
...
X
...
X
...
X
...
"Present": 5,
...
X
...
X
...
X
...
"Unknown": 5
...
X
...
X
...
X
...
},
...
X
...
X
...
X
...
"resources": [
...
array of resources organized by app in the composite-app
...
X
...
X
...
X
...
{
...
X
...
X
...
X
...
"app-name": "collectd",
...
X
...
X
...
X
...
"clusters": [
...
array of clusters in the app
...
X
...
X
...
X
...
{
...
X
...
X
...
X
...
"name": "cluster1",
...
X
...
X
...
X
...
"resources": [
...
array of resources in the cluster
...
X
...
X
...
X
...
{
...
X
...
X
...
X
...
"GVK": {
...
The GVK will come from the AppContext except for cases where the resource is only a cluster resource - e.g. an rsync resource that is a Deployment can result in cluster resources of both Deployment and Pod(s).
...
X
...
X
...
X
...
"Group": "<group>",
...
X
...
X
...
X
...
"Version": "<version>",
...
X
...
X
...
X
...
"Kind": "<kind>"
...
X
...
X
...
X
...
},
...
X
...
X
...
X
...
"Name": "<resource name>",
...
The name of the resource from the AppContext (or cluster as described for GVK)
...
X
...
X
...
X
...
"metadata": { },
...
The 'metadata' element of the cluster resource as received in the ResourceBundleState CR. In the case of an 'output=rsync' query, this will be the 'metadata' element from the AppContext.
...
X
...
X
...
"spec": { },
...
The 'spec' element of the cluster resource as received in the ResourceBundleState CR. In the case of an 'output=rsync' query, this will be the 'metadata' element from the AppContext.
...
X
...
X
...
"status": { },
...
The 'status' element of the k8s resource as received in the ResourceBundleState CR (if the resource has one). In the case of an 'output=rsync' query, this will not be present.
...
X
...
"rsync-status": "[ Pending | Instantiated | Failed | Retrying |Terminated ]",
...
"Pending" - is set by orchestrator before issuing an Instantiate command to rsync
"Instantiate" - means rsync has successfully invoked deployment of the resource to the cluster
"Failed" - means rsync got an explicit failure when invoking to the cluster
"Retrying" - means connection to cluster is temporarily unavailable, rsync will continue to retry - applies to both instantiate and terminate sequences (initial thought is to detect this condition at a cluster level - but mark each resource)
"Terminated" - means rsync has successfully invoked termination of the resource to the cluster
...
X
...
X
...
X
Included with the StateInfo maintained in the Cluster or DeploymentIntentGroup record is the State, the time when that State was requested (via the API), and the identifier of the AppContext associated with the specific instance. A history of StateInfo data is maintained. For example:
Code Block | ||||
---|---|---|---|---|
| ||||
"state": {
"Actions": [
{
"State": "Created",
"ContextId": "",
"TimeStamp": "2020-08-21T05:36:54.186Z"
},
{
"State": "Approved",
"ContextId": "",
"TimeStamp": "2020-08-21T05:37:28.748Z"
},
{
"State": "Instantiated",
"ContextId": "2621114006130701074",
"TimeStamp": "2020-08-21T05:37:28.947Z"
},
{
"State": "Terminated",
"ContextId": "2621114006130701074",
"TimeStamp": "2020-08-21T05:38:51.061Z"
},
{
"State": "Approved",
"ContextId": "",
"TimeStamp": "2020-08-21T05:39:34.436Z"
},
{
"State": "Instantiated",
"ContextId": "2755581958183303505",
"TimeStamp": "2020-08-21T05:39:34.64Z"
}
]
}, |
AppContext State
The rsync process will maintain a top level state for the AppContext. The states are:
- Instantiating: Once rsync is invoked to instantiate an AppContext, the state will be set to Instantiating.
- Instantiated: Rsyinc will set the .AppContext state to Instantiated after all Resources in the AppContext have been Applied.
- InstantiateFailed: This indicates that one or more Resources Failed to be applied.
- Terminating: When terminate is invoked, this state will be entered directly if the AppContext is in Instantiated or InstantiateFailed state. If a previous Instantiating sequence is still running, the transition process includes shutting down the instantiating sequence before entering the Terminating state.
- Terminated: This indicates that rsync has successfully Deleted all resources.
- TerminateFailed: This indicates that rsync has received a failure response from one or more Resources when attempting to delete them.
PlantUML Macro |
---|
@startuml
hide empty description
title AppContext State Transition
state Instantiated #LightBlue
state Instantiating #LightBlue
state InstantiateFailed #LightBlue
state Terminating #LightBlue
state Terminated #LightBlue
state TerminateFailed #LightBlue
[*] -> Instantiating : **gRPC instantiate**
Instantiating -> InstantiateFailed : //any resource//\n//Apply Failed//
Instantiating -> Instantiated : //all resources//\n//Applied//
Instantiating --> Terminating : **gRPC terminate**\n//old instantiate//\n//go routines//\n//cleaned up//
Instantiated --> Terminating : **gRPC terminate**
InstantiateFailed --> Terminating : **gRPC terminate**
note left of Terminating
Transition to Terminating
is allowed to occur when the controlling
EMCO resource (e.g. DeploymentIntentGroup
or Cluster) is in the Instantiated state
end note
Terminating -> Terminated : //all resources//\n//Deleted//
Terminating --> TerminateFailed : //some resources//\n//Delete Failed//
@enduml |
Rsync resource state values
The state of the rsync resource is maintained in the EMCO AppContext by rsync. The status values are defined as follows.
- Pending: Upon initial creation by ncm (for cluster network intents) or orchestrator (for DIGs), the rsync resources will be initialized to a Pending status to indicate they have not been handled by rsync yet.
- Applied: This indicates that rsync has successfully applied the rsync resource to a destination cluster. This does not indicate anything about the actual status of the corresponding cluster resource(s) - other than it is expected that the cluster resource does exist.
- Failed: This indicates that rsync has received a failure response when either attempting to apply or delete the rsync resource from the destination cluster. rsync is taking no further action with this resource.
- Retrying: This indicates that rsync is continuing to attempt to apply or delete the rsync resource from the destination cluster. This may occur because connectivity to the destination cluster is currently unavailable, but may resume at a later time. This will continue until a different lifecyle state is invoked on the controlling EMCO resource.
- Terminated: This indicates that rsync has successfully deleted the rsync resource from the destination cluster. This does not indicate anything about the actual status of the corresponding cluster resource(s) - other than it is expected to that the cluster resource does exist or is in the process of terminating.
PlantUML Macro |
---|
@startuml
hide empty description
title Rsync Resource State Transition (with corresponding AppContext states)
[*] -left-> Instantiating #LightBlue : **gRPC**\n**instantiate**
[*] -right-> Pending : **AppContext**\n**changes to**\n**Instantiating**
Pending -> Applied : Successful\napply
Pending --> Failed : Failed apply\nor delete
Pending --> Retrying
Failed --> Deleted : **AppContext**\n**changes to**\n**Terminating**
Pending --> Deleted : **AppContext**\n**changes to**\n**Terminating**
Retrying --> Deleted : **AppContext**\n**changes to**\n**Terminating**
Applied --> Pending : **AppContext**\n**changes to**\n**Terminating**
Pending --> Deleted : successful\ndelete
Retrying -> Applied
Retrying --> Retrying
Retrying --> Deleted : successful\ndelete
Retrying --> Failed : Exceed\nretry time
Instantiating -left-> Terminating #LightBlue : **//gRPC//**\n**//terminate//**\n//old instantiate threads//\n//etc. cleaned up//
X #LightBlue -up-> Terminating : **//gRPC//**\n**//terminate//**\n//from InstantiateFailed//\n//or Instantiated//
Failed --> InstantiateFailed #LightBlue : //any resource//
Failed --> TerminateFailed #LightBlue : //any resource//
Applied --> Instantiated #LightBlue : //all resources//
Deleted --> Terminated #LightBlue : //all resources//
@enduml |
Cluster resource status
The status of resources deployed by rsync to clusters is detected as follows.
- When rsync instantiates rsync resources, it will also instantiate a ResourceBundleState CR to the cluster. For a given Composite Application, a ResourceBundleState CR will be deployed for each App (in the Composite App) per cluster. A label will be applied by rsync to all cluster resources of a given App and will be matched by label to the corresponding ResourceBundleState CR in the cluster. The label format is: "emco/deployment-id: <AppContext identifier>-<app-name>"
- A 'monitor' pod is present in each cluster is monitors all resources with the "emco/deployement-id" label. When it detects changes to those resources, it will update the matching ResourceBundleState CR with the details of the resource. In the example ResourceBundleState CR below, for example, all pod resources that are labeled with emco/deployment-id: 171887448792644816-sink will be included in the 'podStatuses' array.
A Watcher thread is started per cluster by rsync to watch for changes to ResourceBundleState CRs in the cluster. When an updated CR is detected, the Watcher retrieves it and saves it into the corresponding AppContext per App/Cluster so it is available to provide information for cluster resource queries.
Code Block | ||
---|---|---|
| ||
apiVersion: k8splugin.io/v1alpha1
kind: ResourceBundleState
metadata:
labels:
emco/deployment-id: 171887448792644816-sink
name: sink-171887448792644816
namespace: default
spec:
selector:
matchLabels:
emco/deployment-id: 171887448792644816-sink
status:
ready: false
resourceCount: 0
configMapStatuses: []
daemonSetStatuses: []
deploymentStatuses: []
ingressStatuses: []
jobStatuses: []
podStatuses: []
secretStatuses: []
serviceStatuses: []
statefulSetStatuses: [] |
The cluster resource status is provided in two forms.
- The actual status{} portion of the cluster resource (if present) is made available in the information returned via the ResourceBundleState CR.
- Summarized in a value as follows.
- Unknown: The unknown status represents the case where a ResourceBundleState CR has not been received yet, or that the ResourceBundleState CR does not support that resource type.
- NotPresent: For resource types (Kinds) that are supported by the ResourceBundleState CR, if an rsync resource does not have a corresponding cluster resource in the CR, then the cluster status of the resource is NotPresent.
- Present: For rsync resources with a corresponding cluster resource in the ResourceBundleState CR, the clustesr status is present.
- TBD: Further work can be done to summarize the Status{} portion of cluster resources to identify the status more precisely than Present - such as: Active, Ready, Not Ready, etc.
Instantiate Sequence
This illustrates the Deployment Intent Group instantiate sequence
PlantUML Macro |
---|
@startuml
title Deploymenet Intent Group Instantiation Sequence
actor Admin
box "Orchestrator"
participant scheduler
participant generic_placement_controller
end box
box "EMCO DB"
database mongo
end box
box "AppContext"
participant etcd
end box
box "resource synchronizer"
participant rsync
participant appcontext_watcher
collections worker_thread
collections cluster_watcher
end box
box "Edge Cluster"
participant API_Server
participant cluster_etcd
participant monitor
end box
Admin -> scheduler : POST instantiate\nDeployment Intent Group
activate scheduler
activate scheduler #Red
scheduler -> mongo : retrieve Deployment Intent Group
scheduler -> Admin : ERROR if wrong state
deactivate scheduler #Red
scheduler -> generic_placement_controller : create AppContext
activate generic_placement_controller
generic_placement_controller -> mongo : get generic placement intent
generic_placement_controller -> etcd : create AppContext and metadata
generic_placement_controller ->scheduler : done
deactivate generic_placement_controller
scheduler -> scheduler : invoke all other\nplacement and action\ncontrollers (not shown)
scheduler -> rsync : gRPC call to instantiate AppContext
activate rsync
activate rsync #Red
rsync -> etcd : Initialize Appcontext state=Instantiating\nSet Resources state=Pending
rsync -> etcd : Initialize StopFlag=False
rsync -> scheduler : Return Error if on occurs
deactivate rsync #Red
scheduler --> Admin : Return ERROR
create appcontext_watcher
rsync -> appcontext_watcher: invoke thread to instantiate AppContext
activate appcontext_watcher
appcontext_watcher -> etcd : start watching StopFlag
rsync -> scheduler : return OK
deactivate rsync
scheduler -> Admin : Return OK
deactivate scheduler
create worker_thread
loop for each app
appcontext_watcher -> worker_thread : start thread to handle app
loop for each cluster
worker_thread -> worker_thread : start thread to handle cluster
create cluster_watcher
worker_thread -> cluster_watcher : invoke Watcher for\nResourceBundleState CRs\n(may already exist)
loop until all resources are Applied or a Failure occurs
worker_thread -> API_Server : Send Resource Apply to Cluster
worker_thread -> etcd : If success Resource State = Applied\nElse If Connectivity Error Resource State = Retrying\nElse Resource State = Failed
end
worker_thread -> API_Server : Apply ResourceBundleState CR
end
end
worker_thread -> appcontext_watcher : all worker threads complete
appcontext_watcher -> etcd : if any Resource = Failed, AppContext State = Failed\nElse All Resources=Applied, AppContext State = Instantiated
deactivate appcontext_watcher
@enduml |
Terminate Sequence
This illustrates the Deployment Intent Group terminate sequence
PlantUML Macro |
---|
@startuml
title Deployment Intent Group Termination Sequence
actor Admin
box "Orchestrator"
participant scheduler
participant generic_placement_controller
end box
box "EMCO DB"
database mongo
end box
box "AppContext"
participant etcd
end box
box "resource synchronizer"
participant rsync
participant old_appcontext_watcher
participant appcontext_watcher
collections old_worker_thread
collections worker_thread
collections cluster_watcher
end box
box "Edge Cluster"
participant API_Server
participant cluster_etcd
participant monitor
end box
activate old_appcontext_watcher
note right of old_appcontext_watcher : may be a thread still\nstill trying to instantiate\nthe appcontext
activate old_worker_thread
note left of old_worker_thread : may be thread(s) still trying\nto instantiate resources
old_appcontext_watcher <- etcd : watching StopFlag
Admin -> scheduler : POST terminate\nDeployment Intent Group
activate scheduler
activate scheduler #Red
scheduler -> mongo : retrieve Deployment Intent Group
scheduler -> Admin : ERROR if wrong state
deactivate scheduler #Red
scheduler -> rsync : gRPC call to terminate AppContext
activate rsync
create appcontext_watcher
rsync -> appcontext_watcher : invoke thread to terminate AppContext
appcontext_watcher -> etcd : Set StopFlag = True
etcd -> appcontext_watcher ++ #Red : start watching until AppContext==Instantiated OR StopFlag==False
etcd -> old_appcontext_watcher ++ #Red : detects StopFlag=True
old_appcontext_watcher -> old_worker_thread : terminate any running Instantiate threads
rsync -> scheduler : Return Error if one occurs at this stage
scheduler --> Admin : Return ERROR
rsync -> scheduler : return OK
deactivate rsync
scheduler -> Admin : Return OK
deactivate scheduler
old_worker_thread -> old_appcontext_watcher : all old threads return
deactivate old_worker_thread
etcd <- old_appcontext_watcher -- #Red : Set StopFlag=False
rsync <- old_appcontext_watcher : old instantiate thread returns
deactivate old_appcontext_watcher
etcd <- appcontext_watcher -- : Detects AppContext==Instantiated OR StopFlag==False
etcd <- appcontext_watcher : Set AppContext=Terminating
activate appcontext_watcher #LightGreen
loop initialize all resources states for terminate handling
appcontext_watcher -> etcd : If Resource = Applied, Resource= Pending\nelseif Resource = (Pending, Failed, Retrying), Resource=Deleted
end
loop for each app
create worker_thread
appcontext_watcher -> worker_thread : start thread to handle app
loop for each cluster
worker_thread -> worker_thread : start thread to handle cluster
loop until all resources are Deleted or Failed
worker_thread -> API_Server : Send Resource Delete to Cluster
worker_thread -> etcd : If success Resource State = Deleted\nElse If Connectivity Error Resource State = Retrying\nElse Resource State = Failed
end
loop waiting until deleted resources disappear from ResourceBundleState CR
monitor -> cluster_etcd : Update ResourceBundleState CR\nas resources get deleted
cluster_watcher <- cluster_etcd : Notifications of changes in\nResourcebundleState CR
cluster_watcher -> etcd : Update Status at app-cluster\nin AppContext
worker_thread -> etcd : wait until ResourceBundleState CR is empty
end
worker_thread -> API_Server : Delete ResourceBundleState CR
end
end
worker_thread -> appcontext_watcher : all worker threads complete
appcontext_watcher -> etcd : if any Resource = Failed, AppContext State = Failed\nElse All Resources=Deleted, AppContext State = Terminated
deactivate appcontext_watcher
@enduml |
Status Query Sequence
This illustrates a status query sequence
PlantUML Macro |
---|
@startuml
title Status / State Update Sequence
actor Admin
box "Orchestrator"
participant status_api
end box
box "EMCO DB"
database mongo
end box
box "AppContext"
participant etcd
end box
box "resource synchronizer"
participant rsync
participant appcontext_watcher
collections worker_thread
collections cluster_watcher
end box
box "Edge Cluster"
participant API_Server
participant cluster_etcd
participant monitor
end box
worker_thread -> API_Server : App Resources Applied
worker_thread -> API_Server : Create ResourceBundleState CR
loop monitoring for lifetime of ResourceBundleStateCR
monitor <- cluster_etcd : watches for changes on labeled resources
monitor -> cluster_etcd : updates associated ResourceBundleStateCR
cluster_watcher <- cluster_etcd : watches for changes on ResourceBundleStateCR
cluster_watcher -> etcd : Update app/cluster Status object in AppContext
end
Admin -> status_api : GET Deployment Intent\nGroup Status
status_api -> mongo : get Deployment Intent Group
status_api -> etcd : access AppContext State\n(from Deployment Intent Group state
loop for all requested apps/clusters/resources
status_api -> etcd : access AppContext Cluster Status
status_api -> etcd : access AppContext Resource State
end
status_api -> status_api : prepare response
status_api -> Admin : return Status query results
@enduml |
Status Query
The status query, and variations with query parameters, on an EMCO resource will present the information described previously to the caller. The basic status query for the two EMCO resources discussed above will look like the following:
URL: GET /v2/cluster-providers/{cluster-provider-name}/clusters/{cluster-name}/status
URL: GET /v2/projects/{project-name}/composite-apps/{composite-app-name}/{version}/deployment-intent-groups/{deployment-intent-group-name}/status
The following query parameters will be available:
type=<rsync | cluster>
default type is 'rsync'
rsync: gathers status based on the rsync resources.
cluster: gathers status based on cluster resource information received in the ResourceBundleState CRs received from the cluster(s)
output=<summary | all | detail >
default output value is: 'all'
summary: will just show the top level EMCO resource state and status along with aggregated resource statuses but no resource detail information
any filters added will affect the aggregated resource status results, although resource details will not be displayed
all: will include a list of resources, organized by App and Cluster, showing basic resource identification (Group Version Kind) and resource statuses
detail: includes in the resource list the metadata, spec, and status of the resource as received in the ResourceBundleState CR
The following query parameters filter the results returned. Aggregated status results at the top level are relative to the filter parameters supplied
app=<appname>
default is all apps
This will filter the results of the query to show results only for the resources of the specified App(s).
cluster=<cluster>
default is all clusters
This will filter the results of the query to show results only for the specified cluster(s)
resource=<resource name>
default is all resources
This will filter the results of the query to show results only for the specified resource(s)
Output of Status Query
The following table shows the essential structure of the status query response with a description of which elements are present based on the 'output' parameter.
Format of the status query response | Description | 'summary' query | 'all' query | 'detail' query | 'rsync' query |
{ | |||||
"name": "<name>", | The name of the Deployment Intent Group or Cluster | X | X | X | X |
"project": "<project name>", | Present for the Deployment Intent Group | X | X | X | X |
"composite-app-name": "<composite-app-name>", | Present for the Deployment Intent Group | X | X | X | X |
"composite-app-version": "<composite-app-version>", | Present for the Deployment Intent Group | X | X | X | X |
"composite-profile-name": "<composite-profile-name>", | Present for the Deployment Intent Group | X | X | X | X |
"state": "[ Created, Approved, Instantiated, Terminated ]", | 'state' is the action made by the user - the format of this field is the list of stateInfo (with state, instance id, time) as show above | X | X | X | X |
"rsync-status": { | 'rsync-status' is the aggregated rsync-status of the resources - subject to query parameter filters | X | X | X | X |
"Pending": 0, | elements with zero can be dropped | X | X | X | X |
"Applied": 5, | X | X | X | X | |
"Failed": 2, | X | X | X | X | |
"Retrying": 3, | X | X | X | X | |
"Terminated": 0 | X | X | X | X | |
}, | X | X | X | X | |
"cluster-status" : { | 'cluster-status' is the aggregated cluster-status of the resources - subject to query parameter filters | X | X | X | |
"NotPresent": 0, | X | X | X | ||
"Present": 5, | Note: current implementation only uses the 'Present' status | X | X | X | |
"Unknown": 5 | X | X | X | ||
}, | X | X | X | ||
"resources": [ | array of resources organized by app in the composite-app | X | X | X | |
{ | X | X | X | ||
"app-name": "collectd", | X | X | X | ||
"clusters": [ | array of clusters in the app | X | X | X | |
{ | X | X | X | ||
"name": "cluster1", | X | X | X | ||
"resources": [ | array of resources in the cluster | X | X | X | |
{ | X | X | X | ||
"GVK": { | The GVK will come from the AppContext except for cases where the resource is only a cluster resource - e.g. an rsync resource that is a Deployment can result in cluster resources of both Deployment and Pod(s). | X | X | X | |
"Group": "<group>", | X | X | X | ||
"Version": "<version>", | X | X | X | ||
"Kind": "<kind>" | X | X | X | ||
}, | X | X | X | ||
"Name": "<resource name>", | The name of the resource from the AppContext (or cluster as described for GVK) | X | X | X | |
"metadata": { }, | The 'metadata' element of the cluster resource as received in the ResourceBundleState CR. In the case of an 'output=rsync' query, this will be the 'metadata' element from the AppContext. | X | X | ||
"spec": { }, | The 'spec' element of the cluster resource as received in the ResourceBundleState CR. In the case of an 'output=rsync' query, this will be the 'metadata' element from the AppContext. | X | X | ||
"status": { }, | The 'status' element of the k8s resource as received in the ResourceBundleState CR (if the resource has one). In the case of an 'output=rsync' query, this will not be present. | X | |||
"rsync-status": "[ Pending | Instantiated | Failed | Retrying |Terminated ]", | "Pending" - is set by orchestrator before issuing an Instantiate command to rsync "Instantiate" - means rsync has successfully invoked deployment of the resource to the cluster "Failed" - means rsync got an explicit failure when invoking to the cluster "Retrying" - means connection to cluster is temporarily unavailable, rsync will continue to retry - applies to both instantiate and terminate sequences (initial thought is to detect this condition at a cluster level - but mark each resource) "Terminated" - means rsync has successfully invoked termination of the resource to the cluster | X | X | X | |
| Currently not include in 'cluster' type queries - any resource shown is inherently 'Present'
|
|
| ||
} | X | X | X | ||
] | X | X | X | ||
} | X | X | X | ||
] | X | X | X | ||
} | X | X | X | ||
] | X | X | X | ||
} | X | X | X |
Examples of queries and outputs:
Basic default query of a composite application - ('type' is 'all')
Code Block | ||||
---|---|---|---|---|
| ||||
{
"project": "testvfw",
"composite-app-name": "compositevfw",
"composite-app-version": "v1",
"composite-profile-name": "vfw_composite-profile",
"name": "vfw_deployment_intent_group",
"state": {
"Actions": [
{
"State": "Created",
"ContextId": "",
"TimeStamp": "2020-08-21T05:36:54.186Z"
},
{
"State": "Approved",
"ContextId": "",
"TimeStamp": "2020-08-21T05:37:28.748Z"
},
{
"State": "Instantiated",
"ContextId": "2621114006130701074",
"TimeStamp": "2020-08-21T05:37:28.947Z"
},
{
"State": "Terminated",
"ContextId": "2621114006130701074",
"TimeStamp": "2020-08-21T05:38:51.061Z"
},
{
"State": "Approved",
"ContextId": "",
"TimeStamp": "2020-08-21T05:39:34.436Z"
},
{
"State": "Instantiated",
"ContextId": "2755581958183303505",
"TimeStamp": "2020-08-21T05:39:34.64Z"
}
]
},
"status": "Instantiated",
"rsync-status": {
"Applied": 12
},
"apps": [
{
"name": "packetgen",
"clusters": [
{
"cluster-provider": "vfw-cluster-provider",
"cluster": "edge01",
"resources": [
{
"GVK": {
"Group": "apps",
"Version": "v1",
"Kind": "Deployment"
},
"name": "fw0-packetgen",
"rsync-status": "Applied"
},
{
"GVK": {
"Group": "",
"Version": "v1",
"Kind": "Service"
},
"name": "packetgen-service",
"rsync-status": "Applied"
}
]
},
{
"cluster-provider": "vfw-cluster-provider",
"cluster": "edge02",
"resources": [
{
"GVK": {
"Group": "apps",
"Version": "v1",
"Kind": "Deployment"
},
"name": "fw0-packetgen",
"rsync-status": "Applied"
},
{
"GVK": {
"Group": "",
"Version": "v1",
"Kind": "Service"
},
"name": "packetgen-service",
"rsync-status": "Applied"
}
]
}
]
},
{
"name": "firewall",
"clusters": [
{
"cluster-provider": "vfw-cluster-provider",
"cluster": "edge01",
"resources": [
{
"GVK": {
"Group": "apps",
"Version": "v1",
"Kind": "Deployment"
},
"name": "fw0-firewall",
"rsync-status": "Applied"
}
]
},
{
"cluster-provider": "vfw-cluster-provider",
"cluster": "edge02",
"resources": [
{
"GVK": {
"Group": "apps",
"Version": "v1",
"Kind": "Deployment"
},
"name": "fw0-firewall",
"rsync-status": "Applied"
}
]
}
]
},
{
"name": "sink",
"clusters": [
{
"cluster-provider": "vfw-cluster-provider",
"cluster": "edge01",
"resources": [
{
"GVK": {
"Group": "apps",
"Version": "v1",
"Kind": "Deployment"
},
"name": "fw0-sink",
"rsync-status": "Applied"
},
{
"GVK": {
"Group": "",
"Version": "v1",
"Kind": "ConfigMap"
},
"name": "sink-configmap",
"rsync-status": "Applied"
},
{
"GVK": {
"Group": "",
"Version": "v1",
"Kind": "Service"
},
"name": "sink-service",
"rsync-status": "Applied"
}
]
},
{
"cluster-provider": "vfw-cluster-provider",
"cluster": "edge02",
"resources": [
{
"GVK": {
"Group": "apps",
"Version": "v1",
"Kind": "Deployment"
},
"name": "fw0-sink",
"rsync-status": "Applied"
},
{
"GVK": {
"Group": "",
"Version": "v1",
"Kind": "ConfigMap"
},
"name": "sink-configmap",
"rsync-status": "Applied"
},
{
"GVK": {
"Group": "",
"Version": "v1",
"Kind": "Service"
},
"name": "sink-service",
"rsync-status": "Applied"
}
]
}
]
}
]
} |
Query just the 'summary'
Code Block | ||||
---|---|---|---|---|
| ||||
{
"project": "testvfw",
"composite-app-name": "compositevfw",
"composite-app-version": "v1",
"composite-profile-name": "vfw_composite-profile",
"name": "vfw_deployment_intent_group",
"state": {
"Actions": [
{
"State": "Created",
"ContextId": "",
"TimeStamp": "2020-08-21T05:36:54.186Z"
},
{
"State": "Approved",
"ContextId": "",
"TimeStamp": "2020-08-21T05:37:28.748Z"
},
{
"State": "Instantiated",
"ContextId": "2621114006130701074",
"TimeStamp": "2020-08-21T05:37:28.947Z"
},
{
"State": "Terminated",
"ContextId": "2621114006130701074",
"TimeStamp": "2020-08-21T05:38:51.061Z"
},
{
"State": "Approved",
"ContextId": "",
"TimeStamp": "2020-08-21T05:39:34.436Z"
},
{
"State": "Instantiated",
"ContextId": "2755581958183303505",
"TimeStamp": "2020-08-21T05:39:34.64Z"
}
]
},
"status": "Instantiated",
"rsync-status": {
"Applied": 12
}
} |
Query with type=cluster, output=detail, app=sink, resource=sink-configmap
Code Block | ||||
---|---|---|---|---|
| ||||
{
"project": "testvfw",
"composite-app-name": "compositevfw",
"composite-app-version": "v1",
"composite-profile-name": "vfw_composite-profile",
"name": "vfw_deployment_intent_group",
"state": {
"Actions": [
{
"State": "Created",
"ContextId": "",
"TimeStamp": "2020-08-21T05:36:54.186Z"
},
{
"State": "Approved",
"ContextId": "",
"TimeStamp": "2020-08-21T05:37:28.748Z"
},
{
"State": "Instantiated",
"ContextId": "2621114006130701074",
"TimeStamp": "2020-08-21T05:37:28.947Z"
},
{
"State": "Terminated",
"ContextId": "2621114006130701074",
"TimeStamp": "2020-08-21T05:38:51.061Z"
},
{
"State": "Approved",
"ContextId": "",
"TimeStamp": "2020-08-21T05:39:34.436Z"
},
{
"State": "Instantiated",
"ContextId": "2755581958183303505",
"TimeStamp": "2020-08-21T05:39:34.64Z"
}
]
},
"status": "Instantiated",
"cluster-status": {
"Present": 2
},
"apps": [
{
"name": "sink",
"clusters": [
{
"cluster-provider": "vfw-cluster-provider",
"cluster": "edge01",
"resources": [
{
"GVK": {
"Group": "",
"Version": "v1",
"Kind": "ConfigMap"
},
"name": "sink-configmap",
"detail": {
"kind": "ConfigMap",
"apiVersion": "v1",
"metadata": {
"name": "sink-configmap",
"namespace": "default",
"selfLink": "/api/v1/namespaces/default/configmaps/sink-configmap",
"uid": "07f9b01f-c26d-462a-8eee-8e4f4553dda5",
"resourceVersion": "152135",
"creationTimestamp": "2020-08-21T05:39:34Z",
"labels": {
"emco/deployment-id": "2755581958183303505-sink"
},
"annotations": {
"kubectl.kubernetes.io/last-applied-configuration": "{\"apiVersion\":\"v1\",\"data\":{\"protected_net_gw\":\"192.168.20.100\",\"protected_private_net_cidr\":\"192.168.10.0/24\"},\"kind\":\"ConfigMap\",\"metadata\":{\"annotations\":{},\"labels\":{\"emco
/deployment-id\":\"2755581958183303505-sink\"},\"name\":\"sink-configmap\",\"namespace\":\"default\"}}\n"
}
}
}
}
]
},
{
"cluster-provider": "vfw-cluster-provider",
"cluster": "edge02",
"resources": [
{
"GVK": {
"Group": "",
"Version": "v1",
"Kind": "ConfigMap"
},
"name": "sink-configmap",
"detail": {
"kind": "ConfigMap",
"apiVersion": "v1",
"metadata": {
"name": "sink-configmap",
"namespace": "default",
"selfLink": "/api/v1/namespaces/default/configmaps/sink-configmap",
"uid": "fb9d25ed-5e51-4450-b0d4-7ceb1ec5daae",
"resourceVersion": "120014",
"creationTimestamp": "2020-08-21T05:39:35Z",
"labels": {
"emco/deployment-id": "2755581958183303505-sink"
},
"annotations": {
"kubectl.kubernetes.io/last-applied-configuration": "{\"apiVersion\":\"v1\",\"data\":{\"protected_net_gw\":\"192.168.20.100\",\"protected_private_net_cidr\":\"192.168.10.0/24\"},\"kind\":\"ConfigMap\",\"metadata\":{\"annotations\":{},\"labels\":{\"emco
/deployment-id\":\"2755581958183303505-sink\"},\"name\":\"sink-configmap\",\"namespace\":\"default\"}}\n"
}
}
}
}
]
}
]
}
]
} |
Query instance, multiple resources
Code Block | ||||
---|---|---|---|---|
| ||||
{
"project": "testvfw",
"composite-app-name": "compositevfw",
"composite-app-version": "v1",
"composite-profile-name": "vfw_composite-profile",
"name": "vfw_deployment_intent_group",
"state": {
"Actions": [
{
"State": "Created",
"ContextId": "",
"TimeStamp": "2020-08-21T05:36:54.186Z"
},
{
"State": "Approved",
"ContextId": "",
"TimeStamp": "2020-08-21T05:37:28.748Z"
},
{
"State": "Instantiated",
"ContextId": "2621114006130701074",
"TimeStamp": "2020-08-21T05:37:28.947Z"
},
{
"State": "Terminated",
"ContextId": "2621114006130701074",
"TimeStamp": "2020-08-21T05:38:51.061Z"
},
{
"State": "Approved",
"ContextId": "",
"TimeStamp": "2020-08-21T05:39:34.436Z"
},
{
"State": "Instantiated",
"ContextId": "2755581958183303505",
"TimeStamp": "2020-08-21T05:39:34.64Z"
}
]
},
"status": "Terminated",
"rsync-status": {
"Deleted": 4
},
"apps": [
{
"name": "packetgen",
"clusters": [
{
"cluster-provider": "vfw-cluster-provider",
"cluster": "edge01",
"resources": [
{
"GVK": {
"Group": "apps",
"Version": "v1",
"Kind": "Deployment"
},
"name": "fw0-packetgen",
"rsync-status": "Deleted"
}
]
},
{
"cluster-provider": "vfw-cluster-provider",
"cluster": "edge02",
"resources": [
{
"GVK": {
"Group": "apps",
"Version": "v1",
"Kind": "Deployment"
},
"name": "fw0-packetgen",
"rsync-status": "Deleted"
}
]
}
]
},
{
"name": "sink",
"clusters": [
{
"cluster-provider": "vfw-cluster-provider",
"cluster": "edge01",
"resources": [
{
"GVK": {
"Group": "",
"Version": "v1",
"Kind": "ConfigMap"
},
"name": "sink-configmap",
"rsync-status": "Deleted"
}
]
},
{
"cluster-provider": "vfw-cluster-provider",
"cluster": "edge02",
"resources": [
{
"GVK": {
"Group": "",
"Version": "v1",
"Kind": "ConfigMap"
},
"name": "sink-configmap",
"rsync-status": "Deleted"
}
]
}
]
}
]
} |
Query by cluster - note, the cluster is provided as <cluster provider>+<cluster> - where the '+' character is sent in the URL as '%2B'
...
"cluster-status": "[ Unknown | NotPresent | Present | (tbd) ]"
...
Summary status for the resource from info obtained from the cluster (e.g. via the ResourceBundleState CR)
"Unknown" - means a ResourceBundleState has not yet been received or the resource type is not supported in the ResourceBundleState CR
"NotPresent" - the resource is not in the ResourceBundleState CR and it is supported.
"Present" - the resource is present in the ResourceBundleState CR
"tbd" - further status values can be derived as analysis of how to represent the full status {} object from the resource can be interpreted - e.g. 'Ready', 'Failed', 'Pending', etc.
...
X
...
X
...
}
...
X
...
X
...
X
...
]
...
X
...
X
...
X
...
}
...
X
...
X
...
X
...
]
...
X
...
X
...
X
...
}
...
X
...
X
...
X
...
]
...
X
...
X
...
X
...
}
...
X
...
X
...
X
Examples of queries and outputs:
Basic default query of a composite application - ('type' is 'all')
Code Block | |||||||
---|---|---|---|---|---|---|---|
| |||||||
{ { "name": "dig-xyz", "project": "myprojecttestvfw", "composite-app-name": "composite-app-xyzcompositevfw", "composite-app-version": "v1.0", "state": "Instantiated", "rsync-status": { "Instantiated": 4 }, "cluster-status" : { "Present": 4 }, "resources": [ { "app-name": "app1", "clusters": [ { composite-profile-name": "vfw_composite-profile", "name": "cluster1vfw_deployment_intent_group", "resourcesstate": [{ { "GVKActions": [ { "GroupState": "Created", "VersionContextId": "v1", "KindTimeStamp": "Service" 2020-08-21T18:17:03.503Z" }, "Name": "mongo", "rsync-status": "Instantiated", "cluster-status": "Present" } ] }, { "name { "State": "cluster2Approved", "resources": [ { "GVK": { "Group "ContextId": "", "VersionTimeStamp": "v1", "Kind2020-08-21T18:17:18.037Z" }, { "State": "ServiceInstantiated" }, "Name, "ContextId": "mongo3983785381246146818", "rsync-statusTimeStamp": "Instantiated", "cluster-status": "Present" } ] } ] 2020-08-21T18:17:18.275Z" } ] }, { "app-namestatus": "app2Instantiated", "clustersrsync-status": [{ { "nameApplied": "cluster1" 6 }, "resourcesapps": [ { "GVK": { "Group "name": "packetgen", "Versionclusters": "v1", "Kind": "Pod" }, "Name": "app2pod", "rsync-status": "Instantiated", "cluster-status": "Present" } ] }, { "name": "cluster2", [ { "cluster-provider": "vfw-cluster-provider", "cluster": "edge02", "resources": [ { "GVK": { "Group": "", "Version": "v1", "Kind": "Deployment" }, "Name "Group": "app2deploymentapps", "rsync-status "Version": "Instantiatedv1", "cluster-status "Kind": "PresentDeployment" } ] } ] } ] } |
Query by cluster2
Code Block | ||||
---|---|---|---|---|
| ||||
{ }, "name": "digfw0-xyzpacketgen", "projectrsync-status": "myproject", "composite-app-name": "composite-app-xyz", "composite-app-versionApplied" }, { "GVK": { "Group": "", "Version": "v1.0", "stateKind": "InstantiatedService", "rsync-status": { "Instantiated": 2 }, "cluster-status" : { "Present": 2 }, "resources": [ { "app- "name": "app1packetgen-service", "clusters": [ { "name "rsync-status": "cluster2Applied", "resources": [ { "GVK": { "Group": "", "Version": "v1", "Kind": "Service" }, "Name": "mongo", "rsync-status": "Instantiated", "cluster-status": "Present" } ] } ] }, { "app-name": "app2", "clusters": [ { "name": "cluster2", } ] } ] }, { "name": "firewall", "clusters": [ { "cluster-provider": "vfw-cluster-provider", "cluster": "edge02", "resources": [ { "GVK": { "Group": "", "Version "Group": "v1apps", "Kind": "Deployment" }, "Name": "app2deployment", "rsync-status": "Instantiated", "cluster-status": "Present" } ] } ] } ] } |
Query by app2
Code Block | ||||
---|---|---|---|---|
| ||||
{ "name "Version": "dig-xyzv1", "projectKind": "myproject", "composite-app-Deployment" }, "name": "compositefw0-app-xyzfirewall", "compositersync-app-versionstatus": "v1.0", "state": "Instantiated", "rsync-status": { "Instantiated": 2 }, "cluster-status" : { "Present": 2 }, "resources": [ { "app-name": "app2", "clusters": [ { "name": "cluster1", Applied" } ] } ] }, { "name": "sink", "clusters": [ { "cluster-provider": "vfw-cluster-provider", "cluster": "edge02", "resources": [ { "GVK": { "Group": "apps", "Version": "v1", "Kind": "PodDeployment" }, "Namename": "app2pod", fw0-sink", "rsync-status": "InstantiatedApplied", "cluster-status": "Present" } ] }, { "name": "cluster2", "resources": [ { "GVK": { "Group": "", "Version": "v1", "Kind": "Deployment" }, "Name": "app2deployment", "rsync-status "Version": "Instantiatedv1", "cluster-status "Kind": "PresentConfigMap" } ] } ] } ] } |
Query by cluster2 and app2
Code Block | ||||
---|---|---|---|---|
| ||||
{ "name": "dig-xyz", "project": "myproject", "composite-app- }, "name": "compositesink-app-xyzconfigmap", "compositersync-app-versionstatus": "v1.0Applied", "state": "Instantiated", "rsync-status": { "Instantiated": 1 }, "cluster-status" : { "Present": 1 }, "resources": [ { "app-name": "app2", "clusters": [ { "name": "cluster2", "resources": [ { "GVK": { "Group": "", "Version": "v1", "Kind": "Deployment" }, "Name": "app2deployment", }, { "GVK": { "Group": "", "Version": "v1", "Kind": "Service" }, "name": "sink-service", "rsync-status": "InstantiatedApplied", "cluster-status": "Present" } ] } ] } ] } |
Concept for query of historical instances EMCO resources
Once an EMCO resource has been instantiated and terminated. It may be instantiated again. On termination, it is possible to save the AppContext of the old instance so that it can be
}
]
}
]
}
]
} |
Example of a cluster network intents status query - all the same query parameters apply.
Note - if the type=cluster query is made, the ResourceBundleState CR currently does not support the Network and ProviderNetwork resources, so no resources will be foundQuery of a previous composite application instance (no instance value specified will list all instances)
Code Block | ||||
---|---|---|---|---|
| ||||
{
"name": "dig-xyz",
"project": "myproject",
"composite-app-name": "composite-app-xyz",
"composite-app-version": "v1",
"instances": [
"8369380046269976283",
"5693804306826847621",
"3393800466288997627",
"2369300486868279620"
]
} |
Query of a previous composite application instance (other query parameters would apply as shown above)
| ||||
{
"name": "vfw-cluster-provider+edge01",
"state": {
"Actions": [
{
"State": "Created",
"ContextId": "",
"TimeStamp": "2020-08-21T18:17:02.464Z"
},
{
"State": "Applied",
"ContextId": "9116473999070524329",
"TimeStamp": "2020-08-21T18:17:11.983Z"
}
]
},
"status": "Instantiated",
"rsync-status": {
"Applied": 3
},
"apps": [
{
"name": "network-intents",
"clusters": [
{
"cluster-provider": "vfw-cluster-provider",
"cluster": "edge01",
"resources": [
{
"GVK": {
"Group": "k8s.plugin.opnfv.org",
"Version": "v1alpha1",
"Kind": "ProviderNetwork"
},
"name": "emco-private-net",
| ||||
Code Block | ||||
---|---|---|---|---|
| ||||
{ "name": "dig-xyz", "project": "myproject", "composite-app-name": "composite-app-xyz", "composite-app-version": "v1", "state": "Instantiated", "rsync-status": { "Terminated": 4 }, "cluster-status" : { "Unknown": 4 }, "resources": [ { "app-name": "app1", "clusters": [ { "name": "cluster1", "resources": [ { "GVK": { "Group": "", "Version": "v1", "Kind": "Service" }, "Name": "mongo", "rsync-status": "Terminated", "cluster-status": "Unknown" } ] }, { "name": "cluster2", "resources": [ { Applied" }, { "GVK": { "Group": "k8s.plugin.opnfv.org", "Version": "v1v1alpha1", "Kind": "Service" }, "NameNetwork" }, "name": "mongoprotected-private-net", "rsync-status": "TerminatedApplied", "cluster-status": "Unknown" } ] } ] }, { "app-name": "app2", "clusters": [ { "name": "cluster1", "resources": [ { "GVK": { }, { "GVK": { "Group": "k8s.plugin.opnfv.org", "Version": "v1", "Kind": "Pod" }, "Name": "app2pod", "rsync-status": "Terminated", "cluster-status": "Unknown" } ] }, { "nameVersion": "cluster2v1alpha1", "resources": [ { "GVK": { "Group": "", "Version": "v1", "Kind": "DeploymentProviderNetwork" }, "Namename": "app2deploymentunprotected-private-net", "rsync-status": "TerminatedApplied", "cluster-status": "Unknown" } ] } ] } } ] } ] } ] } |