The idea of using control loops to automatically (or autonomously) perform network management has been the subject of much research in the Network Management research community, see this paper for some background. However, it is only with the advent of ONAP that we have a platform that supports control loops for network management. Before ONAP, Control Loops have been implemented by hard-coding components together and hard coding logic into components. ONAP has taken a step forward towards automatic implementation of Control Loops by allowing parameterization of Control Loops that work on the premise that the Control Loops use a set of analytic, policy, and control components connected together in set ways.

The goal of the work is to extend and enhance the current ONAP Control Loop support to provide a complete open-source framework for Control Loops. This will enhance the current support to provide TOSCA based Control Loop definition and development, commissioning and run-time management. The participants that comprise a Control Loop and the metadata needed to link the participants together to create a Control Loop are specified in a standardized way using the OASIS TOSCA modelling language. The TOSCA description is then used to commission, instantiate, and manage the Control Loops in the run time system.

1. Terminology

Control Loop: A set of executing elements on which Life Cycle Management (LCM) is executed collectively. For example, a set of microservices may be spawned and executed together to deliver a service. This collection of services is a control loop

Control Loop Element: A single executing entitity, with its Life Cycle being managed as part of the overall control loop. For example, a single microservice that is executing as one microservice in a service.

Participant: A system or framework that runs Control Loop Elements. A participant chooses to partake in control loops, to manage Control Loop Elements for CLAMP, and to receive, send and act on LCM messages for the CLAMP runtime.

Control Loop Type: A definition of a Control Loop in the TOSCA language. This definition describes a certain type of a control loop. The life cycle of instances of a Control Loop Type are managed by CLAMP.

Control Loop Instance: An instance of a Control Loop Type. The life cycle of a Control Loop Instance is managed by CLAMP.

Property: Metadata defined in TOSCA that is associated with a Control Loop, a Control Loop Element, or a Participant.

Property Type: The TOSCA defintion of the type of a property. A property can have a generic type such as string or integer or can have a user defined TOSCA data type.

Property Value: The value of a Property Type. Property values are assigned at run time in CLAMP.

Common Property Type: Property Types that apply to all instances of a Control Loop Type.

Common Property Value: The value of a Property Type. It is assigned at run time once for all instances of a Control Loop Type.

Instance Specific Property Type: Property Types that apply to an individual instance of a Control Loop Type.

Instance Specific Property Value: The value of a Property Type that applies to an individual instance of a Control Loop Type. The value is assigned at run time for each control loop instance.

2. Capabilities

We consider the capabilities of Control Loops at Design Time and Run Time.

At Design Time, two capabilities are supported:

1. Participant Metadata Definition. This capability allows external users and systems (such as SDC or DCAE-MOD) to define participants that can take part in a control loop and to define the metadata that can be used on and configured on a participant when it is taking part in in a control loop. The post condition of an execution of this capability is that a participant is defined in the Control Loop Design Time Catalogue together with sets of metadata that can be used with this participant in control loops.
2. Control Loop Composition. This capability allows users and other systems to create Control Loop Type definitions by connecting a chain of participants together from the participants that are available in the Control Loop Design Time Catalogue. In an execution of this capability, a user defines the control loop chain for the Control Loop Type and defines the connections between participants. The user also selects the correct metadata sets for each participant in the Control Loop Type and defines the overall Control Loop Type metadata. The user also specifies the Common Property Types that apply to all instances of a control loop type and the Instance Specific Property Types that apply to individual instances of a Control Loop Type. The post condition for an execution of this capability is a Control Loop definition in TOSCA stored in the Control Loop Design Time Catalogue.

At Run Time, the following capabilities are supported:

1. Participant Registration. This capability allows participants to register and deregister with CLAMP. The post condition for an execution of this capability is that a participant is available for participation in a control loop. Participants can explicitly register with CLAMP at any point after they start, or they can implicitly register when they respond to a control loop initiation request.
2. Control Loop Commissioning. This capability allows version controlled Control Loop Type definitions to be taken from the Control Loop Design Time Catalogue and be placed in the Control Loop Run Time Inventory. It also allows the values of Common Property Types that apply to all instances of a Control Loop Type to be set. The post condition for an execution of this capability is that the Control Loop definition is in the Control Loop Run Time Inventory.
3. Control Loop Priming on Participants. A participant is primed to support a Control Loop Type. The definition of a control loop and the values of Common Property Types that apply to all instances of a control loop type on a participant is sent to a participant. The participant can then take whatever actions it need to do to support the control loop type in question. Control Loop Priming can take place explicitly as a separate operation on participants or can be done implicitly in the instantiation message for the first instance of a Control Loop Type. The post condition for an execution of this capability is that all participants in this control loop type are commissioned, that is they are prepared to run instances of this control loop type.
4. Control Loop Life Cycle Management. This capability allows a Control Loop Instance to be created. The Control Loop Type definition is read from the Control Loop Run Time Inventory and values are assigned to the Instance Specific Property Types defined for instances of the Control Loop Type in the same manner as the existing CLAMP client does. A Control Loop Instance that has been created but has not yet been instantiated on participants is in state UNINITIALIZED. In this state, the Instance Specific Property Type values can be revised and updated as often as the user requires. Once the user is happy with the property values, the Control Loop Instance is instantiated on participants and the Control Loop Elements for this Control Loop Instance are started by participants using the control loop metadata. Once the Control Loop Instance is instantiated on each participant, the Control Loop Instance is set as being in state PASSIVE in the Control Loop Run Time Inventory. The user can now order the participants to change the state of the Control Loop Instance to state RUNNING. Each participant begins accepting and processing control loop events and the Control Loop Instance is set to state RUNNING in the control loop inventory. The post condition for an execution of this capability is that the Control Loop instance is running on participants and is processing events.
5. Control Loop Monitoring. This capability allows Control Loop Instances to be monitored. Users can check the status of Participants, Control Loop Instances, and Control Loop Elements. Participants report their overall status and the status of Control Loop Elements they are running periodically to CLAMP. Clamp aggregates these status reports into an aggregated Control Loop Instance status record, which is available for monitoring. The post condition for an execution of this capability is that Control Loop Instances are being monitored.

When a control loop definition has been commissioned, instances of the control loop can be created, updated, and deleted. The system manages the lifecycle of control loops and control loop elements following the state transition diagram below.

3: Overall Architecture

The diagram below shows an overview of the architecture of TOSCA based Control Loop Management in CLAMP.

Following the ONAP Reference Architecture, the architecture has a Design Time part and a Runtime part.

The Design Time part of the archtiecture allows a user to specify metadata for participants. It also allows users to compose control loops. The Design Time Catalogue contains the metadata primitives and control loop definition primitives for composition of control loops. As shown in the figure above, the Design Time component provides a system where Control Loops can be designed and defined in metadata. This means that a Control Loop can have any arbitrary structure and the Control Loop developers can use whatever analytic, policy, or control participants they like to implement their Control Loop. At composition time, the user parameterises the Control Loop and stores it in the design time catalogue. This catalogue contains the primitive metadata for any participants that can be used to compose a Control Loop. A Control Loop SDK is used to compose a Control Loop by aggregating the metadata for the participants chosen to be used in a Control Loop and by constructing the references between the participants. The architecture of the Control Loop Design Time part will be elaborated in future releases.

Composed Control Loops are commissioned on the run time part of the system, where they are stored in the run time inventory and are available for instantiation. The Commissioning component provides a CRUD REST interface for Control Loop Types, and implements CRUD of Control Loop Types. Commissioning also implements validation and persistence of incoming Control Loop Types. It also guarantees the integrity of updates and deletions of Control Loop Types, such as performing updates accordance with semantic versioning rules and ensuring that deletions are not allowed on Control Loop Types that have instances defined.

The Instantiation component manages the Life Cycle Management of Control Loop Instances and their Control Loop Elements. It publishes a REST interface that is used to create Control Loop Instances and set values for Common and Instance Specific properties. This REST interface is public and is used by the CLAMP GUI. It may also be used by any other client via the public REST interface. the REST interface also allows the state of Control Loop Instances to be changed. A user can change the state of Control Loop Instances as described in the state transition diagram shown in section 2 above. The Instantiation component also implements persistence of Control Loop Instances and their state changes.

When a user wishes to instantiate a Control Loop, they set values for the parameters of the Control Loop. Once the parameterization has been carried out, the Control Loop instantiated, with the metadata and whatever other artifacts are required being passed to the participants in the Control Loop. At runtime, the Control Loop can be monitored and analysed. It can also be updated  as required and can be deleted when it is on longer needed.

1.1: Class Diagrams

1.1.1 Design Time

1.1.2 Runtime 

Comment

1.2: ERD

2: Control Loop Modelling

Joseph O'Leary to pad out this section

A Control Loop service template is made up of several components, those which represent applications, those which represent dynamic config schemas, and the actual node_templates which makes up the loop itself.

Applications can be a DCAE microservice, an operational policy, or any other application as long as it can be modeled, and the targeted ecosystem to has a participant client waiting for the event distributions from CLAMP via DMaaP Message Router.

Dynamic config on the other hand can be a monitoring policy, or any other resource that provides config to parts of the loop, can be updated after the run time phase has started and is supported by the components hosting the applications in the control loop.

2.1: Control Loop TOSCA file definition

2.1.1 Control Loop Component Definition 

A Control Loop Component that can be part of a control loop, it defines the components that partake in a control loop, and are implemented at run time by participants. The control loop component definition is truly dynamic and, as long as the participant that the control loop component definition relates to understands its definition, it can be anything. However, we have designed a base control loop component attribute that's generic and that can act as a good starting point.

node_types:
  org.onap.CL_Component:
    properties:
      component_name:
        type: string
        description: Human readable name for the component.
        required: true
      provider:
        type: string
        description: Provider of the component and of the descriptor.
        required: true
      component_version:
        type: string
        description: Software version of the component.
        required: true
      resource_id:
        type: string
        description: >The ID of the resource, 
          should be provided if the resource was uploaded to the entity's inventory already.
        required: false
      resource_content:
        type: string
        description: the contents of the component resource, to be uploaded during commssioning phase of loop.
        required: false
      monitoring_policy:
        type: string
        description: A reference to the monitoring policy if applicable.
        required: false
    version: 0.0.1
    derived_from: tosca.nodes.Root

2.1.2 Loop Definition

The loop definition is explicit in the node_templates within the topology_template,  a Control Loop node template is specified and any node tempalte specified in the Control Loop node tepolcate is part of the control loop managed by CLAMP.

tosca_definitions_version: tosca_simple_yaml_1_3
data_types:
  onap.datatypes.ToscaConceptIdentifier:
    derived_from: tosca.datatypes.Root
    properties:
      name:
        type: string
        required: true
      version:
        type: string
        required: true
node_types:
  org.onap.policy.clamp.controlloop.Participant:
    version: 1.0.1
    derived_from: tosca.nodetypes.Root
    properties:
      provider:
        type: string
        requred: false
  org.onap.policy.clamp.controlloop.ControlLoopElement:
    version: 1.0.1
    derived_from: tosca.nodetypes.Root
    properties:
      provider:
        type: string
        requred: false
      participant_id:
        type: onap.datatypes.ToscaConceptIdentifier
        requred: true
  org.onap.policy.clamp.controlloop.ControlLoop:
    version: 1.0.1
    derived_from: tosca.nodetypes.Root
    properties:
      provider:
        type: string
        requred: false
      elements:
        type: list
        required: true
        entry_schema:
          type: onap.datatypes.ToscaConceptIdentifier
  org.onap.policy.clamp.controlloop.DCAEMicroserviceControlLoopElement:
    version: 1.0.1
    derived_from: org.onap.policy.clamp.controlloop.ControlLoopElement
    properties:
      dcae_blueprint_id:
        type: onap.datatypes.ToscaConceptIdentifier
        requred: true
  org.onap.policy.clamp.controlloop.PolicyTypeControlLoopElement:
    version: 1.0.1
    derived_from: org.onap.policy.clamp.controlloop.ControlLoopElement
    properties:
      policy_type_id:
        type: onap.datatypes.ToscaConceptIdentifier
        requred: true
  org.onap.policy.clamp.controlloop.CDSControlLoopElement:
    version: 1.0.1
    derived_from: org.onap.policy.clamp.controlloop.ControlLoopElement
    properties:
      cds_blueprint_id:
        type: onap.datatypes.ToscaConceptIdentifier
        requred: true
topology_template:
  node_templates:
    org.onap.dcae.controlloop.DCAEMicroserviceControlLoopParticipant:
      version: 2.3.4
      type: org.onap.policy.clamp.controlloop.Participant
      type_version: 1.0.1
      description: Participant for DCAE microservices
      properties:
        provider: ONAP
    org.onap.policy.controlloop.MonitoringPolicyControlLoopParticipant:
      version: 2.3.1
      type: org.onap.policy.clamp.controlloop.Participant
      type_version: 1.0.1
      description: Participant for DCAE microservices
      properties:
        provider: ONAP
    org.onap.policy.controlloop.OperationalPolicyControlLoopParticipant:
      version: 3.2.1
      type: org.onap.policy.clamp.controlloop.Participant
      type_version: 1.0.1
      description: Participant for DCAE microservices
      properties:
        provider: ONAP
    org.onap.ccsdk.cds.controlloop.CdsControlLoopParticipant:
      version: 2.2.1
      type: org.onap.policy.clamp.controlloop.Participant
      type_version: 1.0.1
      description: Participant for DCAE microservices
      properties:
        provider: ONAP
    org.onap.domain.pmsh.PMSH_DCAEMicroservice:
      version: 1.2.3
      type: org.onap.policy.clamp.controlloop.DCAEMicroserviceControlLoopElement
      type_version: 1.0.0
      description: Control loop element for the DCAE microservice for Performance Management Subscription Handling
      properties:
        provider: Ericsson
        participant_id:
          name: org.onap.dcae.controlloop.DCAEMicroserviceControlLoopParticipant
          version: 2.3.4
        dcae_blueprint_id:
          name: org.onap.dcae.blueprints.PMSHBlueprint
          version: 1.0.0
    org.onap.domain.pmsh.PMSH_MonitoringPolicyControlLoopElement:
      version: 1.2.3
      type: org.onap.policy.clamp.controlloop.PolicyTypeControlLoopElement
      type_version: 1.0.0
      description: Control loop element for the monitoring policy for Performance Management Subscription Handling
      properties:
        provider: Ericsson
        participant_id:
          name: org.onap.policy.controlloop.PolicyControlLoopParticipant
          version: 2.3.1
        policy_type_id:
          name: onap.policies.monitoring.pm-subscription-handler
          version: 1.0.0
    org.onap.domain.pmsh.PMSH_OperationalPolicyControlLoopElement:
      version: 1.2.3
      type: org.onap.policy.clamp.controlloop.PolicyTypeControlLoopElement
      type_version: 1.0.0
      description: Control loop element for the operational policy for Performance Management Subscription Handling
      properties:
        provider: Ericsson
        participant_id:
          name: org.onap.policy.controlloop.PolicyControlLoopParticipant
          version: 2.3.1
        policy_type_id:
          name: onap.policies.operational.pm-subscription-handler
          version: 1.0.0
    org.onap.domain.pmsh.PMSH_CDS_ControlLoopElement:
      version: 1.2.3
      type: org.onap.policy.clamp.controlloop.ControlLoopElement
      type_version: 1.0.0
      description: Control loop element for CDS for Performance Management Subscription Handling
      properties:
        provider: Ericsson
        participant_Id:
          name: org.onap.ccsdk.cds.controlloop.CdsControlLoopParticipant
          version: 3.2.1
        cds_blueprint_id:
          name: org.onap.ccsdk.cds.PMSHCdsBlueprint
          version: 1.0.0
    org.onap.domain.pmsh.PMSHControlLoopDefinition:
      version: 1.2.3
      type: org.onap.policy.clamp.controlloop.ControlLoop
      type_version: 1.0.0
      description: Control loop for Performance Management Subscription Handling
      properties:
        provider: Ericsson
        elements:
        - name: org.onap.domain.pmsh.PMSH_DCAEMicroservice
          version: 1.2.3
        - name: org.onap.domain.pmsh.PMSH_MonitoringPolicyControlLoopElement
          version: 1.2.3
        - name: org.onap.domain.pmsh.PMSH_OperationalPolicyControlLoopElement
          version: 1.2.3
        - name: org.onap.domain.pmsh.PMSH_CDS_ControlLoopElement
          version: 1.2.3

2.1.3 Example of two Control Loop instances

{
    "controlLoops": [
        {
            "name": "PMSHInstance0",
            "version": "1.0.1",
            "definition": {
                "name": "org.onap.domain.pmsh.PMSHControlLoopDefinition",
                "version": "1.0.0"
            },
            "state": "UNINITIALISED",
            "orderedState": "UNINITIALISED",
            "description": "PMSH control loop instance 0",
            "elements": [
                {
                    "id": "709c62b3-8918-41b9-a747-d21eb79c6c20",
                    "definition": {
                        "name": "org.onap.domain.pmsh.PMSH_DCAEMicroservice",
                        "version": "1.2.3"
                    },
                    "participantId": {
                        "name": "DCAEParticipant0",
                        "version": "1.0.0"
                    },
                    "state": "UNINITIALISED",
                    "orderedState": "UNINITIALISED",
                    "description": "DCAE Control Loop Element for the PMSH instance 0 control loop"
                },
                {
                    "id": "709c62b3-8918-41b9-a747-d21eb79c6c21",
                    "definition": {
                        "name": "org.onap.domain.pmsh.PMSH_MonitoringPolicyControlLoopElement",
                        "version": "1.2.3"
                    },
                    "participantId": {
                        "name": "PolicyParticipant0",
                        "version": "1.0.0"
                    },
                    "state": "UNINITIALISED",
                    "orderedState": "UNINITIALISED",
                    "description": "Monitoring Policy Control Loop Element for the PMSH instance 0 control loop"
                },
                {
                    "id": "709c62b3-8918-41b9-a747-d21eb79c6c22",
                    "definition": {
                        "name": "org.onap.domain.pmsh.PMSH_OperationalPolicyControlLoopElement",
                        "version": "1.2.3"
                    },
                    "participantId": {
                        "name": "PolicyParticipant0",
                        "version": "1.0.0"
                    },
                    "state": "UNINITIALISED",
                    "orderedState": "UNINITIALISED",
                    "description": "Operational Policy Control Loop Element for the PMSH instance 0 control loop"
                },
                {
                    "id": "709c62b3-8918-41b9-a747-d21eb79c6c23",
                    "definition": {
                        "name": "org.onap.domain.pmsh.PMSH_CDS_ControlLoopElement",
                        "version": "1.2.3"
                    },
                    "participantId": {
                        "name": "CDSParticipant0",
                        "version": "1.0.0"
                    },
                    "state": "UNINITIALISED",
                    "orderedState": "UNINITIALISED",
                    "description": "CDS Control Loop Element for the PMSH instance 0 control loop"
                }
            ]
        },
        {
            "name": "PMSHInstance1",
            "version": "1.0.1",
            "definition": {
                "name": "org.onap.domain.pmsh.PMSHControlLoopDefinition",
                "version": "1.0.0"
            },
            "state": "UNINITIALISED",
            "orderedState": "UNINITIALISED",
            "description": "PMSH control loop instance 1",
            "elements": [
                {
                    "id": "709c62b3-8918-41b9-a747-e21eb79c6c24",
                    "definition": {
                        "name": "org.onap.domain.pmsh.PMSH_DCAEMicroservice",
                        "version": "1.2.3"
                    },
                    "participantId": {
                        "name": "DCAEParticipant0",
                        "version": "1.0.0"
                    },
                    "state": "UNINITIALISED",
                    "orderedState": "UNINITIALISED",
                    "description": "DCAE Control Loop Element for the PMSH instance 1 control loop"
                },
                {
                    "id": "709c62b3-8918-41b9-a747-e21eb79c6c25",
                    "definition": {
                        "name": "org.onap.domain.pmsh.PMSH_MonitoringPolicyControlLoopElement",
                        "version": "1.2.3"
                    },
                    "participantId": {
                        "name": "PolicyParticipant0",
                        "version": "1.0.0"
                    },
                    "state": "UNINITIALISED",
                    "orderedState": "UNINITIALISED",
                    "description": "Monitoring Policy Control Loop Element for the PMSH instance 1 control loop"
                },
                {
                    "id": "709c62b3-8918-41b9-a747-e21eb79c6c26",
                    "definition": {
                        "name": "org.onap.domain.pmsh.PMSH_OperationalPolicyControlLoopElement",
                        "version": "1.2.3"
                    },
                    "participantId": {
                        "name": "PolicyParticipant0",
                        "version": "1.0.0"
                    },
                    "state": "UNINITIALISED",
                    "orderedState": "UNINITIALISED",
                    "description": "Operational Policy Control Loop Element for the PMSH instance 1 control loop"
                },
                {
                    "id": "709c62b3-8918-41b9-a747-e21eb79c6c27",
                    "definition": {
                        "name": "org.onap.domain.pmsh.PMSH_CDS_ControlLoopElement",
                        "version": "1.2.3"
                    },
                    "participantId": {
                        "name": "CDSParticipant0",
                        "version": "1.0.0"
                    },
                    "state": "UNINITIALISED",
                    "orderedState": "UNINITIALISED",
                    "description": "CDS Control Loop Element for the PMSH instance 1 control loop"
                }
            ]
        }
    ]
}

2.2: Modelling from TOSCA to Commissioned Data in Run Time Inventory

2.3: Modelling from TOSCA to Instance Data Run Time Inventory

2.4: Swagger REST APIs for Control Loop

ControlLoop Runtime Swagger REST APIs:

ControlLoop_Runtime_Swagger_API.yml

Participant Swagger REST APIs:

Participant_Swagger_API.yml

3: APIs and Sequence Diagrams

3.1: Commissioning

Ajay Deep Singh to pad out this section

This section defines Commissioning/CRUD Operations that can be performed on ControlLoops.

A Client, in this case CLAMP, can perform CRUD operations or can commission ControlLoops from DesignTime to RunTime Inventory Database.

DesignTime/RunTime Catalogue/Inventory Database stores ControlLoop definitions, CRUD operations on database supported by REST Endpoints like Get, Delete, Create allowing selection of a particular ControlLoop to be addressed, below sequence diagram will help you understand flow how a client(Clamp) application can initiate Rest call for performing different operations on Database.

API_Gateway Service is for interacting to different database DesignTime/RunTime and should be responsible for responding success or failure status on different operations.

The commissioning of ControlLoops definition from DesignTime Catalogue to RunTime Inventory Database can we achived using the commissioning Rest Endpoint, in this process when a rest request is initiated from a client(Clamp) the API_Gateway Service take cares of fetching ControlLoops metadata from DesignTime and creates in RunTime Inventory Database, Commissioning API ControlLoop Sequence diagram will help you understand the flow.

3.1.1: Commissioning REST API 

openapi: 3.0.3
info:
  description: >-
    The ControlLoop Commissioning API allows users to search and perform
    CRUD action on Controlloop Tosca exported data sets.
  version: 1.0.0
  title: Control Loop Commissioning API
  license:
    name: Apache 2.0
    url: 'http://www.apache.org/licenses/LICENSE-2.0'
  contact:
    email: lego@est.tech
servers:
  - url: '{scheme}://onap/controlloop/v2/'
    variables:
      scheme:
        description: 'The Data Set API is accessible via https and http'
        enum:
          - 'https'
          - 'http'
        default: 'http'
tags:
  - name: Control Loop 
    description: Commissioning API
  - name: design-catalogue
    description: CRUD actions on an designtime Control Loop data sets.
  - name: runtime-inventory
    description: CRUD actions on an runtime Control Loop data sets.
paths:
  /design-catalogue/loops:
    get:
      tags:
        - design-catalogue
      operationId: getAllLoopsUsingGET
      summary: Returns a list of all available Control Loops.
      responses:
        '200':
          description: OK
          content:
            application/yaml:
              schema:
                type: array
                items:
                  $ref: '#/components/schemas/ControlLoop'
                  
  /design-catalogue/loop/{controlLoopId}:
    get:
      tags:
        - design-catalogue
      summary: >-
        Returns a specific control loop.
      description: >-
        This GET API returns a specific control loop.
      operationId: getLoopByIdUsingGET
      parameters:
        - name: controlLoopId
          in: path
          description: 'Id of the control loop.'
          required: true
          example: "org.onap.PM_CDS_Blueprint"
          schema:
            type: string
      responses:
        '200':
          description: OK
          content:
            application/yaml:
              schema:
                $ref: '#/components/schemas/ControlLoop'
  
    delete:
      tags:
        - design-catalogue
      summary: Delete a specific Control Loop by ID.
      operationId: deleteLoopByIdUsingDELETE
      parameters:
        - name: controlLoopId
          in: path
          description: Identity for the control loop
          required: true
          style: simple
          schema:
            type: string
      responses:
        '200':
          description: OK
        '404':
          description: Control Loop not present

    post:
      tags:
        - design-catalogue
      summary: >-
        Accepts a Tosca Service Template representing a Control Loop.
      description: >-
        This API Create a Control Loop.
      operationId: createLoopWithIDUsingPOST
      parameters:
        - name: controlLoopId
          in: path
          description: Identity for the control loop
          required: true
          style: simple
          schema:
            type: string
      responses:
        '200':
          description: OK
        '409':
          description: Control Loop already exist.
     
    put:
      tags:
        - design-catalogue
      summary: Updates a Control Loop.
      operationId: updateLoopWithIDUsingPUT
      parameters:
        - name: controlLoopId
          in: path
          description: Identity for the control loop
          required: true
          style: simple
          schema:
            type: string
      responses:
        '200':
          description: OK
        '404':
          description: Control Loop not found
          
  /runtime-inventory/commission:
    post:
      tags:
        - runtime-inventory
      summary: >-
        Commissions Tosca Service Template in RunTime Inventory.
      description: >-
        This API commissions Tosca Service Template in RunTime Inventory DB.
      operationId: commissionLoopWithPOST
      responses:
        '200':
          description: OK
        '404':
          description: No matching record found for the given criteria.
          
  /runtime-inventory/loop/{controlLoopId}:
    get:
      tags:
        - runtime-inventory
      summary: >-
        Returns a specific control loop.
      description: >-
        This GET API returns a specific control loop.
      operationId: getLoopByIdUsing
      parameters:
        - name: controlLoopId
          in: path
          description: 'Id of the control loop.'
          required: true
          example: "org.onap.PM_CDS_Blueprint"
          schema:
            type: string
      responses:
        '200':
          description: OK
          content:
            application/yaml:
              schema:
                type: array
                items:
                  $ref: '#/components/schemas/ControlLoop'
  
    delete:
      tags:
        - runtime-inventory
      summary: Delete a specific Control Loop by ID.
      operationId: deleteLoopByIdUsing
      parameters:
        - name: controlLoopId
          in: path
          description: Identity for the control loop
          required: true
          style: simple
          schema:
            type: string
      responses:
        '200':
          description: OK
        '404':
          description: Control Loop not present

    post:
      tags:
        - runtime-inventory
      summary: >-
        Accepts a Tosca Service Template representing a Control Loop.
      description: >-
        This API Create a Control Loop.
      operationId: createLoopWithIDUsing
      parameters:
        - name: controlLoopId
          in: path
          description: Identity for the control loop
          required: true
          style: simple
          schema:
            type: string
      responses:
        '200':
          description: OK
        '409':
          description: Control Loop already exist.
     
    put:
      tags:
        - runtime-inventory
      summary: Updates a Control Loop.
      operationId: updateLoopWithIDUsing
      parameters:
        - name: controlLoopId
          in: path
          description: Identity for the control loop
          required: true
          style: simple
          schema:
            type: string
      responses:
        '200':
          description: OK
        '404':
          description: Control Loop not found

components:
  schemas:
    ControlLoop:
      title: ControlLoop
      type: object
      properties:
        id:
          type: string
        ControlLoop:
          type: string

3.1.2: Commissioning Sequence Diagrams

1. GET, DELETE, CREATE API ControlLoop Sequence Diagram

2. Commission API ControlLoop Sequence Diagram 

3.2: Instantiation

Robertas Rimkus to pad out this section

This section refers to Instantiation of a Commissioned control loop. A client, in this case CLAMP (potentially DCAEMOD, etc in the future) will render the commissioned control loops allowing selection of a particular control loop to be instantiated. User will then provide the configurations needed to instantiate the selected control loop which will be sent onto the CL_Instance_Control Service. The service will then distribute the configurations to DMaaP topic. Participants (agents) will pull the event containing the config and pick out their control loop components to be instantiated and start/set up those particular components. CL_Instance_Control Service will be waiting for a response back from all participants involved in the instantiation of the control loop, in regards to the state of instantiation. In successful response case the service will store the CL Instance LCM (Life Cycle management) data into the runtime DB as well as providing a message back to the client of the successful instantiation. In failure to receive the response case, a timeout will be called, which will result in a teardown event being sent to DMaaP. The participants will then receive the event and proceed to teardown the components that were instantiated or check that they have failed to instantiate in the first place and send a Teardown ACK back to the CL_Instance_Control Service. No CL Instance LCM data will be stored and a message indicating failure to instantiate the CL along side with the error will be sent back to the client (CLAMP).

3.2.1: Instantiation REST API

openapi: 3.0.3
info:
  title: Swagger Control Loop Instantiation
  description: ""
  version: 1.0.0
servers:
  - url: '{scheme}://onap/controlloop/v2/'
    variables:
      scheme:
        description: 'The Data Set API is accessible via https and http'
        enum:
          - 'https'
          - 'http'
        default: 'http'
tags:
- name: instantiate
  description: Control Loop Instantiation
paths:
  /runtime-inventory/instantiate:
    post:
      tags:
      - instantiate
      summary: Instantiate a commissioned control loop
      operationId: instantiateLoop
      requestBody:
        description: Control loop that is to be instantiated
        content:
          application/json:
            schema:
              $ref: '#/components/schemas/ControlLoop'
        required: true
      responses:
        200:
          description: OK
          content:            
            application/json:
              schema:
                type: array
                items:
                  $ref: '#/components/schemas/ApiResponse'
        404:
          description: No Control Loop found
        405:
          description: Invalid input
  /runtime-inventory/loops/instantiated:
    get:
      tags:
      - instantiate
      summary: Get a list of instantiated control loops
      responses:
        200:
          description: Success
          content:            
            application/json:
              schema:
                type: array
                items:
                  $ref: '#/components/schemas/ControlLoop'
        404:
          description: No instantiated control loops found
  /runtime-inventory/loops/instantiated/{loop-id}:
    get:
      tags:
      - instantiate
      summary: Get an instantiated control loop by id
      parameters:
      - name: loop-id
        in: path
        description: ID of instantiated loop to return
        required: true
        schema:
          type: string
      responses:
        200:
          description: OK
          content:            
            application/json:
              schema:
                $ref: '#/components/schemas/ControlLoop'
        404:
          description: Control Loop not found
    put:
      tags:
      - instantiate
      summary: Update an instantiated control loop
      parameters:
      - name: loop-id
        in: path
        description: ID of instantiated loop
        required: true
        schema:
          type: string
      requestBody:
        description: Control loop to be updated
        content:
          application/json:
            schema:
              $ref: '#/components/schemas/ControlLoop'
        required: true
        
      responses:
        200:
          description: OK
          content:            
            application/json:
              schema:
                $ref: '#/components/schemas/ControlLoop'
        404:
          description: Control Loop not found
    delete:
      tags:
      - instantiate
      summary: Delete an instantiated control loop
      parameters:
      - name: loop-id
        in: path
        description: ID of instantiated loop
        required: true
        schema:
          type: string
      responses:
        200:
          description: OK
        404:
          description: Control Loop not found
components:
  schemas:
    ControlLoop:
      title: ControlLoop
      type: object
      properties:
        id:
          type: string
        controlLoopTosca:
          type: string
    ApiResponse:
      type: object
      properties:
        code:
          type: integer
          format: int32
        status:
          type: string

3.2.2: Instantiation Sequence Diagrams

3.2.3: Instantiation DMaaP API

Initial Thought for an event to be sent from CL_Instance_Control onto DMaaP for Participants to consume. The event would go onto an output topic which the Participants would be polling/subscribed to

e.g url : https://{{ONAPIP}}:{{DMaaPPort}}/events/CL_INSTANCE_CONTROL_OUTPUT

{
  "instance-id":"myCLInstance1",
  "action":"INSTANTIATE",
  "configurations":[
    {
      "participant-id":"Participant_DCAE",
      "applications":[
        {
          "application-id":"example.pmsh",
          "config":{
            
          }
        },
        {
          "application-id":"example.dfc",
          "config":{
            
          }
        }
      ]
    },
    {
      "participant-id":"Participant_Policy",
      "applications":[
        {
          "application-id":"example.OperationalPolicy",
          "config":{
            
          }
        }
      ]
    }
  ]
}

*Preferred solution is to send TOSCA in the body. Meaning we could reuse the parsing code which is already present and provide it to the participant. 

3.2.4: Instantiation Participant API

*Suggestion was to put JAVA API code in this section for the participant talking to DMaaP. TBD

3.3: Monitoring

In this case it refers to monitoring the data that the participants will provide to DMaaP. Participants will send events to DMaaP which will be pulled by the CL_Supervision_Service in to the runtime database. Monitoring service provides APIs to display the statistics data from runtime database to the Monitoring GUI. The data provided should include a reference id to the control loops that are instantiated on the participant, as well as the applications that have been instantiated as a part of that control loop for that participant. Data should also include the time that the application has started, state of it (running/terminated) and any other critical information which would help to determine the health of an instantiated control loop and its components. Idea is for the participant to provide events every certain period of time, similar to a health-check, in order to provide consistent monitoring.

3.3.1: Monitoring REST API

openapi: 3.0.3
info:
  title: Swagger Control Loop Monitoring
  description: ""
  version: 1.0.0
servers:
  - url: '{scheme}://onap/controlloop/v2/'
    variables:
      scheme:
        description: 'The Data Set API is accessible via https and http'
        enum:
          - 'https'
          - 'http'
        default: 'http'
tags:
- name: monitoring
  description: Control Loop Monitoring
paths:
  /runtime-inventory/monitoring:
    post:
      tags:
      - monitoring
      summary: Input monitoring data into the database
      requestBody:
        description: Monitoring data to be input
        content:
          application/json:
            schema:
              $ref: '#/components/schemas/MonitoringData'
        required: true
      responses:
        200:
          description: OK
        405:
          description: Invalid input
    get:
      tags:
      - monitoring
      summary: Get monitoring data for all participants
      responses:
        200:
          description: Success
          content:            
            application/json:
              schema:
                type: array
                items:
                  $ref: '#/components/schemas/MonitoringData'
        404:
          description: No monitoring data found
  /runtime-inventory/monitoring/{participant-id}:
    get:
      tags:
      - monitoring
      summary: Get monitoring data for a participant by participant id
      parameters:
      - name: participant-id
        in: path
        description: ID of a participant 
        required: true
        schema:
          type: string
      responses:
        200:
          description: OK
          content:            
            application/json:
              schema:
                $ref: '#/components/schemas/MonitoringData'
        404:
          description: Monitoring data for the participant not found
    put:
      tags:
      - monitoring
      summary: Update monitoring data for a participant by participant id
      parameters:
      - name: participant-id
        in: path
        description: ID of a participant
        required: true
        schema:
          type: string
      requestBody:
        description: Monitoring data to be updated
        content:
          application/json:
            schema:
              $ref: '#/components/schemas/MonitoringData'
        required: true
      responses:
        200:
          description: OK
          content:            
            application/json:
              schema:
                $ref: '#/components/schemas/MonitoringData'
        404:
          description: Monitoring data for the participant not found
    delete:
      tags:
      - monitoring
      summary: Delete monitoring data for a participant by participant id
      parameters:
      - name: participant-id
        in: path
        description: ID of a participant
        required: true
        schema:
          type: string
      responses:
        200:
          description: OK
        404:
          description: Monitoring data for the participant not found
components:
  schemas:
    MonitoringData:
      properties:
        participant-id:
          type: string
        control-loops:
          type: array
          items:
            properties:
              cl-instance-id:
                type: string
              applications:
                type: array
                items:
                  properties:
                    application-id:
                      type: string
                    status:
                      type: string
                      enum:
                        - INSTANTIATION_STARTED
                        - INSTANTIATED_RUNNING
                        - INSTANTIATED_FAILED
                    other-data:
                      type: string

3.3.2: Monitoring Sequence Diagrams

3.3.3: Monitoring DMaaP API

Participants will send an event containing monitoring data to a DMaaP topic at a set interval after participant has received an event to instantiate a control loop

e.g url: https://{{ONAPIP}}:{{DMaaPPort}}/events/CL_MONITORING_SERVICE_INPUT

{
  "timestamp":"1605689254",
  "participant-id":"Participant_DCAE",
  "control-loops":[
    {
      "instance-id":"myCLInstance1",
      "applications":[
        {
          "application-id":"example.pmsh",
          "status":"INSTANTIATION_STARTED"
        },
        {
          "application-id":"example.dfc",
          "status":"INSTANTIATION_STARTED"
        }
      ]
    },
    {
      "instance-id":"myCLInstance2",
      "applications":[
        {
          "application-id":"example.pmsh",
          "status":"INSTANTIATED"
        }
      ]
    }
  ]
}

3.3.4: Monitoring Participant API

Presume similar thinking to Instantiation Participant API

*Suggestion was to put JAVA API code in this section for the participant talking to DMaaP. TBD

3.4: Supervision

Supervision is responsible for ensuring that

1. control loops are established once their initiation has been ordered
2. control loops are running correctly once their initiation is completed
3. control loops are correctly removed once their removal has been ordered

3.4.1: Supervision Sequence Diagrams

3.4.2: Supervision APIs to other components

4: Design

4.1: Server Side

4.1.1 Database Schema and JPA

4.1.2: TOSCA Processing

4.1.3: Instance Control

4.1.4: Execution Monitoring

4.2: Participant Side

Participant is a component that acts as a bridge between Runtime and components like Policy-framework, DCAE, Kubernetes cluster etc.
It listens to Dmaap to receive messages from runtime and performs operations towards control loop components.

Every participant has two parts Participant-Intermediary and a Participant-Impl.
Participant-Intermediary is a common component that listens to Dmaap and acts on the messages, participant-impl handles the logic towards
control loop element.

4.2.1: Participant Message handling

Participant handles 4 types of messages

1. Participant State Change : This message handles states of a participant. Runtime can order participant for a state change.
    ParticipantState can be set to one of the following
        UNKNOWN : Control Loop execution is unknown.
        PASSIVE : Control Loop execution is always rejected.
        SAFE : Control Loop execution proceeds, but changes to domain state or context are not carried out.
                   The participant returns an indication that it is running in SAFE mode together with the action it would
                   have performed if it was operating in ACTIVE mode.
        TEST : Control Loop execution proceeds and changes to domain and state are carried out in a test environment.
                   The participant returns an indication that it is running in TEST mode together with the action it has performed
                   on the test environment
        ACTIVE : Control Loop execution is executed in the live environment by the participant.
        TERMINATED : Control Loop execution is terminated and not available.

2. Control Loop Update: This message creates the control loop elements and brings them from UNINITIALIZED to PASSIVE state.
    ControlLoopUpdate message contains full ToscaServiceTemplate describing all components participating in a control loop.
    This acts as a template for any control loop to be created according to the template.
    When participant-intermediary receives this message, it triggers creation of policy-types and policies in Policy-Framework by Policy-Participant,
    and deploys DCAE from DCAE-participant

3. Control Loop State change: This message is used to order a state change in control loop element.
    Runtime can order one of the following ordered states.
         UNINITIALIZED : The control loop or control loop element should become uninitialized on participants, it should not exist on participants.
         PASSIVE : The control loop or control loop element should initialized on the participants and be passive, that is,
                         it is not handling control loop messages yet.
         RUNNING : The control loop or control loop element should running and is executing control loops. Once any of above states are ordered, then control loop element transitions into
         UNINITIALIZED : The control loop or control loop element is not initialized on participants, it does not exist on participants.
         UNINITIALIZED2PASSIVE : The control loop or control loop element is changing from uninitialized to passive,
                         it is being initialized onto participants.
         PASSIVE : The control loop or control loop element is initialized on the participants but is passive, that is, it is not
                         handling control loop messages yet.
         PASSIVE2RUNNING : The control loop or control loop element is changing from passive to running,
                          the participants are preparing to execute control loops.
         RUNNING : The control loop or control loop element is running and is executing control loops.
         RUNNING2PASSIVE : The control loop or control loop element is completing execution of current control loops but
                          will not start running any more control loops and will become passive.
         PASSIVE2UNINITIALIZED : The control loop or control loop element is changing from passive to uninitialized,
                          the control loop is being removed from participants

4. Participant Healthcheck: This message is used to learn the health status of a participant.

As a response to any of the above message participant returns a Participant Status message, holding respective message response.
Runtime receives Participant Status message and stores relevant information in database, Or performs respective actions.

4.2.2: Policy Participant Agent

Policy participant receives messages through participant-intermediary common code, and handles them by invoking REST APIs towards policy-framework.

For example, When a ControlLoopUpdate message is received by policy participant, it contains full ToscaServiceTemplate describing all components
participating in a control loop. When the control loop element state changed from UNINITIALIZED to PASSIVE, Policy-participant triggers creation
of policy-types and policies in Policy-Framework.
When the state changes from PASSIVE to UNINITIALIZED, Policy-Participant deletes the policies, policy-types by invoking REST APIs towards policy-framework.

4.2.4: DCAE Participant Agent

DCAE participant receives messages through participant-intermediary common code, and handles them by invoking CLAMP DCAE methods,
which internally work towards DCAE.

For example, When a ControlLoopUpdate message is received by DCAE participant, it contains full ToscaServiceTemplate describing all components
participating in a control loop. When the control loop element state changed from UNINITIALIZED to PASSIVE, DCAE-participant triggers deploy
of DCAE.
When the state changes from PASSIVE to UNINITIALIZED, DCAE-Participant un-deploys DCAE by invoking methods towards CLAMP.

4.2.5: Kubernetes Participant Agent

Kubernetes participant receives messages through participant-intermediary common code, and handles them by invoking Kubernetes Open API.
For example, When a ControlLoopUpdate message is received by Kubernetes participant, When the control loop element state changed from UNINITIALIZED to PASSIVE, Kubernetes-participant triggers Kubernetes Open API and passes the HELM charts towards cluster.

4.3: Client Side

4.3.1: Client SDK: Composition of Control Loop Tosca

4.3.2: Client User Interface

4.4 Other Considerations

4.4.1 Upgrade

Performing a hot upgrade of the Control Loop at run time as well as handling an upgrade of the software in one or more of the participants in an Control Loop is a particularly challenging issue because upgrading must handle the following cases without tearing down the Control Loop:

• Upgrade and changes of the configuration data of participants
• Addition of or removal of participants in an Control Loop
• Upgrade of software in one or more participants in an Control Loop
• Maintenance of compatibility between participants when an update of more than one participant must be done  together to ensure compatibility, for example, when a protocol being used by two participants to communicate is upgraded

4.4.2 Scalability

The system is designed to be inherently scalable. The control loop runtime server is stateless, all state is preserved in the run time inventory in the database. When the user requests a control loop operation (such as an instantiation, activation, passivation, or an ininitialization) the server broadcasts the request to participants over DMaaP and saves details of the request to the database. The server does not directly wait for responses to requests.

When a request is broadcast on DMaaP, the request is asynchronously picked up by participants of the types required for the control loop instance and those participants manage the life cycle of its control loop elements. Periodically, each participant reports back on the status of operations it has picked up for the control loop elements it controls, together with statistics on the control loop elements over DMaaP. On reception of these participant messages, the server stores this information to its database.

The server periodically runs a supervion function, which checks the status of all existing control loop instances and the status of outstanding requests. It builds a picture of the current status of each control loop instance from the reports on the elements of the control loop instances. Once the server has a full picture, it checks that the control loop instance is in the correct state as requested by the user of the system. If the control loop is not in the correct state, the supervision function can initiate actions such aas performing retries on operations or issuing alarms or notificaitons on control loop instances.

This approach makes it easy to scale control loop LCM. As control loop instance counts increase, more than one runtime server can be deployed and REST/supervision operations on control loop instances can run in parallel. The number of participants can scale because an asynchronous broadcast mechanism is used for server-participant communication and there is no direct connection or communication channel between participants and runtime servers. Participant state, control loop instance state, and control loop element state is held in the database, so any runtime server can handle operations for any participant. Because many participants of a particular type can be deployed and participant instances can load balance control loop element instances for different control loops of many types across themselves using a mechanism such as a Kubernetes cluster.

5: Goals

5.1: MVP

5.2: ControlLoop in Tosca LCM Istanbul Jiras

• Design Time
  • Support design of multiple control loops*
  • Support design of individual control loop component**
  • Support composition of control loops**
• Runtime
  • Participant registration and participant deregistration  
  • Support commissioning of control loops
    • Ingestion with artifact references* 
    • Ingestion with artifact embedded**
  • Support instantiation of control loop
    • Support instantiation of control loop TOSCA to DMaaP MR*
    • Support instantiation of config for the control loop*
  • Support monitoring of control loops
    
    • Receive control loop heartbeat events (heartbeat starts when component of control loop is running)*
  • Support supervision of control loops
    • Periodically check monitored data, and update state of control loop*
• Participants
  • Agent library*
  • Reference(test) participant*
  • CDS participant*
  • DCAE participant*
  • Policy participant*
• Demo**
  • Throwaway Monitoring/Control GUI