This page is Work in Progress

The idea of using control loops to automatically (or autonomously) perform network management has been the subject of much research in the Network Management research community, see this paper for some background. However, it is only with the advent of ONAP that we have a platform that supports control loops for network management. Before ONAP, Control Loops have been implemented by hard-coding components together and hard coding logic into components. ONAP has taken a step forward towards automatic implementation of Control Loops by allowing parameterization of Control Loops that work on the premise that the Control Loops use a set of analytic, policy, and control components connected together in set ways.

The goal of the work is to extend and enhance the current ONAP Control Loop support to provide a complete open-source framework for Control Loops. This will enhance the current support to provide TOSCA based Control Loop definition and development, commissioning and run-time management. The participants that comprise a Control Loop and the metadata needed to link the participants together to create a Control Loop are specified in a standardized way using the OASIS TOSCA modelling language. The TOSCA description is then used to commission, instantiate, and manage the Control Loops in the run time system.

1. Terminology

Control Loop: A set of executing elements on which Life Cycle Management (LCM) is executed collectively. For example, a set of microservices may be spawned and executed together to deliver a service. This collection of services is a control loop

Control Loop Element: A single executing entitity, with its Life Cycle being managed as part of the overall control loop. For example, a single microservice that is executing as one microservice in a service.

Participant: A system or framework that runs Control Loop Elements. A participant chooses to partake in control loops, to manage Control Loop Elements for CLAMP, and to receive, send and act on LCM messages for the CLAMP runtime.

Control Loop Type: A definition of a Control Loop in the TOSCA language. This definition describes a certain type of a control loop. The life cycle of instances of a Control Loop Type are managed by CLAMP.

Control Loop Instance: An instance of a Control Loop Type. The life cycle of a Control Loop Instance is managed by CLAMP.

Property: Metadata defined in TOSCA that is associated with a Control Loop, a Control Loop Element, or a Participant.

Property Type: The TOSCA defintion of the type of a property. A property can have a generic type such as string or integer or can have a user defined TOSCA data type.

Property Value: The value of a Property Type. Property values are assigned at run time in CLAMP.

Common Property Type: Property Types that apply to all instances of a Control Loop Type.

Common Property Value: The value of a Property Type. It is assigned at run time once for all instances of a Control Loop Type.

Instance Specific Property Type: Property Types that apply to an individual instance of a Control Loop Type.

Instance Specific Property Value: The value of a Property Type that applies to an individual instance of a Control Loop Type. The value is assigned at run time for each control loop instance.

2. Capabilities

We consider the capabilities of Control Loops at Design Time and Run Time.

At Design Time, two capabilities are supported:

Participant Metadata Definition. This capability allows external users and systems (such as SDC or DCAE-MOD) to define participants that can take part in a control loop and to define the metadata that can be used on and configured on a participant when it is taking part in in a control loop. The post condition of an execution of this capability is that a participant is defined in the Control Loop Design Time Catalogue together with sets of metadata that can be used with this participant in control loops.
Control Loop Composition. This capability allows users and other systems to create Control Loop Type definitions by connecting a chain of participants together from the participants that are available in the Control Loop Design Time Catalogue. In an execution of this capability, a user defines the control loop chain for the Control Loop Type and defines the connections between participants. The user also selects the correct metadata sets for each participant in the Control Loop Type and defines the overall Control Loop Type metadata. The user also specifies the Common Property Types that apply to all instances of a control loop type and the Instance Specific Property Types that apply to individual instances of a Control Loop Type. The post condition for an execution of this capability is a Control Loop definition in TOSCA stored in the Control Loop Design Time Catalogue.

At Run Time, the following capabilities are supported:

Participant Registration. This capability allows participants to register and deregister with CLAMP. The post condition for an execution of this capability is that a participant is available for participation in a control loop. Participants can explicitly register with CLAMP at any point after they start, or they can implicitly register when they respond to a control loop initiation request.
Control Loop Commissioning. This capability allows version controlled Control Loop Type definitions to be taken from the Control Loop Design Time Catalogue and be placed in the Control Loop Run Time Inventory. It also allows the values of Common Property Types that apply to all instances of a Control Loop Type to be set. The post condition for an execution of this capability is that the Control Loop definition is in the Control Loop Run Time Inventory.
Control Loop Priming on Participants. A participant is primed to support a Control Loop Type. The definition of a control loop and the values of Common Property Types that apply to all instances of a control loop type on a participant is sent to a participant. The participant can then take whatever actions it need to do to support the control loop type in question. Control Loop Priming can take place explicitly as a separate operation on participants or can be done implicitly in the instantiation message for the first instance of a Control Loop Type. The post condition for an execution of this capability is that all participants in this control loop type are commissioned, that is they are prepared to run instances of this control loop type.
Control Loop Life Cycle Management. This capability allows a Control Loop Instance to be created. The Control Loop Type definition is read from the Control Loop Run Time Inventory and values are assigned to the Instance Specific Property Types defined for instances of the Control Loop Type in the same manner as the existing CLAMP client does. A Control Loop Instance that has been created but has not yet been instantiated on participants is in state UNINITIALIZED. In this state, the Instance Specific Property Type values can be revised and updated as often as the user requires. Once the user is happy with the property values, the Control Loop Instance is instantiated on participants and the Control Loop Elements for this Control Loop Instance are started by participants using the control loop metadata. Once the Control Loop Instance is instantiated on each participant, the Control Loop Instance is set as being in state PASSIVE in the Control Loop Run Time Inventory. The user can now order the participants to change the state of the Control Loop Instance to state RUNNING. Each participant begins accepting and processing control loop events and the Control Loop Instance is set to state RUNNING in the control loop inventory. The post condition for an execution of this capability is that the Control Loop instance is running on participants and is processing events.
Control Loop Monitoring. This capability allows Control Loop Instances to be monitored. Users can check the status of Participants, Control Loop Instances, and Control Loop Elements. Participants report their overall status and the status of Control Loop Elements they are running periodically to CLAMP. Clamp aggregates these status reports into an aggregated Control Loop Instance status record, which is available for monitoring. The post condition for an execution of this capability is that Control Loop Instances are being monitored.

When a control loop definition has been commissioned, instances of the control loop can be created, updated, and deleted. The system manages the lifecycle of control loops and control loop elements following the state transition diagram below.

Zu Qiang (Ericsson) Let's update the state diagram below.

3: Overall Architecture

The diagram below shows an overview of the architecture of TOSCA based Control Loop Management in CLAMP.

Following the ONAP Reference Architecture, the architecture has a Design Time part and a Runtime part.

The Design Time part of the archtiecture allows a user to specify metadata for participants. It also allows users to compose control loops. The Design Time Catalogue contains the metadata primitives and control loop definition primitives for composition of control loops. As shown in the figure above, the Design Time component provides a system where Control Loops can be designed and defined in metadata. This means that a Control Loop can have any arbitrary structure and the Control Loop developers can use whatever analytic, policy, or control participants they like to implement their Control Loop. At composition time, the user parameterises the Control Loop and stores it in the design time catalogue. This catalogue contains the primitive metadata for any participants that can be used to compose a Control Loop. A Control Loop SDK is used to compose a Control Loop by aggregating the metadata for the participants chosen to be used in a Control Loop and by constructing the references between the participants. The architecture of the Control Loop Design Time part will be elaborated in future releases.

Composed Control Loops are commissioned on the run time part of the system, where they are stored in the run time inventory and are available for instantiation. The Commissioning component provides a CRUD REST interface for Control Loop Types, and implements CRUD of Control Loop Types. Commissioning also implements validation and persistence of incoming Control Loop Types. It also guarantees the integrity of updates and deletions of Control Loop Types, such as performing updates accordance with semantic versioning rules and ensuring that deletions are not allowed on Control Loop Types that have instances defined.

The Instantiation component manages the Life Cycle Management of Control Loop Instances and their Control Loop Elements. It publishes a REST interface that is used to create Control Loop Instances and set values for Common and Instance Specific properties. This REST interface is public and is used by the CLAMP GUI. It may also be used by any other client via the public REST interface. the REST interface also allows the state of Control Loop Instances to be changed. A user can change the state of Control Loop Instances as described in the state transition diagram shown in section 2 above. The Instantiation component issues update and state change messages via DMaaP to participants so that they can update and mange the state of the Control Loop Elements they are responsible for. The Instantiation component also implements persistence of Control Loop Instances, control loop elements, and their state changes.

The Monitoring component reads updates sent by participants. Participants report on the state of their Control Loop Elements periodically and in response to a message they have received from the Instantiation component. The Monitoring component reads the contents of the participant messages and persists their state updates and statistics records. It also publishes a REST interface that publishes the current state of all Participants, Control Loop Instances and their Control Loop Elements, as well as publishing Participant and Control Loop statistics.

The Supervision component is responsible for checking that Control Loop Instances are correctly instantiated and are in the correct state (UNINITIALIZED/READY/RUNNING). It also handles timeouts and on state changes to Control Loop Instances, and retries and rolls back state changes where state changes failed.

A Participant is an executing component that partakes in control loops. More explicitly, a Participant is something that implements the Participant Instantiation and Participant Monitoring messaging protocol over DMaaP for Life Cycle management of Control Loop Elements. A Participant runs Control Loop Elements and manages and reports on their life cycle following the instructions it gets from the CLAMP runtime in messages delivered over DMaaP.

In the figure above, five participants are shown. A Configuration Perisistence Participant manages Control Loop Elements that interact with the ONAP Configuration Persistence Service to store common data. The DCAE Participant runs Control Loop Elements that manage DCAE microservices. The Kubernetes Participant hosts the Control Loop Elements that are managing the life cycle of microservices in control loops that are in a Kubernetes ecosystem. The Policy Participant handles the Control Loop Elements that interact with the Policy Framework to manage policies for control loops. A Controller Participant such as the CDS Participant runs Control Loop Elements that load metadata and configure controllers so that they can partake in control loops. Any third party Existing System Participant can be developed to run Control Loop Elements that interact with any existing system (such as an operator's analytic, machine learning, or artificial intelligence system) so that those systems can partake in control loops.

4. Concepts and Relationships on CLAMP Control Loops

5. Design and Implementation

The design and implementation of TOSCA Control Loops in CLAMP is described in the following pages

This page is updated for Istanbul to this point, the information below this point may or may not be correct for Istanbul.

2.4: Swagger REST APIs for Control Loop

ControlLoop Runtime Swagger REST APIs:

ControlLoop_Runtime_Swagger_API.yml

Participant Swagger REST APIs:

Participant_Swagger_API.yml

3: APIs and Sequence Diagrams

3.1: Commissioning

Ajay Deep Singh to pad out this section

This section defines Commissioning/CRUD Operations that can be performed on ControlLoops.

A Client, in this case CLAMP, can perform CRUD operations or can commission ControlLoops from DesignTime to RunTime Inventory Database.

DesignTime/RunTime Catalogue/Inventory Database stores ControlLoop definitions, CRUD operations on database supported by REST Endpoints like Get, Delete, Create allowing selection of a particular ControlLoop to be addressed, below sequence diagram will help you understand flow how a client(Clamp) application can initiate Rest call for performing different operations on Database.

API_Gateway Service is for interacting to different database DesignTime/RunTime and should be responsible for responding success or failure status on different operations.

The commissioning of ControlLoops definition from DesignTime Catalogue to RunTime Inventory Database can we achived using the commissioning Rest Endpoint, in this process when a rest request is initiated from a client(Clamp) the API_Gateway Service take cares of fetching ControlLoops metadata from DesignTime and creates in RunTime Inventory Database, Commissioning API ControlLoop Sequence diagram will help you understand the flow.

In future commissioning Rest Endpoint might be updated to push ControlLoops not only in RunTime Database but to the participants involved in ControlLoop.

3.1.1: Commissioning REST API

3.1.2: Commissioning Sequence Diagrams

GET, DELETE, CREATE API ControlLoop Sequence Diagram
Commission API ControlLoop Sequence Diagram

3.2: Instantiation

Robertas Rimkus to pad out this section

This section refers to Instantiation of a Commissioned control loop. A client, in this case CLAMP (potentially DCAEMOD, etc in the future) will render the commissioned control loops allowing selection of a particular control loop to be instantiated. User will then provide the configurations needed to instantiate the selected control loop which will be sent onto the CL_Instance_Control Service. The service will then distribute the configurations to DMaaP topic. Participants (agents) will pull the event containing the config and pick out their control loop components to be instantiated and start/set up those particular components. CL_Instance_Control Service will be waiting for a response back from all participants involved in the instantiation of the control loop, in regards to the state of instantiation. In successful response case the service will store the CL Instance LCM (Life Cycle management) data into the runtime DB as well as providing a message back to the client of the successful instantiation. In failure to receive the response case, a timeout will be called, which will result in a teardown event being sent to DMaaP. The participants will then receive the event and proceed to teardown the components that were instantiated or check that they have failed to instantiate in the first place and send a Teardown ACK back to the CL_Instance_Control Service. No CL Instance LCM data will be stored and a message indicating failure to instantiate the CL along side with the error will be sent back to the client (CLAMP).

3.2.1: Instantiation REST API

3.2.2: Instantiation Sequence Diagrams

3.2.3: Instantiation DMaaP API

Initial Thought for an event to be sent from CL_Instance_Control onto DMaaP for Participants to consume. The event would go onto an output topic which the Participants would be polling/subscribed to

e.g url : https://{{ONAPIP}}:{{DMaaPPort}}/events/CL_INSTANCE_CONTROL_OUTPUT

*Preferred solution is to send TOSCA in the body. Meaning we could reuse the parsing code which is already present and provide it to the participant.

3.2.4: Instantiation Participant API

*Suggestion was to put JAVA API code in this section for the participant talking to DMaaP. TBD

3.3: Monitoring

In this case it refers to monitoring the data that the participants will provide to DMaaP. Participants will send events to DMaaP which will be pulled by the CL_Supervision_Service in to the runtime database. Monitoring service provides APIs to display the statistics data from runtime database to the Monitoring GUI. The data provided should include a reference id to the control loops that are instantiated on the participant, as well as the applications that have been instantiated as a part of that control loop for that participant. Data should also include the time that the application has started, state of it (running/terminated) and any other critical information which would help to determine the health of an instantiated control loop and its components. Idea is for the participant to provide events every certain period of time, similar to a health-check, in order to provide consistent monitoring.

3.3.1: Monitoring REST API

3.3.2: Monitoring Sequence Diagrams

3.3.3: Monitoring DMaaP API

Participants will send an event containing monitoring data to a DMaaP topic at a set interval after participant has received an event to instantiate a control loop

e.g url: https://{{ONAPIP}}:{{DMaaPPort}}/events/CL_MONITORING_SERVICE_INPUT

3.3.4: Monitoring Participant API

Presume similar thinking to Instantiation Participant API

*Suggestion was to put JAVA API code in this section for the participant talking to DMaaP. TBD

3.4: Supervision

Supervision is responsible for ensuring that

control loops are established once their initiation has been ordered
control loops are running correctly once their initiation is completed
control loops are correctly removed once their removal has been ordered

3.4.1: Supervision Sequence Diagrams

3.4.2: Supervision APIs to other components

4: Design

4.1: Server Side

4.1.1 Database Schema and JPA

4.1.2: TOSCA Processing

4.1.3: Instance Control

4.1.4: Execution Monitoring

4.2:

4.3: Client Side

4.3.1: Client SDK: Composition of Control Loop Tosca

4.3.2: Client User Interface

4.4 Other Considerations

4.4.1 Upgrade

Performing a hot upgrade of the Control Loop at run time as well as handling an upgrade of the software in one or more of the participants in an Control Loop is a particularly challenging issue because upgrading must handle the following cases without tearing down the Control Loop:

Upgrade and changes of the configuration data of participants
Addition of or removal of participants in an Control Loop
Upgrade of software in one or more participants in an Control Loop
Maintenance of compatibility between participants when an update of more than one participant must be done together to ensure compatibility, for example, when a protocol being used by two participants to communicate is upgraded

4.4.2 Scalability

The system is designed to be inherently scalable. The control loop runtime server is stateless, all state is preserved in the run time inventory in the database. When the user requests a control loop operation (such as an instantiation, activation, passivation, or an ininitialization) the server broadcasts the request to participants over DMaaP and saves details of the request to the database. The server does not directly wait for responses to requests.

When a request is broadcast on DMaaP, the request is asynchronously picked up by participants of the types required for the control loop instance and those participants manage the life cycle of its control loop elements. Periodically, each participant reports back on the status of operations it has picked up for the control loop elements it controls, together with statistics on the control loop elements over DMaaP. On reception of these participant messages, the server stores this information to its database.

The server periodically runs a supervion function, which checks the status of all existing control loop instances and the status of outstanding requests. It builds a picture of the current status of each control loop instance from the reports on the elements of the control loop instances. Once the server has a full picture, it checks that the control loop instance is in the correct state as requested by the user of the system. If the control loop is not in the correct state, the supervision function can initiate actions such aas performing retries on operations or issuing alarms or notificaitons on control loop instances.

This approach makes it easy to scale control loop LCM. As control loop instance counts increase, more than one runtime server can be deployed and REST/supervision operations on control loop instances can run in parallel. The number of participants can scale because an asynchronous broadcast mechanism is used for server-participant communication and there is no direct connection or communication channel between participants and runtime servers. Participant state, control loop instance state, and control loop element state is held in the database, so any runtime server can handle operations for any participant. Because many participants of a particular type can be deployed and participant instances can load balance control loop element instances for different control loops of many types across themselves using a mechanism such as a Kubernetes cluster.

5: Goals

5.1: MVP

Key	Summary	Assignee	P	Status

Loading...

Refresh

5.2: ControlLoop in Tosca LCM Istanbul Jiras

Key	Summary	Assignee	P	Status