Versions Compared

Key

  • This line was added.
  • This line was removed.
  • Formatting was changed.

Create 'bulk' version for org.onap.cps.ncmp.rest.controller.NetworkCmProxyController#getResourceDataForCmHandle to support multiple cm-handles (~ multiple anchors in in CPS-Core)

Table of Contents

References

  1. CPS-1515 - Spike: Support multiple CM-Handles for NCMP Get Operation
  2. CPS-NCMP ↔ DMI-Plugin Interface Details Jakarta-R10
  3. CPS Events Structure

Requirements

Functional

#InterfaceRequirementAdditional Information
1REST CPS(-NCMP)-E-05Support batch read operation (new) using an asynchronous response on a client specified topicpayload includes list of cm handle ids
2REST DMI-I-01Support batch read operation (new) using an asynchronous response to an internal topicpayload includes list of (associated for this plugin) cm handles ids with their private (additional) properties

Error Handling

#Error ScenarioExpected behavior
1DMI Not respond to initial synchronous request with normal HTTP timeoutSpecial Error message (Kafka event)  including affected, and reason for all the handles in the message cm handles send to client topic detailing cm-handles
(per dmi plugin)
2Topic not supplied on CPS-E-05Return HTTP 501 (not implemented)
3Client specided Topic is not configuredLog error message only
4Non-existing cm-handle idSimilar message but different reason as specified #1 above

Capabilities

Excerpt


#ParameterExpectationNotes
1Response Time 1 Batch request<2 seconds (average)
  • Async response available on client topic
  • No delay in DMI PLugin Plugin (tested/measured using stub DMI Plugin)
2Batch-size200 cm handlesNo hardcoded limit
3Response payload size~2 KB per cm handlesPerformance test for capability should be tested with this average response size
4Maximum registered #cm handles 20,000This will effect the internal query time
5Supported # DMI PLugins10This might effect processing times
6Test Environment


Expand
  1. CPS and NCMP

requests:
    cpu: 2000m
    memory: 2Gi
limits:
    memory: 3Gi
    cpu: 3000m

2. Postgres

requests:
    cpu: 4000m
    memory: 1Gi
 limits:
    memory: 3Gi
    cpu: 6000m



7Concurrent request1212 clients requests toward 1 NCMP simultaneously
8Request Frequency100 request/minShould not affect performance, does not need to be tested


Out-of-scope

  1. Support for multiple resource identifiers in one batch operation
  2. Support cached data batch requests: only passthrough datastores will be supported (see decision #2)
  3. NCMP does NOT keep track of request status to see if  it is completed, amalgamate responses or anything like that. It wil simply forward responses from the internal topic to the client topic.
  4. Access control

Assumptions

#

Assumption

Notes

1

Proprietary options or not in the scope of this analysis.

agreed with kieran mccarthy ; these are optional parameters (name-value pairs) not interpreted by NCMP but can be interpreted by proprietary plugins
In the future "scope" might become standardized instead of proprietary but that wil be achieved through a separate requirement 

2

same xpath (resourceIdentifierInQuery) for all cm handles or different for each cm handle

agreed with kieran mccarthy ; if different resources are required on the same cm-handle the client wil send another (batch) request

Issues & Decisions

3
  • options, resourceIdentifier is optional for bulk operations.
  • operation, datastore and cmhandleIds are mandatory fields

agreed with CPS team.

4We are not merging any duplicate cm handles while sending request to dmi-plugin

agreed  

5

Get batch operation does not support includeDescendants as it is impl. for 

pass through datastore only

ncmp-datastore:passthrough-running and
ncmp-datastore:passthrough-operational

agreed on  

Issues & Decisions

#

Issue

Notes 

Decision

1Which operation(s) need support for multiple cm handles?
  1. Get
  2. Create
  3. Update (

#

Issue

Notes 

Decision

1Which operation(s) need support for multiple cm handles?
  • Get
  • Create
  • Update (
    1. Put)
    2. Patch
    3. Delete

    if many what is the priority?

    agreed with kieran mccarthy 

    Only Get (read)

    (in future other operations might be support batch option too)

    2

    Which datasources should be supported?

    Do we need to support passthrough-only no-cached() data only ?
    (maybe just start with that, support cached data bulk request later)

    agreed with kieran mccarthy :

    all passthrough datastores will be supported

    Not implemented (yet) response, for non passthrough datastores 

    3URL  pattern for NCMP bulk endpoints

    Existing : /v1/ch/{cm-handle}/data/ds/{datastore-name}

    CPS Proposed : /v1/batch/data/ds/{datastore-name}

          ( include cm handles into payload / body and leave datastore into url)

    We had an internal review with some of our rApp colleagues around some of the recent proposed NCMP batch interface and they came back with some valid comment.  The proposal is that we should not distinguish batch from bulk or other flavors of read/write. 

    The aim is to only have a single flavor of interface for read or write for clients.  Therefore the proposal is to drop “batch” from the interface URL and just act toward “data”

    Agreed with team kieran mccarthy 

    POST http://localhost:8080/ncmp/v1/data&topic=my-topic-name

    agreed with kieran mccarthy : Follow the existing interfaces as much as possible for consistency and efficiency

    4keep datastore, topic and optional parameters in the URL itself instead into body.

    CPS prefers keep interface similar as single cm handle interface (consistency and cost)

    Existing : ...&topic=topicParamInQuery

    agreed with kieran mccarthy : Follow the existing interfaces as much as possible for consistency and efficiency

    5

    support in ONAP DMI-plugin


    agreed with Toine Siebelink : ONAP plugin can respond with not implemented yet code,

    6

    Response always Async ie. topic is compulsory ?

    Assume topic is compulsory (defined in OPenApi) → Response therefore wil be 400 if not supplied

    Agreed with kieran mccarthy :

    Topic is optional but system will respond with 'Not implemented (yet when not specified or blank

    7Should NCMP Amalgamate Async responses from DMI-Plugin before forwarding ?step 6 in flow diagram

    Agreed with kieran mccarthy : NCMP wil only forward to client topic no handling tracking or any responses or status of request

    8Handle non responding dmi-plugin

    Agreed with kieran mccarthy :
    No response for 4b then send an error response to the topic given by client

    9Should (can) NCMP check if 'MyTopic' specified by client existConsider Access Control too. For now NCMP can log error. Client is responsible for topic setup

    Agreed with kieran mccarthy :
    NCMP can log error when forward to 'client topic' Security not in scope (yet)

    10How to handle non-existing CM-Handles (id)

    Suggestions

    1. Silently ignore
    2. (initial) error response
      1. should we combine all errors in one message?

    Agreed with kieran mccarthy :

    Additional error (messages) response with all cm-handles that cannot be resolved, also a separate error message wil be sent for each failed DMI

    11

    Overlap/clash with Deutsche Telekom user story:

    Jira
    serverONAP Jira
    serverId425b2b0a-557c-3c0c-b515-579789cceedb
    keyCPS-1377
    ?!


    discussed in weekly ONAP meeting the DT user story is affect CPS-Core interface (not NCMP) and the requirement is to execute a query over ALL cm-handles (cached  only?!) instead of a given list of cm-handles (~anchors)

    12

    Schema of bulk response

    Jira
    serverONAP Jira
    columnIdsissuekey,summary,issuetype,created,updated,duedate,assignee,reporter,priority,status,resolution
    columnskey,summary,type,created,updated,due,assignee,reporter,priority,status,resolution
    serverId425b2b0a-557c-3c0c-b515-579789cceedb
    keyCPS-1556

    1) Agreement required on the structure of the response.  Please see response structure below.
    2) Does 'eventTime' field which holds the timestamp of the bulk response event,  required or can it be dropped?

    Need to follow schema structure there in  #15 under Issues & Decisions section and it  is  agreed on    

    13

    Schema for Bulk Response event forwarding to client specified  topic

    Image AddedCPS-1557 - NCMP : forward bulk response messages to client topic

    Not keeping the 'eventTarget' which comes from the (BulkResponseEvent(In progress of the structure agreement)).

    Note:  Will consider  'eventTarget', If it finalized the schema from

    Jira
    serverONAP Jira
    columnIdsissuekey,summary,issuetype,created,updated,duedate,assignee,reporter,priority,status,resolution
    columnskey,summary,type,created,updated,due,assignee,reporter,priority,status,resolution
    serverId425b2b0a-557c-3c0c-b515-579789cceedb
    keyCPS-1556

    Agreed on to keep one schema for both events (DMI → NCMP) and (NCMP → ClientApps)

    14

    How NCMP would forward response? (Response data : ref.  message flow #6)

    Jira
    serverONAP Jira
    columnIdsissuekey,summary,issuetype,created,updated,duedate,assignee,reporter,priority,status,resolution
    columnskey,summary,type,created,updated,due,assignee,reporter,priority,status,resolution
    serverId425b2b0a-557c-3c0c-b515-579789cceedb
    keyCPS-1556

        1. Do we need to send only one response containing all the requested cm handles ?
        2. Send single response for each cm handle? This could lead to to much data in the event for large batch requests. Csaba Kocsis , Sourabh Sourabh and Toine Siebelink proposed to us single event for each cm handle
        3. Single response message per DMI plugin? Just is a list of IDS should be OK.

    kieran mccarthy

    kieran mccarthy  Confirmed in meeting  

    15Message format for the batch interfaceWe have had some internal discussions including with some O-RAN standards representatives and one of the outcomes is that it would be good if we aligned with the some community standards for event header definitions.  IT is proposed (initially from AT&T) that we should follow Cloud Native Computing Foundation (CNCF) specification as defined in their cloudevents incubator project.
    Some non-standard headers wil need to be implemented as 'extensions' and names to be confirmed
    • correlationid
    • target
    16NCMP will send only one request to each DMI

    topic and requestId will be send as a path param from NCMP to DMI-plugin.

    NCMP batch endpoint : http://localhost:8080/ncmp/v1/data?&topic=my-topic-name

    DMI-Plugin batch endpoint : http://172.26.202.25:8783/dmi/v1/data?topic=my-topic-name&requestId=e6fa4d26-4dc1-4877-aa3c-45e99f840708

    • Mandatory Fields:
        • "operation": "read"
        • "operationId": "12"
          "datastore": "ncmp-datastore:passthrough-operational"
        • "targetIds": [ "0df4d39af6514d99b816758148389cfd", "ec2e9495679a43c58659c07d87025e72" ]
    • Optional Fields: 
        • "options": "(fields=schemas/schema)"
        • "resourceIdentifier": "parent/child"

    Agreed with Csaba Kocsis  and kieran mccarthy




    17Feature Name (events)

    event type proposal : from BatchDataEvent  to

    DataOperationEvent

    For example : DataOperationResponseEvent 

     'eventType' value as below 'org.onap.cps.ncmp.events.BatchDataEvent

    proposed Operation like :

    'org.onap.cps.ncmp.events.DataOperationEvent

    'batch' was the keyword use in the URL for this feature (now gone) but much code (classnames, variable names) etc uses this name still...

    kieran mccarthy and Team agreed on and some cost (1-2 days) is acceptable 

    18What would be the field name of response data?data.responses[0].data

    kieran mccarthy please confirm : we are using 'responseContent' as name of this field as 'data' is already used for cloud event standard payload.

    Now it is like : data.responses[0].responseContent

    Sourabh Sourabh Suggest to use "result".  This is after all where the actual result output.  the other tags in responses[x] are status related to the request  :

                            data.responses[0].result

    19

    Exact format of timestamp of cloud events.


    CloudEvent documentation (https://github.com/cloudevents/spec/blob/v1.0.2/cloudevents/spec.md#time) refers on ISO8601 standard

    (see org.onap.cps.ncmp.api.impl.utils.EventDateTimeFormatter

    Toine Siebelink Although not clearly communicated before ISO8601 is the standard to be used and it seems the current implementation in NCMP conforms to this standard 

    Event Format Definitions

    Event Headers

    Excerpt Include
    CPS Events Structure
    CPS Events Structure

    Event Body (data)

    • The current proposal is to use the CNFC Event structure and library as detail in CPS Events Structure#CNCFCloudEventalignment
    • To prevent problems with large data request multiple response event will be sent for a successful read for each cm handle (id)
    • Error messages can contain information about single cm handle ids


    Field 

    Type

    Description

    Mandatory?

    Notes

    data
    EventThe payload of an event

    Mandatory

    Cloud event standard (data
    data.responses[0, 1, 2, ...]
    Array contains an array or batch response that includes both success and failure.

    Mandatory


    data.responses[0].operationId
    Stringspecified to distinguish multiple operations using same cmhandleId

    Mandatory


    data.responses[0].ids
    Stringcmhandle-ids

    Mandatory

    Example : ["0df4d39af6514d99b816758148389cfd"]
    Note: Ids array should contain only a single element in the array in case of success messages and In case of error it can have any number of elements.

    data.responses[0].statusCode
    String

    Mandatory

    Common NCMP defined error codes:

    • status-code 0-99 is reserved for any success response
    • status-code from 100 to 199 is reserved for any failed response. 
    data.responses[0].statusMessage
    String

    Mandatory

    Examples for code & message :

    statusCodestatusMessage
    0
    Successfully applied changes.
    100
    cm handle id(s) is(are) not found.
    101
    cm handle id(s) is(are) in non ready state.


    data.responses[0].dataObject

    Optional

    1. In case of success :
      • Optional,  for write operations then no need to return configurations
        application/yang-patch+json | application/yang-data+json
    2. In case of failure :
      • Optional, any supplementary error data matching the error status-code

    Implementation

    Bulk Request Message Flow

    Gliffy Diagram
    macroId5966a68a-1641-430a-8c78-27c938f06e15
    displayNameCm Handle Batch Message Flow
    nameCm Handle Batch Interface Design
    pagePin21

    Message Flow details

    Flow Step

    Short description

    Message Details

    Notes

    1

    Bulk Get Request


    Code Block
    languagebash
    titleNCMP batch endpoint
    collapsetrue
    curl --location 'http: //localhost:8080/ncmp/v1/data?topic=my-topic-name' \
    --header 'Content-Type: application/json' \
    --header 'Authorization: Basic Y3BzdXNlcjpjcHNyMGNrcyE=' \
    --data '{
        "operations": [
            {
                "operation": "read",
                "operationId": "12",
                "datastore": "ncmp-datastore:passthrough-operational",
                "options": "some option",
                "resourceIdentifier": "parent/child",
                "targetIds": [
                    "836bb62201f34a7aa056a47bd95a81ed",
                    "202acb75b4a54e43bb1ff8c0c17a8e08"
                ]
            },
            {
                "operation": "read",
                "operationId": "14",
                "datastore": "ncmp-datastore:passthrough-running",
                "targetIds": [
                    "ec2e9495679a43c58659c07d87025e72",
                    "0df4d39af6514d99b816758148389cfd"
                ]
            }
        ]
    }'


    Define new get operation "getResourceDataForCmHandles" into ncmp.yml

    2Ack client Request


    Code Block
    languagebash
    titleNCMP batch endpoint response (202->Accepted)
    collapsetrue
    {
        "requestId": "4753fc1f-7de2-449a-b306-a6204b5370b3"
    }



    3DMI Bulk Request


    Code Block
    languagebash
    titleDMI service batch endpoint
    collapsetrue
    DMI Service 1 (POST): http://172.26.202.25:8783/dmi/v1/data?topic=my-topic-name&requestId=4753fc1f-7de2-449a-b306-a6204b5370b3 -> 
    
    [
        {
            "operationType": "read",
    Code Block
    titleSchema of bulk response
    collapsetrue
    { "$schema": "https://json-schema.org/draft/2019-09/schema", "$id": "urn:cps:org.onap.cps.ncmp.events:bulk-response-event-schema:v1", "$ref": "#/definitions/BulkResponseEvent", "definitions": { "BulkResponseEvent": { "description": "The payload for bulk response event.", "type": "object", "properties": {
    
            "
    eventId
    operationId": 
    {
    "14",
            "datastore": "ncmp-datastore:passthrough-running",
            "
    description
    cmHandles": 
    "The unique id identifying the event generated by DMI for this bulk response event.",
    [
                {
                    "
    type
    id": "
    string
    ec2e9495679a43c58659c07d87025e72",
            
    },
            "
    eventCorrelationId
    cmHandleProperties": {
              
    "description
              "neType": "
    The request id passed by NCMP for this bulk response event.",
    RadioNode"
                    }
       
    "type":
     
    "string"
            },
            
    "eventTime":
     
    {
       {
           
    "description":
     
    "The time of
     
    bulk
     
    response
     
    event.
     
    The
     
    expected
     
    format
     
    is 'yyyy-MM-dd'T'HH:mm:ss.SSSZ'.
     "id": "0df4d39af6514d99b816758148389cfd",
              
    "type
          "cmHandleProperties": 
    "string"
    {
              
    },
              "
    eventType
    neType": 
    {
    "RadioNode"
              
    "description":
     
    "The
     
    type
     
    of
     
    bulk
     
    response
     
    event.",
    }
              
    "type": "string"
      }
            
    },
    ]
        }
    ]
    
    DMI Service 2 (POST) 
    "eventSchema": {
    : http://172.26.202.26:8783/dmi/v1/data?topic=my-topic-name&requestId=4753fc1f-7de2-449a-b306-a6204b5370b3 -> 
    
       [
        {
      
    "description":
     
    "The
     
    event
     
    schema
     
    for
     
    bulk
     
    response event.
    "operationType": "read",
    
            
    "
    type
    operationId": "
    string
    12",
            
    }
    "datastore": "ncmp-datastore:passthrough-operational",
            "
    eventSchemaVersion
    options": 
    {
    "some option",
            
    "
    description
    resourceIdentifier": 
    "The event schema version for bulk response event.",
    "parent/child",
            "cmHandles": [
              
    "type": "string"
      {
            
    },
            "
    event
    id": 
    {
    "836bb62201f34a7aa056a47bd95a81ed",
            
    "$ref":
     
    "#/definitions/Event"
           "cmHandleProperties": 
    }
    {
          
    },
          
    "required":
     
    [
           "neType": "
    eventId
    RadioNode"
    ,
    
            
    "eventCorrelationId",
            
    "eventTime",
    }
            
    "eventType",
        },
        
    "eventSchema",
            
    "eventSchemaVersion"
    {
          
    ]
          
    },
        "
    Event
    id": 
    {
    "202acb75b4a54e43bb1ff8c0c17a8e08",
    
         
    "description":
     
    "The
     
    bulk
     
    response
     
    event
     
    content.",
          "
    type
    cmHandleProperties": 
    "object",
    {
          
    "existingJavaType": "java.lang.Object"
           
    }
      
    } } Code Block
    titleSample bulk response
    collapsetrue
    {
      
    "eventId":
     
    "4cb32729-85e3-44d1-aa6e-c923b9b059a5",
      "
    eventCorrelationId
    neType": "
    68f15800-8ed4-4bae-9e53-27a9e03e1911",
    RadioNode"
      
    "eventTime": "2023-03-28T14:29:23.876+0000", "eventType": "org.onap.cps.ncmp.event.model.BulkResponseEvent", "eventSchema": "urn:cps:org.onap.cps.ncmp.event.model.BulkResponseEvent", "eventSchemaVersion": "v1", "event": { "payload": "response of batch cm handles" } }
                  }
                }
            ]
        }
    ]


        New JIRA

    Jira
    serverONAP Jira
    serverId425b2b0a-557c-3c0c-b515-579789cceedb
    keyCPS-1833
    is created to address it.

    Discussed with Csaba Kocsis and agreed to send identical data request body as NCMP where data operation requat body is as follows:


    Client App→ NCMPNCMP→ DMI Agreed 
    operationoperationTypeoperation
    operation details are wrapped into parent attribute "operations"  like "operations": [ ...operation details ar not wrapped wrap operation details into parent attribute "operations"



    NCMP→ DMICurrent NCMP→ DMIProposed NCMP→ DMI

    {
        "operations": [
            {
                "operation""read",
                "operationId""operational-10",
                "datastore""ncmp-datastore:passthrough-running",
                "resourceIdentifier""parent/child",
                "targetIds": [
                    "cd4473a732724092a97cebe157604a83"
                ]
            }
        ]
    }

    [
        {
            "operationType""read",
            "operationId""operational-10",
            "datastore""ncmp-datastore:passthrough-running",
            "cmHandles": [
                {
                    "id""cd4473a732724092a97cebe157604a83",
                    "cmHandleProperties": {
                        "neType""RadioNode"
                    }
                }
            ]
        }
    ]

    {
        "operations": [
            {
                "operation""read",
                "operationId""operational-10",
                "datastore""ncmp-datastore:passthrough-running",
                "resourceIdentifier""parent/child",
                "cmHandles": [
                    {
                        "id""cd4473a732724092a97cebe157604a83",
                        "cmHandleProperties": {
                            "neType""RadioNode"
                        }
                    }
                ]
            }
        ]
    }


    4Ack NCMP Request


    Code Block
    languagejava
    titleResponse 202
    collapsetrue


    202 is non-committal, meaning that there is no way for the HTTP to later send an asynchronous response indicating the outcome of processing the request. It is intended for cases where another process or server handles the request, or for batch processing.
    5Kafka Messages from DMI to NCMP


    Code Block
    titleBatch Event Payload Schema
    collapsetrue
    {
      "$schema": "https://json-schema.org/draft/2019-09/schema",
      "$id": "urn:cps:org.onap.cps.ncmp.events.async:batch-event-schema:1.0.0",
      "$ref": "#/definitions/BatchDataResponseEvent",
      "definitions": {
        "BatchDataResponseEvent
    13

    Schema for Bulk Response event forwarding to client specified  topic

    Image RemovedCPS-1557 - NCMP : forward bulk response messages to client topic

    Not keeping the 'eventTarget' which comes from the BulkResponseEvent.

    Note:  Will consider  'eventTarget', If it finalized the schema from

    Jira
    serverONAP Jira
    columnIdsissuekey,summary,issuetype,created,updated,duedate,assignee,reporter,priority,status,resolution
    columnskey,summary,type,created,updated,due,assignee,reporter,priority,status,resolution
    serverId425b2b0a-557c-3c0c-b515-579789cceedb
    keyCPS-1556

    Code Block
    titleSchema of bulk response
    collapsetrue
    { "$schema": "https://json-schema.org/draft/2019-09/schema", "$id": "urn:cps:org.onap.cps.ncmp.events:async-bulk-response-event-schema:v1", "$ref": "#/definitions/AsyncBulkResponseEvent", "definitions": { "AsyncBulkResponseEvent": { "description": "The payload for bulk response event.", "type": "object", "properties": { "eventId": { "description": "The unique id identifying the event generated by DMI for this bulk response event.", "type": "string" }, "eventCorrelationId": { "description": "The request id passed by NCMP for this bulk response event.", "type": "string" }, "eventTime": { "description": "The time of bulk response event. The expected format is 'yyyy-MM-dd'T'HH:mm:ss.SSSZ'.", "type": "string" }, "eventType": { "description": "The type of bulk response event.", "type": "string" }, "eventSchema": { "description": "The event schema for bulk response event.", "type": "string" }, "eventSchemaVersion
    ": {
          
    "description": "The 
    event
    payload 
    schema version for bulk response
    of batch event.",
    
          "type": "
    string
    object",
          "javaType" 
    }
    : "org.onap.cps.ncmp.events.async.v1.BatchEvent",
          
    "
    event
    properties": {
            "
    $ref
    event": 
    "#/definitions/Event"
    {
            
    }
      "description": "The content of Batch 
    }
    event.",
              "
    required
    type": 
    [
    "object",
              "existingJavaType": "
    eventId
    java.lang.Object",
              "
    eventCorrelationId
    additionalProperties"
    ,
    : false
            
    "eventTime",
    }
          
    "eventType"
    },
          
    "
    eventSchema
    required"
    ,
    : [
            "
    eventSchemaVersion
    event"
          ],
      
    },
        "
    Event
    additionalProperties": 
    {
    false
        }
      
    "description": "The bulk response event content.", "type": "object",
    }
    }  


    Code Block
    titleBatch Event Payload
    collapsetrue
    {     
      "event": {
         
    "
    existingJavaType
    payload": "
    java.lang.Object" }
    response of batch cm handles"
      } 
    }



    6Kafka Message(s) from NCMP to Client 
    Table



    Code Block
    title
    Sample bulk response
    Batch Event Payload
    collapsetrue
    {
    "eventId": "4cb32729-85e3-44d1-aa6e-c923b9b059a5", "eventCorrelationId": "68f15800-8ed4-4bae-9e53-27a9e03e1911", "eventTime": "2023-03-28T14:29:23.876+0000", "eventType": "org.onap.cps.ncmp.event.model.AysncBulkResponseEvent", "eventSchema": "urn:cps:org.onap.cps.ncmp.event.model.AysncBulkResponseEvent", "eventSchemaVersion": "v1", "event": { "payload": "response of batch cm handles" } }

    <Note. use green for closed issues, yellow for important ones if needed>

    Implementation

    Bulk Request Message Flow

    Gliffy Diagram
    macroId5966a68a-1641-430a-8c78-27c938f06e15
    displayNameCm Handle Batch Message Flow
    nameCm Handle Batch Interface Design
    pagePin21

    Message Flow details

    ...

    Flow Step

    ...

    Short description

    ...

    Message Details

    ...

    Notes

    ...

    Bulk Get Request

    Code Block
    title/v1/batch/data/ds/{datastore-name}
    collapsetrue
    body:
    ["cm-1",...,"cm-n"]
    Code Block
    titleExample CURL cmd : curl --location --request POST
    collapsetrue
    'http://localhost:8080/ncmp/v1/batch/data/ds/ncmp-datastore:passthrough-running?resourceIdentifier=parent/child%26options=(a=1,b=2)&topic=my-topic-name&options=(fields=schemas/schema)' \
    --header 'Authorization: Basic Y3BzdXNlcjpjcHNyMGNrcyE=' \
    --header 'Content-Type: application/json' \
    --data '[ "40137a9771f84459affa795fa1d633ab", "f5a92ec7a7db4d6fbb0e0ce2803a86cc" ]'

    ...

    Define new get operation "getResourceDataForCmHandles" into ncmp.yml

         
      "event": {
         "payload": "response of batch cm handles"
      } 
    }


    Single response format for all scenarios bot positive and error, just using optional fields instead
    7Error for 4/5 → Non responding DMI, non existing CM Handles or any other error. NCMP will have to create error message detailing all cm-handle ids affect by that error.


    Code Block
    titleNon responding DMI-plugin 
    collapsetrue
    {
      "timestamp":"2023-03-01T23:00:00.345-0400", 
      "requestId":123,
      "error": "<error-message>",
      "Cmhandles":["cm-1",...,"cm-n"]
      }


    Single response format for all scenarios bot positive and error, just using optional fields instead
    See decision # 8

    Proposed JIRAs :

    PriorityComponentDescriptionJIRAsEstimateStatus
    1DMI, NCMPData operation response  event (DMI → NCMP) to Comply with Cloud Events

    Jira
    serverONAP Jira
    columnIdsissuekey,summary,issuetype,created,updated,duedate,assignee,reporter,priority,status,resolution
    columnskey,summary,type,created,updated,due,assignee,reporter,priority,status,resolution
    serverId425b2b0a-557c-3c0c-b515-579789cceedb
    keyCPS-1723

    5 Days (M)

    Done

    2NCMPData operation response event (NCMP → Client App)

    Jira
    serverONAP Jira
    columnIdsissuekey,summary,issuetype,created,updated,duedate,assignee,reporter,priority,status,resolution
    columnskey,summary,type,created,updated,due,assignee,reporter,priority,status,resolution
    serverId425b2b0a-557c-3c0c-b515-579789cceedb
    keyCPS-1724

    TBD 

    Done

    3DMI-PluginAccept  datastore name as param into URL 

    Jira
    serverONAP Jira
    serverId425b2b0a-557c-3c0c-b515-579789cceedb
    keyCPS-1550

    5 DaysDone
    4NCMP Expose REST endpoint to accept collection of cm handles for GET operation (Passthrough only)

    Jira
    serverONAP Jira
    serverId425b2b0a-557c-3c0c-b515-579789cceedb
    keyCPS-1553

    15 DaysDone
    5DMI-PluginExpose endpoint for ONAP not impl. and Stub impl. for testing/demo

    Jira
    serverONAP Jira
    serverId425b2b0a-557c-3c0c-b515-579789cceedb
    keyCPS-1555

    5 DaysDone
    6NCMPNCMP: Update existing REST endpoint that accepts data operation request for GET operation

    Jira
    serverONAP Jira
    columnIdsissuekey,summary,issuetype,created,updated,duedate,assignee,reporter,priority,status,resolution
    columnskey,summary,type,created,updated,due,assignee,reporter,priority,status,resolution
    serverId425b2b0a-557c-3c0c-b515-579789cceedb
    keyCPS-1635

    5 Days

    Done

    7DMI-PluginDMI-Plugin : Update endpoint to accept data operation request

    Jira
    serverONAP Jira
    columnIdsissuekey,summary,issuetype,created,updated,duedate,assignee,reporter,priority,status,resolution
    columnskey,summary,type,created,updated,due,assignee,reporter,priority,status,resolution
    serverId425b2b0a-557c-3c0c-b515-579789cceedb
    keyCPS-1636

    3 Days

    Done

    8Stubbed DMI-PluginInclude code to send response messages to internal kafka topic with delay 

    Jira
    serverONAP Jira
    serverId425b2b0a-557c-3c0c-b515-579789cceedb
    keyCPS-1556

    10 Days

    Done

    9NCMP Forward response messages to client given kafka topic

    Jira
    serverONAP Jira
    serverId425b2b0a-557c-3c0c-b515-579789cceedb
    keyCPS-1557

    5 Days

    Done

    10NCMP Handle non-existing cm handles

    Jira
    serverONAP Jira
    columnIdsissuekey,summary,issuetype,created,updated,duedate,assignee,reporter,priority,status,resolution
    columnskey,summary,type,created,updated,due,assignee,reporter,priority,status,resolution

    ...

    Code Block
    titleResponse 200
    collapsetrue
    {"requestId":"123"}

    ...

    Code Block
    title/v1/ch/batch/data/ds/{datastore-name}&topic=topicParamInQuery
    collapsetrue
    body: {"Cmhandles":["cm-1",...,"cm-n"],"requestId":123}

    The DMI PLugin should be told (included in request) the client topic so that NCMP does not have to 'remember' to relation between request id and client topic!

    ...

    titleResponse 202
    collapsetrue

    ...

    Code Block
    titleTopic: ncmp-async-m2m
    collapsetrue
    Message Details TBD

    ...

    Code Block
    titleTopic: <client topic>
    collapsetrue
    Message Details TBD

    ...

    Response message structure ? (Flow no. 5)

    Non responding DMI-plugin 

    Code Block
    titleNon responding DMI-plugin 
    collapsetrue
    {
      "timestamp":"2023-03-01T23:00:00.345-0400", 
      "requestId":123,
      "error": "DMI Service Unavailable, {service-name}",
      "Cmhandles":["cm-1",...,"cm-n"]
      }

    ...

    Response message structure ? (Flow no. 5)

    Non existing cm handles

    Code Block
    titleNon existing cm handles
    collapsetrue
    {
    	"timestamp":"2023-03-01T23:00:00.345-0400",
        "requestId":123,
    	"error":"Cm-Handle not found",
    	"Cmhandles":["cm-1",...,"cm-n"]
    }

    ...

    Code Block
    titleNon READY existing cm handles
    collapsetrue
    {
    	"timestamp":"2023-03-01T23:00:00.345-0400",
        "requestId":123,
    	"error":"Cm-Handle not in READY state.",
    	"Cmhandles":["cm-1",...,"cm-n"]
    }

    ...

    Existing DMI endpoints are : 

    /v1/ch/{cmHandle}/data/ds/{datastore-name}

    datastore-name:

    1. ncmp-datastore:passthrough-operational
    2. ncmp-datastore:passthrough-running

     ...&topic=topicParamInQuery

    CPS Proposed :

    /v1/ch/batch/data/ds/{datastore-name}

     ...&topic=topicParamInQuery

    cm handle ids and requestid into body

    Proposed JIRAs :

    #ComponentDescriptionJIRAsEstimate1DMI-PluginAccept  datastore name as param into URL  JiraserverONAP Jira

    serverId425b2b0a-557c-3c0c-b515-579789cceedb
    keyCPS-

    1550

    1583

    5 Days
    2

    Done

    11NCMP 
    Expose REST endpoint to accept collection of cm handles for GET operation (Passthrough only)
    Error handling for non-ready cm handle state

    Jira
    serverONAP Jira
    serverId425b2b0a-557c-3c0c-b515-579789cceedb
    keyCPS-

    1553

    1614

    15
    5 Days

    Done

    3
    12NCMP Handle non responding DMI-Plugin
    Expose endpint for ONAP not impl. and Stub impl. for testing/demo

    Jira
    serverONAP Jira
    serverId425b2b0a-557c-3c0c-b515-579789cceedb
    keyCPS-

    1555

    1558

    5 Days
    4Stubbed DMI-PluginInclude code to send response messages to internal kafka topic with delay 

    Done

    13CSIT test for demo

    Jira
    serverONAP Jira
    serverId425b2b0a-557c-3c0c-b515-579789cceedb
    keyCPS-

    1556

    1560

    10
    5 Days
    5NCMP Forward response messages to client given kafka topic

    Done

    14Update RTD 

    Jira
    serverONAP Jira
    serverId425b2b0a-557c-3c0c-b515-579789cceedb
    keyCPS

    -1557

    -1785

    4 Daya
    15Test performanceCapabilities
    5 Days6NCMP Handle non-existing cm handles

    Jira
    serverONAP Jira
    columnIdsissuekey,summary,issuetype,created,updated,duedate,assignee,reporter,priority,status,resolution
    columnskey,summary,type,created,updated,due,assignee,reporter,priority,status,resolution
    serverId425b2b0a-557c-3c0c-b515-579789cceedb
    keyCPS-1583

    5 Days7NCMP Error handling for non-ready cm handle state

    Jira
    serverONAP Jira
    serverId425b2b0a-557c-3c0c-b515-579789cceedb
    key

    CPS-16145 Days

    CPS-1786

    3 Days

    Done

    (NCMP: Read data operation resource API Performance for multiple cm-handles)

    16NCMPModify DMI data operation request body. 
    8NCMP Handle non responding DMI-Plugin

    Jira
    serverONAP Jira
    serverId425b2b0a-557c-3c0c-b515-579789cceedb
    keyCPS-

    1558

    1833

    5 Days9CSIT test for demo

    Jira
    serverONAP Jira
    serverId425b2b0a-557c-3c0c-b515-579789cceedb
    keyCPS-1560

    5 Days

    Planning :

    2 DaysDone

    Planning :

    • Allow for 2 more user stories each may take 1 weeks
    • Based on 2 resource working parallelly may take approx.. 8 weeks from 29 March 2023  
    • (updated after additional items where added) The estimated date for completion is   
    • The release for cps-and-ncmp 3.3.3 version is completed on Friday,  

      Release tag : https://gerrit.onap.org/r/gitweb?p=cps.git;a=tag;h=refs%2Ftags%2F3.3.3

    • Demo recording can be referred here : https://wiki.onap.org/display/DW/CPS+User+Story+Demos
    • Allow for 2 more user stories each may take 1 weeks
    • Based on 2 resource working  parallelly may take àpprox. 6 weeks from 29 March 2023  
    • The estimated date for completion is 19th May ' 2023.