Create 'bulk' version for org.onap.cps.ncmp.rest.controller.NetworkCmProxyController#getResourceDataForCmHandle to support multiple cm-handles (~ multiple anchors in in CPS-Core)
Table of Contents
References
CPS-1515- Spike: Support multiple CM-Handles for NCMP Get Operation
Support batch read operation (new) using an asynchronous response on a client specified topic
payload includes list of cm handle ids
2
REST DMI-I-01
Support batch read operation (new) using an asynchronous response to an internal topic
payload includes list of (associated for this plugin) cm handles ids with their private (additional) properties
Error Handling
#
Error Scenario
Expected behavior
1
DMI Not respond to initial synchronous request with normal HTTP timeout
Special Error message (Kafka event) including affected, and reason for all the handles in the message cm handles send to client topic detailing cm-handles (per dmi plugin)
2
Topic not supplied on CPS-E-05
Return HTTP 501 (not implemented)
3
Client specided Topic is not configured
Log error message only
4
Non-existing cm-handle id
Similar message but different reason as specified #1 above
Capabilities
Excerpt
#
Parameter
Expectation
Notes
1
Response Time 1 Batch request
<2 seconds (average)
Async response available on client topic
No delay in DMI PLugin Plugin (tested/measured using stub DMI Plugin)
2
Batch-size
200 cm handles
No hardcoded limit
3
Response payload size
~2 KB per cm handles
Performance test for capability should be tested with this average response size
Should not affect performance, does not need to be tested
Out-of-scope
Support for multiple resource identifiers in one batch operation
Support cached data batch requests: only passthrough datastores will be supported (see decision #2)
NCMP does NOT keep track of request status to see if it is completed, amalgamate responses or anything like that. It wil simply forward responses from the internal topic to the client topic.
Access control
Assumptions
#
Assumption
Notes
1
Proprietary options or not in the scope of this analysis.
agreed with kieran mccarthy ; these are optional parameters (name-value pairs) not interpreted by NCMP but can be interpreted by proprietary plugins In the future "scope" might become standardized instead of proprietary but that wil be achieved through a separate requirement
2
same xpath (resourceIdentifierInQuery) for all cm handles or different for each cm handle
agreed with kieran mccarthy ; if different resources are required on the same cm-handle the client wil send another (batch) request
3
options, resourceIdentifier is optional for bulk operations.
operation, datastore and cmhandleIds are mandatory fields
agreed with CPS team.
4
We are not merging any duplicate cm handles while sending request to dmi-plugin
agreed
5
Get batch operation does not support includeDescendants as it is impl. for
pass through datastore only
ncmp-datastore:passthrough-running and
ncmp-datastore:passthrough-operational
agreed on
Issues & Decisions
#
Issue
Notes
Decision
1
Which operation(s) need support for multiple cm handles?
Not implemented (yet) response, for non passthrough datastores
3
URL pattern for NCMP bulk endpoints
We had an internal review with some of our rApp colleagues around some of the recent proposed NCMP batch interface and they came back with some valid comment. The proposal is that we should not distinguish batch from bulk or other flavors of read/write.
The aim is to only have a single flavor of interface for read or write for clients. Therefore the proposal is to drop “batch” from the interface URL and just act toward “data”
Additional error (messages) response with all cm-handles that cannot be resolved, also a separate error message wil be sent for each failed DMI
11
Overlap/clash with Deutsche Telekom user story:
Jira
server
ONAP Jira
serverId
425b2b0a-557c-3c0c-b515-579789cceedb
key
CPS-1377
?!
discussed in weekly ONAP meeting the DT user story is affect CPS-Core interface (not NCMP) and the requirement is to execute a query over ALL cm-handles (cached only?!) instead of a given list of cm-handles (~anchors)
1) Agreement required on the structure of the response. Please see response structure below. 2) Does 'eventTime' field which holds the timestamp of the bulk response event, required or can it be dropped?
Need to follow schema structure there in #15 under Issues & Decisions section and it is agreed on
13
Schema for Bulk Response event forwarding to client specified topic
1. Do we need to send only one response containing all the requested cm handles ? 2. Send single response for each cm handle? This could lead to to much data in the event for large batch requests. Csaba Kocsis , Sourabh Sourabh and Toine Siebelink proposed to us single event for each cm handle 3. Single response message per DMI plugin? Just is a list of IDS should be OK.
We have had some internal discussions including with some O-RAN standards representatives and one of the outcomes is that it would be good if we aligned with the some community standards for event header definitions. IT is proposed (initially from AT&T) that we should follow Cloud Native Computing Foundation (CNCF) specification as defined in their cloudevents incubator project. Some non-standard headers wil need to be implemented as 'extensions' and names to be confirmed
'eventType' value as below 'org.onap.cps.ncmp.events.BatchDataEvent'
proposed Operation like :
'org.onap.cps.ncmp.events.DataOperationEvent'
'batch' was the keyword use in the URL for this feature (now gone) but much code (classnames, variable names) etc uses this name still...
kieran mccarthy and Team agreed on and some cost (1-2 days) is acceptable
18
What would be the field name of response data?
data.responses[0].data
kieran mccarthyplease confirm : we are using 'responseContent' as name of this field as 'data' is already used for cloud event standard payload.
Now it is like : data.responses[0].responseContent
Sourabh Sourabh Suggest to use "result". This is after all where the actual result output. the other tags in responses[x] are status related to the request :
(see org.onap.cps.ncmp.api.impl.utils.EventDateTimeFormatter )
Toine Siebelink Although not clearly communicated before ISO8601 is the standard to be used and it seems the current implementation in NCMP conforms to this standard
To prevent problems with large data request multiple response event will be sent for a successful read for each cm handle (id)
Error messages can contain information about single cm handle ids
Field
Type
Description
Mandatory?
Notes
data
Event
The payload of an event
Mandatory
Cloud event standard (data)
data.responses[0, 1, 2, ...]
Array
contains an array or batch response that includes both success and failure.
Mandatory
data.responses[0].operationId
String
specified to distinguish multiple operations using same cmhandleId
Mandatory
data.responses[0].ids
String
cmhandle-ids
Mandatory
Example : ["0df4d39af6514d99b816758148389cfd"] Note: Ids array should contain only a single element in the array in case of success messages and In case of error it can have any number of elements.
data.responses[0].statusCode
String
Mandatory
Common NCMP defined error codes:
status-code 0-99 is reserved for any success response
status-code from 100 to 199 is reserved for any failed response.
data.responses[0].statusMessage
String
Mandatory
Examples for code & message :
statusCode
statusMessage
0
Successfully applied changes.
100
cm handle id(s) is(are) not found.
101
cm handle id(s) is(are) in non ready state.
data.responses[0].data
Object
Optional
In case of success :
Optional, for write operations then no need to return configurations application/yang-patch+json | application/yang-data+json
In case of failure :
Optional, any supplementary error data matching the error status-code
We had an internal review with some of our rApp colleagues around some of the recent proposed NCMP batch interface and they came back with some valid comment.
The proposal is that we should not distinguish batch from bulk or other flavours of read/write.
The aim is to only have a single flavour of interface for read or write for clients. Therefore the proposal is to drop “batch” from the interface URL and just act toward “data” (reads/writes/actions)
Additional error (messages) response with all cm-handles that cannot be resolved, also a separate error message wil be sent for each failed DMI
11
Overlap/clash with Deutsche Telekom user story:
Jira
server
ONAP Jira
serverId
425b2b0a-557c-3c0c-b515-579789cceedb
key
CPS-1377
?!
discussed in weekly ONAP meeting the DT user story is affect CPS-Core interface (not NCMP) and the requirement is to execute a query over ALL cm-handles (cached only?!) instead of a given list of cm-handles (~anchors)
1) Agreement required on the structure of the response. Please see response structure below. 2) Does 'eventTime' field which holds the timestamp of the bulk response event, required or can it be dropped?
1. Do we need to send only one response containing all the requested cm handles ? 2. Send single response for each cm handle? 3. Single response message per DMI plugin? kieran mccarthy - Please share your inputs on the above listed questions.
We have had some internal discussions including with some O-RAN standards representatives and one of the outcomes is that it would be good if we aligned with the some community standards for event header definitions. IT is proposed (initially from AT&T) that we should follow Cloud Native Computing Foundation (CNCF) specification as defined in their cloudevents incubator project. More details see CPS Events Structure#CNCFCloudEventalignment
To prevent problems with large data request a single response event will be sent for a successful read for each cm handle (id)
Error message can contain information about multiple cm handle ids
...
Field
...
Type
...
Description
...
Mandatory?
...
Notes
...
event
...
Mandatory
...
event.responses[0, 1, 2, ...]
...
Mandatory
...
event.responses[0].operationId
...
Mandatory
...
event.responses[0].ids
...
Mandatory
...
Example : ["0df4d39af6514d99b816758148389cfd"] Note: Ids array should contain only a single element in the array in case of success messages and In case of error it can have any number of elements.
}
]
New JIRA
Jira
server
ONAP Jira
serverId
425b2b0a-557c-3c0c-b515-579789cceedb
key
CPS-1833
is created to address it.
Discussed with Csaba Kocsis and agreed to send identical data request body as NCMP where data operation requat body is as follows:
Client App→ NCMP
NCMP→ DMI
Agreed
operation
operationType
operation
operation details are wrapped into parent attribute "operations" like "operations": [ ...
operation details ar not wrapped
wrap operation details into parent attribute "operations"
202 is non-committal, meaning that there is no way for the HTTP to later send an asynchronous response indicating the outcome of processing the request. It is intended for cases where another process or server handles the request, or for batch processing.
The DMI PLugin should be told (included in request) the client topic so that NCMP does not have to 'remember' to relation between request id and client topic!
4
Ack NCMP Request
Code Block
title
Response 202
collapse
true
202 is non-committal, meaning that there is no way for the HTTP to later send an asynchronous response indicating the outcome of processing the request. It is intended for cases where another process or server handles the request, or for batch processing.
Expose REST endpoint to accept collection of cm handles for GET operation (Passthrough only)DMI-Plugin : Update endpoint to accept data operation request