Create 'bulk' version for org.onap.cps.ncmp.rest.controller.NetworkCmProxyController#getResourceDataForCmHandle to support multiple cm-handles (~ multiple anchors in in CPS-Core)
Table of Contents
References
CPS-1515- Spike: Support multiple CM-Handles for NCMP Get Operation
Support batch read operation (new) using an asynchronous response on a client specified topic
payload includes list of cm handle ids
2
REST DMI-I-01
Support batch read operation (new) using an asynchronous response to an internal topic
payload includes list of (associated for this plugin) cm handles ids with their private (additional) properties
Error Handling
#
Error Scenario
Expected behavior
1
DMI Not respond to initial synchronous request with normal HTTP timeout
Special Error message (Kafka event) including affected, and reason for all the handles in the message cm handles send to client topic detailing cm-handles (per dmi plugin)
2
Topic not supplied on CPS-E-05
Return HTTP 501 (not implemented)
3
Client specided Topic is not configured
Log error message only
4
Non-existing cm-handle id
Similar message but different reason as specified #1 above
Capabilities
Excerpt
#
Parameter
Expectation
Notes
1
Response Time 1 Batch request
<2 seconds (average)
Async response available on client topic
No delay in DMI PLugin Plugin (tested/measured using stub DMI Plugin)
2
Batch-size
200 cm handles
No hardcoded limit
3
Response payload size
~2 KB per cm handles
Performance test for capability should be tested with this average response size
Should not affect performance, does not need to be tested
Out-of-scope
Support for multiple resource identifiers in one batch operation
Support cached data batch requests: only passthrough datastores will be supported (see decision #2)
NCMP does NOT keep track of request status to see if it is completed, amalgamate responses or anything like that. It wil simply forward responses from the internal topic to the client topic.
Access control
Assumptions
#
Assumption
Notes
1
Proprietary options or not in the scope of this analysis.
agreed with kieran mccarthy ; these are optional parameters (name-value pairs) not interpreted by NCMP but can be interpreted by proprietary plugins In the future "scope" might become standardized instead of proprietary but that wil be achieved through a separate requirement
2
same xpath (resourceIdentifierInQuery) for all cm handles or different for each cm handle
agreed with kieran mccarthy ; if different resources are required on the same cm-handle the client wil send another (batch) request
3
options, resourceIdentifier is optional for bulk operations.
operation, datastore and cmhandleIds are mandatory fields
agreed with CPS team.
4
We are not merging any duplicate cm handles while sending request to dmi-plugin
agreed
5
Get batch operation does not support includeDescendants as it is impl. for
pass through datastore only
ncmp-datastore:passthrough-running and
ncmp-datastore:passthrough-operational
agreed on
Issues & Decisions
#
Issue
Notes
Decision
1
Which operation(s) need support for multiple cm handles?
Not implemented (yet) response, for non passthrough datastores
3
URL pattern for NCMP bulk endpoints
We had an internal review with some of our rApp colleagues around some of the recent proposed NCMP batch interface and they came back with some valid comment. The proposal is that we should not distinguish batch from bulk or other flavors of read/write.
The aim is to only have a single flavor of interface for read or write for clients. Therefore the proposal is to drop “batch” from the interface URL and just act toward “data”
Additional error (messages) response with all cm-handles that cannot be resolved, also a separate error message wil be sent for each failed DMI
11
Overlap/clash with Deutsche Telekom user story:
Jira
server
ONAP Jira
serverId
425b2b0a-557c-3c0c-b515-579789cceedb
key
CPS-1377
?!
discussed in weekly ONAP meeting the DT user story is affect CPS-Core interface (not NCMP) and the requirement is to execute a query over ALL cm-handles (cached only?!) instead of a given list of cm-handles (~anchors)
1) Agreement required on the structure of the response. Please see response structure below. 2) Does 'eventTime' field which holds the timestamp of the bulk response event, required or can it be dropped?
Need to follow schema structure there in #15 under Issues & Decisions section and it is agreed on
13
Schema for Bulk Response event forwarding to client specified topic
1. Do we need to send only one response containing all the requested cm handles ? 2. Send single response for each cm handle? This could lead to to much data in the event for large batch requests. Csaba Kocsis , Sourabh Sourabh and Toine Siebelink proposed to us single event for each cm handle 3. Single response message per DMI plugin? Just is a list of IDS should be OK.
We have had some internal discussions including with some O-RAN standards representatives and one of the outcomes is that it would be good if we aligned with the some community standards for event header definitions. IT is proposed (initially from AT&T) that we should follow Cloud Native Computing Foundation (CNCF) specification as defined in their cloudevents incubator project. Some non-standard headers wil need to be implemented as 'extensions' and names to be confirmed
'eventType' value as below 'org.onap.cps.ncmp.events.BatchDataEvent'
proposed Operation like :
'org.onap.cps.ncmp.events.DataOperationEvent'
'batch' was the keyword use in the URL for this feature (now gone) but much code (classnames, variable names) etc uses this name still...
kieran mccarthy and Team agreed on and some cost (1-2 days) is acceptable
18
What would be the field name of response data?
data.responses[0].data
kieran mccarthyplease confirm : we are using 'responseContent' as name of this field as 'data' is already used for cloud event standard payload.
Now it is like : data.responses[0].responseContent
Sourabh Sourabh Suggest to use "result". This is after all where the actual result output. the other tags in responses[x] are status related to the request :
(see org.onap.cps.ncmp.api.impl.utils.EventDateTimeFormatter )
Toine Siebelink Although not clearly communicated before ISO8601 is the standard to be used and it seems the current implementation in NCMP conforms to this standard
Additional error (messages) response with all cm-handles that cannot be resolved, also a separate error message wil be sent for each failed DMI
11
Overlap/clash with Deutsche Telekom user story:
Jira
server
ONAP Jira
serverId
425b2b0a-557c-3c0c-b515-579789cceedb
key
CPS-1377
?!
discussed in weekly ONAP meeting the DT user story is affect CPS-Core interface (not NCMP) and the requirement is to execute a query over ALL cm-handles (cached only?!) instead of a given list of cm-handles (~anchors)
1) Agreement required on the structure of the response. Please see response structure below. 2) Does 'eventTime' field which holds the timestamp of the bulk response event, required or can it be dropped?
Code Block
title
Batch Event Headers Schema
collapse
true
{
"$schema": "https://json-schema.org/draft/2019-09/schema",
"$id": "urn:cps:org.onap.cps.ncmp.events.async:batch-event-headers:1.0.0",
"$ref": "#/definitions/BatchEventHeaders",
"definitions": {
"BatchEventHeaders": {
"description": "The header information of the Batch event.",
"type": "object",
"javaType" : "org.onap.cps.ncmp.events.async.v1.BatchEventHeaders",
"properties": {
"eventId": {
"description": "The unique id for identifying the event.",
"type": "string"
},
"eventCorrelationId": {
"description": "The request id received by NCMP as an acknowledgement.",
"type": "string"
},
"eventTime": {
"description": "The time of the event. It should be in RFC format ('yyyy-MM- dd'T'HH:mm:ss.SSSZ').",
"type": "string"
},
"eventTarget": {
"description": "The destination topic to forward the consumed event.",
"type": "string"
},
"eventSource": {
"description": "The source of the event.",
"type": "string"
},
"eventType": {
"description": "The type of the Batch event.",
"type": "string"
},
"eventSchema": {
"description": "The schema of the Batch event payload.",
"type": "string"
},
"eventSchemaVersion": {
"description": "The schema version of the Batch event payload.",
"type": "string"
}
},
"required": [
"eventId",
"eventCorrelationId",
"eventTarget",
"eventType",
"eventSchema",
"eventSchemaVersion"
],
"additionalProperties": false
}
}
}
1. Do we need to send only one response containing all the requested cm handles ? 2. Send single response for each cm handle? 3. Single response message per DMI plugin? kieran mccarthy - Please share your inputs on the above listed questions.
The DMI PLugin should be told (included in request) the client topic so that NCMP does not have to 'remember' to relation between request id and client topic!
202 is non-committal, meaning that there is no way for the HTTP to later send an asynchronous response indicating the outcome of processing the request. It is intended for cases where another process or server handles the request, or for batch processing.
{
"event": {
"payload": "response of batch cm handles"
}
}
Single response format for all scenarios bot positive and error, just using optional fields instead
7
Error for 4/5 → Non responding DMI, non existing CM Handles or any other error. NCMP will have to create error message detailing all cm-handle ids affect by that error.
Code Block
title
Non responding DMI-plugin
Code Block
title
Batch Event Payload
collapse
true
{
"timestamp":"2023-03-01T23:00:00.345-0400",
"
event
requestId":
{
123,"error":"<error-message>",
"
payload
Cmhandles":
"response of batch cm handles"
}
}
7
Alternative for 4/5 → Non responding DMI. NCMP will have to create error message detailing cm-handles
See decision # 8 and 9
8
Response message structure ? (Flow no. 5)
Non responding DMI-plugin
Code Block
title
Non responding DMI-plugin
collapse
true
{
"timestamp":"2023-03-01T23:00:00.345-0400",
"requestId":123,
"error": "DMI Service Unavailable, {service-name}",
"Cmhandles":["cm-1",...,"cm-n"]
}
9
Response message structure ? (Flow no. 5)
Non existing cm handles
Code Block
title
Non existing cm handles
collapse
true
{
"timestamp":"2023-03-01T23:00:00.345-0400",
"requestId":123,
"error":"Cm-Handle not found",
"Cmhandles":["cm-1",...,"cm-n"]
}
10
Non Ready cm handles
Code Block
title
Non READY existing cm handles
collapse
true
{
"timestamp":"2023-03-01T23:00:00.345-0400",
"requestId":123,
"error":"Cm-Handle not in READY state.",
"Cmhandles":["cm-1",...,"cm-n"]
}
11
URL pattern for DMI-Plugin bulk endpoints
Existing DMI endpoints are :
/v1/ch/{cmHandle}/data/ds/{datastore-name}
datastore-name:
ncmp-datastore:passthrough-operational
ncmp-datastore:passthrough-running
...&topic=topicParamInQuery
CPS Proposed :
/v1/ch/batch/data/ds/{datastore-name}
...&topic=topicParamInQuery
cm handle ids and requestid into body
Proposed JIRAs :
["cm-1",...,"cm-n"]
}
Single response format for all scenarios bot positive and error, just using optional fields instead See decision # 8
Proposed JIRAs :
Priority
Component
Description
JIRAs
Estimate
Status
1
DMI-Plugin
Accept datastore name as param into URL
Jira
server
ONAP Jira
Priority
Component
Description
JIRAs
Estimate
Status
1
DMI, NCMP
Data operation response event (DMI → NCMP) to Comply with Cloud Events
Allow for 2 more user stories each may take 1 weeks
Based on 2 resource working working parallelly may take àpproxapprox.. 6 8 weeks from 29 March 2023 The estimated date for completion is 31st May ' 2023. 2023
(updated after additional items where added) The estimated date for completion is
The release for cps-and-ncmp 3.3.3 version is completed on Friday,
CPS-1515 - Spike: Support multiple CM-Handles for NCMP Get Operation