Create 'bulk' version for org.onap.cps.ncmp.rest.controller.NetworkCmProxyController#getResourceDataForCmHandle to support multiple cm-handles (~ multiple anchors in in CPS-Core)
Table of Contents |
---|
References
- 515 CPS-1 - Spike: Support multiple CM-Handles for NCMP Get Operation
- CPS-NCMP ↔ DMI-Plugin Interface Details Jakarta-R10
- CPS Events Structure
Requirements
Functional
# | Interface | Requirement | Additional Information |
---|---|---|---|
1 | REST CPS(-NCMP)-E-05 | Support batch read operation (new) using an asynchronous response on a client specified topic | payload includes list of cm handle ids |
2 | REST DMI-I-01 | Support batch read operation (new) using an asynchronous response to an internal topic | payload includes list of (associated for this plugin) cm handles ids with their private (additional) properties |
Error Handling
# | Error Scenario | Expected behavior |
---|---|---|
1 | DMI Not respond to initial synchronous request with normal HTTP timeout | Special Error message (Kafka event) including affected, and reason for all the handles in the message cm handles send to client topic detailing cm-handles (per dmi plugin) |
2 | Topic not supplied on CPS-E-05 | Return HTTP 501 (not implemented) |
3 | Client specided Topic is not configured | Log error message only |
4 | Non-existing cm-handle id | Similar message but different reason as specified #1 above |
Capabilities
Excerpt | ||||||||||||||||||||||||||||||||||||||
---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
|
Out-of-scope
- Support for multiple resource identifiers in one batch operation
- Support cached data batch requests: only passthrough datastores will be supported (see decision #2)
- NCMP does NOT keep track of request status to see if it is completed, amalgamate responses or anything like that. It wil simply forward responses from the internal topic to the client topic.
- Access control
Assumptions
# | Assumption | Notes |
---|---|---|
1 | Proprietary options or not in the scope of this analysis. | agreed with kieran mccarthy ; these are optional parameters (name-value pairs) not interpreted by NCMP but can be interpreted by proprietary plugins |
2 | same xpath (resourceIdentifierInQuery) for all cm handles or different for each cm handle | agreed with kieran mccarthy ; if different resources are required on the same cm-handle the client wil send another (batch) request |
3 |
| agreed with CPS team. |
4 | We are not merging any duplicate cm handles while sending request to dmi-plugin | agreed |
5 | Get batch operation does not support includeDescendants as it is impl. for pass through datastore only ncmp-datastore:passthrough-running and ncmp-datastore:passthrough-operational | agreed on |
Issues & Decisions
...
Event Body (data)
- The current proposal is to use the CNFC Event structure and library as detail in CPS Events Structure#CNCFCloudEventalignment
- To prevent problems with large data request multiple response event will be sent for a successful read for each cm handle (id)
- Error messages can contain information about single cm handle ids
Field
Type
Description
Mandatory?
Notes
data
Mandatory
data
) data
.responses[0, 1, 2, ...]
Mandatory
# | Issue | Notes | Decision | ||||||||||||
---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
1 | Which operation(s) need support for multiple cm handles? |
if many what is the priority? | agreed with kieran mccarthy Only Get (read) (in future other operations might be support batch option too) | ||||||||||||
2 | Which datasources should be supported? | Do we need to support passthrough-only no-cached() data only ? | agreed with kieran mccarthy : all passthrough datastores will be supported Not implemented (yet) response, for non passthrough datastores | ||||||||||||
3 | URL pattern for NCMP bulk endpoints | We had an internal review with some of our rApp colleagues around some of the recent proposed NCMP batch interface and they came back with some valid comment. The proposal is that we should not distinguish batch from bulk or other flavors of read/write. The aim is to only have a single flavor of interface for read or write for clients. Therefore the proposal is to drop “batch” from the interface URL and just act toward “data” | Agreed with team kieran mccarthy | ||||||||||||
4 | keep datastore, topic and optional parameters in the URL itself instead into body.
| CPS prefers keep interface similar as single cm handle interface (consistency and cost) Existing : ...&topic=topicParamInQuery | agreed with kieran mccarthy : Follow the existing interfaces as much as possible for consistency and efficiency | ||||||||||||
5 | support in ONAP DMI-plugin | agreed with Toine Siebelink : ONAP plugin can respond with not implemented yet code, | |||||||||||||
6 | Response always Async ie. topic is compulsory ? | Assume topic is compulsory (defined in OPenApi) → Response therefore wil be 400 if not supplied | Agreed with kieran mccarthy : Topic is optional but system will respond with 'Not implemented (yet when not specified or blank | ||||||||||||
7 | Should NCMP Amalgamate Async responses from DMI-Plugin before forwarding ? | step 6 in flow diagram | Agreed with kieran mccarthy : NCMP wil only forward to client topic no handling tracking or any responses or status of request | ||||||||||||
8 | Handle non responding dmi-plugin | Agreed with kieran mccarthy : | |||||||||||||
9 | Should (can) NCMP check if 'MyTopic' specified by client exist | Consider Access Control too. For now NCMP can log error. Client is responsible for topic setup | Agreed with kieran mccarthy : | ||||||||||||
10 | How to handle non-existing CM-Handles (id) | Suggestions
| Agreed with kieran mccarthy : Additional error (messages) response with all cm-handles that cannot be resolved, also a separate error message wil be sent for each failed DMI | ||||||||||||
11 | Overlap/clash with Deutsche Telekom user story:
| discussed in weekly ONAP meeting the DT user story is affect CPS-Core interface (not NCMP) and the requirement is to execute a query over ALL cm-handles (cached only?!) instead of a given list of cm-handles (~anchors) | |||||||||||||
12 | Schema of bulk response
| 1) Agreement required on the structure of the response. Please see response structure below. | Need to follow schema structure there in #15 under Issues & Decisions section and it is agreed on | ||||||||||||
13 |
CPS-1557 - NCMP : forward bulk response messages to client topic |
| Agreed on to keep one schema for both events (DMI → NCMP) and (NCMP → ClientApps) | ||||||||||||
14 | How NCMP would forward response? (Response data : ref. message flow #6)
| 1. Do we need to send only one response containing all the requested cm handles ? | kieran mccarthy Confirmed in meeting | ||||||||||||
15 | Message format for the batch interface | We have had some internal discussions including with some O-RAN standards representatives and one of the outcomes is that it would be good if we aligned with the some community standards for event header definitions. IT is proposed (initially from AT&T) that we should follow Cloud Native Computing Foundation (CNCF) specification as defined in their cloudevents incubator project. Some non-standard headers wil need to be implemented as 'extensions' and names to be confirmed
|
| ||||||||||||
16 | NCMP will send only one request to each DMI |
NCMP batch endpoint : http://localhost:8080/ncmp/v1/data?&topic=my-topic-name DMI-Plugin batch endpoint : http://172.26.202.25:8783/dmi/v1/data?topic=my-topic-name&requestId=e6fa4d26-4dc1-4877-aa3c-45e99f840708
| Agreed with Csaba Kocsis and kieran mccarthy | ||||||||||||
17 | Feature Name (events) | event type proposal : from DataOperationEvent For example : DataOperationResponseEvent 'eventType' value as below 'org.onap.cps.ncmp.events. proposed Operation like : 'org.onap.cps.ncmp.events.DataOperationEvent' 'batch' was the keyword use in the URL for this feature (now gone) but much code (classnames, variable names) etc uses this name still... | kieran mccarthy and Team agreed on and some cost (1-2 days) is acceptable |
Event Format Definitions
Event Headers
...
18 | What would be the field name of response data? | data .responses[0].data | kieran mccarthy please confirm : we are using 'responseContent' as name of this field as 'data' is already used for cloud event standard payload. Now it is like : Sourabh Sourabh Suggest to use "result". This is after all where the actual result output. the other tags in responses[x] are status related to the request :
|
|
result | |||
19 | Exact format of timestamp of cloud events. | CloudEvent documentation (https://github.com/cloudevents/spec/blob/v1.0.2/cloudevents/spec.md#time) refers on ISO8601 standard (see org.onap.cps.ncmp.api.impl.utils.EventDateTimeFormatter ) | Toine Siebelink Although not clearly communicated before ISO8601 is the standard to be used and it seems the current implementation in NCMP conforms to this standard |
Event Format Definitions
Event Headers
Excerpt Include | ||||
---|---|---|---|---|
|
Event Body (data)
- The current proposal is to use the CNFC Event structure and library as detail in CPS Events Structure#CNCFCloudEventalignment
- To prevent problems with large data request multiple response event will be sent for a successful read for each cm handle (id)
- Error messages can contain information about single cm handle ids
Field | Type | Description | Mandatory? | Notes | |||||||||||||
---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| Event | The payload of an event | Mandatory | Cloud event standard (data ) | |||||||||||||
| Array | contains an array or batch response that includes both success and failure. | Mandatory | ||||||||||||||
| String | specified to distinguish multiple operations using same cmhandleId | Mandatory | ||||||||||||||
| String | cmhandle-ids | Mandatory | Example : ["0df4d39af6514d99b816758148389cfd"] | String | specified to distinguish multiple operations using same cmhandleId | Mandatory | ||||||||||
| String | cmhandle-ids | Mandatory | Example : ["0df4d39af6514d99b816758148389cfd"] | |||||||||||||
| String | Mandatory | Common NCMP defined error codes:
|
| String | Mandatory |
statusCode | statusMessage |
---|---|
1 | "Successfully applied changes" |
101 | "cmHandle(s) do not exist" |
data
.responses[0].dataOptional
- In case of success :
- Optional, for write operations then no need to return configurations
application/yang-patch+json | application/yang-data+json
- Optional, for write operations then no need to return configurations
- In case of failure :
- Optional, any supplementary error data matching the error status-code
kieran mccarthy please confirm : we are using 'responseContent' as name of this field as 'data' is already used for cloud event standard payload.
Now it is like :data
.responses[0].
responseContentstatusCode
Implementation
Bulk Request Message Flow
Gliffy Diagram | ||||||||
---|---|---|---|---|---|---|---|---|
|
Message Flow details
Mandatory | Common NCMP defined error codes:
| |||||||||||
| String | Mandatory | Examples for code & message :
| |||||||||
data .responses[0].data | Object | Optional |
|
Implementation
Bulk Request Message Flow
Gliffy Diagram | ||||||||
---|---|---|---|---|---|---|---|---|
|
Message Flow details
Flow Step | Short description | Message Details | Notes | ||||||||
---|---|---|---|---|---|---|---|---|---|---|---|
1 | Bulk Get Request |
|
Flow Step
Short description
Message Details
Notes
Bulk Get Request
language | bash |
---|---|
title | NCMP batch endpoint |
collapse | true |
| Define new get operation "getResourceDataForCmHandles" into ncmp.yml | |||||||||||
2 | Ack client Request |
| ||||||||||
3 | DMI Bulk Request |
|
language | java |
---|---|
title | Response 202 |
collapse | true |
Code Block | ||||
---|---|---|---|---|
| ||||
{
"$schema": "https://json-schema.org/draft/2019-09/schema",
"$id": "urn:cps:org.onap.cps.ncmp.events.async:batch-event-schema:1.0.0",
"$ref": "#/definitions/BatchDataResponseEvent",
"definitions": {
"BatchDataResponseEvent": {
"description": "The payload of batch event.",
"type": "object",
"javaType" : "org.onap.cps.ncmp.events.async.v1.BatchEvent",
"properties": {
"event": {
"description": "The content of Batch event.",
"type": "object",
"existingJavaType": "java.lang.Object",
"additionalProperties": false
}
},
"required": [
"event"
],
"additionalProperties": false
}
}
} |
Code Block | ||||
---|---|---|---|---|
| ||||
{
"event": {
"payload": "response of batch cm handles"
}
} |
Table
Code Block | ||||
---|---|---|---|---|
| ||||
{
"event": {
"payload": "response of batch cm handles"
}
} |
Code Block | ||||
---|---|---|---|---|
| ||||
{
"timestamp":"2023-03-01T23:00:00.345-0400",
"requestId":123,
"error": "<error-message>",
"Cmhandles":["cm-1",...,"cm-n"]
} |
See decision # 8
Proposed JIRAs :
New JIRA
Discussed with Csaba Kocsis and agreed to send identical data request body as NCMP where data operation requat body is as follows:
| ||||||||||||||||||||||||
4 | Ack NCMP Request |
| 202 is non-committal, meaning that there is no way for the HTTP to later send an asynchronous response indicating the outcome of processing the request. It is intended for cases where another process or server handles the request, or for batch processing. | |||||||||||||||||||||
5 | Kafka Messages from DMI to NCMP |
| ||||||||||||||||||||||
6 | Kafka Message(s) from NCMP to Client Table |
| Single response format for all scenarios bot positive and error, just using optional fields instead | |||||||||||||||||||||
7 | Error for 4/5 → Non responding DMI, non existing CM Handles or any other error. NCMP will have to create error message detailing all cm-handle ids affect by that error. |
| Single response format for all scenarios bot positive and error, just using optional fields instead See decision # 8 |
Proposed JIRAs :
Priority | Component | Description | JIRAs | Estimate | Status | |||||||||||||||||||||||||
---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
1 | DMI, NCMP | Data operation response event (DMI → NCMP) to Comply with Cloud Events | ||||||||||||||||||||||||||||
Priority | Component | Description | JIRAs | Estimate | Status | |||||||||||||||||||||||||
1 | DMI, NCMP | Data operation response event (DMI → NCMP) to Comply with Cloud Events |
| 5 Days (M) | New Requirement. | |||||||||||||||||||||||||
2 | NCMP | Data operation response event (NCMP → Client App) |
| TBD | Done | |||||||||||||||||||||||||
3 | DMI-Plugin | Accept datastore name as param into URL |
| 5 Days | Done | |||||||||||||||||||||||||
4 | NCMP | Expose REST endpoint to accept collection of cm handles for GET operation (Passthrough only) | ||||||||||||||||||||||||||||
Jira | ||||||||||||||||||||||||||||||
server | ONAP Jira | serverId |
|
| 1553
| 15 5 Days (M) | Done | |||||||||||||||||||||||
2 | NCMP | Data operation response event (NCMP → Client App) | 5 | DMI-Plugin | Expose endpoint for ONAP not impl. and Stub impl. for testing/demo |
| ||||||||||||||||||||||||
serverId | 425b2b0a-557c-3c0c-b515-579789cceedb | |||||||||||||||||||||||||||||
key | CPS-1555 | 5 Days | Done | 6 | NCMP | NCMP: Update existing REST endpoint that accepts data operation request for GET operation |
| 5 Days | new because Requirements changed +5 days | TBD | Done | |||||||||||||||||||
3 | 7DMI-Plugin | DMI-Plugin : Update endpoint to accept data operation requestAccept datastore name as param into URL |
| columnIds|||||||||||||||||||||||||||
columns | key,summary,type,created,updated,due,assignee,reporter,priority,status,resolution | issuekey,summary,issuetype,created,updated,duedate,assignee,reporter,priority,status,resolution |
| 1636
| 3 5 Days | ClosedDone | ||||||||||||||||||||||||
4 | NCMP | Expose REST endpoint to accept collection of cm handles for GET operation (Passthrough only) | 8 | Stubbed DMI-Plugin | Include code to send response messages to internal kafka topic with delay |
| 10 15 Days | OpenDone | ||||||||||||||||||||||
95 | NCMP | DMI-Plugin | Expose endpoint for ONAP not impl. and Stub impl. for testing/demoForward response messages to client given kafka topic |
| 5 Days | ClosedDone | ||||||||||||||||||||||||
106 | NCMP | NCMP | NCMP: Update existing REST endpoint that accepts data operation request for GET operationHandle non-existing cm handles |
| 5 Days |
| 5 Days | Done | ||||||||||||||||||||||
7 | DMI-Plugin | DMI-Plugin : Update endpoint to accept data operation request |
| 3 Days | Done | |||||||||||||||||||||||||
8 | Stubbed DMI-Plugin | Include code to send response messages to internal kafka topic with delay |
| 10 Days | Done | |||||||||||||||||||||||||
9 | NCMP | Forward response messages to client given kafka topic |
| 5 Days | Done | |||||||||||||||||||||||||
10 | NCMP | Handle non-existing cm handles |
| 5 Days | Done | |||||||||||||||||||||||||
11 | NCMP | Error handling for non-ready cm handle state |
| 5 Days | Done | |||||||||||||||||||||||||
12 | NCMP | Handle non responding DMI-Plugin |
| 5 Days | Done | |||||||||||||||||||||||||
13 | CSIT test for demo |
| 5 Days | Done | ||||||||||||||||||||||||||
14 | Update RTD |
| 4 Daya | |||||||||||||||||||||||||||
15 | Test performance | Capabilities | 11 | NCMP | Error handling for non-ready cm handle state |
| 5 Days |
| 3 Days | Done (NCMP: Read data operation resource API Performance for multiple cm-handles) | ||||||||||||||||||||
16 | NCMP | Modify DMI data operation request body. | 12 | NCMP | Handle non responding DMI-Plugin |
| 1558
| 5 Days | 13 | CSIT test for demo |
| 5 Days |
Planning :
2 Days | Done |
Planning :
- Allow for 2 more user stories each may take 1 weeks
- Based on 2 resource working parallelly may take approx..
8 weeks from 29 March 2023 - (updated after additional items where added) The estimated date for completion is
The release for cps-and-ncmp 3.3.3 version is completed on Friday,
Release tag : https://gerrit.onap.org/r/gitweb?p=cps.git;a=tag;h=refs%2Ftags%2F3.3.3
- Demo recording can be referred here : https://wiki.onap.org/display/DW/CPS+User+Story+Demos
- Allow for 2 more user stories each may take 1 weeks
- Based on 2 resource working parallelly may take approx..
8 weeks from 29 March 2023 - (updated after additional items where added) The estimated date for completion is