...
# | Issue | Notes | Decision | ||||||||||||||||||||||||||||||||||||||||
---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
1 | Which operation(s) need support for multiple cm handles? |
if many what is the priority? | agreed with kieran mccarthy Only Get (read) (in future other operations might be support batch option too) | ||||||||||||||||||||||||||||||||||||||||
2 | Which datasources should be supported? | Do we need to support passthrough-only no-cached() data only ? | agreed with kieran mccarthy : all passthrough datastores will be supported Not implemented (yet) response, for non passthrough datastores | ||||||||||||||||||||||||||||||||||||||||
3 | URL pattern for NCMP bulk endpoints |
----------------------------------------------------------------------------------------------------------------------- Meeting : kafka message schema & batch interface extension
{
|
------------------------------------------------------------------------------------------------------------ We had an internal review with some of our rApp colleagues around some of the recent proposed NCMP batch interface and they came back with some valid comment. The proposal is that we should not distinguish batch from bulk or other flavours of read/write. The aim is to only have a single flavour of interface for read or write for clients. Therefore the proposal is to drop “batch” from the interface URL and just act toward “data” (reads/writes/actions)
| ||||||||||||||||||||||||||||||||||||||||
4 | keep datastore, topic and optional parameters in the URL itself instead into body.
| CPS prefers keep interface similar as single cm handle interface (consistency and cost) Existing : ...&topic=topicParamInQuery | agreed with kieran mccarthy : Follow the existing interfaces as much as possible for consistency and efficiency | ||||||||||||||||||||||||||||||||||||||||
5 | support in ONAP DMI-plugin | agreed with Toine Siebelink : ONAP plugin can respond with not implemented yet code, | |||||||||||||||||||||||||||||||||||||||||
6 | Response always Async ie. topic is compulsory ? | Assume topic is compulsory (defined in OPenApi) → Response therefore wil be 400 if not supplied | Agreed with kieran mccarthy : Topic is optional but system will respond with 'Not implemented (yet when not specified or blank | ||||||||||||||||||||||||||||||||||||||||
7 | Should NCMP Amalgamate Async responses from DMI-Plugin before forwarding ? | step 6 in flow diagram | Agreed with kieran mccarthy : NCMP wil only forward to client topic no handling tracking or any responses or status of request | ||||||||||||||||||||||||||||||||||||||||
8 | Handle non responding dmi-plugin | Agreed with kieran mccarthy : | |||||||||||||||||||||||||||||||||||||||||
9 | Should (can) NCMP check if 'MyTopic' specified by client exist | Consider Access Control too. For now NCMP can log error. Client is responsible for topic setup | Agreed with kieran mccarthy : | ||||||||||||||||||||||||||||||||||||||||
10 | How to handle non-existing CM-Handles (id) | Suggestions
| Agreed with kieran mccarthy : Additional error (messages) response with all cm-handles that cannot be resolved, also a separate error message wil be sent for each failed DMI | ||||||||||||||||||||||||||||||||||||||||
11 | Overlap/clash with Deutsche Telekom user story:
| discussed in weekly ONAP meeting the DT user story is affect CPS-Core interface (not NCMP) and the requirement is to execute a query over ALL cm-handles (cached only?!) instead of a given list of cm-handles (~anchors) | |||||||||||||||||||||||||||||||||||||||||
12 | Schema of bulk response
| 1) Agreement required on the structure of the response. Please see response structure below.
| Need to follow schema structure there in #15 under Issues & Decisions section and it is agreed on | ||||||||||||||||||||||||||||||||||||||||
13 |
CPS-1557 - NCMP : forward bulk response messages to client topic |
| Agreed on to keep one schema for both events (DMI → NCMP) and (NCMP → ClientApps) | ||||||||||||||||||||||||||||||||||||||||
14 | How NCMP would forward response? (Response data : ref. message flow #6)
| 1. Do we need to send only one response containing all the requested cm handles ? | |||||||||||||||||||||||||||||||||||||||||
15 | Message format for the batch interface | We have had some internal discussions including with some O-RAN standards representatives and one of the outcomes is that it would be good if we aligned with the some community standards for event header definitions. IT is proposed (initially from AT&T) that we should follow Cloud Native Computing Foundation (CNCF) specification as defined in their cloudevents incubator project. | More details see CPS Events Structure#CNCFCloudEventalignment | ||||||||||||||||||||||||||||||||||||||||
16 | NCMP wil send only one request to each DMI |
| |||||||||||||||||||||||||||||||||||||||||
Batch interface message events structure | |||||||||||||||||||||||||||||||||||||||||||
kafka Headers | |||||||||||||||||||||||||||||||||||||||||||
|
|
|
|
|
| ||||||||||||||||||||||||||||||||||||||
eventId | string | The unique id identifying the event | M | Generated by DMI-Plugin | Agreed on with kieran mccarthy Deided field name as "id" | eventCorrelationId | string | The request id passed by NCMP | M | It's requestId that NCMP sent to client as an ACK. Example : “request-1234" (UUID) | Agreed with kieran mccarthyeventTime | string | The timestamp when original event occurred | O | Example of Internet date/time format. 1985-04-12T23:20:50.52Z This represents 20 minutes and 50.52 seconds after the 23rd hour of April 12th, 1985 in UTC. | ||||||||||||||||||||||||||||
eventTarget | string | The destination topic of the client | M | Example : my-topic | Agreed with kieran mccarthy | ||||||||||||||||||||||||||||||||||||||
eventSource | string | The source of the event | O | Example : dmi-plugin:enm-1 (dmi service name) | Agreed with kieran mccarthy Deided field name as "source" | ||||||||||||||||||||||||||||||||||||||
eventType | string | The type of the event | M |
| kieran mccarthy confirmed : event type proposed name :
proposed DataOperationXXXEvent For example : DataOperationResponseEvent 'eventType' value as below 'org.onap.cps.ncmp.events. proposed Operation like : 'org.onap.cps.ncmp.events.DataOperationXXXEvent' Deided field name as "type" | eventSchema | string | The schema of the Batch event payload. | M |
| schemaVersion | string | The schema version of the Batch event payload. | M | On versioning O-RAN follows https://semver.org. kieran mccarthy confirmed "eventSchemaVersion" | specversion (default 1.0) | String | kieran mccarthy ?? | |||||||||||||||||||||||||
Event Payload | event | Event | The payload of an event | M | Agreed on with kieran mccarthy | event.responses[0, 1, 2, ...] | Array | contains an array or batch response that includes both success and failure. | M | Agreed on with kieran mccarthy NCMP will not amalgamate any data | event.responses[0].operationId | String | specified to distinguish multiple operations using same cmhandleId | M | Agreed on with kieran mccarthy NCMP wil not check uniqueness of 'operationId' | ||||||||||||||||||||||||||||
event.responses[0].ids | String | cmhandle-ids | M | Example : ["0df4d39af6514d99b816758148389cfd"] | kieran mccarthy confirmed on Question : If read there will only be a single cmhandle in this array ?? Ans : Ids array should contain only a single element in the array in case of success messages and In case of error it can have any number of elements. | event.responses[0].status-code | String | M | Common NCMP defined error codes:
| event.responses[0].status-message | String | M |
status-code | status-message |
---|---|
1 | "Successfully applied changes" |
101 | "cmHandle(s) do not exist" |
Agreed with kieran mccarthy
O
- In case of success :
- Optional, for write operations then no need to return configurations application/yang-patch+json | application/yang-data+json
- In case of failure :
- Optional, any supplementary error data matching the error status-code
Code Block | ||||||
---|---|---|---|---|---|---|
| ||||||
curl --location 'http: //localhost:8080/ncmp/v1/data?topic=my-topic-name' \ --header 'Content-Type: application/json' \ --header 'Authorization: Basic Y3BzdXNlcjpjcHNyMGNrcyE=' \ --data '{ "operations": [ { "operation": "read", { "operationId": "12", "datastoreid": "ncmp-datastore:passthrough-operationalec2e9495679a43c58659c07d87025e72", "optionscmHandleProperties": "(fields=schemas/schema)", { "resourceIdentifierneType": "parent/childRadioNode", "targetIds": [ } "836bb62201f34a7aa056a47bd95a81ed"}, "202acb75b4a54e43bb1ff8c0c17a8e08"{ ] }, "id": "0df4d39af6514d99b816758148389cfd", { "operationcmHandleProperties": "read", { "operationId": "14", "datastoreneType": "ncmp-datastore:passthrough-running",RadioNode" "targetIds": [ } "ec2e9495679a43c58659c07d87025e72",} ] } ] DMI Service 2 "0df4d39af6514d99b816758148389cfd" (POST) : http://172.26.202.26:8783/dmi/v1/data?topic=my-topic-name&requestId=4753fc1f-7de2-449a-b306-a6204b5370b3 -> [ { ] }"operationType": "read", ] }' | ||||||
Code Block | ||||||
| ||||||
{ "requestIdoperationId": "4753fc1f-7de2-449a-b306-a6204b5370b3" } | ||||||
Code Block | ||||||
| ||||||
DMI Service 1 (POST): http://172.26.202.25:8783/dmi/v1/data?topic=my-topic-name&requestId=4753fc1f-7de2-449a-b306-a6204b5370b3 -> [ {"12", "datastore": "ncmp-datastore:passthrough-operational", "operationTypeoptions": "read(fields=schemas/schema)", "operationIdresourceIdentifier": "14parent/child", "datastore": "ncmp-datastore:passthrough-running", "cmHandles": [ { "id": "ec2e9495679a43c58659c07d87025e72836bb62201f34a7aa056a47bd95a81ed", "cmHandleProperties": { "neType": "RadioNode" } }, { "id": "0df4d39af6514d99b816758148389cfd202acb75b4a54e43bb1ff8c0c17a8e08", "cmHandleProperties": { "neType": "RadioNode" } } ] } ] DMI Service 2 (POST) : |
Agreed with Csaba Kocsis and kieran mccarthy
requestId would be send as a path param from NCMP to DMI-plugin.
Example
NCMP batch endpoint : http://
172.26.202.26:8783/localhost:8080/ncmp/v1/data?&topic=my-topic-name
DMI-Plugin batch endpoint : http://172.26.202.25:8783/dmi/v1/data?topic=my-topic-name&requestId=
4753fc1f7de2449ab306-a6204b5370b3 -> [ { "operationType": "read",- Mandatory Fileds :
- "operation": "read"
- "operationId":
- "12
- "
"datastore": "ncmp-datastore:passthrough-operational"
- "
- "targetIds": [ "0df4d39af6514d99b816758148389cfd", "ec2e9495679a43c58659c07d87025e72" ]
- Optional Fields :
- "options":
- "(fields=schemas/schema)"
- "resourceIdentifier":
- "parent/child"
Agreed with Csaba Kocsis and kieran mccarthy
requestId would be send as a path param from NCMP to DMI-plugin.
Example
NCMP batch endpoint : http://localhost:8080/ncmp/v1/data?&topic=my-topic-name
DMI-Plugin batch endpoint : http://172.26.202.25:8783/dmi/v1/data?topic=my-topic-name&requestId=e6fa4d26-4dc1-4877-aa3c-45e99f840708
- Mandatory Fileds :
- "operation": "read"
- "operationId": "12"
"datastore": "ncmp-datastore:passthrough-operational" - "targetIds": [ "0df4d39af6514d99b816758148389cfd", "ec2e9495679a43c58659c07d87025e72" ]
- Optional Fields :
- "options": "(fields=schemas/schema)"
- "resourceIdentifier": "parent/child"
Csaba Kocsis proposed to use 'dataOperation' for the events but this is inconsistent with URL and affect a lot off other stuff under implementation like classnames, documentation schemas etc.
'batch' is the keyword use in the UIRL for this feature
Csaba Kocsis proposed to use 'dataOperation' for the events but this is inconsistent with URL and affect a lot off other stuff under implementation like classnames, documentation schemas etc.
Event Header and Data Definitions
Batch interface message events structure | |||||||||||
---|---|---|---|---|---|---|---|---|---|---|---|
kafka Headers | |||||||||||
|
|
|
|
|
| ||||||
eventId | string | The unique id identifying the event | M | Generated by DMI-Plugin | Agreed on with kieran mccarthy Deided field name as "id" | ||||||
eventCorrelationId | string | The request id passed by NCMP | M | It's requestId that NCMP sent to client as an ACK. Example : “request-1234" (UUID) |
| ||||||
eventTime | string | The timestamp when original event occurred | O | Example of Internet date/time format. 1985-04-12T23:20:50.52Z This represents 20 minutes and 50.52 seconds after the 23rd hour of April 12th, 1985 in UTC. | Agreed with kieran mccarthy The timestamp should follow that on https://www.rfc-editor.org/rfc/rfc3339#section-5. This follows ISO 8601 and is what is used/referenced in 3GPP standards Deided field name as "time" | ||||||
eventTarget | string | The destination topic of the client | M | Example : my-topic | Agreed with kieran mccarthy | ||||||
eventSource | string | The source of the event | O | Example : dmi-plugin:enm-1 (dmi service name) | Agreed with kieran mccarthy Deided field name as "source" | ||||||
eventType | string | The type of the event | M |
| kieran mccarthy confirmed : event type proposed name :
proposed DataOperationXXXEvent For example : DataOperationResponseEvent 'eventType' value as below 'org.onap.cps.ncmp.events. proposed Operation like : 'org.onap.cps.ncmp.events.DataOperationXXXEvent' Deided field name as "type" | ||||||
eventSchema | string | The schema of the Batch event payload. | M |
| kieran mccarthy confirmed : 'org.onap.cps.ncmp.events: Csaba Kocsiswould update ???? Deided field name as "datacontenttype" | ||||||
schemaVersion | string | The schema version of the Batch event payload. | M | For example a version might be 1.0.0 (without “v”) | On versioning O-RAN follows https://semver.org. kieran mccarthy confirmed "eventSchemaVersion" | ||||||
specversion (default 1.0) | String | kieran mccarthy ?? | |||||||||
Event Payload | |||||||||||
event | Event | The payload of an event | M | Agreed on with kieran mccarthy | |||||||
event.responses[0, 1, 2, ...] | Array | contains an array or batch response that includes both success and failure. | M | Agreed on with kieran mccarthy NCMP will not amalgamate any data | |||||||
event.responses[0].operationId | String | specified to distinguish multiple operations using same cmhandleId | M | Agreed on with kieran mccarthy NCMP wil not check uniqueness of 'operationId' | |||||||
event.responses[0].ids | String | cmhandle-ids | M | Example : ["0df4d39af6514d99b816758148389cfd"] | kieran mccarthy confirmed on Question : If read there will only be a single cmhandle in this array ?? Ans : Ids array should contain only a single element in the array in case of success messages and In case of error it can have any number of elements. | ||||||
event.responses[0].status-code | String | M | Common NCMP defined error codes:
| Agreed with kieran mccarthy | |||||||
event.responses[0].status-message | String | M | Example :
| Agreed with kieran mccarthy | |||||||
event.responses[0].data | Object | O |
| Agreed with kieran mccarthy |
Implementation
Bulk Request Message Flow
...
Flow Step | Short description | Message Details | Notes | ||||||||||||||
---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
1 | Bulk Get Request |
| Define new get operation "getResourceDataForCmHandles" into ncmp.yml | ||||||||||||||
2 | Ack clent Request |
| |||||||||||||||
3 | DMI Bulk Request |
| The DMI PLugin should be told (included in request) the client topic so that NCMP does not have to 'remember' to relation between request id and client topic! | ||||||||||||||
4 | Ack NCMP Request |
| 202 is non-committal, meaning that there is no way for the HTTP to later send an asynchronous response indicating the outcome of processing the request. It is intended for cases where another process or server handles the request, or for batch processing. | ||||||||||||||
5 | Kafka Messages from DMI to NCMP |
| |||||||||||||||
6 | Kafka Message(s) from NCMP to Client Table |
| |||||||||||||||
7 | Alternative for 4/5 → Non responding DMI. NCMP will have to create error message detailing cm-handles | See decision # 8 and 9 | |||||||||||||||
8 | Response message structure ? (Flow no. 5) Non responding DMI-plugin |
| |||||||||||||||
9 | Response message structure ? (Flow no. 5) Non existing cm handles |
| |||||||||||||||
10 | Non Ready cm handles |
| |||||||||||||||
11 | URL pattern for DMI-Plugin bulk endpoints | Existing DMI endpoints are : /v1/ch/{cmHandle}/data/ds/{datastore-name} datastore-name:
...&topic=topicParamInQuery CPS Proposed : /v1/ch/batch/data/ds/{datastore-name} ...&topic=topicParamInQuery cm handle ids and requestid into body |
Proposed JIRAs :
Priority | Component | Description | JIRAs | Estimate | Status | ||||||||||||
---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
1 | DMI, NCMP | Batch Response Event (DMI → NCMP) to Comply with Cloud Events |
| 5 Days (M) | New Requirement. | ||||||||||||
2 | NCMP | Batch Response Event (NCMP → Client App) |
| TBD | New Requirement. | ||||||||||||
3 | DMI-Plugin | Accept datastore name as param into URL |
| 5 Days | Done | ||||||||||||
4 | NCMP | Expose REST endpoint to accept collection of cm handles for GET operation (Passthrough only) |
| 15 Days | Pending Doc. only | ||||||||||||
5 | DMI-Plugin | Expose endpoint for ONAP not impl. and Stub impl. for testing/demo |
| 5 Days | Done | ||||||||||||
6 | NCMP | NCMP: Update existing REST endpoint that accepts bulk request for GET operation |
| 5 Days | new because Requirements changed +5 days | ||||||||||||
7 | DMI-Plugin | DMI-Plugin : Update endpoint to accept bulk request |
| 3 Days | new because Requirements changed | ||||||||||||
8 | Stubbed DMI-Plugin | Include code to send response messages to internal kafka topic with delay |
| 10 Days | |||||||||||||
9 | NCMP | Forward response messages to client given kafka topic |
| 5 Days | delayed because Schema/Headers issue +4 days | ||||||||||||
10 | NCMP | Handle non-existing cm handles |
| 5 Days | |||||||||||||
11 | NCMP | Error handling for non-ready cm handle state |
| 5 Days | |||||||||||||
12 | NCMP | Handle non responding DMI-Plugin |
| 5 Days | |||||||||||||
13 | CSIT test for demo |
| 5 Days |
...