...
# | Issue | Notes | Decision | ||||||||||||
|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| 1 | Which operation(s) need support for multiple cm handles? |
if many what is the priority? | agreed with kieran mccarthy Only Get (read) (in future other operations might be support batch option too) | ||||||||||||
| 2 | Which datasources should be supported? | Do we need to support passthrough-only no-cached() data only ? | agreed with kieran mccarthy : all passthrough datastores will be supported Not implemented (yet) response, for non passthrough datastores | ||||||||||||
| 3 | URL pattern for NCMP bulk endpoints
----------------------------------------------------------------------------------------------------------------------- Meeting : kafka message schema & batch interface extension
{
POST http://localhost:8080/ncmp/v1/data&topic=my-topic-name
------------------------------------------------------------------------------------------------------------ We had an internal review with some of our rApp colleagues around some of the recent proposed NCMP batch interface and they came back with some valid comment. The proposal is that we should not distinguish batch from bulk or other flavours of read/write. The aim is to only have a single flavour of interface for read or write for clients. Therefore the proposal is to drop “batch” from the interface URL and just act toward “data” (reads/writes/actions)
| 4 | keep datastore, topic and optional parameters in the URL itself instead into body.CPS prefers keep interface similar as single cm handle interface (consistency and cost) Existing : ...&topic=topicParamInQuery | agreed with kieran mccarthy : Follow the existing interfaces as much as possible for consistency and efficiency | 5 | support in ONAP DMI-plugin | agreed with Toine Siebelink : ONAP plugin can respond with not implemented yet code, | ||||||||
| 6 | Response always Async ie. topic is compulsory ? | Assume topic is compulsory (defined in OPenApi) → Response therefore wil be 400 if not supplied | Agreed with kieran mccarthy : Topic is optional but system will respond with 'Not implemented (yet when not specified or blank | ||||||||||||
| 7 | Should NCMP Amalgamate Async responses from DMI-Plugin before forwarding ? | step 6 in flow diagram | Agreed with kieran mccarthy : NCMP wil only forward to client topic no handling tracking or any responses or status of request | 8 | Handle non responding dmi-plugin | Agreed with kieran mccarthy : | |||||||||
| 9 | Should (can) NCMP check if 'MyTopic' specified by client exist | Consider Access Control too. For now NCMP can log error. Client is responsible for topic setup | Agreed with kieran mccarthy : | ||||||||||||
| 10 | How to handle non-existing CM-Handles (id) | Suggestions
| Agreed with kieran mccarthy : Additional error (messages) response with all cm-handles that cannot be resolved, also a separate error message wil be sent for each failed DMI | 11 | Overlap/clash with Deutsche Telekom user story:
| discussed in weekly ONAP meeting the DT user story is affect CPS-Core interface (not NCMP) and the requirement is to execute a query over ALL cm-handles (cached only?!) instead of a given list of cm-handles (~anchors) | 12 |
| Jira | ||||||||||
|---|---|---|---|---|---|---|---|---|---|---|
|
1) Agreement required on the structure of the response. Please see response structure below.
2) Does 'eventTime' field which holds the timestamp of the bulk response event, required or can it be dropped?
| title | Batch Event Payload Schema |
|---|---|
| collapse | true |
We had an internal review with some of our rApp colleagues around some of the recent proposed NCMP batch interface and they came back with some valid comment. The proposal is that we should not distinguish batch from bulk or other flavours of read/write.
The aim is to only have a single flavour of interface for read or write for clients. Therefore the proposal is to drop “batch” from the interface URL and just act toward “data”
Agreed with team kieran mccarthy
CPS prefers keep interface similar as single cm handle interface (consistency and cost)
Existing : ...&topic=topicParamInQuery
agreed with kieran mccarthy : Follow the existing interfaces as much as possible for consistency and efficiency
support in ONAP DMI-plugin
agreed with Toine Siebelink : ONAP plugin can respond with not implemented yet code,
Response always Async ie. topic is compulsory ?
Agreed with kieran mccarthy :
Topic is optional but system will respond with 'Not implemented (yet when not specified or blank
Agreed with kieran mccarthy : NCMP wil only forward to client topic no handling tracking or any responses or status of request
Agreed with kieran mccarthy :
No response for 4b then send an error response to the topic given by client
Agreed with kieran mccarthy :
NCMP can log error when forward to 'client topic' Security not in scope (yet)
Suggestions
- Silently ignore
- (initial) error response
- should we combine all errors in one message?
Agreed with kieran mccarthy :
Additional error (messages) response with all cm-handles that cannot be resolved, also a separate error message wil be sent for each failed DMI
Overlap/clash with Deutsche Telekom user story:
| Jira | ||||||
|---|---|---|---|---|---|---|
|
discussed in weekly ONAP meeting the DT user story is affect CPS-Core interface (not NCMP) and the requirement is to execute a query over ALL cm-handles (cached only?!) instead of a given list of cm-handles (~anchors)
Schema of bulk response
| Jira | ||||||||||
|---|---|---|---|---|---|---|---|---|---|---|
|
1) Agreement required on the structure of the response. Please see response structure below.
2) Does 'eventTime' field which holds the timestamp of the bulk response event, required or can it be dropped?
Need to follow schema structure there in #15 under Issues & Decisions section and it is agreed on
Schema for Bulk Response event forwarding to client specified topic
CPS-1557 - NCMP : forward bulk response messages to client topic
Not keeping the 'eventTarget' which comes from the (BulkResponseEvent(In progress of the structure agreement)).
Note: Will consider 'eventTarget', If it finalized the schema from
| Jira | ||||||||||
|---|---|---|---|---|---|---|---|---|---|---|
|
Agreed on to keep one schema for both events (DMI → NCMP) and (NCMP → ClientApps)
How NCMP would forward response? (Response data : ref. message flow #6)
| Jira | ||||||||||
|---|---|---|---|---|---|---|---|---|---|---|
|
1. Do we need to send only one response containing all the requested cm handles ?
2. Send single response for each cm handle?
3. Single response message per DMI plugin?
kieran mccarthy - Please share your inputs on the above listed questions.
Some non-standard headers wil need to be implemented as 'extensions' and names to be confirmed
correlationidtarget
More details see CPS Events Structure#CNCFCloudEventalignment
topic and requestId will be send as a path param from NCMP to DMI-plugin.
NCMP batch endpoint : http://localhost:8080/ncmp/v1/data?&topic=my-topic-name
DMI-Plugin batch endpoint : http://172.26.202.25:8783/dmi/v1/data?topic=my-topic-name&requestId=e6fa4d26-4dc1-4877-aa3c-45e99f840708
- Mandatory Fields:
- "operation": "read"
- "operationId": "12"
"datastore": "ncmp-datastore:passthrough-operational" - "targetIds": [ "0df4d39af6514d99b816758148389cfd", "ec2e9495679a43c58659c07d87025e72" ]
- Optional Fields:
- "options": "(fields=schemas/schema)"
- "resourceIdentifier": "parent/child"
Agreed with Csaba Kocsis and kieran mccarthy
event type proposal : from BatchDataXXXEvent to
DataOperationXXXEvent
For example : DataOperationResponseEvent
'eventType' value as below 'org.onap.cps.ncmp.events.BatchDataXXXEvent'
proposed Operation like :
'org.onap.cps.ncmp.events.DataOperationXXXEvent'
'batch' was the keyword use in the URL for this feature (now gone) but much code (classnames, variable names) etc uses this name still...
Event Format Definitions
Event Headers
| Excerpt Include | ||||||
|---|---|---|---|---|---|---|
|
Event Body (data)
- The current proposal is to use the CNFC Event structure and library as detail in CPS Events Structure#CNCFCloudEventalignment
- To prevent problems with large data request a single response event will be sent for a successful read for each cm handle (id)
- Error message can contain information about multiple cm handle ids
Field | Type | Description | Mandatory? | Notes | ||||||
|---|---|---|---|---|---|---|---|---|---|---|
event | Event | The payload of an event | Mandatory | |||||||
event.responses[0, 1, 2, ...] | Array | contains an array or batch response that includes both success and failure. | Mandatory | |||||||
event.responses[0].operationId | String | specified to distinguish multiple operations using same cmhandleId | Mandatory | |||||||
event.responses[0].ids | String | cmhandle-ids | Mandatory | Example : ["0df4d39af6514d99b816758148389cfd"] | ||||||
event.responses[0].statusCode | String | Mandatory | Common NCMP defined error codes:
| |||||||
event.responses[0].statusMessage | String | Mandatory | Examples for code & message :
| |||||||
| event.responses[0].data | Object | Optional |
|
Implementation
Bulk Request Message Flow
| Gliffy Diagram | ||||||||
|---|---|---|---|---|---|---|---|---|
|
Message Flow details
Flow Step | Short description | Message Details | Notes | ||||||||
|---|---|---|---|---|---|---|---|---|---|---|---|
| 1 | Bulk Get Request |
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| title | Sample Batch Event Payload |
|---|---|
| collapse | true |
|
|
|
|
|
Need to follow schema structure there in #15 under Issues & Decisions section and it is agreed on
Schema for Bulk Response event forwarding to client specified topic
CPS-1557 - NCMP : forward bulk response messages to client topic
Not keeping the 'eventTarget' which comes from the (BulkResponseEvent(In progress of the structure agreement)).
Note: Will consider 'eventTarget', If it finalized the schema from
| Jira | ||||||||||
|---|---|---|---|---|---|---|---|---|---|---|
|
Agreed on to keep one schema for both events (DMI → NCMP) and (NCMP → ClientApps)
How NCMP would forward response? (Response data : ref. message flow #6)
| Jira | ||||||||||
|---|---|---|---|---|---|---|---|---|---|---|
|
2. Send single response for each cm handle?
3. Single response message per DMI plugin?
kieran mccarthy - Please share your inputs on the above listed questions.
| Define new get operation "getResourceDataForCmHandles" into ncmp.yml | |||||||||||
| 2 | Ack client Request |
| ||||||||||
| 3 | DMI Bulk Request |
|
correlationidtarget
We have had some internal discussions including with some O-RAN standards representatives and one of the outcomes is that it would be good if we aligned with the some community standards for event header definitions. IT is proposed (initially from AT&T) that we should follow Cloud Native Computing Foundation (CNCF) specification as defined in their cloudevents incubator project.
More details see CPS Events Structure#CNCFCloudEventalignment
| Code Block | ||||||
|---|---|---|---|---|---|---|
| ||||||
curl --location 'http: //localhost:8080/ncmp/v1/data?topic=my-topic-name' \
--header 'Content-Type: application/json' \
--header 'Authorization: Basic Y3BzdXNlcjpjcHNyMGNrcyE=' \
--data '{
"operations": [
{
"operation": "read",
"operationId": "12",
"datastore": "ncmp-datastore:passthrough-operational",
"options": "(fields=schemas/schema)",
"resourceIdentifier": "parent/child",
"targetIds": [
"836bb62201f34a7aa056a47bd95a81ed",
"202acb75b4a54e43bb1ff8c0c17a8e08"
]
},
{
"operation": "read",
"operationId": "14",
"datastore": "ncmp-datastore:passthrough-running",
"targetIds": [
"ec2e9495679a43c58659c07d87025e72",
"0df4d39af6514d99b816758148389cfd"
]
}
]
}' |
| Code Block | ||||||
|---|---|---|---|---|---|---|
| ||||||
{
"requestId": "4753fc1f-7de2-449a-b306-a6204b5370b3"
} |
| language | bash |
|---|---|
| title | DMI service batch endpoint |
| collapse | true |
|
Agreed with Csaba Kocsis and kieran mccarthy
requestId would be send as a path param from NCMP to DMI-plugin.
Example
NCMP batch endpoint : http://localhost:8080/ncmp/v1/data?&topic=my-topic-name
DMI-Plugin batch endpoint : http://172.26.202.25:8783/dmi/v1/data?topic=my-topic-name&requestId=e6fa4d26-4dc1-4877-aa3c-45e99f840708
- Mandatory Fileds :
- "operation": "read"
- "operationId": "12"
"datastore": "ncmp-datastore:passthrough-operational" - "targetIds": [ "0df4d39af6514d99b816758148389cfd", "ec2e9495679a43c58659c07d87025e72" ]
- Optional Fields :
- "options": "(fields=schemas/schema)"
- "resourceIdentifier": "parent/child"
event type proposal : from BatchDataXXXEvent to
DataOperationXXXEvent
For example : DataOperationResponseEvent
'eventType' value as below 'org.onap.cps.ncmp.events.BatchDataXXXEvent'
proposed Operation like :
'org.onap.cps.ncmp.events.DataOperationXXXEvent'
'batch' was the keyword use in the URL for this feature (now gone) but much code (classnames, variable names) etc uses this name still...
Event Format Definitions
Event Headers
...
Event Body (data)
- The current proposal is to use the CNFC Event structure and library as detail in CPS Events Structure#CNCFCloudEventalignment
- To prevent problems with large data request a single response event will be sent for a successful read for each cm handle (id)
- Error message can contain information about multiple cm handle ids
...
Field
...
Type
...
Description
...
Mandatory?
...
Notes
...
event...
Mandatory
...
event.responses[0, 1, 2, ...]...
Mandatory
...
event.responses[0].operationId...
Mandatory
...
event.responses[0].ids...
Mandatory
...
Example : ["0df4d39af6514d99b816758148389cfd"]
Note: Ids array should contain only a single element in the array in case of success messages and In case of error it can have any number of elements.
| ||||||||||
| 4 | Ack NCMP Request |
| 202 is non-committal, meaning that there is no way for the HTTP to later send an asynchronous response indicating the outcome of processing the request. It is intended for cases where another process or server handles the request, or for batch processing. | |||||||
| 5 | Kafka Messages from DMI to NCMP |
|
...
event.responses[0].statusCode...
Mandatory
...
Common NCMP defined error codes:
- status-code 0-99 is reserved for any success response
- status-code from 100 to 199 is reserved for any failed response.
...
event.responses[0].statusMessage...
Mandatory
Examples for code & message :
| statusCode | statusMessage |
|---|---|
| 1 | "Successfully applied changes" |
| 101 | "cmHandle(s) do not exist" |
...
Optional
...
- In case of success :
- Optional, for write operations then no need to return configurations
application/yang-patch+json | application/yang-data+json
- Optional, for write operations then no need to return configurations
- In case of failure :
- Optional, any supplementary error data matching the error status-code
Implementation
Bulk Request Message Flow
| Gliffy Diagram | ||||||||
|---|---|---|---|---|---|---|---|---|
|
Message Flow details
Flow Step | Short description | Message Details | Notes | ||||||||||||||||||||||||||||||||||
|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| 1 | Bulk Get Request |
| Code Block | ||||
|---|---|---|---|---|
| ||||
body:
["cm-1",...,"cm-n"] |
| Code Block | ||||
|---|---|---|---|---|
| ||||
'http://localhost:8080/ncmp/v1/batch/data/ds/ncmp-datastore:passthrough-running?resourceIdentifier=parent/child%26options=(a=1,b=2)&topic=my-topic-name&options=(fields=schemas/schema)' \
--header 'Authorization: Basic Y3BzdXNlcjpjcHNyMGNrcyE=' \
--header 'Content-Type: application/json' \
--data '[ "40137a9771f84459affa795fa1d633ab", "f5a92ec7a7db4d6fbb0e0ce2803a86cc" ]' |
Define new get operation "getResourceDataForCmHandles" into ncmp.yml
| Code Block | ||||
|---|---|---|---|---|
| ||||
{"requestId":"123"} |
| Code Block | ||||
|---|---|---|---|---|
| ||||
body: {"Cmhandles":["cm-1",...,"cm-n"],"requestId":123} |
The DMI Plugin should be told (included in request) the client topic so that NCMP does not have to 'remember' to relation between request id and client topic!
| title | Response 202 |
|---|---|
| collapse | true |
| Code Block | ||||
|---|---|---|---|---|
| ||||
{
"eventId": "4cb32729-85e3-44d1-aa6e-c923b9b059a5",
"eventCorrelationId": "68f15800-8ed4-4bae-9e53-27a9e03e1911",
"eventTime": "2023-03-28T14:29:23.876+0000",
"eventTarget": "client-topic"
"eventSource": "dmi-plugin:enm-1"(dmi service name)
"eventType": "org.onap.cps.ncmp.event.model.BulkResponseEvent",
"eventSchema": "urn:cps:org.onap.cps.ncmp.events.async:batch-event-schema:1.0.0",
"schemaVersion": "1.0.0",
} |
| Code Block | ||||
|---|---|---|---|---|
| ||||
{
"event": {
"payload": "response of batch cm handles"
}
} |
Table
| Code Block | ||||
|---|---|---|---|---|
| ||||
{
"eventId": "4cb32729-85e3-44d1-aa6e-c923b9b059a5",
"eventCorrelationId": "68f15800-8ed4-4bae-9e53-27a9e03e1911",
"eventTime": "2023-03-28T14:29:23.876+0000",
"eventSource": "dmi-plugin:enm-1"(dmi service name)
"eventType": "org.onap.cps.ncmp.event.model.BulkResponseEvent",
"eventSchema": "urn:cps:org.onap.cps.ncmp.events.async:batch-event-schema:1.0.0",
"schemaVersion": "1.0.0",
} |
| Code Block | ||||
|---|---|---|---|---|
| ||||
{
"event": {
"payload": "response of batch cm handles"
}
} |
Response message structure ? (Flow no. 5)
Non responding DMI-plugin
| Code Block | ||||
|---|---|---|---|---|
| ||||
{
"timestamp":"2023-03-01T23:00:00.345-0400",
"requestId":123,
"error": "DMI Service Unavailable, {service-name}",
"Cmhandles":["cm-1",...,"cm-n"]
} |
Response message structure ? (Flow no. 5)
Non existing cm handles
| Code Block | ||||
|---|---|---|---|---|
| ||||
{
"timestamp":"2023-03-01T23:00:00.345-0400",
"requestId":123,
"error":"Cm-Handle not found",
"Cmhandles":["cm-1",...,"cm-n"]
} |
| Code Block | ||||
|---|---|---|---|---|
| ||||
{
"timestamp":"2023-03-01T23:00:00.345-0400",
"requestId":123,
"error":"Cm-Handle not in READY state.",
"Cmhandles":["cm-1",...,"cm-n"]
} |
Existing DMI endpoints are :
/v1/ch/{cmHandle}/data/ds/{datastore-name}
datastore-name:
- ncmp-datastore:passthrough-operational
- ncmp-datastore:passthrough-running
...&topic=topicParamInQuery
CPS Proposed :
/v1/ch/batch/data/ds/{datastore-name}
...&topic=topicParamInQuery
cm handle ids and requestid into body
...