Create 'bulk' version for org.onap.cps.ncmp.rest.controller.NetworkCmProxyController#getResourceDataForCmHandle to support multiple cm-handles (~ multiple anchors in in CPS-Core)
References
- CPS-1515 - Spike: Support multiple CM-Handles for NCMP Get Operation
- CPS-NCMP ↔ DMI-Plugin Interface Details Jakarta-R10
Requirements
Functional
# | Interface | Requirement | Additional Information |
---|---|---|---|
1 | REST CPS(-NCMP)-E-05 | Support batch read operation (new) using an asynchronous response on a client specified topic | payload includes list of cm handle ids |
2 | REST DMI-I-01 | Support batch read operation (new) using an asynchronous response to an internal topic | payload includes list of (associated for this plugin) cm handles ids with their private (additional) properties |
Error Handling
# | Error Scenario | Expected behavior |
---|---|---|
1 | DMI Not respond to initial synchronous request with normal HTTP timeout | Special Error message (Kafka event) including affected, and reason for all the handles in the message cm handles send to client topic detailing cm-handles (per dmi plugin) |
2 | Topic not supplied on CPS-E-05 | Return HTTP 501 (not implemented) |
3 | Client specided Topic is not configured | Log error message only |
4 | Non-existing cm-handle id | Similar message but different reason as specified #1 above |
Capabilities
# | Parameter | Expectation | Notes |
---|---|---|---|
1 | Response Time 1 Batch request | <2 seconds (average) |
|
2 | Batch-size | 200 cm handles | No hardcoded limit |
3 | Response payload size | ~2 KB per cm handles | Performance test for capability should be tested with this average response size |
4 | Maximum registered #cm handles | 20,000 | This will effect the internal query time |
5 | Supported # DMI PLugins | 10 | This might effect processing times |
6 | Test Environment | ||
7 | Concurrent request | 12 | 12 clients requests toward 1 NCMP simultaneously |
8 | Request Frequency | 100 request/min | Should not affect performance, does not need to be tested |
Out-of-scope
- Support for multiple resource identifiers in one batch operation
- Support cached data batch requests: only passthrough datastores will be supported (see decision #2)
- NCMP does NOT keep track of request status to see if it is completed, amalgamate responses or anything like that. It wil simply forward responses from the internal topic to the client topic.
- Access control
Assumptions
# | Assumption | Notes |
---|---|---|
1 | Proprietary options or not in the scope of this analysis. | agreed with kieran mccarthy ; these are optional parameters (name-value pairs) not interpreted by NCMP but can be interpreted by proprietary plugins |
2 | same xpath (resourceIdentifierInQuery) for all cm handles or different for each cm handle | agreed with kieran mccarthy ; if different resources are required on the same cm-handle the client wil send another (batch) request |
3 |
| agreed with CPS team. |
4 | We are not merging any duplicate cm handles while sending request to dmi-plugin | agreed |
5 | Get batch operation does not support includeDescendants as it is impl. for pass through datastore only ncmp-datastore:passthrough-running and ncmp-datastore:passthrough-operational | agreed on |
Issues & Decisions
# | Issue | Notes | Decision | ||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||
---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
1 | Which operation(s) need support for multiple cm handles? |
if many what is the priority? | agreed with kieran mccarthy Only Get (read) (in future other operations might be support batch option too) | ||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||
2 | Which datasources should be supported? | Do we need to support passthrough-only no-cached() data only ? | agreed with kieran mccarthy : all passthrough datastores will be supported Not implemented (yet) response, for non passthrough datastores | ||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||
3 | URL pattern for NCMP bulk endpoints |
----------------------------------------------------------------------------------------------------------------------- Meeting : kafka message schema & batch interface extension POST http://localhost:8080/ncmp/v1/batch/data&topic=my-topic-name { |
------------------------------------------------------------------------------------------------------------ | ||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||
4 | keep datastore, topic and optional parameters in the URL itself instead into body.
| CPS prefers keep interface similar as single cm handle interface (consistency and cost) Existing : ...&topic=topicParamInQuery | agreed with kieran mccarthy : Follow the existing interfaces as much as possible for consistency and efficiency | ||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||
5 | support in ONAP DMI-plugin | agreed with Toine Siebelink : ONAP plugin can respond with not implemented yet code, | |||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||
6 | Response always Async ie. topic is compulsory ? | Assume topic is compulsory (defined in OPenApi) → Response therefore wil be 400 if not supplied | Agreed with kieran mccarthy : Topic is optional but system will respond with 'Not implemented (yet when not specified or blank | ||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||
7 | Should NCMP Amalgamate Async responses from DMI-Plugin before forwarding ? | step 6 in flow diagram | Agreed with kieran mccarthy : NCMP wil only forward to client topic no handling tracking or any responses or status of request | ||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||
8 | Handle non responding dmi-plugin | Agreed with kieran mccarthy : | |||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||
9 | Should (can) NCMP check if 'MyTopic' specified by client exist | Consider Access Control too. For now NCMP can log error. Client is responsible for topic setup | Agreed with kieran mccarthy : | ||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||
10 | How to handle non-existing CM-Handles (id) | Suggestions
| Agreed with kieran mccarthy : Additional error (messages) response with all cm-handles that cannot be resolved, also a separate error message wil be sent for each failed DMI | ||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||
11 | discussed in weekly ONAP meeting the DT user story is affect CPS-Core interface (not NCMP) and the requirement is to execute a query over ALL cm-handles (cached only?!) instead of a given list of cm-handles (~anchors) | ||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||
12 | 1) Agreement required on the structure of the response. Please see response structure below. | Need to follow schema structure there in #15 under Issues & Decisions section and it is agreed on | |||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||
13 |
CPS-1557 - NCMP : forward bulk response messages to client topic | Agreed on to keep one schema for both events (DMI → NCMP) and (NCMP → ClientApps) | |||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||
14 | 1. Do we need to send only one response containing all the requested cm handles ? | ||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||
15 | Message format for the batch interface |
| |||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||
16 | NCMP wil send only one request to each DMI | Agreed with Csaba Kocsis and kieran mccarthy requestId would be send as a path param from NCMP to DMI-plugin. Example NCMP batch endpoint : http://localhost:8080/ncmp/v1/batch?include-descendants=true&topic=my-topic-name DMI-Plugin batch endpoint : http://172.26.202.25:8783/dmi/v1/batch?topic=my-topic-name&requestId=e6fa4d26-4dc1-4877-aa3c-45e99f840708 |
<Note. use green for closed issues, yellow for important ones if needed>
Implementation
Bulk Request Message Flow
Message Flow details
Flow Step | Short description | Message Details | Notes |
---|---|---|---|
1 | Bulk Get Request | Define new get operation "getResourceDataForCmHandles" into ncmp.yml | |
2 | Ack Client Request | ||
3 | DMI Bulk Request | The DMI PLugin should be told (included in request) the client topic so that NCMP does not have to 'remember' to relation between request id and client topic! | |
4 | Ack Client Request | ||
5 | Kafka Messages from DMI to NCMP | ||
6 | Kafka Message(s) from NCMP to Client Table | ||
7 | Alternative for 4/5 → Non responding DMI. NCMP will have to create error message detailing cm-handles | See decision # 8 and 9 | |
8 | Response message structure ? (Flow no. 5) Non responding DMI-plugin | ||
9 | Response message structure ? (Flow no. 5) Non existing cm handles | ||
10 | Non Ready cm handles | ||
11 | URL pattern for DMI-Plugin bulk endpoints | Existing DMI endpoints are : /v1/ch/{cmHandle}/data/ds/{datastore-name} datastore-name:
...&topic=topicParamInQuery CPS Proposed : /v1/ch/batch/data/ds/{datastore-name} ...&topic=topicParamInQuery cm handle ids and requestid into body |
Proposed JIRAs :
Priority | Component | Description | JIRAs | Estimate |
---|---|---|---|---|
1 | DMI-Plugin | Accept datastore name as param into URL | 5 Days | |
2 | NCMP | Expose REST endpoint to accept collection of cm handles for GET operation (Passthrough only) | 15 Days | |
3 | DMI-Plugin | Expose endpint for ONAP not impl. and Stub impl. for testing/demo | 5 Days | |
4 | NCMP | NCMP: Update existing REST endpoint that accepts bulk request for GET operation | 5 Days | |
5 | DMI-Plugin | DMI-Plugin : Update endpoint to accept bulk request | 3 Days | |
6 | Stubbed DMI-Plugin | Include code to send response messages to internal kafka topic with delay | 10 Days | |
7 | NCMP | Forward response messages to client given kafka topic | 5 Days | |
8 | NCMP | Handle non-existing cm handles | 5 Days | |
9 | NCMP | Error handling for non-ready cm handle state | 5 Days | |
10 | NCMP | Handle non responding DMI-Plugin | 5 Days | |
11 | CSIT test for demo | 5 Days |
Planning :
- Allow for 2 more user stories each may take 1 weeks
- Based on 2 resource working parallelly may take àpprox. 6 weeks from 29 March 2023
- The estimated date for completion is 31st May ' 2023.