You are viewing an old version of this page. View the current version.

Compare with Current View Page History

« Previous Version 53 Next »

Create 'bulk' version for org.onap.cps.ncmp.rest.controller.NetworkCmProxyController#getResourceDataForCmHandle to support multiple cm-handles (~ multiple anchors in in CPS-Core)

References

  1. CPS-1515 - Spike: Support multiple CM-Handles for NCMP Get Operation
  2. CPS-NCMP ↔ DMI-Plugin Interface Details Jakarta-R10

Requirements

Functional

#InterfaceRequirementAdditional Information
1REST CPS(-NCMP)-E-05Support batch read operation (new) using an asynchronous response on a client specified topicpayload includes list of cm handle ids
2REST DMI-I-01Support batch read operation (new) using an asynchronous response to an internal topicpayload includes list of (associated for this plugin) cm handles ids with their private (additional) properties

Error Handling

#Error ScenarioExpected behavior
1DMI Not respond to initial synchronous request with normal HTTP timeoutSpecial Error message (Kafka event)  including affected, and reason for all the handles in the message cm handles send to client topic detailing cm-handles
(per dmi plugin)
2Topic not supplied on CPS-E-05Return HTTP 501 (not implemented)
3Client specided Topic is not configuredLog error message only
4Non-existing cm-handle idSimilar message but different reason as specified #1 above

Capabilities

#ParameterExpectationNotes
1Response Time 1 Batch request<2 seconds (average)
  • Async response available on client topic
  • No delay in DMI PLugin (tested/measured using stub DMI Plugin)
2Batch-size200 cm handlesNo hardcoded limit
3Response payload sizeTBD KBPerformance test for cabilitie should be tested with this average response size
4Maximum registered #cm handles 20,000This will effect the internal query time
5Supported # DMI PLugins10This might effect processing times
6Test Environment
  1. CPS and NCMP

requests:
    cpu: 2000m
    memory: 2Gi
limits:
    memory: 3Gi
    cpu: 3000m

2. Postgres

requests:
    cpu: 4000m
    memory: 1Gi
 limits:
    memory: 3Gi
    cpu: 6000m


7Concurrent request1212 clients requests toward 1 NCMP simultaneously
8Request Frequency100 request/minShould not affect performance, does not need to be tested

Out-of-scope

  1. Support for multiple resource identifiers in one batch operation
  2. Support cached data batch requests: only passthrough datastores will be supported (see decision #2)
  3. NCMP does NOT keep track of request status to see if  it is completed, amalgamate responses or anything like that. It wil simply forward responses from the internal topic to the client topic.
  4. Access control

Assumptions

#

Assumption

Notes

1

Proprietary options or not in the scope of this analysis.

agreed with kieran mccarthy ; these are optional parameters (name-value pairs) not interpreted by NCMP but can be interpreted by proprietary plugins
In the future "scope" might become standardized instead of proprietary but that wil be achieved through a separate requirement 

2

same xpath (resourceIdentifierInQuery) for all cm handles or different for each cm handle

agreed with kieran mccarthy ; if different resources are required on the same cm-handle the client wil send another (batch) request

Issues & Decisions

#

Issue

Notes 

Decision

1Which operation(s) need support for multiple cm handles?
  1. Get
  2. Create
  3. Update (Put)
  4. Patch
  5. Delete

if many what is the priority?

agreed with kieran mccarthy 

Only Get (read)

(in future other operations might be support batch option too)

2

Which datasources should be supported?

Do we need to support passthrough-only no-cached() data only ?
(maybe just start with that, support cached data bulk request later)

agreed with kieran mccarthy :

all passthrough datastores will be supported

Not implemented (yet) response, for non passthrough datastores 

3URL  pattern for NCMP bulk endpoints

Existing : /v1/ch/{cm-handle}/data/ds/{datastore-name}

CPS Proposed : /v1/batch/data/ds/{datastore-name}

      ( include cm handles into payload / body and leave datastore into url)

agreed with kieran mccarthy : Follow the existing interfaces as much as possible for consistency and efficiency

4keep datastore, topic and optional parameters in the URL itself instead into body.

CPS prefers keep interface similar as single cm handle interface (consistency and cost)

Existing : ...&topic=topicParamInQuery

agreed with kieran mccarthy : Follow the existing interfaces as much as possible for consistency and efficiency

5

support in ONAP DMI-plugin


agreed with Toine Siebelink : ONAP plugin can respond with not implemented yet code,

6

Response always Async ie. topic is compulsory ?

Assume topic is compulsory (defined in OPenApi) → Response therefore wil be 400 if not supplied

Agreed with kieran mccarthy :

Topic is optional but system will respond with 'Not implemented (yet when not specified or blank

7Should NCMP Amalgamate Async responses from DMI-Plugin before forwarding ?step 6 in flow diagram

Agreed with kieran mccarthy : NCMP wil only forward to client topic no handling tracking or any responses or status of request

8Handle non responding dmi-plugin

Agreed with kieran mccarthy :
No response for 4b then send an error response to the topic given by client

9Should (can) NCMP check if 'MyTopic' specified by client existConsider Access Control too. For now NCMP can log error. Client is responsible for topic setup

Agreed with kieran mccarthy :
NCMP can log error when forward to 'client topic' Security not in scope (yet)

10How to handle non-existing CM-Handles (id)

Suggestions

  1. Silently ignore
  2. (initial) error response
    1. should we combine all errors in one message?

Agreed with kieran mccarthy :

Additional error (messages) response with all cm-handles that cannot be resolved, also a separate error message wil be sent for each failed DMI

11

Overlap/clash with Deutsche Telekom user story: CPS-1377 - Getting issue details... STATUS ?!


discussed in weekly ONAP meeting the DT user story is affect CPS-Core interface (not NCMP) and the requirement is to execute a query over ALL cm-handles (cached  only?!) instead of a given list of cm-handles (~anchors)

<Note. use green for closed issues, yellow for important ones if needed>

Implementation

Bulk Request Message Flow

Cm Handle Batch Interface Design

Message Flow details

Flow Step

Short description

Message Details

Notes

1

Bulk Get Request

/v1/batch/data/ds/{datastore-name} 
body:
{"cmhandleIds":["cm-1",...,"cm-n"]

Define new get operation "getResourceDataForCmHandles" into ncmp.yml

2Ack Client Request
Response 200 
{"requestId":"123"}

3DMI Bulk Request
/v1/ch/batch/data/ds/{datastore-name}&topic=topicParamInQuery 
body: {"Cmhandles":["cm-1",...,"cm-n"],"requestId":123}

The DMI PLugin should be told (included in request) the client topic so that NCMP does not have to 'remember' to relation between request id and client topic!



4Ack Client Request
Response 202

5Kafka Messages from DMI to NCMP
Topic: ncmp-async-m2m 
Message Details TBD

6Kafka Message(s) from NCMP to Client 
Table

Topic: <client topic> 
Message Details TBD

7Alternative for 4/5 → Non responding DMI. NCMP will have to create error message detailing cm-handles
See decision # 8 and 9
8

Response message structure ? (Flow no. 5)

Non responding DMI-plugin 

Non responding DMI-plugin  
{
"timestamp":"2023-03-01T23:00:00.345-0400",
  "status": 503, 
"requestId":123,
  "message": "DMI Service Unavailable, {service-name}"
 }

9

Response message structure ? (Flow no. 5)

Non existing cm handles

Non existing cm handles 
{
	"timestamp":"2023-03-01T23:00:00.345-0400",
"status":404,
"requestId":123,
	"error":"cm handle not found",
	"message":{"Cmhandles":["cm-1",...,"cm-n"]}
}

10URL  pattern for DMI-Plugin bulk endpoints

Existing DMI endpoints are : 

/v1/ch/{cmHandle}/data/ds/{datastore-name}


datastore-name:

  1. ncmp-datastore:passthrough-operational
  2. ncmp-datastore:passthrough-running

 ...&topic=topicParamInQuery


CPS Proposed :

/v1/ch/batch/data/ds/{datastore-name}

 ...&topic=topicParamInQuery

cm handle ids and requestid into body


Proposed JIRAs :

#ComponentDescriptionJIRAs
1DMI-PluginAccept  datastore name as param into URL 

CPS-1550 - Getting issue details... STATUS

2NCMP Expose REST endpoint to accept collection of cm handles for GET operation (Passthrough only)

CPS-1553 - Getting issue details... STATUS

3DMI-PluginExpose endpint for ONAP not impl. and Stub impl. for testing/demo

CPS-1555 - Getting issue details... STATUS

4Stubbed DMI-PluginInclude code to send response messages to internal kafka topic with delay 

CPS-1556 - Getting issue details... STATUS

5NCMP Forward response messages to client given kafka topic

CPS-1557 - Getting issue details... STATUS

6NCMP Handle non responding DMI-Plugin and non-existing cm handles

CPS-1558 - Getting issue details... STATUS

7CSIT test for demo

CPS-1560 - Getting issue details... STATUS

  • No labels