Table of Contents |
---|
References
Jira server ONAP Jira columnIds issuekey,summary,issuetype,created,updated,duedate,assignee,reporter,priority,status,resolution columns key,summary,type,created,updated,due,assignee,reporter,priority,status,resolution serverId 425b2b0a-557c-3c0c-b515-579789cceedb key CPS-1616 Jira server ONAP Jira columnIds issuekey,summary,issuetype,created,updated,duedate,assignee,reporter,priority,status,resolution columns key,summary,type,created,updated,due,assignee,reporter,priority,status,resolution serverId 425b2b0a-557c-3c0c-b515-579789cceedb key CPS-1615 Jira server ONAP Jira columnIds issuekey,summary,issuetype,created,updated,duedate,assignee,reporter,priority,status,resolution columns key,summary,type,created,updated,due,assignee,reporter,priority,status,resolution serverId 425b2b0a-557c-3c0c-b515-579789cceedb key CPS-1812 - Impl. Proposal CM Event Subscription LCM: Create
- Impl. Proposal CM Event Subscription LCM: Delete
Requirements
Functional
Interface | Requirement | Additional Information | Signoff | |||||||||
---|---|---|---|---|---|---|---|---|---|---|---|---|
1 | CPS-E-08e | NCMP will forward 'CM Notifications Subscription' LCM Create events to the appropriate DMI Plugins responsible for the CM Handles (ids) defined in the LCM Events. The message sent to DMI should include the private properties for each CM-Handle | ||||||||||
2 | CPS-E-08e | NCMP will forward 'CM Notifications Subscription' LCM Delete events to the appropriate DMI Plugins responsible for the CM Handles is defined in the LCM Events. The message sent to DMI should include the private properties for each CM-Handle | 'Basic' delete we be implemented followed by a more advanced delete operation depending on Merge functionality:
| |||||||||
3 | CPS-E-08e | NCMP responds using Kafka message(s) with the result of the LCM operation These messages wil use agreed status fields:
| Action: Define error details later | |||||||||
4 | CPS-E-08e | All events should use Cloud Events format | https://cloudevents.io/ | |||||||||
5 | CPS-E-08e | Messages should be processed in the order they are created (received) | It will depend on DMIs responding within 30s. Subscription-id should be used as Kafka 'key' to ensure this. |
| ||||||||
6 | CPS-E-08.e | NCMP is to merge CM Notification Subscriptions create request and forward those to DMI plugin. (Interface to be added). |
| kieran mccarthy | ||||||||
7 | CPS-E-08.e | Last lights out: upon subscription Delete request only when there is no more subscription for a cm-handle & xpath & datastore combination a subscription-delete request will be sent to the relevant DMI(s). |
| |||||||||
8 | CPS-E-08.e | A single client subscription request should result into a maximum of one request per DMI. Of course there can be several messages if more than 1 DMI is involved. |
| |||||||||
9 | CPS-E-08.e | Amalgamate response should include rejected/accepted/pending DMI responses received within 30 seconds. A client shall be notified of available DMI subscription information after 30 seconds. Subsequent DMI subscription updates shall be notified to clients as they become available. |
|
| ||||||||
10 | CPS-NCMP-I-01 | CM Handle deletion should NOT update subscription details. | Do NOT delete dmi-subscription entry until owning subscription is deleted, see issue #4 below |
| ||||||||
11 | CPS-E-08.e | LCM and Merge-functionality are now closely interlinked and no separate delivery f LCM without merging support wil be done |
Error Handling
Error Scenario | Expected behavior | Sign-off | |
---|---|---|---|
1 | Non responding DMIs (configurable timeout) | An error message listing the 'pending' CM Handle IDs and statusMessage as "partially applied subscription" with relevant statuscode. | |
2 | Non Existing or non-valid CM Handle Ids | An error message listing the 'rejected' CM Handle IDs | |
3 | DMI Plugin without CM Event subscription support | An error message listing the 'rejected' CM Handle IDs | |
4 | Late Response (after timeout defined in #1) from DMI | A message contain the status message as "fully applied subscription" with relevant status code. | |
5 | Create for existing subscription id | ||
6 | delete for non-existing cm handle id | (silently?) ignore | |
7 | delete for non-existing subscription id | ||
8 | error upon error | An error scenario on a second subscription for the same cm-handle/xpath as a previous subscription which did not complete successfully (yet) to be discussed, see |
section below |
Characteristics
...
Parameter | Expectation | Notes |
---|
Sign-off | |
---|---|
Frequency of LCM Events | No high volume of such events is expected For testing purposed 1 LCM event per minute would suffice |
Out-of-scope
- Working with the wildcards, only list of specific CM Handle IDs are supported
- Subscription Update scenarios for an ongoing subscription.
Assumptions
...
#
...
Assumption
...
Notes
...
Developing based on the schemas we previously agreed.
Any change in the schemas will impact in code change.
Issues & Decisions
...
#
...
Issue
...
Notes
...
Decision
...
ncmp-datastore:passthrough-running or ncmp-datastore:passthrough-operational ??
or both?
...
Maximum number of cm data subscriptions | 200 | kieran mccarthy confirmed |
|
---|---|---|---|
Maximum, Average CM Handles per subscription | Avg: 200 Max: 500 (document only) | kieran mccarthy to confirm |
|
Maximum, Average XPaths per CmHandle | Avg: 10 |
| |
Status update processing time limit | Based on the numbers above working out the current state of each cm handle/xpath combination could be slow! | kieran mccarthy order of 30 seconds (NCMP side) |
|
Parameter | Expectation | Notes | Sign-off | |
---|---|---|---|---|
1 | Frequency of LCM Events | No high volume of such events is expected For testing purposed 1 LCM event per minute would suffice | ||
2 | Number of Subscriptions | Max: 200 | kieran mccarthy confirmed |
|
3 | CM Handles per Subscription | Avg: 200 Max: 500 (document only) | kieran mccarthy to confirm |
|
4 | Xpaths per Cm Handle | Avg: 10 |
| |
5 | Status update processing time | (Order of 30) seconds (NCMP side) |
|
Out-of-scope
- Subscription Update scenarios for an ongoing subscription.
- CM Notification Forwarding Check: When forwarding CM Notification NCMP will not check the content to see if the is a valid active subscription. It is assumed that the DMI Plugin as acted on the 'delete subscription' request (that request is NCMPs responsibility). And of course there wil be timing issues it also possible a CM Notification was send just after the subscription-delete was send (from client) but before the whole change had acted upon that.
- Retry: NCMP will only report when actions are pending or rejected. NCMP will not implement a retry mechanism
- Wildcards: Wildcards or similar functionality where one string represent 0 or more xpaths is not covered as part of this requirement but it should be kept in mind as a future possibility
- Dynamic Topic: Topic for CM Data Notifications back to client will be hardcode for now.
Study should consider compatibility with an 'ALL' cm handles options
Assumptions
Assumption | Notes | |
---|---|---|
1 | Targets is a collection of xpaths | can be implemented as a json array of strings |
2 | for the DMI plugin cm-handle-id, datastore and xpath are the key to a unique subscription | the client-id and subscription name are not relevant for the DMI (should not be included in the request) as this would break the concept of merging! |
Issues & Decisions
Issue | Notes | Decision | |
---|---|---|---|
1 | Is it intended that CM Notification subscription request cover (all) descendants of the given xpath too?! | e.g.. if a child is removed and there is a subscription for the parent node, will a notification be send (grandchild, child leaf updates etc.) I hope NOT! Consider:
| kieran mccarthy descendant not covered by 'basic' paths Wildcard will cover this kind of function in future |
2 | Could xpath point to an element that does not exist (yet) | if not how, how can I client be informed about a create event? | kieran mccarthy yes, xpath can point to things that don't exist yet (not even in the curernt model when an upgraded is pending) |
3 | Should NCMP support re-homing, moving of a CM Handle from one DMI to another? | assume only trough delete & create | kieran mccarthy out of scope |
4 | CM Handle Delete: Should DMI or Clients be sent a subscription update | do NOT delete dmi-subscription entry until owning subscription is deleted Note. LCM is already broadcast (today) |
|
5 | Validation of xpath | options order of implement and also performance cost!
| kieran mccarthy not required right now |
6 | can DMI plugin 'reject' a subscription create (for a given cm-handle-xpath combination) | As NCMP might not validate as per issue#5 the DMI=plugin or component further down might have to reject an invalid xpath... | yes currently DMI can use response to say which cm handles are not accepted i.e. rejected' (but not 'pending') kieran mccarthyconfirms DMI can reject whole subscription |
7 | implementation question: should 'rejected' DMI-subscriptions be stored | not needed as whole subscription should be rejected | kieran mccarthyconfirms |
8 | Dimensioning of DB depends on #cm handles, #subscriptions and #xpaths per subscription, this could be too big for fast processing of updates! | Need to agree maximum and possibly realistic average/total number of entries based on the characteristics above The team is blocked until this becomes clear as it wil affect the way the data needs to be modelled exactly | ongoing but not completed! |
9 | Maximum (error) message size | theoretically all cm handles and all xpaths combinations could be rejected or pending leading to a very large error message! | Toine Siebelink no longer relevant given and max decided |
10 | can each CM-Handle have different set of xpath(s) per subscription | the current 'basic' solution only supports a common set of datastore/xpaths (filter) | kieran mccarthy Yes, clients will create only one subscription and they might have different needs for different nodes (cm handles) |
11 | can the same cm handle/xpath have different subscriptions with different datastores, does that affect the cm data notifications send (which datastore applies) |
| kieran mccarthy datastore included. Although this wiki-page will not be updated where cmHandle+xpath is mentioned as unique entry etc. this should now include a third field: datastore as well. |
12 | Will migration from 'basic' be supported | Preferred to ask customers to create new subscriptions | kieran mccarthy no need to support (migration) of 'basic' solution. Development of 'basic' solution can be stopped! |
13 | list v list instances filtering | /p/c1 | kieran mccarthy yes, from NCMP point of view a list and a list entry are different xpaths |
14 | confirm subscription id (currently subscription name + client id) | just subscription-id (string) | |
15 | what subscription details should be send when there is a change (in the union) |
| (after initial meeting) kieran mccarthy & Toine Siebelink agreed that because of the possible size of the union (200x200x10=400,000) it is only feasible to send the delta ie option 1. |
16 | one DMI rejects whole (see decision #6) subscription (affected cmhandles) but other DMI accepts the same subscription, is this possible how to handle | kieran mccarthy only sends message back to client about rejected DMIs i.e. the subscription can be partially 'active' | |
17 | What all datastores are supported ? | ncmp-datastore:passthrough-running or ncmp-datastore:passthrough-operational ?? or both? |
Solution Proposal
State handling for initial (basic)subscription create/delete use cases (not using concept of 'merging')
Expand |
---|
Scenario 1 NCMP Send message to client: Statsu OK (no list with Rejected/Pending)
NCMP Send message to client: Pending [CH-1, CH-2] Scenario 3 NCMP Send message to client: Rejected [CH-2] Scenario 4: Scenario 4-part a NCMP -> Client : Rejected [CH-4], Pending [ CH-5, CH-6, CH-7, CH-8 ] 4-part b 4-part c |
Note. The above algorithm depends ons storing (in DB using yang modelled data) a cm-handle status for each cm-handle for each subscription!
Merge → Split
- NCMP treats a Client Subscription as one or more (small) DMI Subscriptions, each of which wil have there own state.
Each DMI Subscription is related to 1 or more client subscriptions.
If there is no more related client subscription the DMI Subscription can be deleted (once accepted by DMI Plugin)!
Data Model (not a great diagram...)
draw.io Diagram | ||||||||||||||||||
---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
|
Create Combinations
Below examples demonstrate what should happen when two separate subscriptions operations are performed:
- an operation on subscription 'A' for client id '10
- an operation on subscription 'B' for client id '52'
# | Existing Subscription A-10 | Client Create Subscription B-52 Request | DMI Sub Create Request | Data Model Before (2nd) Create | Data Model After Create | |||||||||||||||||||||||||||||||||||||||||||||||||||||||||
---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
1 | CH-1, [ /p/c1, /p/c2 ] | CH-1, [ /p/c1 ] | None |
|
| |||||||||||||||||||||||||||||||||||||||||||||||||||||||||
2 | CH-1, [ /p/c1, p/c2 ] | CH-1, [ /p/c2, /p/c3 ] | CH-1, [ /p/c2, /p/c3 ] |
|
| |||||||||||||||||||||||||||||||||||||||||||||||||||||||||
3 | CH-1, [ /p/c1] CH-2, [ /p/c1] | CH-2, [ /p/c1] CH-3, [ /p/c1] | CH-3, [ /p/c1] |
|
| |||||||||||||||||||||||||||||||||||||||||||||||||||||||||
4 | CH-1, [ /p/c1] | CH-1, [ /p/c1/gc1 ] | CH-1, [ /p/c1/gc1 ] *as per decision 1 the xpath to a parent does NOT include any children... |
|
|
Create 'Merge' Diagram
Below diagram shows and example for two subscriptions with party overlapping CM Handles and XPaths.
Gliffy Diagram | ||||||||
---|---|---|---|---|---|---|---|---|
|
* Note 1: given the possible combinations the message to DMI needs to be able to specify different xpaths per cm-handle. So a more complex structure is needed for this even ie. an array of CM Handles objects each having their own list of (target) xpaths!
Note 2: as per decision #11'datastore' should be included in all messages as well
Delete Combinations
# | Existing Subscription A-10 | Existing Subscription B-52 | Client Delete A-10 Request | DMI Sub update/delete request | Data Model* Before Before | Data Model* After Delete | |||||||||||||||||||||||||||||||||||||||||||||||||||||||||
---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
1 | CH-1, [ /p/c1, p/c2 ] | CH-1, [ /p/c1 ] | CH-1, [ p/c2 ] |
|
| ||||||||||||||||||||||||||||||||||||||||||||||||||||||||||
2 | CH-1, [ /p/c1] CH-2, [ /p/c1] | CH-2, [ /p/c1] CH-3, [ /p/c1] | CH-1, [ /p/c1] |
|
| ||||||||||||||||||||||||||||||||||||||||||||||||||||||||||
3 | CH-1, [ /p/c1] | CH-1, [ /p/c1] CH-2, [ /p/c1] | None |
|
|
* Note: as per decision #11'datastore' should be included in the 'DMI susbcription' data model too now.
Error-Upon-Error Combinations
Previous Interaction | Current Interaction | Expectation | Notes | |
---|---|---|---|---|
1 | any operation on rejected for non-existing cm-handle | operation for same non-existing cm-handle | listed in 'rejected' immediately | behavior as normal |
2 | create operation rejected by DMI | create for same cm-handle/xpath | try again ?! | |
3 | create pending | create for same cm-handle/xpath | TBD send again or just remain pending?! | |
4 | create pending | delete for same cm-handle/xpath | ||
5 | delete pending | delete for same cm-handle/xpath | ||
6 | delete pending | create for same cm-handle/xpath | ||
7 |
Client-Schema Update
Based on Issue#10 , the incoming schema from DME to NCMP:
- A single subscription contains multiple predicates
- Predicates should be an array of Predicates ( currently a single instance )
- Each predicate contains an array of targets ( cmhandles) - as is.
- Each predicate contains a single datastore (mandatory)
- datastore-xpath-filter should become an array of xpath-filters (instead of pipe separated single string)
Performance considerations
Below options (partly) explored dring CPS teem meeting could explore further once we know the characteristics, see issue #8 !
- DMI record could reach in millions of instances
- In memory solution (ie hazelcast) can handle 10s - 100s of Megabytes (add some sample calc)
- still need persistence if we don't have enough instances
- use many small messages reduce need for amalgamation
- amalgamated string-based solution....
Solution Proposal - Storage & Merging
Storage
We have the below configuration managed by CPS-NCMP
# cmhandle dmi-plugin 1 ch-1 dmi-1 2 ch-2 dmi-1 3 ch-3 dmi-2 4 ch-4 dmi-2 5 ch-5 non-responding-dmi
We get a Subscription Create Request with unique subscription-id and a list of predicates(list of targets , datastore and list of filters)
Code Block title Subscription Create Request { "subscriptionId" : "A-10", "predicates" : [ { "targets": [ch-1,ch-2], "datastore" : "ncmp-datastore:passthrough-operational", "datastore-xpath-filter" : ["p1/c1","p2/c2"] }, { "targets": [ch-3], "datastore" : "ncmp-datastore:passthrough-operational", "datastore-xpath-filter" : ["p3/c3"] } ] }
- We persist the incoming request as is in our persistent store using a relevant subscription model ( this model also needs to be revised )
We also maintain a distributed datastructure in Hazelcast to keep track of the request and response to the client.
SubscriptionId dmi-plugin affectedCmHandles status A-10 dmi-1 ["ch-1","ch-2"] PENDING A-10 dmi-2 ["ch-3"] PENDING - Read the requests from the distributed data structure and form the request for the DMI-Plugin and publish it to the internal kafka channel.
- If the DMI-Plugin responds back within the configured time , so the request will either be ACCEPTED or REJECTED.
- if whole subscription request is ACCEPTED , update the distributed data structure and then store the subscription to the ActiveSubscriptions in cps-cache.
- if whole subscription request is REJECTED , update the distributed data structure and send back the response to the client.
if the DMI Plugin fails to respond then the status would remain PENDING only as there is no change.
SubscriptionId dmi-plugin affectedCmHandles status A-10 dmi-1 ["ch-1","ch-2"] ACCEPTED A-10 dmi-2 ["ch-3"] REJECTED
We store the "ActiveSubscriptions" using a new model which will look like below.
SubscriptionId cmHandle filter datastore A-10 ch-1 p1/c1 ncmp-datastore:passthrough-operational A-10 ch-1 p2/c2 ncmp-datastore:passthrough-operational A-10 ch-2 p1/c1 ncmp-datastore:passthrough-operational A-10 ch-2 p2/c2 ncmp-datastore:passthrough-operational Now based on whatever we have in the distributed map , we will send the response to the clients and may be clear the in-memory structure if required.
Merging ( in-progress )
We get a new subscription create request.
Code Block { "subscriptionId" : "B-52", "predicates" : [ { "targets": [ch-2], "datastore" : "ncmp-datastore:passthrough-operational", "datastore-xpath-filter" : ["p2/c2" , "p3/c3"] } ] }
We now look at the "ActiveSubscriptions" if we already have an active subscription either for a combination of "cmhandle and xpath filter and the datastore".
Current view of ActiveSubscriptionsSubscriptionId cmHandle filter datastore A-10 ch-1 p1/c1 ncmp-datastore:passthrough-operational A-10 ch-1 p2/c2 ncmp-datastore:passthrough-operational A-10 ch-2 p1/c1 ncmp-datastore:passthrough-operational A-10 ch-2 p2/c2 ncmp-datastore:passthrough-operational
Check if we have ch-2 , p2/c2 and ncmp-datastore:passthrough-operational then directly add to the ActiveSubscription.Similarly check for ch-3 , p3/c3 and ncmp-datastore:passthrough-operational ( which in this case will not be directly to the active subscription)Now the state of ActiveSubscription is :
SubscriptionId cmHandle filter datastore A-10 ch-1 p1/c1 ncmp-datastore:passthrough-operational A-10 ch-1 p2/c2 ncmp-datastore:passthrough-operational A-10 ch-2 p1/c1 ncmp-datastore:passthrough-operational A-10 ch-2 p2/c2 ncmp-datastore:passthrough-operational B-52 ch-2 p2/c2 ncmp-datastore:passthrough-operational and state of the in-memory datastructure is as follows.
SubscriptionId dmi-plugin affectedCmHandles status B-52 dmi-1 ["ch-2"] PENDING - Now we send the request to DMI plugin (dmi-1) with just the private properties of the ch-2 with the delta xpath.
- DMI Plugin could either apply the whole subscription request or reject the whole subscription. i.e respond with accepted/rejected
Based on the response from DMI plugin the in-memory structure is updated and response is sent back to the clients.
if DMI says it is accepted then :SubscriptionId dmi-plugin affectedCmHandles status B-52 dmi-1 ["ch-2"] ACCEPTED
if DMI says it is rejected then :SubscriptionId dmi-plugin affectedCmHandles status B-52 dmi-1 ["ch-2"] REJECTED
if we get no response within the configured time then :SubscriptionId dmi-plugin affectedCmHandles status B-52 dmi-1 ["ch-2"] PENDING
But we still send some response within the pre-configured time.if the above request was accepted then we should add it to the list of "ActiveSubscriptions" as well.
SubscriptionId cmHandle filter datastore A-10 ch-1 p1/c1 ncmp-datastore:passthrough-operational A-10 ch-1 p2/c2 ncmp-datastore:passthrough-operational A-10 ch-2 p1/c1 ncmp-datastore:passthrough-operational A-10 ch-2 p2/c2 ncmp-datastore:passthrough-operational B-52 ch-2 p2/c2 ncmp-datastore:passthrough-operational B-52 ch-2 p3/c3 ncmp-datastore:passthrough-operational - If the status was accepted/rejected then after sending the response to the client we can clear the in-memory structure for subscriptionId B-52.
Overview
...
Kafka Topics
# | Topic Name | Environment Variable | Source | Destination | Notes |
---|---|---|---|---|---|
1 | subscription | NCMP_CM_AVC_SUBSCRIPTION | DME | DME/Client-Apps | |
2 | ncmp-dmi-cm-avc-subscription-<dmi-name> | NCMP_FORWARD_CM_AVC_SUBSCRIPTION | NCMP | DMI Plugin | |
3 | dmi-ncmp-cm-avc-subscription | NCMP_RESPONSE_CM_AVC_SUBSCRIPTION | DMI Plugin | NCMP | |
4 | subscription-response | NCMP_OUTCOME_CM_AVC_SUBSCRIPTION | NCMP | DME/Client-Apps | |
5 | dmi-cm-events | DMI_CM_EVENTS_TOPIC | DMI Plugin | NCMP | actual data notifications as per the subscriptions |
6 | cm-events | NCMP_CM_EVENTS_TOPIC | NCMP | DME/Client-Apps | NCMP to forward the notifications to the client-apps. |
...