Table of Contents
References
Jira | ||||||
---|---|---|---|---|---|---|
|
...
Jira |
---|
...
|
...
|
...
Notes
...
Decisions
|
Assumptions
# | Issue | Notes | Decisions |
---|---|---|---|
1 |
Issues & Decisions
...
Issues & Decisions
# | Issue | Notes | Decisions |
---|---|---|---|
1 | Whether we need to persist the raw request for auditing purpose? | ||
2 | Keep track of the correlation id | Request to and from DMI Plugin has to be tracked hence we need to have a correlation id for this. ( to match the request and response ) |
Overview
This page only contains the details used for starting the development on the Merging of Subscriptions use case. Just the straight forward scenarios will be covered and the overall solution will be build on top of it.
Schema Definitions
DME to NCMP Subscription Request
codeName
id : random-UUID version : "1.0" source : "DME" type : "subscriptionCreateRequest", dataschema :title DME to NCMP Subscription Request linenumbers true collapse true Parent
Type
Example
Mandatory ? Notes
1 id header String Y 2 version header String 1.0 Y 3 source
header String DME Y 4 type
header String subscriptionCreateRequest Y 5 dataschema
header URI org.onap.ncmp.cm.subscription:1.0.0 Y 6 data
Object Y 7 subscriptionId
data String Y 8 predicates
data Array N 9 targetFilter
predicates Array of String Y 10 scopeFilter
predicates Object N 11 datastore
scopeFilter String Y 12 xpathFilter
scopeFilter Array of String Y
Example
NCMP to DMI-Plugin Subscription RequestCode Block title DME to NCMP Subscription Request linenumbers true collapse true id : random-UUID version : "1.0" source : "DME" type : "subscriptionCreateRequest", dataschema : org.onap.ncmp.cm.subscription:1.0.0 { "data": { "subscriptionId": "unique subscription id", { "data": { "subscriptionId": "unique subscription id", // mandatory "predicates": [ { “targetFilter” : list of cmhandles, // mandatory "scopeFilter" : { "datastore": “ncmp-datastore:passthrough-operational or ncmp-datastore:passthrough-running", // optional. default is passthrough-operational "xpath-filter": list of valid xpaths // mandatory "predicates": [ { } } ] }
DMI-Plugin to NCMP Subscription ResponseCode Block title NCMP to DMI-Plugin Subscription Request linenumbers true collapse true id : random UUID version“targetFilter” : 1.0 source : "NCMP" type : "subscriptionCreateRequest" dataschema : org.onap.ncmp.dmi.cm.subscription:1.0.0 correlationId ( concatenation of subscriptionId and dmi-plugin-name with a separator) { "data": { "cmhandles" : [ // mandatory { "cmHandleId": "cmhandle1", "private-properties": {...} }, { "cmHandleId": "cmhandle2", "private-properties": {...} }, { "cmHandleId": "cmhandle3", "private-properties": {...} } ] "predicates": [ { “targetFilter” : [“cmHandle1”, “cmHandle2”, "cmhandle3"], // mandatory "scopeFilter" : { "datastore": “ncmp-datastore:passthrough-operational or ncmp-datastore:passthrough-running", // optional. default is passthrough-operational "xpath-filter": list of valid xpaths // mandatory } }, { “targetFilter” : [“cmHandle1”, “cmHandle2”, "cmhandle3"], "scopeFilter" : { "datastore": “ncmp-datastore:passthrough-operational or ncmp-datastore:passthrough-running", // optional. default is passthrough-operational "xpath-filter": list of valid xpaths * } } ] }
Code Block title DMI-Plugin to NCMP Subscription Response linenumbers true collapse true id : random UUID version : "1.0" source : <dmi-plugin-name> type : "subcriptionCreateResponse" dataschema : org.onap.ncmp.dmi.cm.subscription:1.0.0 correlation-id : ( concatenation of subscriptionId and dmi-plugin-name with a separator) { "data" : { "statusCode": "1", // mandatory "statusMessage" : "accepted/rejected" // mandatory } }
list of cmhandles, // mandatory "scopeFilter" : { "datastore": “ncmp-datastore:passthrough-operational or ncmp-datastore:passthrough-running", // optional. default is passthrough-operational "xpath-filter": list of valid xpaths // mandatory } } ] }
NCMP to DMI-Plugin Subscription Request
Name
Parent
Type
Example
Mandatory ?
Notes
1 id header String Y 2 version header String 1.0 Y 3 source
header String DME Y 4 type
header String subscriptionCreateRequest Y 5 dataschema
header URI org.onap.ncmp.dmi.cm.subscription:1.0.0 Y 6 correlationid
header String subscriptionId#dmiPluginName Y Concatenation of subscriptionId and dmi plugin name 7 data
Object Y 8 cmHandles
data Array Y 9 cmhandleId
cmHandles String Y 10 privateProperties
cmHandles Map N 11 predicates
data Array N 12 targetFilter
predicates Array of String Y list of cm handle ids 13 scopeFilter
predicates Object N 14 datastore
scopeFilter String Y ncmp-datastore:passthrough-operational OR ncmp-datastore:passthrough-running 15 xpathFilter
scopeFilter Array of String Y Example
Code Block title NCMP to DMI-Plugin Subscription Request linenumbers true collapse true id : random UUID version : 1.0 source : "NCMP" type : "subscriptionCreateRequest" dataschema : org.onap.ncmp.dmi.cm.subscription:1.0.0 correlationId ( concatenation of subscriptionId and dmi-plugin-name with a separator) { "data": { "cmhandles" : [ // mandatory { "cmHandleId": "cmhandle1", "privateProperties": {...} }, { "cmHandleId": "cmhandle2", "privateProperties": {...} }, { "cmHandleId": "cmhandle3", "privateProperties": {...} } ] "predicates": [ { “targetFilter” : [“cmHandle1”, “cmHandle2”, "cmhandle3"], // mandatory "scopeFilter" : { "datastore": “ncmp-datastore:passthrough-operational or ncmp-datastore:passthrough-running", // optional. default is passthrough-operational "xpathFilter": list of valid xpaths // mandatory } }, { “targetFilter” : [“cmHandle1”, “cmHandle2”, "cmhandle3"], "scopeFilter" : { "datastore": “ncmp-datastore:passthrough-operational or ncmp-datastore:passthrough-running", // optional. default is passthrough-operational "xpathFilter": list of valid xpaths * } } ] }
DMI-Plugin to NCMP Subscription Response
Name
Parent
Type
Example
Mandatory ?
Notes
1 id header String Y 2 version header String 1.0 Y 3 source
header String DME Y 4 type
header String subscriptionCreateResponse Y 5 dataschema
header URI org.onap.ncmp.dmi.cm.subscription:1.0.0 Y 6 correlationid
header String subscriptionId#dmiPluginName Y Concatenation of subscriptionId and dmi plugin name 7 data
Object Y 8 statusCode
data String Y 1 - Accept the subscription request
104 - Reject the subscription request9 statusMessage
data String Y ACCEPTED
REJECTED
ExampleCode Block title DMI-Plugin to NCMP Subscription Response linenumbers true collapse true id : random UUID version : "1.0" source : <dmi-plugin-name> type : "subcriptionCreateResponse" dataschema : org.onap.ncmp.dmi.cm.subscription:1.0.0 correlationid : ( concatenation of subscriptionId and dmi-plugin-name with a separator) { "data" : { "statusCode": "1/104", // mandatory "statusMessage" : "ACCEPTED/REJECTED" // mandatory } }
NCMP to DME Subscription Response
Name
Parent
Type
Example
Mandatory ?
Notes
1 id header String Y 2 version header String 1.0 Y 3 source
header String DME Y 4 type
header String subscriptionDeleteResponse Y 5 dataschema
header URI org.onap.ncmp.cm.subscription:1.0.0 Y 6 correlationid
header String subscriptionId Y 7 data
Object Y 8 subscriptionId
data String Y 9 acceptedTargets
data Array of String N 10 rejectedTargets
data Array of String N 11 pendingTargets
data Array of String N
ExampleNCMP to DME Subscription ResponseCode Block title NCMP to DME Subscription Response linenumbers true collapse true id : random UUID version : "1.0" source : "NCMP" type : "subcriptionCreateResponse", dataschema : org.onap.ncmp.cm.subscription:1.0.0, correlationId : <subscriptionId> { "data": { "subscriptionId": "sample-subscription-id", // mandatory "accepted-targets" : ["ch-1", ...], // optional "rejected-targets" : ["ch-1", ...], // optional "pending-targets" : ["ch-1", ...], // optional } }
Storage Solutions
Proposed Data Storage
Use a combination of
- Database (yang modelled) for permanent subscription details, subscribers, predicates
- In-Memory (hazelcast) for transient subscription details: state (acceptep, rejected, pending)
Example Flow 1: First Subscription
We have the below configuration managed by CPS-NCMP
#
cmhandle
dmi-plugin
1 ch-1 dmi-1 2 ch-2 dmi-1 3 ch-3 dmi-2 4 ch-4 dmi-2 5 ch-5 non-responding-dmi
We get a Subscription Create Request with unique subscription-id and a list of predicates(list of targets , datastore and list of filters)
We persist the incoming request as is in our persistent store using a relevant subscription model ( this model also needs to be revised )Code Block title Subscription Create Message linenumbers true collapse true { "subscriptionId" : "A-10", "predicates" : [ { "targets": [ch-1,ch-2], "datastore" : "ncmp-datastore:passthrough-operational", "datastore-xpath-filter" : ["p1/c1","p2/c2"] }, { "targets": [ch-3], "datastore" : "ncmp-datastore:passthrough-operational", "datastore-xpath-filter" : ["p3/c3"] } ] }
We also maintain a distributed datastructure in Hazelcast to keep track of the request and response to the client.
SubscriptionId
dmi-plugin
affectedCmHandles
status
A-10 dmi-1 ["ch-1","ch-2"] PENDING A-10 dmi-2 ["ch-3"] PENDING - Read the requests from the distributed data structure and form the request for the DMI-Plugin and publish it to the internal kafka channel.
- If the DMI-Plugin responds back within the configured time , so the request will either be ACCEPTED or REJECTED.
- if whole subscription request is ACCEPTED , update the distributed data structure and then store the subscription to the ActiveSubscriptions in cps-cachethe database.
- if whole subscription request is REJECTED , update the distributed data structure and send back the response to the clientthe response to the client and no need to store anything to database.
if the DMI Plugin fails to respond then the status would remain PENDING only in the distributed datastructure as there is no change.
SubscriptionId
dmi-plugin
affectedCmHandles
status
A-10 dmi-1 ["ch-1","ch-2"] ACCEPTED A-10 dmi-2 ["ch-3"] REJECTED
If there are no more PENDING requests , then we can right away send the response without updating the cache.
We store the "ActiveSubscriptions" using a new model which will look like below.
SubscriptionId
cmHandle
filter
datastore
A-10 ch-1 p1/c1 ncmp-datastore:passthrough-operational A-10 ch-1 p2/c2 ncmp-datastore:passthrough-operational A-10 ch-2 p1/c1 ncmp-datastore:passthrough-operational A-10 ch-2 p2/c2 ncmp-datastore:passthrough-operational - Now based on whatever we have in the distributed map , we will send the response to the clients and may be clear the in-memory structure if required.
Example Flow 2: Second Subscription (merge)
We get a new subscription create request.
Code Block title 2nd Subscription Create Message linenumbers true collapse true { "subscriptionId" : "B-52", "predicates" : [ { "targets": [ch-2], "datastore" : "ncmp-datastore:passthrough-operational", "datastore-xpath-filter" : ["p2/c2" , "p3/c3"] } ] }
We now look at the "ActiveSubscriptions" in the database if we already have an active ongoing subscription either for a combination of "cmhandle and xpath filter and the datastore".
Current view of ActiveSubscriptionsSubscriptionId
cmHandle
filter
datastore
A-10 ch-1 p1/c1 ncmp-datastore:passthrough-operational A-10 ch-1 p2/c2 ncmp-datastore:passthrough-operational A-10 ch-2 p1/c1 ncmp-datastore:passthrough-operational A-10 ch-2 p2/c2 ncmp-datastore:passthrough-operational Check if we have ch-2 , p2/c2 and ncmp-datastore:passthrough-operational then directly add to the ActiveSubscription. Similarly check for ch-3 , p3/c3 and ncmp-datastore:passthrough-operational ( which in this case will not be directly to the active subscription)
Now the state of ActiveSubscription is :
SubscriptionId
cmHandle
filter
datastore
A-10 ch-1 p1/c1 ncmp-datastore:passthrough-operational A-10 ch-1 p2/c2 ncmp-datastore:passthrough-operational A-10 ch-2 p1/c1 ncmp-datastore:passthrough-operational A-10 ch-2 p2/c2 ncmp-datastore:passthrough-operational B-52 ch-2 p2/c2 ncmp-datastore:passthrough-operational and state of the in-memory data structure is as follows.
SubscriptionId
dmi-plugin
affectedCmHandles
status
B-52 dmi-1 ["ch-2"] PENDING - Now we send the request to DMI plugin (dmi-1) with just the private properties of the ch-2 with the delta xpath.
- DMI Plugin could either apply the whole subscription request or reject the whole subscription. i.e respond with accepted/rejected
Based on the response from DMI plugin the in-memory structure is updated and database also updated if the request is ACCEPTED and response is sent back to the clients.
if DMI says it is accepted then :SubscriptionId
dmi-plugin
affectedCmHandles
status
B-52 dmi-1 ["ch-2"] ACCEPTED
if DMI says it is rejected then :SubscriptionId
dmi-plugin
affectedCmHandles
status
B-52 dmi-1 ["ch-2"] REJECTED
if we get no response within the configured time then :SubscriptionId
dmi-plugin
affectedCmHandles
status
B-52 dmi-1 ["ch-2"] PENDING
But we still send some response within the pre-configured time.if the above request was accepted then we should add it to the list of "ActiveSubscriptions" as well in the database.
SubscriptionId
cmHandle
filter
datastore
A-10 ch-1 p1/c1 ncmp-datastore:passthrough-operational A-10 ch-1 p2/c2 ncmp-datastore:passthrough-operational A-10 ch-2 p1/c1 ncmp-datastore:passthrough-operational A-10 ch-2 p2/c2 ncmp-datastore:passthrough-operational B-52 ch-2 p2/c2 ncmp-datastore:passthrough-operational B-52 ch-2 p3/c3 ncmp-datastore:passthrough-operational - If the status was accepted/rejected then after sending the response to the client we can clear the in-memory structure for subscriptionId B-52.
Storage Solution Discussion
We had an internal meeting and concluded to do a POC with the below YANG schema to efficiently store the subscriptions.
Summary for Grooming
Note : This is the latest approach we are following. (Priyank Maheshwari as of )
Tracking Subscriptions in memory