...
- DMI record could reach in millions of instances
- In memory solution (ie hazelcast) can handle 10s - 100s of Megabytes (add some sample calc)
- still need persistence if we don't have enough instances
- use many small messages reduce need for amalgamation
- amalgamated string-based solution....
Solution Proposal - Storage & Merging
Storage
We have the below configuration managed by CPS-NCMP
# cmhandle dmi-plugin 1 ch-1 dmi-1 2 ch-2 dmi-1 3 ch-3 dmi-2 4 ch-4 dmi-2 5 ch-5 non-responding-dmi
We get a Subscription Create Request with unique subscription-id and a list of predicates(list of targets , datastore and list of filters)
Code Block title Subscription Create Request { "subscriptionId" : "A-10", "predicates" : [ { "targets": [ch-1,ch-2], "datastore" : "ncmp-datastore:passthrough-operational", "datastore-xpath-filter" : ["p1/c1","p2/c2"] }, { "targets": [ch-3], "datastore" : "ncmp-datastore:passthrough-operational", "datastore-xpath-filter" : ["p3/c3"] } ] }
- We persist the incoming request as is in our persistent store using a relevant subscription model ( this model also needs to be revised )
We also maintain a distributed datastructure in Hazelcast to keep track of the request and response to the client.
SubscriptionId dmi-plugin affectedCmHandles status A-10 dmi-1 ["ch-1","ch-2"] PENDING A-10 dmi-2 ["ch-3"] PENDING - Read the requests from the distributed data structure and form the request for the DMI-Plugin and publish it to the internal kafka channel.
- If the DMI-Plugin responds back within the configured time , so the request will either be ACCEPTED or REJECTED.
- if whole subscription request is ACCEPTED , update the distributed data structure and then store the subscription to the ActiveSubscriptions in cps-cache.
- if whole subscription request is REJECTED , update the distributed data structure and send back the response to the client.
if the DMI Plugin fails to respond then the status would remain PENDING only as there is no change.
SubscriptionId dmi-plugin affectedCmHandles status A-10 dmi-1 ["ch-1","ch-2"] ACCEPTED A-10 dmi-2 ["ch-3"] REJECTED
We store the "ActiveSubscriptions" using a new model which will look like below.
SubscriptionId cmHandle filter datastore A-10 ch-1 p1/c1 ncmp-datastore:passthrough-operational A-10 ch-1 p2/c2 ncmp-datastore:passthrough-operational A-10 ch-2 p1/c1 ncmp-datastore:passthrough-operational A-10 ch-2 p2/c2 ncmp-datastore:passthrough-operational Now based on whatever we have in the distributed map , we will send the response to the clients and may be clear the in-memory structure if required.
Merging
We get a new subscription create request.
Code Block { "subscriptionId" : "B-52", "predicates" : [ { "targets": [ch-2], "datastore" : "ncmp-datastore:passthrough-operational", "datastore-xpath-filter" : ["p2/c2" , "p3/c3"] } ] }
We now look at the "ActiveSubscriptions" if we already have an active subscription either for a combination of "cmhandle and xpath filter and the datastore".
Current view of ActiveSubscriptionsSubscriptionId cmHandle filter datastore A-10 ch-1 p1/c1 ncmp-datastore:passthrough-operational A-10 ch-1 p2/c2 ncmp-datastore:passthrough-operational A-10 ch-2 p1/c1 ncmp-datastore:passthrough-operational A-10 ch-2 p2/c2 ncmp-datastore:passthrough-operational
Check if we have ch-2 , p2/c2 and ncmp-datastore:passthrough-operational then directly add to the ActiveSubscription.
Similarly check for ch-3 , p3/c3 and ncmp-datastore:passthrough-operational ( which in this case will not be directly to the active subscription)Now the state of ActiveSubscription is :
SubscriptionId cmHandle filter datastore A-10 ch-1 p1/c1 ncmp-datastore:passthrough-operational A-10 ch-1 p2/c2 ncmp-datastore:passthrough-operational A-10 ch-2 p1/c1 ncmp-datastore:passthrough-operational A-10 ch-2 p2/c2 ncmp-datastore:passthrough-operational B-52 ch-2 p2/c2 ncmp-datastore:passthrough-operational and state of the in-memory datastructure is as follows.
SubscriptionId dmi-plugin affectedCmHandles status B-52 dmi-1 ["ch-2"] PENDING
Proposed JIRAs
Priority | Component | Description | JIRA | Estimates |
---|---|---|---|---|
1 | ||||
2 | ||||
3 | ||||
4 | ||||
5 | ||||
6 | ||||
7 | ||||
8 |
...