Versions Compared

Key

  • This line was added.
  • This line was removed.
  • Formatting was changed.

...

Excerpt


#ParameterExpectationNotes
1Request Frequency
Upgrade to NEW moduleSetTags
3 upgrades  / second → 180 upgrades / minute

The assumption is there are 100 modules in each moduleSet and a moduleSet is available from a dmi plugin within 0.5 seconds. 
Also assume there is a 90% overlap in modules across all CM Handles.
Across 20,000 Nodes there are 10 different ModuleSetTags

2Request Frequency
Upgrade to already existing moduleSetTags
6 upgrades  / second → 360 upgrades / minute
3Test Environment


Expand
  1. CPS and NCMP

requests:
    cpu: 2000m
    memory: 2Gi
limits:
    memory: 3Gi
    cpu: 3000m

2. Postgres

requests:
    cpu: 4000m
    memory: 1Gi
 limits:
    memory: 3Gi
    cpu: 6000m



4Concurrent request12 clients requests toward 1 NCMP simultaneously
5Number of CM Handles in one request

Performance is posted here :  Performance test for updating YANG schema set API

Out-of-scope

  • Upgrade of models for cached data. "ncmp-datastore:operational" is out-of-scope.  

...

  1. Need to use a new (shared) Hazelcast map with ModuleSetTags as key (value list of module refs) that have been processed (but not saved yet) to be used both inside a batch and different instance to prevent unnecessary trips to DMI/Node
  2. Watchdog needs to use Lock state AND lock reason to determine what node need to be upgraded
  3. Initial inventory and upgrade is not likely to happen at the same time but watchdog can handle both, of course performance would be affected if that does occur
  4. CACHE(s) needs to be cleared or updates as algorithm wil re-use existing schema set name
  5. Create new upgrade specific methods and reuse/slightly duplicate initial inventory methods
  6. Legacy (and new) checks for lock need to check the lock-reason too now! To differentiate between failed initial inventory and upgrade
  7. Probably need more specific (new) failure reasons to differentiate between initial inventory and upgrade failures
  8. as usual: small commits, early reviews to introduce all this functionality are advised, posisbel steps
    1. set lock state and reason upon request
    2. watch dog just list to be upgrade node (and does not mix them up with failed initial inventory)
    3. perform first upgrade for a new ModuleSetTag
    4. introduce and use new Hazelcast map
    5. perform upgrade of an node with a modueleSetTag that is already in cache
    6. perform upgrade of an node with a modueleSetTag that is already in DB (ie introduce DB query)
    7. handle failure of upgrade (re-use same retry mechanism as initial inventory but with different lock reason!)
    8. etc.

Proposed JIRAs

Use-Case Overview (Sync in watchDog)



OperationTag
Provided
Tag
Cached
Tag
In DB
(other cm handle)
Steps
1CreateNoN/AN/A
  1. get modules (delta) from Node (DMI)
  2. create schema set 
  3. create anchor
2CreateYesNoNo
  1. get modules (delta) from Node (DMI)
  2. create schema set
  3. create anchor
3CreateYesNoYes
  1. get modules from DB (other cm handle) No delta
  2. create schema set (blank map for new resources)
  3. create anchor
4CreateYesYesN/A
  1. get modules from Cache 
  2. create schema set ( blank map)
  3. create anchor
  4. cache tag details
5UpgradeNoN/AN/A
  1. get modules from Node (DMI)
  2. update schema set 
6UpgradeYesNoNo
  1. get modules from Node (DMI)
  2. update schema set 
7UpgradeYesNoYes
  1. get modules from DB (other cm handle)
  2. update schema set (blank map)
8UpgradeYesYesN/A
  1. get modules from Cache 
  2. update schema set (blank map)
  3. cache tag details

Note. Error handling like invalid IDs are handled during the Synchronous part of registration and not part of this use-case overview. 
Upgrade to the same Tag should be captured in synchronous pre-processing.

re-usable methods

  1. Gget modules (delta) from Node (DMI)
  2. Create schemaset. Combination of exiting Refs and new yang resources
    New yang resources empty for known module sets (known tag)
  3. Create anchor
  4. Update schemaset. Combination of exiting Refs and new yang resources
    New yang resources empty for known module sets (known tag)

Changes agreed between Daniel Hanrahan , Sourabh Sourabh and Toine Siebelink 


ChangeNotes
1No need for ModuleSetTagCachereduce complexity, no significant impact on performance
Can be re-introduced in a better way later if needed but not expected
2Use same method for New and Known Schema Sets (module set tags)
  • NEW SchemaSet → 0 or More NEW YangResources, 1 OR more ModuleReferences
  • KNOWN SchemaSet → 0 NEW YangResources, 1 OR more ModuleReferences


Code Block
languagejava
titlePseudo Code
linenumberstrue
void processCreate() {
   if (tagProvided && tag in DB) {
          allModuleRefences = referencesFromDb
          newYangResources = []
       else {
          delta = getModulesDelta()
          newYangResources = delta.newYangResources 
          allModuleRefences = delta.allModuleRefences 
    }
    creatSchemaSet(newYangResources,  allModuleRefences)
    createAnchor()
}       
     
void processUpgrade() {
    if (tagProvided && tag in DB) {
          allModuleRefences = referencesFromDb
          newYangResources = []
       else {
          delta = getModulesDelta()
          newYangResources = delta.newYangResources 
          allModuleRefences = delta.allModuleRefences 
    }
    udpateSchemaSet(newYangResources,  allModuleRefences)
}





Proposed JIRAs


Component DescriptionJIRAEstimates
1CPS CoreExpose a java interface to update schema set

Jira
serverONAP Jira
columnIdsissuekey,summary,issuetype,created,

Component DescriptionJIRAEstimates1CPS CoreExpose a java interface to update schema set

Jira
serverONAP Jira
columnIdsissuekey,summary,issuetype,created,updated,duedate,assignee,reporter,priority,status,resolution
columnskey,summary,type,created,updated,due,assignee,reporter,priority,status,resolution
serverId425b2b0a-557c-3c0c-b515-579789cceedb
keyCPS-1800

5 Days
2NCMPUpdate existing REST endpoint add operation to upgrade YANG schema set using moduleSetTag

Jira
serverONAP Jira
columnIdsissuekey,summary,issuetype,created,updated,duedate,assignee,reporter,priority,status,resolution
columnskey,summary,type,created,updated,due,assignee,reporter,priority,status,resolution
serverId425b2b0a-557c-3c0c-b515-579789cceedb
keyCPS-1798

15 Days
3NCMPUpgrade inventory YANG model to store moduleSetTag

Jira
serverONAP Jira
columnIdsissuekey,summary,issuetype,created,updated,duedate,assignee,reporter,priority,status,resolution
columnskey,summary,type,created,updated,due,assignee,reporter,priority,status,resolution
serverId425b2b0a-557c-3c0c-b515-579789cceedb
keyCPS-1804

5 Days
4NCMPUse moduleSetTagCache to be cleared or updates as algorithm wil re-use existing schema set namealgorithm wil re-use existing schema set name

Jira
serverONAP Jira
serverId425b2b0a-557c-3c0c-b515-579789cceedb
keyCPS-1821

5 Days
5NCMPUpdate cmhandle state (lock) during YANG model upgrade

Jira
serverONAP Jira
columnIdsissuekey,summary,issuetype,created,updated,duedate,assignee,reporter,priority,status,resolution
columnskey,summary,type,created,updated,due,assignee,reporter,priority,status,resolution
Jira
serverONAP Jira
serverId425b2b0a-557c-3c0c-b515-579789cceedb
keyCPS-18211801

5 Days
56NCMPUpdate cmhandle state (lock) during YANG model upgradeModify existing inventory operations using moduleSetTag as optional attribute

Jira
serverONAP Jira
columnIdsissuekey,summary,issuetype,created,updated,duedate,assignee,reporter,priority,status,resolution
columnskey,summary,type,created,updated,due,assignee,reporter,priority,status,resolution
serverId425b2b0a-557c-3c0c-b515-579789cceedb
keyCPS-18011803

5 15 Days
67NCMPModify existing inventory operations using moduleSetTag as optional attributeIdentify and test error scenarios 

Jira
serverONAP Jira
columnIdsissuekey,summary,issuetype,created,updated,duedate,assignee,reporter,priority,status,resolution
columnskey,summary,type,created,updated,due,assignee,reporter,priority,status,resolution
serverId425b2b0a-557c-3c0c-b515-579789cceedb
keyCPS-18031802

15 5 Days
78NCMPIdentify and test error scenarios CSIT test and demo 

Jira
serverONAP Jira
columnIdsissuekey,summary,issuetype,created,updated,duedate,assignee,reporter,priority,status,resolution
columnskey,summary,type,created,updated,due,assignee,reporter,priority,status,resolution
serverId425b2b0a-557c-3c0c-b515-579789cceedb
keyCPS-18021806

5 Days
89NCMPCSIT test and demo Test and document performance of updating YANG schema set API.

Jira
serverONAP Jira
columnIdsissuekey,summary,issuetype,created,updated,duedate,assignee,reporter,priority,status,resolution
columnskey,summary,type,created,updated,due,assignee,reporter,priority,status,resolutionpriority,status,resolution
serverId425b2b0a-557c-3c0c-b515-579789cceedb
keyCPS-1805

10 Days

10NCMPHandle yangTextSchemaSourceSetCache for module set tag

Jira
serverONAP Jira
serverId425b2b0a-557c-3c0c-b515-579789cceedb
keyCPS-18061995

5 Days

911NCMPTest and document performance of updating YANG schema set API.Schema object cache not distributed
Jira
serverONAP Jira
columnIdsissuekey,summary,issuetype,created,updated,duedate,assignee,reporter,priority,status,resolution
columnskey,summary,type,created,updated,due,assignee,reporter,priority,status,resolution
serverId425b2b0a-557c-3c0c-b515-579789cceedb
keyCPS-
1805

10 Days

10NCMPHandle yangTextSchemaSourceSetCache for module set tag

Jira
serverONAP Jira
serverId425b2b0a-557c-3c0c-b515-579789cceedb
keyCPS-1995

5 Days

2000



12NCMPImprove unit test and dmi plugin csit stub11NCMPSchema object cache not distributed

Jira
serverONAP Jira
columnIdsissuekey,summary,issuetype,created,updated,duedate,assignee,reporter,priority,status,resolution
columnskey,summary,type,created,updated,due,assignee,reporter,priority,status,resolution
serverId425b2b0a-557c-3c0c-b515-579789cceedb
keyCPS-

1995

2006


1213NCMP/DMIAdd moduleSetTag to the request towards dmi plugin if moduleSetTag is set for the cmHandleImprove unit test and dmi plugin csit stub

Jira
serverONAP Jira
columnIdsissuekey,summary,issuetype,created,updated,duedate,assignee,reporter,priority,status,resolution
columnskey,summary,type,created,updated,due,assignee,reporter,priority,status,resolution
serverId425b2b0a-557c-3c0c-b515-579789cceedb
keyCPS-20062030


Planning:

  • Allow for 2 more user stories each may take 1 week. 
  • Estimated date of completion is   based on 1 person working. There is a possibility that 1 more person will join in between for implementation but there is no clear visibility on when the person can start contributing as of now. 
  • These are estimates and should not be interpreted as commitments. 

...