References
CPS-2146 - Getting issue details... STATUS
Assumptions
| # | Assumption | Notes |
|---|---|---|
| 1 | Proposed solution is not a quick fix, but allows for future scaling of NCMP. |
Issues & Decisions
| # | Issue | Notes | Decision |
|---|---|---|---|
| 1 | This is an open issue | ||
| 2 | Do we need a analysis template? | is convention for (new) developers to guide them | Luke Gleeson and Toine Siebelink agreed we do to have consistent study pages |
| 3 | This is a very important (blocking issue) |
<Note. use green for closed issues, yellow for important ones if needed>
Background
CPS and NCMP have much higher memory consumption than required. Regarding NCMP specifically, it has some in-memory data structures that grow linearly with the number of CM-handles.
Regarding CPS-core, there is a more fundamental problem in that CPS path queries could return any amount of data - it will be unknown to the application until a query is executed (even if pagination on DataNodes were used, since CPS DataNodes represent tree-structures, it is not known how deep the tree is). Further analysis is needed here, however simple mitigations will be recommended (e.g. extending public APIs to allow limiting maximum results).
This study and implementation proposal will target simple and concrete steps to reduce memory consumption.
Analysis
A number of issues leading to high memory usage have been identified.
Hazelcast
The use of Hazelcast (an In-Memory Data Grid) has been identified as a particular source of high memory usage. Some points of interest:
- In NCMP, Hazelcast is not used as a cache, so idle eviction is not used, and the structures are configured to have 3 backups. It follows that scaling up the deployment (e.g. Kubernetes auto-scaling) would not help in a low-memory situation, as the new instances would have also be storing the whole structure.
- Given Hazelcast is configured for synchronous operation, it is likely to have worse performance than a database solution.
- There are additional reasons to avoid Hazelcast, since as a distributed asynchronous system, it cannot give strong consistency guarantees like an ACID database - it is prone to split horizon among other issues.
- I strongly advise against the use of Hazelcast for future development.
The following is an overview of Hazelcast structures in CPS and NCMP, along with recommendations.
| Component | Hazelcast Structure | Type | Purpose | Recommendation | Implementation Proposal | Notes |
|---|---|---|---|---|---|---|
| CPS | anchorDataCache | Map<String, AnchorDataCacheEntry> | Likely low risk, but needs further analysis | |||
| NCMP | moduleSyncWorkQueue | BlockingQueue<DataNode> | Remove immediately | TBC | Entire CM handles are stored in work queue for module sync. This creates very high memory usage during CM handle registration. | |
| NCMP | moduleSyncStartedOnCmHandles | Map<String, Object> | Remove immediately | TBC | ||
| NCMP | dataSyncSemaphores | Map<String, Boolean> | Remove later | Low priority - this map is only populated if data sync is enabled for a CM handle. If the feature is used, it will store one entry per CM handle with data sync enabled. | ||
| NCMP | trustLevelPerCmHandle | Map<String, TrustLevel> | Remove immediately | TBC | One entry is stored in memory per CM handle. This is directly implicated in logs supplied in investigation of out-of-memory errors in CPS-2146 | |
| NCMP | trustLevelPerDmiPlugin | Map<String, TrustLevel> | Low risk, See notes | Low priority - there are only small number of DMIs, so this structure will not grow so large. However, if trustLevelPerCmHandle is being removed, this structure may be removed as part of the same solution. | ||
| NCMP | cmNotificationSubscriptionCache | Map<String, Map<String, DmiCmNotificationSubscriptionDetails>> | Will need further analysis in future; see notes | This is low priority, as the CM subscription feature is not yet implemented, thus it is not in use. It is unclear how much data will be stored in the structure. It is presumed to be low, as this structure will only hold pending subscriptions. |
CPS Path Queries
The current public Java API for CPS (cps-service) returns Collection<DataNode> for read operations. At a minimum, it is advised to allow limiting max results for these operations. This would be useful in NCMP when fetching CM handles by state during module sync, so a single batch can be fetched. For example, the current CpsQueryService::queryDataNodes
Collection<DataNode> queryDataNodes(String dataspaceName, String anchorName, String cpsPath, FetchDescendantsOption fetchDescendantsOption);
An additional API can be defined limiting maxResults:
Collection<DataNode> queryDataNodes(String dataspaceName, String anchorName, String cpsPath, FetchDescendantsOption fetchDescendantsOption, int maxResults);
Depending on use case, pagination may be required. A more sophisticated solution returning Stream<DataNode> would be ideal, a custom Stream implementation would allow more control over memory use.
ModuleSetTag lookup
As ModuleSetTag is a relatively new feature, and performance testing of that feature is not yet complete (in progress, see CPS-1805), there is some risk of this causing high memory consumption.
Particularly the method ModuleSyncService::getAnyReadyCmHandleByModuleSetTag use a Cps Path Query that will return all CM handles with a given moduleSetTag. An alternate solution is recommended.