...
Jira server ONAP Jira serverId 425b2b0a-557c-3c0c-b515-579789cceedb key CPS-2146
...
Purpose
The read APIs in CPS Core (cps-service and cps-ri) return Collection<DataNode>:
Code Block | ||
---|---|---|
| ||
Collection<DataNode> queryDataNodes(String dataspaceName, String anchorName, String cpsPath, FetchDescendantsOption fetchDescendantsOption);
Collection<DataNode> getDataNodes(String dataspaceName, String anchorName, String xpath, FetchDescendantsOption fetchDescendantsOption);
Collection<DataNode> getDataNodesForMultipleXpaths(String dataspaceName, String anchorName, Collection<String> xpaths, FetchDescendantsOption fetchDescendantsOption); |
Additionally, internal APIs in CPS Reference Implementation (cps-ri) use List<FragmentEntity>, e.g.
Code Block | ||
---|---|---|
| ||
List<FragmentEntity> findByAnchorAndCpsPath(AnchorEntity anchorEntity, CpsPathQuery cpsPathQuery); |
When a CPS path query is run, this will result in a List<FragmentEntity> which needs to be converted to a Collection<DataNode>. Thus, the Fragment Entities cannot be garbage collected until the list is converted to Data Nodes. This doubles the memory usage.
Additionally, NCMP uses CPS path queries, e.g. to find CM handles in a given state. NCMP will then convert Collection<DataNode> to Collection<YangModelCmHandle>. Again, the Collection<DataNode> cannot be garbage collected until fully converted to YangModelCmHandles. This again results in doubling of memory usage.
Similar applies when converting to NcmpServiceCmHandle.
NCMP also contains many queries where only partial results are needed, making a Streams API ideal.
Additionally, all Rest APIs returning query results return Lists. Spring framework allows returning Streams, eliminating memory overhead.
long-term goal of this Study & Implementation Proposal is reduce overall memory consumption in CPS and NCMP by over 99% - actually from linear O(N) to constant O(1).
The immediate objective is to fix Out Of Memory Errors encountered in NCMP while performing CM handles searches.
Proposed Solution
It is proposed to create an end-to-end streaming solution, from Repository (database) to Rest interface. It is speculated to achieve more than 99% reduction in memory usage. A POC will be constructed to investigate.
...
Code Block | ||
---|---|---|
| ||
@Override public Stream<String> searchCmHandleIds(final CmHandleQueryParameters cmHandleQueryParameters) { final CmHandleQueryServiceParameters cmHandleQueryServiceParameters = ncmpRestInputMapper.toCmHandleQueryServiceParameters(cmHandleQueryParameters); return networkCmProxyDataService.executeCmHandleIdSearchForInventory(cmHandleQueryServiceParameters); } |
Details of current memory consumption
The read APIs in CPS Core (cps-service and cps-ri) return Collection<DataNode>:
Code Block | ||
---|---|---|
| ||
Collection<DataNode> queryDataNodes(String dataspaceName, String anchorName, String cpsPath, FetchDescendantsOption fetchDescendantsOption);
Collection<DataNode> getDataNodes(String dataspaceName, String anchorName, String xpath, FetchDescendantsOption fetchDescendantsOption);
Collection<DataNode> getDataNodesForMultipleXpaths(String dataspaceName, String anchorName, Collection<String> xpaths, FetchDescendantsOption fetchDescendantsOption); |
Additionally, internal APIs in CPS Reference Implementation (cps-ri) use List<FragmentEntity>, e.g.
Code Block | ||
---|---|---|
| ||
List<FragmentEntity> findByAnchorAndCpsPath(AnchorEntity anchorEntity, CpsPathQuery cpsPathQuery); |
When a CPS path query is run, this will result in a List<FragmentEntity> which needs to be converted to a Collection<DataNode>. Thus, the Fragment Entities cannot be garbage collected until the list is converted to Data Nodes. This doubles the memory usage.
Additionally, NCMP uses CPS path queries, e.g. to find CM handles in a given state. NCMP will then convert Collection<DataNode> to Collection<YangModelCmHandle>. Again, the Collection<DataNode> cannot be garbage collected until fully converted to YangModelCmHandles. This again results in doubling of memory usage.
Similar applies when converting to NcmpServiceCmHandle.
NCMP also contains many queries where only partial results are needed, making a Streams API ideal.
Additionally, all Rest APIs returning query results return Lists. Spring framework allows returning Streams, eliminating memory overhead.