Table of Contents |
---|
References
Jira server ONAP Jira serverId 425b2b0a-557c-3c0c-b515-579789cceedb key CPS-2146
Purpose
The long-term goal of this Study & Implementation Proposal is gain precise control over memory consumption in CPS and NCMP, going from linear O(N) to constant O(1) space complexity.
The immediate objective is to fix Out Of Memory Errors encountered in NCMP while performing CM handles searches.
Summary of Problem
CPS and NCMP receive queries (CPS path queries and NCMP CM handle queries respectively) for which it will not be known how much data will be returned. Since internal and external APIs return Collections, these Collections containing arbitrary amounts of data are being held in memory. Additionally, these Collections undergo many transformations. Each collection cannot be garbage collected until fully processed/transformed.
...
draw.io Diagram | ||||||||||||||||||||
---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
|
Proposed Solution
It is proposed to create an end-to-end streaming solution, from Persistence layer to Controller. A Proof of Concept will be constructed to document challenges and investigate performance characteristics.
An important observation as to why this solution will achieve O(1) constant space complexity is that the streams will not be terminated until they leave the Rest controller.
Streaming all the way
This will require adding Stream versions of CPS Core read operations, e.g.
...
Code Block | ||
---|---|---|
| ||
// In CPS core Stream<DataNode> queryDataNodesAsStream(String dataspaceName, String anchorName, String cpsPath, FetchDescendantsOption fetchDescendantsOption) { return fragmentRepository.streamByAnchorAndCpsPath(getAnchor(dataspaceName, anchorName), cpsPath) .map(fragment -> fetchDescendants(fragment, fetchDescendantsOption)) .map(fragment -> convertToDataNode(fragment)); } // In NCMP private YangModelCmHandle getAnyReadyCmHandleByModuleSetTag(final String moduleSetTag) { return cmHandleQueries.queryNcmpRegistryByCpsPath("/dmi-registry/cm-handles[@module-set-tag='" + moduleSetTag + "']", DIRECT_CHILDREN_ONLY) .map(YangDataConverter::convertCmHandleToYangModel) .filter(cmHandle -> cmHandle.getCompositeState().getCmHandleState() == CmHandleState.READY) .findFirst() .orElse(null); } |
CPS and NCMP Rest APIs
Instead of returning Collections from Rest APIs, a Stream may be returned, reducing memory pressure on the server.
...
Code Block | ||
---|---|---|
| ||
@Override public Stream<String> searchCmHandleIds(final CmHandleQueryParameters cmHandleQueryParameters) { final CmHandleQueryServiceParameters cmHandleQueryServiceParameters = ncmpRestInputMapper.toCmHandleQueryServiceParameters(cmHandleQueryParameters); return networkCmProxyDataService.executeCmHandleIdSearchForInventory(cmHandleQueryServiceParameters); } |
Additional details of current memory consumption - data conversions
The read APIs in CPS Core (cps-service and cps-ri) return Collection<DataNode>:
...