Page History

...

The long-term goal of this Study & Implementation Proposal is reduce overall gain precise control over memory consumption in CPS and NCMP by over 99% - actually , going from linear O(N) to constant O(1) space complexity.

The immediate objective is to fix Out Of Memory Errors encountered in NCMP while performing CM handles searches.

...

CPS and NCMP receive queries (CPS path queries and NCMP CM handle queries respectively) for which it will not be known how much data will be returned. Since the internal and external APIs return Collections, those these Collections containing arbitrary amounts of data are being held in memory. Additionally, these Collections undergo many transformations. Each collection cannot be garbage collected until fully processed/transformed.

In many cases, such as CM handle search, NCMP may be viewed as simply transforming lists of data. Here is an example flow (there are many more such flows):in NCMP showing getting all CM handle IDs, GET http://ncmp-1/ncmpInventory/v1/ch/cmHandles?dmi-plugin-identifier=http://dmi-1

...

This Stream will implement pagination when fetching data from the FragmentRepository. (This may be implemented in a variety of ways, using Spring JpaRepository Pageable interface, or alternately Spring Data supports streaming from a repository directly, but this needs to be investigated for suitability.) The below example shows the Stream<FragmentEntity> using pagination internally to control memory usage:

Here is a proposed end-to-end flow - a simplified example to show how greatly this can reduce memory usagesome source code showing how the streams API would be used:

// In CPS core
Stream<DataNode> queryDataNodesAsStream(String dataspaceName, String anchorName, String cpsPath, FetchDescendantsOption fetchDescendantsOption) {
	return fragmentRepository.streamByAnchorAndCpsPath(getAnchor(dataspaceName, anchorName), cpsPath)
			.map(fragment -> fetchDescendants(fragment, fetchDescendantsOption))
			.map(fragment -> convertToDataNode(fragment));
}

// In NCMP
private YangModelCmHandle getAnyReadyCmHandleByModuleSetTag(final String moduleSetTag) {
    return cmHandleQueries.queryNcmpRegistryByCpsPath("/dmi-registry/cm-handles[@module-set-tag='" + moduleSetTag + "']", DIRECT_CHILDREN_ONLY)
			.map(YangDataConverter::convertCmHandleToYangModel)
            .filter(cmHandle -> cmHandle.getCompositeState().getCmHandleState() == CmHandleState.READY)
            .findFirst()
            .orElse(null);
}

...

    @Override
    public Stream<String> searchCmHandleIds(final CmHandleQueryParameters cmHandleQueryParameters) {
        final CmHandleQueryServiceParameters cmHandleQueryServiceParameters = ncmpRestInputMapper.toCmHandleQueryServiceParameters(cmHandleQueryParameters);
        return networkCmProxyDataService.executeCmHandleIdSearchForInventory(cmHandleQueryServiceParameters);
    }

...

Additional details of current memory consumption -

...

data conversions

The read APIs in CPS Core (cps-service and cps-ri) return Collection<DataNode>:

...