Page History

...

NCMP also contains many queries where only partial results are needed, making a Streams API ideal.

Additionally, all Rest APIs returning query results return Lists. Spring framework allows returning Streams, eliminating memory overhead.

Proposed Solution

It is proposed to create an end-to-end streaming solution, from Repository (database) to Rest interface. It is speculated to achieve more than 99% reduction in memory usage. A POC will be constructed to investigate.

This will require adding Stream versions A Streams API version of CPS Core read operations:, e.g.

Stream<DataNode> queryDataNodesAsStream(String dataspaceName, String anchorName, String cpsPath, FetchDescendantsOption fetchDescendantsOption);

...

Instead of returning Collections from Rest APIs, a Stream may be returned, reducing memory pressure on the server.

Current code looks like this:

    @Override
    public ResponseEntity<List<String>> searchCmHandleIds(final CmHandleQueryParameters cmHandleQueryParameters) {
        final CmHandleQueryServiceParameters cmHandleQueryServiceParameters = ncmpRestInputMapper.toCmHandleQueryServiceParameters(cmHandleQueryParameters);
        final Collection<String> cmHandleIds = networkCmProxyDataService.executeCmHandleIdSearchForInventory(cmHandleQueryServiceParameters);
        return ResponseEntity.ok(List.copyOf(cmHandleIds));
    }

It is proposed to stream out of the Rest API, something like this:

    @Override
    public Stream<String> searchCmHandleIds(final CmHandleQueryParameters cmHandleQueryParameters) {
        final CmHandleQueryServiceParameters cmHandleQueryServiceParameters = ncmpRestInputMapper.toCmHandleQueryServiceParameters(cmHandleQueryParameters);
        return networkCmProxyDataService.executeCmHandleIdSearchForInventory(cmHandleQueryServiceParameters);
    }