CPS-2146: Analysis of Out of Memory and related Errors in NCMP

References

Assumptions

Issues & Decisions

Long term solutions

<Note. use green for closed issues, yellow for important ones if needed>

Decisions for Short Term

Background

CPS and NCMP have much higher memory consumption than required. Regarding NCMP specifically, it has some in-memory data structures that grow linearly with the number of CM-handles.

Regarding CPS-core, there is a more fundamental problem in that CPS path queries could return any amount of data - it will be unknown to the application until a query is executed. Some solutions will be proposed for CPS path queries to reduce memory use.

This study and implementation proposals will target concrete steps to reduce memory consumption.

Analysis

A number of issues leading to high memory usage have been identified.

NCMP CM Handle Queries

NCMP CM Handle Queries are directly implicated in CPS-2146, as the Out Of Memory errors occurs during NCMP Search and ID Search functions.

See CPS-2146 Using Java Streams to reduce memory consumption in CPS and NCMP for analysis & solution to reduce memory consumption during these operations.

Recent performance improvements

One avenue worth further investigation is a series of recent performance improvements to CPS and NCMP introduced around 3.4.2:

Cumulatively, both read and write speeds are up to 10x faster than previous versions, and overhead on Rest requests is over 10x lower. It is very possible that these improvements are adversely affecting memory usage during load tests.

Postgresql configuration does not have appropriate values for memory allocation

CPS-2156 - Getting issue details... STATUS

As I checked the root cause of the PSQL error is most probably the wrong configuration of shared_buffers (https://www.postgresql.org/docs/13/runtime-config-resource.html#GUC-SHARED-BUFFERS ) according to the documentation the optimal value for this configuration should be between 25% and 40% of the memory usage of the DB (request is 1GB, limit is 3GB), and the default value 128 MB.

Due to the system load increasing from recent performance improvements, the database is also under increased load. The DB configuration will need to updated to reflect this (note: the current and previous DB configuration was incorrect).

Hazelcast

The use of Hazelcast (an In-Memory Data Grid) has been identified as a particular source of high memory usage. Some points of interest:

• In NCMP, Hazelcast is not used as a cache, so idle eviction is not used, and the structures are configured to have 3 backups. It follows that scaling up the deployment (e.g. Kubernetes auto-scaling) would not help in a low-memory situation, as the new instances would have also be storing the whole structure.
• Given Hazelcast is configured for synchronous operation, it is likely to have worse performance than a database solution.
• There are additional reasons to avoid Hazelcast, since as a distributed asynchronous system, it cannot give strong consistency guarantees like an ACID database - it is prone to split brain among other issues.
• I advise against the use of Hazelcast for future development in NCMP - CPS API should be used.

The following is an overview of Hazelcast structures in CPS and NCMP, along with recommendations.

Use of Postgres Arrays in Repository methods

Use of Postgres arrays in JpaRepository methods may be using too much memory. Though it is currently unclear how much of a contributor this is to Out Of Memory errors, it appears in the logs from CPS-2146.

See CPS-1574: Remove 32K limit from DB operations for history of this implementation choice - an alternate solution using batching was proposed.

For example, from the logs of CPS-2146, see this stack trace:

2024-02-28T05:18:25.049Z@eric-oss-ncmp-04@ncmp@Connection leak detection triggered for org.postgresql.jdbc.PgConnection@b358fc9 on thread qtp1699794502-7604, stack trace follows, logger: com.zaxxer.hikari.pool.ProxyLeakTask, thread_name: CpsDatabasePool housekeeper, stack_trace: java.lang.Exception: Apparent connection leak detected
 org.onap.cps.spi.repository.YangResourceRepository.findAllModuleReferencesByDataspaceAndModuleNames(YangResourceRepository.java:111)
 org.onap.cps.spi.impl.CpsAdminPersistenceServiceImpl.validateDataspaceAndModuleNames(CpsAdminPersistenceServiceImpl.java:206)
 org.onap.cps.spi.impl.CpsAdminPersistenceServiceImpl.queryAnchors(CpsAdminPersistenceServiceImpl.java:143)
 org.onap.cps.api.impl.CpsAnchorServiceImpl.queryAnchorNames(CpsAnchorServiceImpl.java:90)
 org.onap.cps.ncmp.api.impl.inventory.InventoryPersistenceImpl.getCmHandleIdsWithGivenModules(InventoryPersistenceImpl.java:174)
 org.onap.cps.ncmp.api.impl.NetworkCmProxyCmHandleQueryServiceImpl.executeModuleNameQuery(NetworkCmProxyCmHandleQueryServiceImpl.java:167)
 org.onap.cps.ncmp.api.impl.NetworkCmProxyCmHandleQueryServiceImpl.executeQueries(NetworkCmProxyCmHandleQueryServiceImpl.java:256)
 org.onap.cps.ncmp.api.impl.NetworkCmProxyCmHandleQueryServiceImpl.queryCmHandleIds(NetworkCmProxyCmHandleQueryServiceImpl.java:71)
 org.onap.cps.ncmp.api.impl.NetworkCmProxyCmHandleQueryServiceImpl.queryCmHandles(NetworkCmProxyCmHandleQueryServiceImpl.java:95)
 org.onap.cps.ncmp.api.impl.NetworkCmProxyDataServiceImpl.executeCmHandleSearch(NetworkCmProxyDataServiceImpl.java:215)
 org.onap.cps.ncmp.rest.controller.NetworkCmProxyController.searchCmHandles(NetworkCmProxyController.java:253)

Note: the message about Connection leak in postgres may also indicate a memory issue in the DB.

The code causing the exception in YangResourceRepository is:

    default Set<YangResourceModuleReference> findAllModuleReferencesByDataspaceAndModuleNames(
        final String dataspaceName, final Collection<String> moduleNames) {
        return findAllModuleReferencesByDataspaceAndModuleNames(dataspaceName, moduleNames.toArray(new String[0]));
    }

OpenDaylight Yang Parser & YangTextSchemaSourceSetCache

It was previously suspected that the 3rd party OpenDaylight Yang Parser may have a memory leak. See comments on CPS-2000:

CPS-2000 - Getting issue details... STATUS

While CPS has a cache called YangTextSchemaSourceSetCache to avoid invoking the OpenDaylight Yang Parser, the Yang Parser's internal cache may be causing a memory leak. This requires immediate investigation.

Hibernate Entity Cache

Hibernate has an Entity Cache, which can grow large during transactions. While most of CPS-core's Spring JpaRepository methods are using Native SQL, the Entity Manager is still caching in some cases. I propose the removal of Hibernate be investigated as part of a long term solution. (This is not as much work as it sounds: CPS is not directly reliant on Hibernate/JPA - rather Spring Data JPA is used. This could be replaced with Spring Data JDBC with relatively small code changes.)

Note this change is blocked by CPS-1673. The use of OneToMany mapping in FragmentEntity appears to be only place where CPS is currently reliant on functionality provided by JPA.
CPS-1673 - Getting issue details... STATUS

Memory usage during UPDATE operations

CPS current implementation of update operations involves reading the existing data from DB, applying changes, and storing again. As update operations involve reading data, the current implementations of update operations should be reviewed to ensure they are internally breaking large requests into batches to restrict memory use.

Memory usage of Yang Repository SQL

The SQL statement to get Yang Resources from Module References (org.onap.cps.spi.repository.YangResourceNativeRepositoryImpl#getResourceIdsByModuleReferences) is dynamically generated by UNIONing many SELECT statements together into one large statement, e.g.

SELECT id FROM yang_resource WHERE module_name='ietf-netconf' and revision='2011-06-01' UNION ALL 
SELECT id FROM yang_resource WHERE module_name='ietf-inet-types' and revision='2019-11-04' UNION ALL 
SELECT id FROM yang_resource WHERE module_name='ietf-netconf-acm' and revision='2018-02-14' UNION ALL 
SELECT id FROM yang_resource WHERE module_name='ietf-yang-types' and revision='2019-11-04' UNION ALL 
...

In the logs during an Out Of Memory situation, it was observed that up to 200 such statements will be chained into a single statement, with such statements being executed per CM-handle (e.g. 20,000 times) during Module Sync.