Page History

...

Regarding CPS-core, there is a more fundamental problem in that CPS path queries could return any amount of data - it will be unknown to the application until a query is executed (even if pagination on DataNodes were used, since CPS DataNodes represent tree-structures, it is not known how deep the tree is). Some solutions will be proposed for CPS path queries to reduce memory use.

This study and implementation proposal proposals will target concrete steps to reduce memory consumption.

...

• In NCMP, Hazelcast is not used as a cache, so idle eviction is not used, and the structures are configured to have 3 backups. It follows that scaling up the deployment (e.g. Kubernetes auto-scaling) would not help in a low-memory situation, as the new instances would have also be storing the whole structure.
• Given Hazelcast is configured for synchronous operation, it is likely to have worse performance than a database solution.
• There are additional reasons to avoid Hazelcast, since as a distributed asynchronous system, it cannot give strong consistency guarantees like an ACID database - it is prone to split brain among other issues.
• I strongly advise against the use of Hazelcast for future development .

Side note - this was seen in logs of CPS-2146:

...

• in NCMP - CPS API should be used.

The following is an overview of Hazelcast structures in CPS and NCMP, along with recommendations.

...

Use of Postgres arrays in JpaRepository methods may be using too much memory. Though it is currently unclear how much of a contributor this is to Out Of Memory errors, it appears in the logs from CPS-2146.

See CPS-1574: Remove 32K limit from DB operations for history of this implementation choice - an alternate solution using batching was proposed.

For example, from the logs of CPS-2146, see this partial stack trace:

2024-02-28T05:18:25.049Z@eric-oss-ncmp-04@ncmp@Connection leak detection triggered for org.postgresql.jdbc.PgConnection@b358fc9 on thread qtp1699794502-7604, stack trace follows, logger: com.zaxxer.hikari.pool.ProxyLeakTask, thread_name: CpsDatabasePool housekeeper, stack_trace: java.lang.Exception: Apparent connection leak detected
 org.onap.cps.spi.repository.YangResourceRepository.findAllModuleReferencesByDataspaceAndModuleNames(YangResourceRepository.java:111)
 org.onap.cps.spi.impl.CpsAdminPersistenceServiceImpl.validateDataspaceAndModuleNames(CpsAdminPersistenceServiceImpl.java:206)
 org.onap.cps.spi.impl.CpsAdminPersistenceServiceImpl.queryAnchors(CpsAdminPersistenceServiceImpl.java:143)
 org.onap.cps.api.impl.CpsAnchorServiceImpl.queryAnchorNames(CpsAnchorServiceImpl.java:90)
 org.onap.cps.ncmp.api.impl.inventory.InventoryPersistenceImpl.getCmHandleIdsWithGivenModules(InventoryPersistenceImpl.java:174)
 org.onap.cps.ncmp.api.impl.NetworkCmProxyCmHandleQueryServiceImpl.executeModuleNameQuery(NetworkCmProxyCmHandleQueryServiceImpl.java:167)
 org.onap.cps.ncmp.api.impl.NetworkCmProxyCmHandleQueryServiceImpl.executeQueries(NetworkCmProxyCmHandleQueryServiceImpl.java:256)
 org.onap.cps.ncmp.api.impl.NetworkCmProxyCmHandleQueryServiceImpl.queryCmHandleIds(NetworkCmProxyCmHandleQueryServiceImpl.java:71)
 org.onap.cps.ncmp.api.impl.NetworkCmProxyCmHandleQueryServiceImpl.queryCmHandles(NetworkCmProxyCmHandleQueryServiceImpl.java:95)
 org.onap.cps.ncmp.api.impl.NetworkCmProxyDataServiceImpl.executeCmHandleSearch(NetworkCmProxyDataServiceImpl.java:215)
 org.onap.cps.ncmp.rest.controller.NetworkCmProxyController.searchCmHandles(NetworkCmProxyController.java:253)

...

Hibernate Entity Cache

Hibernate has an entity cacheEntity Cache, which can grow large during transactions. While most of CPS-core's Spring JpaRepository methods are using Native SQL, the Entity Manager is still caching in some cases. I proposed that propose the removal of Hibernate be investigated as part of a long term solution. (Note This is not as much work as it sounds: CPS is not directly reliant on Hibernate/JPA - rather Spring Data JPA is used. This could be replaced with Spring Data JDBC with relatively small code changes.)

Note this change is blocked by:by CPS-1673. The use of OneToMany mapping in FragmentEntity appears to be only place where CPS is currently reliant on functionality provided by JPA.