...
Looking more closely at the ArrayLists, we see one contains many thousands of Postgres Tuples, while the other contains CPS FragmentEntities:
This illustrates the core problem that large collections are stored in memory, and the full collections cannot be garbage collected until the collection is fully processed/transformed.
(Note in the above case, the system ran out of memory before the Tuples were fully converted to FragmentEntity, so peak memory requirement is larger than illustrated above, 50MB per 10K nodes - in reality, actual memory used will depend on the complexity of the data, e.g. number of public properties per CM-handle.), as well as how many search parameters are used (as each search parameter results in an additional DB query).
This illustrates the core problem that large collections are stored in memory, and the full collections cannot be garbage collected until the collection is fully processed/transformed.
Details of Test Setup
In a test deployment using a single instance of NCMP run using docker (with resources limited to 2 CPUs and 1GB memory), 20000 CM-handles were registered with some public properties (10K using different properties):
...