Versions Compared

Old Version 54

changes.mady.by.user Daniel Hanrahan

Saved on

compared with

New Version 55

changes.mady.by.user Daniel Hanrahan

Saved on

Key

  • This line was added.
  • This line was removed.
  • Formatting was changed.

...

Csaba Kocsis ETH will test and report back to CPS #4

IssueNotes Decision
1

Increase memory resources of NCMP (helm chart)

Memory resources of CPS/NCMP pod should be increased to 4GB, 5GB, etc. to determine if the OOME for CPS-2146 is fixed.

Csaba Kocsis ETH will test and report back to CPS

2

Increase shared_buffer allocation in Postgres config

There is preliminary confirmation that this alone fixes CPS-2156, more testing is in progress. This was confirmed to fix CPS-2156.

ETH has already implemented fix.
EST has updated CPS documentation: https://gerrit.onap.org/r/c/cps/+/137534

3

NCMP will implement throttling / rate limiting for Rest API (e.g. 503 HTTP response)

Requires determining maximum request rate, e.g. compare previous successful versus failing tests (e.g. 3.4.2 vs 3.4.6) to determine throttling.

A poc of rate limiting has been created: 20747:
WIP Rate limit NCMP Rest requests | https://gerrit.nordix.org/c/onap/cps/+/20747

  1. Daniel Hanrahan will report statistics of previous passing test (# request per seconds for 20K registration)
    and compare with request rate after performance improvements and failing user cases
  2. Daniel Hanrahan  will investigate how server can report/reject when requests load is too high
  3. Need to agree with stakeholder acceptable limits of request load (per interface?)
4

Rest client (for load tests) will throttle


Depend on outcome of #3 above
5

Lower thread count for Module Sync

This can be done using variable NCMP_MODULES_SYNC_WATCHDOG_ASYNC_EXECUTOR_PARALLELISM_LEVEL (default 10)

Csaba Kocsis ETH will test and report back to CPS

6

Review Hazelcast configuration

Hazelcast is configured to have multiple backups which are not needed in a deployment with only 2 NCMP instances (2 instances requires only 2 copies across the cluster). Testing has shown that having appropriate amount of backups to suit cluster size reduces heap usage by around 100MB during  20K CM handle registration.

Daniel Hanrahanhas provided a patch to reduce memory consumption: https://gerrit.onap.org/r/c/cps/+/137517

Background

CPS and NCMP have much higher memory consumption than required. Regarding NCMP specifically, it has some in-memory data structures that grow linearly with the number of CM-handles.

...