Versions Compared

Old Version 49

changes.mady.by.user Daniel Hanrahan

Saved on

compared with

New Version 50

changes.mady.by.user Daniel Hanrahan

Saved on

Key

  • This line was added.
  • This line was removed.
  • Formatting was changed.

...


IssueNotes Decision
1

Increase memory resources of NCMP (helm chart)


Csaba Kocsis ETH will test and report back to CPS

2

Increase shared_buffer allocation in Postgres config

There is preliminary confirmation that this alone fixes CPS-2156, more testing is in progress

Csaba Kocsis ETH will test and report back to CPS

3

NCMP will implement throttling / rate limiting for Rest API (e.g. 503 HTTP response)

Requires determining maximum request rate, e.g. compare previous successful versus failing tests (e.g. 3.4.2 vs 3.4.6) to determine throttling.

A poc of rate limiting has been created: 20747:
WIP Rate limit NCMP Rest requests | https://gerrit.nordix.org/c/onap/cps/+/20747

  1. Daniel Hanrahan will report statistics of previous passing test (# request per seconds for 20K registration)
    and compare with request rate after performance improvements and failing user cases
  2. Daniel Hanrahan  will investigate how server can report/reject when requests load is too high
  3. Need to agree with stakeholder acceptable limits of request load (per interface?)
4

Rest client (for load tests) will throttle


Depend on outcome of #4 above
5

Lower thread count for Module Sync

This can be done using variable NCMP_MODULES_SYNC_WATCHDOG_ASYNC_EXECUTOR_PARALLELISM_LEVEL (default 10)

Csaba Kocsis ETH will test and report back to CPS

6

Review Hazelcast configuration

Hazelcast is configured to have multiple backups which may not be needed given a deployment with only 2 NCMP instances (given 2 instances, only 1 backup would be needed).

Testing has shown this reduces heap usage by around 100MB during  20K CM handle registration.

Daniel Hanrahanhas provided a patch to reduce memory consumption: https://gerrit.onap.org/r/c/cps/+/137517Daniel Hanrahan will report findings

Background

CPS and NCMP have much higher memory consumption than required. Regarding NCMP specifically, it has some in-memory data structures that grow linearly with the number of CM-handles.

...