...
| Note |
|---|
Please note this section was added long after the implementation and focuses on characteristic only. |
Characteristics - WIP
It is proposed that reported characteristics will be used as a baseline for NCMP when agreed and sign-off.
...
Expected duration
...
100 ms to get module references
1000 ms to get module resources
...
N/A
...
N/A
...
N/A
...
5
...
20
...
TDB e.g. 5 KB
...
25 op/sec
...
20
...
N/A
...
within 1 sec
...
20
...
N/A
...
13 ops/sec
...
20
...
Within 2 sec
Note
- This is for mixed TCs
- Single KPIs will be monitored in NCMP owned pipeline with our performance every day(2 hrs interval) - Performance
- New BATCH KPI - TBD
- Organise a call to walk through the code and TCs with //. Want to see the code of the Test Cases. Csaba Szabó to walkthrough TCs with CPS team
- Check Registration & de-reg agree with //
Open Questions
- Concurrent which number are we agreeing to test, 10 parallel op. according to the input load, the average response time as mentioned in comment is 20 parallel op
- DMI delay - We need feed back from ETH on this
- Better to have R, W use cases separated as in the FS for both throughput and response time or should we merge this on NCMP
- Check Registration - Kolawole Adebisi-Adeolokun Refer to Michelle's email
- Check DE-Registration if this was ever agreed
Ongoing Discussion
- Share Test cases per KPI with CPS. AP @Csaba Szabó X to walkthrough TCs with Toine & Daniel(CPS)
- Functional Specification document.
- Confluence: Configuration Handling Functional Specification.
i. Current content shows mixed testcases, all TCs happening in parallel. AP @Csaba Kocsis Clarify what is being run in parallel and update table accordingly. - Waiting for feedback
ii. AP @Csaba Kocsis update UseCase_0005_IdSearch_NoFilter & UseCase_0008_Search_NoFilter to merge the Input Load column to show total of 5 parallel requests. - Waiting for feedback
iii. AP @Csaba Kocsis Notify Product Engineering (Dagda) about the change. - Waiting for feedback
- AP , @Csaba Kocsis , @Toine Siebelink, @Kolawole Adeolokun: Review the FS and identify if additional information is required. ONGOING
...
Issues & Decisions
| Issues | Notes | Decisions | |
|---|---|---|---|
| 1 | KPI for De-registration of 100 CM-handles | This was mentioned. Was this ever agreed, is this a valid use case that needs to be covered together with the Registration ? | Not priority for now, but acceptable if we match the registration req. |
| 2 | DMI delay | Could we get some feedback on DMI-delays for other use cases as not mentioned in FS document | Awaiting for ETH feedback AP On Kolawole Adebisi-Adeolokun and Csaba Kocsis |
Characteristics - WIP
It is proposed that reported characteristics will be used as a baseline for NCMP when agreed and sign-off.
| Operation | Concurrent requests/parallel | DMI Delay | Response size | Performance Requirement (Blue Stone tablet KPI) | Notes | Sign-Off | |
|---|---|---|---|---|---|---|---|
| 1 | Registration of 20,000 CM-handles (in batches of 100) | 1 (requests are sequential) | 100 ms to get module references | N/A |
|
| |
| 2 | De-registration of 100 CM-handles | 1 (requests are sequential) | No Module delays | N/A |
| De-registration is currently not mentioned in Stone Tablet KPI or FS, however we have agreed to match the performance of registration for now as de-reg is also not a priority at this point in time | |
| 3 | CM-handle ID search with Module filter | 2.5 | N/A | 20,000 CM Handles i.e. 100*20.000 = 2MB | As provided byCsaba Kocsis 0.625 seconds/Operation | ||
| 4 | CM-handle search with Module filter | 2.5 Run in parallel with #3 | N/A | 20,000 CM Handles i.e. 500*20.000 = 10MB | As provided by Csaba Kocsis 13 seconds/Operation | ||
| 5 | Synchronous single CM-handle pass-through read | 10 | 300 ms Csaba Kocsis | 5 KB | 25 (parallel) request/sec | Read are done in parallel with Write | |
| 6 | Synchronous single CM-handle pass-through write (CUD) | 10 | 670 msCsaba Kocsis | 5 KB | 13 (parallel) request/sec | No response is expected |
Notes
- This is for mixed TCs
- Single KPIs will be monitored in NCMP owned pipeline with our performance every day(2 hrs interval) - Performance
- As confirmed, test was carried out on 2 instances in ETH env. CPS validated 2 replica instances have show considerably better test performance - CPS-2038
...
- Question about FS and Stability test strategy – what is expected to be supported in parallel or what is expected to be tested in series? How is that decided? AP Michelle & @Csaba Kocsis - Waiting for feedback
- Can FS to be updated to define single / individual workload. Where can it be tested? AP @Kieran Mccarthy A, @Csaba Kocsis, @Toine Siebelink - Waiting for feedback
Synchronous single cm-handle pass-through (read) requests
...