References

  1. CPS-2155 - Getting issue details... STATUS
  2. Name convention for Managed Objects (3GPP TS 32.300 version 11.2.0 Release 11)

Open Discussion  

#IssueNotesDecision
1MVP (Minimum Viable Product) for DCM
  1. Expected Delivery to EIC  Q3(August) to EIC. (Fossing and Testing)

  2. Write Only Should be prioritized for target delivery (Q3). It should include
    - Batch write
    - NCMP should only check the cmhanlde of the first operation 
    - Should Databag be populated or not kieran mccarthy / Csaba Kocsis ?
  3. NCMP should prioritize CPS-2009 Update remaining existing/legacy NCMP APIs to support alternate Id (FDN) - Developer Wiki - Confluence (onap.org) - over datajob Read. New Epic to be created for DataJob Read sperate this from 1964 - > CPS-2189
  1. EIC Expected Delivery  Q3(August) 
2DataJob Read v Write prioritizationWrite should be prioritized over read

 

3Use of '/ ' causes some issues

kieran mccarthy Csaba Kocsis  to review this impact on querying of FDN

 

4

Prioritize CPS-2009 Update remaining existing/legacy NCMP APIs to support alternate Id (FDN) - Developer Wiki - Confluence (onap.org) - over Read.

New Epic requested for DataJob Read sperate this from 1964.

Expected even earlier than q3

Agreed to move higher on R14 - #17

 

Issues & Decisions


IssueNotes Decision
1dataaccepttype Do we need to consider this parameter in our NCMP internal Java interface?The controller should accept/reject dataaccepttype. From the client application

The Controller will validate and set the Default as necessary kieran mccarthy Sourabh Sourabh Kolawole Adebisi-Adeolokun Toine Siebelink 

2datacontenttype Do we need to consider this parameter in our NCMP internal Java interface?NCMP should maintain the content type as received and not modify datacontenttype.

NCMP should maintain what content type as received and not modify   kieran mccarthy Sourabh Sourabh Kolawole Adebisi-Adeolokun Toine Siebelink 

3

attributes This has plural but all examples contain a single String, why?

kieran mccarthy & Rafael to come back with a 'definition' and 'example

Kolawole Adebisi-Adeolokun to follow up with stakeholders on decision by 19/03/2024

4

The study mentions 'fields and attributes". None of the examples use 'fields'. We can see only 'value' Objects (in the write use cases)

kieran mccarthy & Rafael to come back with a 'definition' and 'example


Read and Write data jobs are 2 deff. objects.  


Kolawole Adebisi-Adeolokun to follow up with stakeholders on decision by 19/03/2024

5

Combine all java api parameters into 1 parameter object?

There is a limit (sonar quality check) of 7 parameter max for a method

Team 
follow common practice and  have a few parameters ie. dataJobId is NOT part of the client request but generated in DCM so it seems logical to keep this separates

6

Confirm that validation is the responsibility of the REST?


Yes, the rest controller will do input validation. via email 26/03/2024 Csaba Kocsis 

7

Is Status likely to change in the future? Considering using string for flexibility.


Java interfaces shall be 'strings'  Csaba Kocsis
Kolawole Adebisi-Adeolokun

8

Is ScopeType likely to change in the future? Don't want to use an enum if change is likely


Java interfaces shall be 'strings Csaba Kocsis Kolawole Adebisi-Adeolokun 

9

Can one request contain both read and write operations?


Writes and reads will not mix. But different writes operations can mix, like update or create. via email 26/03/2024 Csaba Kocsis 

10

Was there a decision made on whether write requests were using kafka or is there any discussion ongoing?


Retrieving the results for the write is done through HTTP synchronously.

Csaba Kocsis 

11

name of privateProperties (CMHandle private properties for the alternateID in the path.)

Can keep this var name as "cmHandleProperties" for consistency?

Agreed to use "cmHandleProperties" Csaba Kocsis Kolawole Adebisi-Adeolokun 


Data Structures

                                                         

Proposed Method signature

processDataJob
void processReadDataJob(String dataAcceptType,  String dataContentType, String dataJobId, List<ReadOperation> readOperations)
void processWriteDataJob(String dataAcceptType,  String dataContentType, String dataJobId, List<WriteOperation> writeOperations, Map<String,String> metadata)

Notes

  1. The order of Operations is important and needs to be maintained hence the use of 'List'
  2. The output is 'void' for the scope in this user story, it will be defined later
  3. dataaccepttype and datacontenttype might be needed too, see issue #1 and #2. If required we could combine them to reduce the number of parameter in an object like RestProtocolParameters

Proposed DMI REST Interface & Forwarded DMI Java Request Data Structure from NCMP

openapi.yaml 
openapi: 3.0.3
info:
  description: Support datajobs through one or more subjob for each DMI and Data Producer
    Identifier combination
  title: NCMP Data Subjob API
  version: 1.0.0
servers:
- url: /dmi
tags:
- description: DMI plugin rest apis
  name: dmi-datajob
paths:
  /v1/dataJob/{requestId}:
    post:
      description: Create a read request
      operationId: createReadRequest
      parameters:
      - description: Identifier for the overall Datajob
        explode: false
        in: path
        name: requestId
        required: true
        schema:
          example: some-identifier
          type: string
        style: simple
      requestBody:
        content:
          application/3gpp-json-patch+json:
            schema:
              $ref: '#/components/schemas/SubjobRequest'
        description: Operation body
      responses:
        "200":
          content:
            application/json:
              schema:
                $ref: '#/components/schemas/createReadRequest_200_response'
          description: Response for subjob
      tags:
      - dmi-datajob
components:
  parameters:
    requestIdInPath:
      description: Identifier for the overall Datajob
      explode: false
      in: path
      name: requestId
      required: true
      schema:
        example: some-identifier
        type: string
      style: simple
  schemas:
    SubjobRequest:
      properties:
        dataAcceptType:
          description: Defines the data response accept type
          example: application/vnd.3gpp.object-tree-hierarchical+json
          type: string
        dataContentType:
          description: Defines the data request content type
          example: application/3gpp-json-patch+json
          type: string
        dataProducerId:
          description: ID of the producer registered by DMI for the alternateIds in
            the operations in this request
          example: my-data-producer-identifier
          type: string
        data:
          $ref: '#/components/schemas/SubjobRequest_data'
      required:
      - data
      - dataProducerId
      type: object
    ReadOperation:
      example:
        op: read
        operationId: 1
        path: SubNetwork=Europe/SubNetwork=Ireland/MeContext=NR03gNodeBRadio00003/ManagedElement=NR03gNodeBRadio00003/GNBCUCPFunction=2
        attributes: userLabel
        scope:
          scopeTyp: BASE_ONLY
      items:
        $ref: '#/components/schemas/ReadOperation_inner'
      type: array
    WriteOperation:
      example:
        op: add
        path: SubNetwork=Europe/SubNetwork=Ireland/MeContext=NR03gNodeBRadio00003/ManagedElement=NR03gNodeBRadio00003/GNBCUCPFunction=1/EUtraNetwork=1/EUtranFrequency=12
        value:
          id: 12
          attributes:
            userLabel: label12
      items:
        $ref: '#/components/schemas/WriteOperation_inner'
      type: array
    CmHandleProperties:
      type: object
    Resource:
      properties:
        id:
          description: Identifier of the resource object
          example: resource-identifier
          type: string
        attributes:
          additionalProperties:
            example: "userLabel: label11"
            type: string
          description: Key value map representing the objects class attributes and
            values
          type: object
      type: object
    ActionParameters:
      additionalProperties:
        type: string
      description: The input of the action in the form of key value pairs
      type: object
    Object:
      type: object
    createReadRequest_200_response:
      example:
        dataProducerJobId: dataProducerJobId
      properties:
        dataProducerJobId:
          description: The data job ID.
          type: string
      type: object
    SubjobRequest_data:
      oneOf:
      - $ref: '#/components/schemas/ReadOperation'
      - $ref: '#/components/schemas/WriteOperation'
    ReadOperation_inner:
      properties:
        path:
          description: Defines the resource on which operation is executed
          example: SubNetwork=Europe/SubNetwork=Ireland/MeContext=NR03gNodeBRadio00003/ManagedElement=NR03gNodeBRadio00003
          type: string
        op:
          description: Describes the operation to execute
          example: read
          type: string
        operationId:
          description: Unique identifier for the operation within the request
          example: "1"
          type: string
        attributes:
          description: This parameter specifies the attributes of the scoped resources
            that are returned
          items:
            example: cellId
            type: string
          type: array
        fields:
          description: This parameter specifies the attribute fields of the scoped
            resources that are returned
          items:
            type: string
          type: array
        filter:
          description: This parameter is used to filter the scoped Managed Objects.
            Only Managed Objects passing the filter criteria will be fetched
          example: NRCellDU/attributes/administrativeState==LOCKED
          type: string
        scopeType:
          description: ScopeType selects MOs depending on relationships with Base
            Managed Object
          example: BASE_ONLY
          type: string
        scopeLevel:
          description: Only used when the scope type is BASE_NTH_LEVEL to specify
            amount of levels to search
          example: 0
          type: integer
        moduleSetTag:
          description: Module set identifier
          example: my-module-set-tag
          type: string
        cmHandleProperties:
          type: object
      required:
      - op
      - path
      type: object
    WriteOperation_inner_value:
      description: Value dependent on the op specified. Resource for an add. Object
        for a replace. ActionParameters for an action.
      oneOf:
      - $ref: '#/components/schemas/Resource'
      - $ref: '#/components/schemas/ActionParameters'
      - $ref: '#/components/schemas/Object'
      type: object
    WriteOperation_inner:
      properties:
        path:
          description: Defines the resource on which operation is executed
          example: SubNetwork=Europe/SubNetwork=Ireland/MeContext=NR03gNodeBRadio00003/ManagedElement=NR03gNodeBRadio00003
          type: string
        op:
          description: Describes the operation to execute
          example: add
          type: string
        operationId:
          description: Unique identifier for the operation within the request
          example: "1"
          type: string
        value:
          $ref: '#/components/schemas/WriteOperation_inner_value'
      required:
      - op
      - path
      type: object




IssueNotesDecision
1


2


3


4


5


Datajob read request

NameDescriptionLocationTypeMandatory

destination

The destination of the results. ( e.g. S3 Bucket)

Query

string

N

dataaccepttype

Define the data response accept type. Passible values:

·        application/vnd.3gpp.object-tree-hierarchical+json (default)

·        application/vnd.3gpp.object-tree-flat+json

request body

enum

N

datacontenttype

Define the data request content type. Passible values:

·        application/3gpp-json-patch+json (default)

request body

enum

N

data

List of operations to be executed. 

request body

List of 3gppReadOperation

Y

3gppReadOperation

NameDescriptionTypeMandatory
path

It is a unique identifier of a managed object (MO) on a network element. 
Defines the resource on which operation is executed. Typically could be Fully Distinguished Name (FDN).

StringY
op

Describes the operation to execute. The value can be: "read" 

StringY
operationId

Unique identifier of the operation within the request

IntegerN
attributes

This parameter specifies the attributes of the scoped resources that are returned. 

List of StringN
fields

This parameter specifies the attribute fields of the scoped resources that are returned.

This should be used if an attribute is a struct and only a subset of its fields should be returned.

List of StringN
filter

The parameter is used to filter the scoped Managed Objects. Only Managed Objects passing the filter criteria will be fetched.

StringN
scopeType

ScopeType selects MOs depending on relationships with Base Managed Object.


enum[BASE_ONLY, BASE_ALL, BASE_NTH_LEVEL, BASE_SUBTREE]N
scopeLevel

Only used when the scope type is BASE_NTH_LEVEL

IntegerN

Datajob write request

NameDescriptionLocationTypeMandatory

destination

The destination of the results. ( e.g. S3 Bucket)

Query

string

N

dataaccepttype

Define the data response accept type. Passible values:

·        application/vnd.3gpp.object-tree-hierarchical+json (default)

·        application/vnd.3gpp.object-tree-flat+json

request body

enum

N

datacontenttype

Define the data request content type. Passible values:

·        application/3gpp-json-patch+json (default)

request body

enum

N

data

List of operations to be executed. 

request body

List of 3gppPatchOperation

Y

3gppPatchOperation

NameDescriptionTypeMandatory
path

It is a unique identifier of a managed object (MO) on a network element. 
Defines the resource on which operation is executed. Typically could be Fully Distinguished Name (FDN).

StringY
op

Describes the operation to execute. The value can be:

"add": creates a new MO with the id and attributes given in the value
"replace":  replaces the value of an attribute specified in the path (attribute is separated with a # from the MO path) with the object in the value
"remove": removes the MO given in the path, value is not applicable
"action": executes an MO action in the path (action is separated with a # from the MO path) with the input parameters in the value

StringY
operationId

Unique identifier of the operation within the request

IntegerN
value

NA if op == remove

Resource if op == add

Object if op == replace

ActionParameters if op == action



ObjectN

Resource

NameDescriptionTypeMandatory
id

Identifier of the resource object

StringN
attributes

Attributes object whose members are the class attributes and values. The object contains key/value map where:

  • key is the attribute name
  • value is the attribute value
ObjectN

ActionParameters

NameDescriptionTypeMandatory
input

The input of the action. Key value pairs.

ObjectN

REST Response from DCM

Name

Description

Type

Mandatory

jobId

The id of the data job

string

Y

status

The status of the jobid    

enum:

[ NOT_STARTED, RUNNING, FINSHED, FAILED, PARTIALLY_FAILED, CANCELLING, CANCELLED ]

Y

statusuri

Status uri for the jobid.

Example: {apiRoot}/ranoam/cm/v1/dataJob/{jobId}

uri

Y

resultsuri

Result uri for the jobid.

Example: {apiRoot}/s3-bucket/{bucketId}

uri

Y

Data SubJob create (NCMP → DMI)

This is a mirror of the Datajob read/write request with the added details needed for the DMI plugins. Below you can see the changes for the READ but the same changes are applicable for the WRITE too.

Method: POST

Path: /dmi/v1/dataJob/{requestId}

DMI Data SubJob read request

NameDescriptionLocationTypeMandatory

destination

The destination of the results. ( e.g. S3 Bucket)

Query

string

N

dataaccepttype

Define the data response accept type. Passible values:

·        application/vnd.3gpp.object-tree-hierarchical+json (default)

·        application/vnd.3gpp.object-tree-flat+json

request body

enum

N

datacontenttype

Define the data request content type. Passible values:

·        application/3gpp-json-patch+json (default)

request body

enum

N

dataProducerId

ID of the producer registered by DMI for the alernateIDs in the operations in this request.

request body

String

Y

requestId

Identifier for the overall Datajob

Path

String

Y

data

List of operations to be executed. 

request body

List of DMI3gppReadOperation

Y

DMI3gppReadOperation

NameDescriptionTypeMandatory
path

It is a unique identifier of a managed object (MO) on a network element. 
Defines the resource on which operation is executed. Typically could be Fully Distinguished Name (FDN).

StringY
op

Describes the operation to execute. The value can be: "read" 

StringY
operationId

Unique identifier of the operation within the request

IntegerN
attributes

This parameter specifies the attributes of the scoped resources that are returned. 

List of StringN
fields

This parameter specifies the attribute fields of the scoped resources that are returned.

This should be used if an attribute is a struct and only a subset of its fields should be returned.

List of StringN
filter

The parameter is used to filter the scoped Managed Objects. Only Managed Objects passing the filter criteria will be fetched.

List of StringN
scopeType

ScopeType selects MOs depending on relationships with Base Managed Object.


enum[BASE_ONLY, BASE_ALL, BASE_NTH_LEVEL, BASE_SUBTREE]N
scopeLevel

Only used when the scope type is BASE_NTH_LEVEL

IntegerN
moduleSetTag

Module set identifier

StringY
privateProperties

CMHandle private properties for the alternateID in the path.

MapY

Response

Status code: 200 - OK

NameDescriptionLocationTypeMandatory
dataProducerJobIdThe ID of the created job by the data producer.response bodyStringY

Data subjob status check (NCMP → DMI)

Request:

Method: GET

Path: /dmi/v1/dataJob/{requestId}/dataProducerJob/{dataProducerJobId}/status

NameDescriptionLocationTypeMandatory

dataProducerId

ID of the producer registered by DMI for the alernateIDs in the operations in this request.

Query

String

Y

requestId

Identifier for the overall Datajob

Path

String

Y

dataProducerJobId

Identifier of the job created by the data producer.

Path

String

Y

Response:

Status code: 200 - OK

NameDescriptionLocationTypeMandatory

status

enum that represents the status of the subjob.
Possible values:		response bodyenum:
[ NOT_STARTED, RUNNING, FINSHED, FAILED, PARTIALLY_FAILED, CANCELLING, CANCELLED ]

Y

Data subjob retrieve result (NCMP → DMI)

Request:

Method: GET

Path: /dmi/v1/dataJob/{requestId}/dataProducerJob/{dataProducerJobId}/result

Parameters:

NameDescriptionLocationTypeMandatory

dataProducerId

ID of the producer registered by DMI for the alernateIDs in the operations in this request.

Query

String

Y

requestId

Identifier for the overall Datajob

Path

String

Y

dataProducerJobId

Identifier of the job created by the data producer.

Path

String

Y

destination

The destination of the results: Kafka topic name or s3 bucket name. This shall be put into the Kafka message headers returned to NCMP

Query

String

Y

Response:

WRITE

Status Code: 200 - Ok

Data part of the message

READ

Status Code: 200 - Ok

NameDescriptionLocationTypeMandatory

resulturi

internal ncmp topic name

response body

String

Y

Examples from DCM Study

DataJob Read request (rAPP -> DCM)

Datajob Write request (rAPP -> DCM)

 Resolve Data Job Operation Path Algorithm

requirement: find the cm handle(id) with the longest match between alternatId and the the input path

Table 1: Registered Cm Handles and their Alternate IDs

CmHandleIdAlternateIdNote
ch-1/SubNetwork=Europe/SubNetwork=IrelandRepresents a group of radio nodes
ch-2/SubNetwork=Europe/SubNetwork=Ireland/SubNetwork=WestMeathRepresents a group of radio nodes
ch-3/SubNetwork=Europe/SubNetwork=Ireland/SubNetwork=WestMeath/ManagedElement=Athlone01Single radio node 
ch-4/SubNetwork=Europe/SubNetwork=Ireland/SubNetwork=WestMeath/ManagedElement=Athlone02Single radio node 
ch-5/SubNetwork=Europe/SubNetwork=Ireland/SubNetwork=LongFordRepresent a group of radio nodes
ch-6/SubNetwork=Europe/SubNetwork=Ireland/SubNetwork=LongFord/ManagedElement=Ballymahon01Single radio node 
ch-7/SubNetwork=Europe/SubNetwork=Ireland/SubNetwork=LongFord/ManagedElement=Ballymahon02Single radio node 

The algorithm knows (3GPP) Path element are separated by / . Pseudo code:

  1. targetCandidate = complete path
  2. cmHandleId = InventoryPersistenceImpl#getCmHandleDataNodeByAlternateId(targetCandidate)
  3. if found: EXIT MATCH
  4. if NOT targetCandidate.CONTAINS('/')  : EXIT WITH NO MATCH
  5. targetCandidate = targetCandidate.SUBSTRING(0,targetCandidate,INDEXOF('/')-1)
  6. repeat step 2 etc.

Table 2: Sample match results

Operation path (matching part in bold)matching cm handle(DMI) resource identifierlookup attemptsNote
/SubNetwork=Europe/SubNetwork=Ireland/SubNetwork=WestMeath/ManagedElement=Athlone01/Function=X/Cell=A123ch-3/Function=X/Cell=A1233
/SubNetwork=Europe/SubNetwork=Ireland/SubNetwork=WestMeathch-2/1This is an operation on a group of radio nodes 
/SubNetwork=Europe/SubNetwork=Ireland/SubNetwork=WestMeath/ManagedElement=Mullingar01/Function=X/Cell=A123ch-2/ManagedElement=Mullingar01/Function=X/Cell=A1234This is probably unintended but the system wil try to execute it anyhow but in the southbound system no matches wil be found

/SubNetwork=Europe/SubNetwork=Belgium

No Match
2

/SubNetwork=Europe/SubNetwork=Belgium/SubNetwork=Brabant/ManagedElement=Antwerpen01/Function=X/Cell-A123

No Match
7


Performance Considerations

  1. For successful matches the performance (number of lookups) wil depend on the number of element under the last subnetwork (the level of subnetworks wil NOT impact performance of positive matches)
  2. For failed match attempts the performance will depend on the total number of elements including the level of subnetworks

Realistic FDN Examples

ExampleURI FDN

4G Cell Relation

SubNetwork=Europe/SubNetwork=Ireland,SubNetwork=NETSimW/ManagedElement=LTE08dg2ERBS00041/ENodeBFunction=1/EUtranCellFDD=LTE08dg2ERBS00041-1/EUtranFreqRelation=1,EUtranCellRelation=17

5G Cell Relation

SubNetwork=Europe/SubNetwork=Ireland/MeContext=NR14gNodeBRadio00018/ManagedElement=NR14gNodeBRadio00018/GNBCUCPFunction=1/NRCellCU=NR14gNodeBRadio00018-1/EUtranCellRelation=1

Proposed JIRAs

Priority

Component 

Description

JIRA

1NCMPCreate implementation proposal for NCMP to provide support for Data jobs qualifier during registration

CPS-2096 - Getting issue details... STATUS

2NCMPNCMP to provide support for data producer identifier during registration

CPS-2105 - Getting issue details... STATUS

3NCMPSupport of Datajobs qualifier during registration

CPS-2084 - Getting issue details... STATUS

4NCMPModify the get cmHandle api to return dataProducerIdentifier, moduleSetTag, and alternateId

CPS-2113 - Getting issue details... STATUS

5NCMPModify lcm events to include dataProducerIdentifier and moduleSetTag

CPS-2114 - Getting issue details... STATUS

6NCMPCreate implementation proposal for NCMP to provide Java interface that can support Data jobs request

CPS-2097 - Getting issue details... STATUS

7NCMPDefine a java based datastructure for DataJob

CPS-2155 - Getting issue details... STATUS

8NCMPHandle Datajob request in java interface

CPS-2142 - Getting issue details... STATUS

9NCMPHandle async datajob response in java interface

CPS-2143 - Getting issue details... STATUS

10
Define DMI REST interface for Datajobs

CPS-2141 - Getting issue details... STATUS

11DMIDefine a java based datastructure dmi request for dmi endpoint

CPS-2163 - Getting issue details... STATUS

12DMIResponse schema for DMI dataJob response

CPS-2154 - Getting issue details... STATUS

13NCMPForward dataJob response from DMI to client topic in header as destination 

CPS-2098 - Getting issue details... STATUS

14DMIStub should be modified to use generated code from OpenAPI

CPS-2145 - Getting issue details... STATUS

15
E2E demo for internal team and stakeholder 

CPS-2144 - Getting issue details... STATUS



  • No labels

3 Comments

  1. kieran mccarthy

    First comment : 

    1. Path will always start with a "/" as below.

            "path": "/SubNetwork=Europe/SubNetwork=Ireland/MeContext=NR03gNodeBRadio00003/ManagedElement=NR03gNodeBRadio00003/GNBCUCPFunction=1/EUtraNetwork=1/EUtranFrequency=12",


           2. operationId is a string


           3. DataJob can be only for read OR write, i.e. operation with "op" : "read"  cannot exist in same DataJob with other op types


           4. We will provide examples with fields, attributes and scope examples.

    1. Toine Siebelink

      the examples are directly from Rafael's study.  I think there are quiet a few inconsistencies hence the need for a chat....

      1. kieran mccarthy

        Agree - action taken to clarify as agreed.

        I have attached the DataJob structures ... might be some small fine tuning but should be pretty much there.