CPS-470: Spike - Add support for data deletion events

CPS-470 - Getting issue details... STATUS

Overview

The CPS-core notification system only sends data-update events when a root data node is created or the anchor is updated. There is no support for delete change. And, the existing supported operations are also not differentiated in the event. To define the operation type, we need to change the existing event structure and the changes on both the cps-core and cps-temporal sides to support the new event version.

Proposal

Operation

Operation	Scenario
CREATE	The root data node is created for the first time
UPDATE	update action on the root 'data node' create, update and delete action on the non-root data nodes
DELETE	Either anchor or root data node is deleted

Event

Event Structure

The event V1 schema definition is strict and it is not possible to add any optional fields without breaking the old schema contract. To have the flexibility of adding optional fields in future with full compatibility, we are setting the value of 'additionalProperties' to true in both 'CpsDataUpdatedEvent' and 'Content' definitions.

Flexible schema definition

{

  "definitions": {
    "CpsDataUpdatedEvent": {
      ...
      "additionalProperties": true
    },
    "Content": {
     ...
      "additionalProperties": true
    },
    ...
  }
}

The new 'operation' field is added to the event in the content section. The field can be marked as

optional: It makes the event v2 backwards compatible and the user can upgrade cps-temporal before cps-core. As it is a field that should be mandatory, it should be set as mandatory in the v3 which gives the user a release to upgrade cps-core to v2.
mandatory: no compatibility with the older version and both cps-core and cps-temporal have to be updated at the same time.

The team decided to mark it as

The existing 'data' section in the content should be set as null for the delete event. As we are deleting a mandatory field, the change is still backwards compatible.

Added operation property in content

{
  	"definitions": {
		...
    	"Content": {
			"properties": {
				"operation": {
          			"description": "The operation supported by the ",
          			"type": "string",
          			"enum": ["CREATE", "UPDATE", "DELETE"]
        			}
				...
			}
		}
	}
}

cps-data-updated-event-schema-v2.json

{

  "$schema": "https://json-schema.org/draft/2019-09/schema",
  "$id": "urn:cps:org.onap.cps:data-updated-event-schema:v2",

  "$ref": "#/definitions/CpsDataUpdatedEvent",

  "definitions": {

    "CpsDataUpdatedEvent": {
      "description": "The payload for CPS data updated event.",
      "type": "object",
      "properties": {
        "schema": {
          "description": "The schema, including its version, that this event adheres to. E.g. 'urn:cps:org.onap.cps:data-updated-event-schema:v99'.",
          "type": "string",
          "format": "uri"
        },
        "id": {
          "description": "The unique id identifying the event for the specified source. Producer must ensure that source + id is unique for each distinct event.",
          "type": "string"
        },
        "source": {
          "description": "The source of the event. Producer must ensure that source + id is unique for each distinct event.",
          "type": "string",
          "format": "uri"
        },
        "type": {
          "description": "The type of the event.",
          "type": "string"
        },
        "content": {
          "$ref": "#/definitions/Content"
        }
      },
      "required": [
        "schema",
        "id",
        "source",
        "type",
        "content"
      ],
      "additionalProperties": true
    },

    "Content": {
      "description": "The event content.",
      "type": "object",
      "properties": {
        "operation": {
          "description": "The operation supported by the ",
          "type": "string",
          "enum": ["CREATE", "UPDATE", "DELETE"]
        },
        "observedTimestamp": {
          "description": "The timestamp when the data has been observed. The expected format is 'yyyy-MM-dd'T'HH:mm:ss.SSSZ'. Ex: '2020-12-01T00:00:00.000+0000'  ",
          "type": "string"
        },
        "dataspaceName": {
          "description": "The name of CPS Core dataspace the data belongs to.",
          "type": "string"
        },
        "schemaSetName": {
          "description": "The name of CPS Core schema set the data adheres to.",
          "type": "string"
        },
        "anchorName": {
          "description": "The name of CPS Core anchor the data is attached to.",
          "type": "string"
        },
        "data": {
          "$ref": "#/definitions/Data"
        }
      },
      "required": [
        "observedTimestamp",
        "dataspaceName",
        "schemaSetName",
        "anchorName"
      ],
      "additionalProperties": true
    },

    "Data": {
      "description": "Data as json object.",
      "type": "object"
    }

  }

}

example: data updated event v2

{
  "schema": "urn:cps:org.onap.cps:data-updated-event-schema:v1",
  "id": "77b8f114-4562-4069-8234-6d059ff742ac",
  "source": "urn:cps:org.onap.cps",
  "type": "org.onap.cps.data-updated-event",
  "content": {
    "observedTimestamp": "2020-12-01T00:00:00.000+0000",
    "dataspaceName": "my-dataspace",
    "schemaSetName": "bookstore-schemaset",
    "anchorName": "chapters",
    "operation": "CREATE",
    "data": {
      "test:bookstore":{
        "bookstore-name": "Chapters",
        "categories": [
          {
            "code": "01",
            "name": "SciFi",
            "books": [
              {
                "authors": [
                  "Iain M. Banks"
                ],
                "lang": "en",
                "price": 895,
                "pub_year": "1994",
                "title": "Feersum Endjinn"
              }
            ]
          }
        ]
      }
    }
  }
}

CPS-Core

cps core should send the event with the 'operation' information for all the events. The scenarios are defined in the Operation section of the proposal.

In the case of the 'delete' event, the 'data' field should not be sent.

Delete event

{
  "schema": "urn:cps:org.onap.cps:data-updated-event-schema:v1",
  "id": "77b8f114-4562-4069-8234-6d059ff742ac",
  "source": "urn:cps:org.onap.cps",
  "type": "org.onap.cps.data-updated-event",
  "content": {
    "observedTimestamp": "2020-12-01T00:00:00.000+0000",
    "dataspaceName": "my-dataspace",
    "schemaSetName": "bookstore-schemaset",
    "anchorName": "chapters",
    "operation": "DELETE"
  }
}

CPS-Temporal

It will store the new 'operation' field in the newly added column in the network_data table. if this field is optional in the event, the default value of UPDATE will be set on the temporal side.

Database Schema

A new field 'operation' of type varchar(20) is added in the network_data table.

Migration

The newly added field 'operation' should have a value of 'UPDATE' for the existing field.

Space shortcuts

Page tree