You are viewing an old version of this page. View the current version.

Compare with Current View Page History

« Previous Version 27 Next »

In Review


Jira item: CPS-459 - Getting issue details... STATUS

Introduction

CPS system is providing a Json Schema for that defines the format of events used to notify about CPS data updates. See CPS-191: Core and Temporal Integration Design for more details.

When CPS system is going to evolve, depending on the evolution made, it might happen that the initially defined schema for Data Updated Event has also to be changed. If such changes are not planned properly they can be complex and risky to implement in a running production system. Event schema changes has to be implemented in both producer and consumers systems and has to be flexible enough so that all components does not have to be updated at the exact same time to respect key micro services design principles: loosely coupled and independently deployable. Then, the system is expected to support a transition period when some components are running with an event schema updated while some other are not.

The purpose of this spike is to propose a strategy to ensure that CPS Data Updated event schema will be able to evolve with future requirements to come.

Compatibility Types

CPS supports one of the 3 following compatibility types between 2 consecutive event schema versions:

Forward Compatibility

Forward compatibility type allows events produced by the new schema to be consumed by the previous schema.

This type has the following constraints:

  • Only changes allowed in the new schema are:
    • Add fields
    • Delete optional fields
  • Producer as to be upgraded first

Backward Compatibility

Backward compatibility type allows events produced by the previous schema to be consumed by the new schema.

This type has the following constraints:

  • Only changes allowed in the new schema are:
    • Delete fields
    • Add optional fields
  • All consumers as to be upgraded first

Full Compatibility

Full compatibility type implies both Forward and Backward compatibility. It means that events produced by the new schema can be consumed by the previous schema and also that events produced by the previous schema can be read by the new schema.

This type has the following constraints:

  • Only changes allowed in the new schema are:
    • Add optional fields
    • Delete optional fields

This type does not have any constraint on the order to complete components upgrades. It gives more flexibility as producer and consumers can be upgraded in any order.


See https://docs.confluent.io/platform/current/schema-registry/avro.html for more details about these compatibility types.

Proposed Solution for Istanbul

Limitations

As described in Complete solution using current event schema implementation, the requirement to support all types of evolution can be fulfilled, but it introduces an additional complexity on the application design because of lacking a common type that all events are implementing to be able to give more flexibility to handle these classes. Then, we need to consider if we should manually write event classes code instead of leveraging "Json Schema 2 Pojo" plugin to have these classes automatically generated. This can be looked during next release.

Meanwhile, for Istanbul, we are proposing to make only minimal changes to support a basic Forward Compatibility change in Jakarta without implementing a complete solution supporting all compatibility types and deployment options yet.

This means that for next release after Istanbul the deployment order will have to be :

  1. Deploy CPS Core new release (events form new schema are produced and still consumed as events from previous schema).
  2. Deploy CPS Temporal new release (event from new schema are produced and now consumed as events from new schema).

Changes

Schema Versioning

Not having is own repository and being part of cps repository, cps-events Maven module is versioned each time cps repository is versioned. Then, having a cps-events Maven release does not means that the event schema is changed. It is possible that it is the exact same as the schema from previous cps-events release. Then, cps-events maven version can not be used to identify event schema version. Another versioning scheme is introduced to keep track of event schema versions. Following is the nomenclature for this versioning:

  • v1
  • v2
  • v3
  • ...

This version number is found in the Json event schema under the schema property.

Schema Property Type

In order for a given event instance to be produced by a previous event schema definition and read by a future event schema definition (backward compatibility), the schema version from the schema definition cannot be a constant anymore. 

In order for a given event instance to be produced by a future event schema definition and still be read by a previous event schema definition (forward compatibility), the schema version from the schema definition cannot be an enum anymore.

Schema version in schema definition is then changed for a simple uri string.


Here are the changed proposed for event schema definition:

diff cps-data-updated-event-schema.json
--- a/cps-events/src/main/resources/schemas/cps-data-updated-event-schema.json
+++ b/cps-events/src/main/resources/schemas/cps-data-updated-event-schema.json
@@ -1,7 +1,7 @@
 {

   "$schema": "https://json-schema.org/draft/2019-09/schema",
-  "$id": "urn:cps:org.onap.cps:data-updated-event-schema:1.1.0-SNAPSHOT",
+  "$id": "urn:cps:org.onap.cps:data-updated-event-schema:v1",

   "$ref": "#/definitions/CpsDataUpdatedEvent",

@@ -12,10 +12,9 @@
       "type": "object",
       "properties": {
         "schema": {
-          "description": "The schema, including its version, that this event adheres to.",
+          "description": "The schema, including its version, that this event adheres to. Ex: 'urn:cps:org.onap.cps:data-updated-event-schema:v1'.",
           "type": "string",
-          "default": "urn:cps:org.onap.cps:data-updated-event-schema:1.1.0-SNAPSHOT",
-          "enum": ["urn:cps:org.onap.cps:data-updated-event-schema:1.1.0-SNAPSHOT"]
+          "format": "uri"
         },
         "id": {
           "description": "The unique id identifying the event for the specified source. Producer must ensure that source + id is unique for each distinct event.",

And an event schema instance starts with:

{
  "schema": "urn:cps:org.onap.cps:data-updated-event-schema:v1",
  "id": "77b8f114-4562-4069-8234-6d059ff742ac",
  ...

Conclusion

The minimal changes proposed above still give the option to make a forward compatible change in next coming release after Istanbul without introducing all the complexity coming with a complete solution right now, with the current design.

During Jakarta release the team would be able to consider other ways to implement event schema classes (currently automatically generated). Being able to rework the design of these classes could simplify the complete solution implementation for schema evolution if needed.


Note:

For future design work related to this subject, here is a Jackson reference to be looked at for event classes hierarchy: https://www.baeldung.com/jackson-inheritance

  • No labels