WIP
Jira item: CPS-459 - Getting issue details... STATUS
Introduction
CPS system is providing a Json Schema for that defines the format of events used to notify about CPS data updates. See CPS-191: Core and Temporal Integration Design for more details.
When CPS system is going to evolve, depending on the evolution made, it might happen that the initially defined schema for Data Updated Event has also to be changed. If such changes are not planned properly they can be complex and risky to implement in a running production system. Event schema changes has to be implemented in both producer and consumers systems and has to be flexible enough so that all components does not have to be updated at the exact same time to respect key micro services design principles: loosely coupled and independently deployable. Then, the system is expected to support a transition period when some components are running with an event schema updated while some other are not.
The purpose of this spike is to propose a strategy to ensure that CPS Data Updated event schema will be able to evolve with future requirements to come.
Compatibility Types
CPS supports one of the 3 following compatibility types between 2 consecutive event schema versions:
Forward Compatibility
Forward compatibility type allows events produced by the new schema to be consumed by the previous schema.
This type has the following constraints:
- Only changes allowed in the new schema are:
- Add fields
- Delete optional fields
- Producer as to be upgraded first
Backward Compatibility
Backward compatibility type allows events produced by the previous schema to be consumed by the new schema.
This type has the following constraints:
- Only changes allowed in the new schema are:
- Delete fields
- Add optional fields
- All consumers as to be upgraded first
Full Compatibility
Full compatibility type implies both Forward and Backward compatibility. It means that events produced by the new schema can be consumed by the previous schema and also that events produced by the previous schema can be read by the new schema.
This type has the following constraints:
- Only changes allowed in the new schema are:
- Add optional fields
- Delete optional fields
This type does not have any constraint on the order to complete components upgrades. It gives more flexibility as producer and consumers can be upgraded in any order.
See https://docs.confluent.io/platform/current/schema-registry/avro.html for more details about these compatibility types.
CPS Events Releases
CPS events are defined (JSON Schema) and generated (Java classes) from cps repository in cps-events module: https://github.com/onap/cps/tree/master/cps-events
Each time a new version of CPS Events schema is published it contains both:
- The new schema definition (N)
- The previously published schema definition (N-1)
It also clearly specifies (in release notes ?) what is the Compatibility type of new schema (N) with previous schema (N-1).
Finally, an EventSchemaMapper is provider to map events from new schema (N) to previous schema (N-1) and the other way around. This mapper needs to have some logic depending on the change made in the schema. For example, If new fields are added in the new schema, they might be given a default value in the previous one, if possible.
Following is an example of v0 and v1 schema definition in cps-events module.
Then, cps-event artifact built contains the generated classes for both new and previous event schema. Both are available to be used by producer and consumers that are importing this specific cps-events jar as a dependency.
cps-events Maven Artifact Version
DataUpdatedEvent Schema Version
Consumers of CPS Events
Compiled Code
When using cps-events artifact, consumers has access to both new and previous cps events classes.
Then, it is possible to implement listeners for both events versions as suggested in the following snippet:
/**
* Consume event from v0 schema.
*/
public void consume(final org.onap.cps.event.model.v0.CpsDataUpdatedEvent eventV0) {
// Map event v0 to v1
org.onap.cps.event.model.v1.CpsDataUpdatedEvent eventV1 = this.eventSchemaMapper.v0ToV1(eventV0);
// Consume event v1
consume(eventV1);
}
/**
* Consume event from v1 schema.
*/
public void consume(final org.onap.cps.event.model.v1.CpsDataUpdatedEvent eventV1) {
// Map event to entity
final var networkData = this.cpsDataUpdatedEventMapper.eventToEntity(eventV1);
// Persist entity
this.networkDataService.addNetworkData(networkData);
}
From a compiled code point of view, supporting both event type versions is in place.
Runtime Configuration
The challenge is now to be able to choose which event version a listener is consuming at startup, by configuration only, without changing the application compiled code.
This is done by enabling only one of the 2 listeners by configuration:
@KafkaListener(
topics = "${app.listener.data-updated.v0.topic}",
errorHandler = "dataUpdatedEventListenerErrorHandler",
autoStartup = "${app.listener.data-updated.v0.autoStartup}")
public void consume(final org.onap.cps.event.model.v0.CpsDataUpdatedEvent eventV0)
@KafkaListener(
topics = "${app.listener.data-updated.v1.topic}",
errorHandler = "dataUpdatedEventListenerErrorHandler",
autoStartup = "${app.listener.data-updated.v1.autoStartup}")
public void consume(final org.onap.cps.event.model.v1.CpsDataUpdatedEvent eventV1)
app:
listener:
data-updated:
v0:
topic: ${CPS_CHANGE_EVENT_V0_LISTENER_TOPIC:cps.dev.null}
autoStartup: ${CPS_CHANGE_EVENT_V0_LISTENER_ENABLED:false}
v1:
topic: ${CPS_CHANGE_EVENT_V1_LISTENER_TOPIC:cps.cfg-state-events}
autoStartup: ${CPS_CHANGE_EVENT_V1_LISTENER_ENABLED:true}
Producer of CPS Events
Compiled Code
The producer is able to send both current and previous versions of events by implementing 2 sets of Notification Service and Notification Publisher using Template Method design pattern.
