Warning |
---|
WIP |
Jira item:
Jira | ||||||
---|---|---|---|---|---|---|
|
...
The purpose of this spike is to propose a strategy to ensure that CPS Data Updated event schema will be able to evolve with future requirements to come.
Compatibility Types
CPS supports one of the 3 following Several compatibility types between 2 consecutive event schema versionsexist when an event schema definition is evolving from one release to the next one:
Forward Compatibility
Forward compatibility type allows events produced by the new schema to be consumed by the previous schema.
...
See https://docs.confluent.io/platform/current/schema-registry/avro.html for more details about these compatibility types.
CPS Events Releases
CPS events are defined (JSON Schema) and generated (Java classes) from cps repository in cps-events module: https://github.com/onap/cps/tree/master/cps-events
Each time a new version of CPS Events schema is published it contains both:
- The new schema definition (N)
- The previously published schema definition (N-1)
It also clearly specifies (in release notes ?) what is the Compatibility type of new schema (N) with previous schema (N-1).
Finally, an EventSchemaMapper is provider to map events from new schema (N) to previous schema (N-1) and the other way around. This mapper needs to have some logic depending on the change made in the schema. For example, If new fields are added in the new schema, they might be given a default value in the previous one, if possible.
Following is an example of v0 and v1 schema definition in cps-events module.
Then, cps-event artifact built contains the generated classes for both new and previous event schema. Both are available to be used by producer and consumers that are importing this specific cps-events jar as a dependency.
cps-events Maven Artifact Version
Note |
---|
To be detailed ... |
DataUpdatedEvent Schema Version
Note |
---|
To be detailed ... |
Consumers of CPS Events
Compiled Code
When using cps-events artifact, consumers has access to both new and previous cps events classes.
Then, it is possible to implement listeners for both events versions as suggested in the following snippet:
Code Block | ||||||
---|---|---|---|---|---|---|
| ||||||
/**
* Consume event from v0 schema.
*/
public void consume(final org.onap.cps.event.model.v0.CpsDataUpdatedEvent eventV0) {
// Map event v0 to v1
org.onap.cps.event.model.v1.CpsDataUpdatedEvent eventV1 = this.eventSchemaMapper.v0ToV1(eventV0);
// Consume event v1
consume(eventV1);
}
/**
* Consume event from v1 schema.
*/
public void consume(final org.onap.cps.event.model.v1.CpsDataUpdatedEvent eventV1) {
// Map event to entity
final var networkData = this.cpsDataUpdatedEventMapper.eventToEntity(eventV1);
// Persist entity
this.networkDataService.addNetworkData(networkData);
}
|
From a compiled code point of view, supporting both event type versions is in place.
Runtime Configuration
The challenge is now to be able to choose which event version a listener is consuming at startup, by configuration only, without changing the application compiled code.
This is done by enabling only one of the 2 listeners by configuration:
Code Block | ||||||
---|---|---|---|---|---|---|
| ||||||
@KafkaListener(
topics = "${app.listener.data-updated.v0.topic}",
errorHandler = "dataUpdatedEventListenerErrorHandler",
autoStartup = "${app.listener.data-updated.v0.autoStartup}")
public void consume(final org.onap.cps.event.model.v0.CpsDataUpdatedEvent eventV0)
@KafkaListener(
topics = "${app.listener.data-updated.v1.topic}",
errorHandler = "dataUpdatedEventListenerErrorHandler",
autoStartup = "${app.listener.data-updated.v1.autoStartup}")
public void consume(final org.onap.cps.event.model.v1.CpsDataUpdatedEvent eventV1) |
Proposed Solution for Istanbul
Limitations
As described in Complete solution using current event schema implementation, the requirement to support all types of evolution can be fulfilled, but it introduces an additional complexity on the application design because of lacking a common type that all events are implementing to be able to give more flexibility to handle these classes. Then, we need to consider if we should manually write event classes code instead of leveraging "Json Schema 2 Pojo" plugin to have these classes automatically generated. This can be looked during next release.
Meanwhile, for Istanbul, we are proposing to make only minimal changes to support a basic Forward Compatibility change in Jakarta without implementing a complete solution supporting all compatibility types and deployment options yet.
This means that for next release after Istanbul the deployment order will have to be :
- Deploy CPS Core new release (events form new schema are produced and still consumed as events from previous schema).
- Deploy CPS Temporal new release (event from new schema are produced and now consumed as events from new schema).
Changes
Schema Versioning
Not having is own repository and being part of cps repository, cps-events Maven module is versioned each time cps repository is versioned. Then, having a cps-events Maven release does not means that the event schema is changed. It is possible that it is the exact same as the schema from previous cps-events release. Then, cps-events maven version can not be used to identify event schema version. Another versioning scheme is introduced to keep track of event schema versions. Following is the nomenclature for this versioning:
- v1
- v2
- v3
- ...
This version number is found in the Json event schema under the schema property.
Schema Property Type
In order for a given event instance to be produced by a previous event schema definition and read by a future event schema definition (backward compatibility), the schema version from the schema definition cannot be a constant anymore.
In order for a given event instance to be produced by a future event schema definition and still be read by a previous event schema definition (forward compatibility), the schema version from the schema definition cannot be an enum anymore.
Schema version in schema definition is then changed for a simple uri string.
Here are the changed proposed for event schema definition:
Code Block | ||||||
---|---|---|---|---|---|---|
| ||||||
--- a/cps-events/src/main/resources/schemas/cps-data-updated-event-schema.json
+++ b/cps-events/src/main/resources/schemas/cps-data-updated-event-schema.json
@@ -1,7 +1,7 @@
{
"$schema": "https://json-schema.org/draft/2019-09/schema",
- "$id": "urn:cps:org.onap.cps:data-updated-event-schema:1.1.0-SNAPSHOT",
+ "$id": "urn:cps:org.onap.cps:data-updated-event-schema:v1",
"$ref": "#/definitions/CpsDataUpdatedEvent",
@@ -12,10 +12,9 @@
"type": "object",
"properties": { | ||||||
Code Block | ||||||
| ||||||
app: listener: data-updated: v0: topic: ${CPS_CHANGE_EVENT_V0_LISTENER_TOPIC:cps.dev.null} autoStartup"schema": ${CPS_CHANGE_EVENT_V0_LISTENER_ENABLED:false} - v1: topic: ${CPS_CHANGE_EVENT_V1_LISTENER_TOPIC:cps.cfg-state-events} "description": "The schema, including its version, that this event adheres to.", + autoStartup: ${CPS_CHANGE_EVENT_V1_LISTENER_ENABLED:true} |
Producer of CPS Events
Compiled Code
The producer is able to send both current and previous versions of events by implementing 2 sets of Notification Service and Notification Publisher using Template Method design pattern.
Following diagrams and snippets are illustrating this implementation:
...
Column | ||
---|---|---|
| ||
Column | ||
---|---|---|
| ||
EventFactory is able to create events for both versions:
Code Block | ||||||
---|---|---|---|---|---|---|
| ||||||
public class CpsDataUpdatedEventFactory { ... private EventSchemaMapper eventSchemaMapper; ... public org.onap.cps.event.model.v0.CpsDataUpdatedEvent createCpsDataUpdatedEventV0( final String dataspaceName, final String anchorName) {"description": "The schema, including its version, that this event adheres to. Ex: 'urn:cps:org.onap.cps:data-updated-event-schema:v1'.", final var eventV1 = this.createCpsDataUpdatedEventV1(dataspaceName, anchorName); "type": "string", - return this.eventSchemaMapper.v1ToV0(eventV1); } public "default": "urn:cps:org.onap.cps.event.model.v1.CpsDataUpdatedEvent createCpsDataUpdatedEventV1( final String dataspaceName, final String anchorName) { final var dataNode = :data-updated-event-schema:1.1.0-SNAPSHOT", - cpsDataService.getDataNode(dataspaceName, anchorName, "/enum", FetchDescendantsOption.INCLUDE_ALL_DESCENDANTS); final var anchor = cpsAdminService.getAnchor(dataspaceName, anchorName); return toCpsDataUpdatedEvent(anchor, dataNode); } ... } |
Abstract Notification and Publisher classes:
Code Block | ||||||
---|---|---|---|---|---|---|
| ||||||
public abstract class NotificationService { ... public void processDataUpdatedEvent(final String dataspaceName, final String anchorName) { ... try { if (shouldSendNotification()) { notify(dataspaceName, anchorName); } } catch (final Exception exception) { notificationErrorHandler.onException("Failed to process cps-data-updated-event.", : ["urn:cps:org.onap.cps:data-updated-event-schema:1.1.0-SNAPSHOT"] + exception, dataspaceName, anchorName); "format": "uri" } } , protected abstract void notify(String dataspaceName, String anchorName); } | ||||||
Code Block | ||||||
| ||||||
public abstract class NotificationPublisher<V> { private final KafkaTemplate<String, V> kafkaTemplate; private final String topicName; "id": { protected NotificationPublisher( final KafkaTemplate<String, V> kafkaTemplate, final @Value("${notification.data-updated.topic}") String topicName) { this.kafkaTemplate = kafkaTemplate; this.topicName = topicName; } protected abstract void sendNotification(@NonNull final V event); }"description": "The unique id identifying the event for the specified source. Producer must ensure that source + id is unique for each distinct event.", |
And an event schema instance starts withConcrete Notification and Publisher classes:
Code Block | |||||||
---|---|---|---|---|---|---|---|
| |||||||
@Service("v1NotificationService") public class V1NotificationService extends NotificationService { private final V1NotificationPublisher notificationPublisher; ... @Override protected void notify(final String dataspaceName, final String anchorName) { final var cpsDataUpdatedEvent = super.getCpsDataUpdatedEventFactory().createCpsDataUpdatedEventV1(dataspaceName, anchorName); notificationPublisher.sendNotification(cpsDataUpdatedEvent); } } | |||||||
Code Block | |||||||
| |||||||
"schema": "urn:cps:org.onap.cps:data-updated-event-schema:v1",
"id": "77b8f114-4562-4069-8234-6d059ff742ac",
... |
Consumer Ignores Type From Message Header
Event message header contains the type used by the producer when sending the message:
Header Key: __TypeId__
Header Value: org.onap.cps.event.model.CpsDataUpdatedEvent
In order to ensure that the listener would always be able to consume events produced with other future types, the consumer should not use this type id from the header to force the type to be used for reading.
Then following property is added to application properties:
Code Block | ||||||
---|---|---|---|---|---|---|
| ||||||
spring: kafka: consumer:.v1.CpsDataUpdatedEvent; public class V1NotificationPublisher extends NotificationPublisher<CpsDataUpdatedEvent> { public V1NotificationPublisher( final KafkaTemplate<String, CpsDataUpdatedEvent> kafkaTemplate, final @Value("${notification.data-updated.topic}") String topicName) { super(kafkaTemplate, topicName); } protected void sendNotification(@NonNull final CpsDataUpdatedEvent cpsDataUpdatedEvent) { final var messageKey = properties: cpsDataUpdatedEventspring.getContent().getDataspaceName() + "," + cpsDataUpdatedEvent.getContent().getAnchorName(); super.getKafkaTemplate().send(super.getTopicName(), messageKey, cpsDataUpdatedEvent); } } |
Runtime Configuration
The event schema version to be published is specified and configurable in application properties. The specified value is used by Spring when the application is starting to instantiate the expected concrete classes for Notification Service and Publisher.
Code Block | ||||||
---|---|---|---|---|---|---|
| ||||||
notification:
data-updated:
enabled: false
topic: ${CPS_CHANGE_EVENT_TOPIC:cps.cfg-state-events}
event-schema-version: ${CPS_CHANGE_EVENT_SCHEMA_VERSION:v1} |
The specific requested Notification Service is injected into Data Service:
Code Block | ||||||
---|---|---|---|---|---|---|
| ||||||
public class CpsDataServiceImpl implements CpsDataService {
...
@Value("${notification.event-schema-version}NotificationService")
private String notificationServiceQualifier;
private NotificationService notificationService;
@Autowired
public void setNotificationService(final ApplicationContext applicationContext) {
this.notificationService = (NotificationService) applicationContext.getBean(notificationServiceQualifier);
}
...
} |
Conclusion
Proposed solution above gives flexibility to deploy new releases of the components (producer or consumers) independently, in any order.
json.use.type.headers: false |
The type to be used by the consumer is already provided in the existing 'spring.json.value.default.type' property.
Default Topic Name
A good practice of Event Driven Architecture is to avoid having multiple kind of events in the same topic. It means that the topic defined for Cps Data Updated Events should be dedicated to those events and is not supposed to receive any other kind of events. To reflect this intention, it is proposed to rename the default topic name with the name of the events that are published in the topic. Then, its name is changed from "cps.cps-cfg-state-events" to "cps.data-updated-events". This default name can still be changed by configuration using "CPS_CHANGE_EVENT_TOPIC" environment variable, if needed.
Gerrit Changes (WIP)
Conclusion
The minimal changes proposed above still give the option to make a forward compatible change in next coming release after Istanbul without introducing all the complexity coming with a complete solution right now, with the current design.
During Jakarta release the team would be able to consider other ways to implement event schema classes (currently automatically generated). Being able to rework the design of these classes could simplify the complete solution implementation for schema evolution if needed.
Note:
For future design work related to this subject, here is a Jackson reference to be looked at for event classes hierarchy: https://www.baeldung.com/jackson-inheritanceWhen deploying them, the event schema configuration provided ensures that producer and consumers can still remain compliant with specific Compatibility type (forward or backward), if any.