...
Using PostgreSQL special type as "json" or "jsonb" rather than "text", all applications will lose any compatibility with H2 and MariaDB. They will be mentioned in this document but not recommended.
Requirements
- Policy types/Policies/Node Types/Node Templates are first order items
- Data Types have a scope of a first order item, so a data type definition only applies in the scope of a policy type or node type definition
- We should keep our current APIs, all changes should be internal
- We must provide an upgrade path to the new data structure and a rollback to the current structure
ORM Layer using Document Storage
ORM layer using document storage (PostgreSQL or MongoDB) could be organized in two layer:
- Document layer (Domain Model to be converted in Json) - implementation has no dependency from DB
- Persistence layer - (Domain Model depend of the DB used): Entities for PostgreSQL, Documents for MongoDB
An implementation on the Document layer could be found here: (https://gerrit.nordix.org/c/onap/policy/models/+/13633)
Example
In the example below DocToscaServiceTemplate should be serialized to Json.
Type | DataStore | Validation | Support Query | Index | Preserve Order |
---|---|---|---|---|---|
longtext | MariaDB | SELECT | Yes | No | Yes |
PostgreSQL | No | No | No | Yes | |
json | PostgreSQL | INSERT / UPDATE | Yes | No | Yes |
jsonb | PostgreSQL | INSERT / UPDATE | Yes | Yes | No |
Requirements
- Policy types/Policies/Node Types/Node Templates are first order items
- Data Types have a scope of a first order item, so a data type definition only applies in the scope of a policy type or node type definition
- We should keep our current APIs, all changes should be internal
- We must provide an upgrade path to the new data structure and a rollback to the current structure
ORM Layer using Document Storage
ORM layer using document storage (PostgreSQL or MongoDB) could be organized in two layer:
- Document layer (Domain Model to be converted in Json) - implementation has no dependency from DB
- Persistence layer - (Domain Model depend of the DB used): Entities for PostgreSQL, Documents for MongoDB
An implementation on the Document layer could be found here: (https://gerrit.nordix.org/c/onap/policy/models/+/13633)
Example
In the example below DocToscaServiceTemplate should be serialized to Json.
Code Block | ||||||
---|---|---|---|---|---|---|
| ||||||
@Data
@EqualsAndHashCode(callSuper = true)
@NoArgsConstructor
public class DocToscaServiceTemplate extends DocToscaEntity<ToscaServiceTemplate> {
@SerializedName("data_types")
private Map<String, DocToscaDataType> dataTypes;
-------
public DocToscaServiceTemplate(ToscaServiceTemplate authorativeConcept) { | ||||||
Code Block | ||||||
| ||||||
@Data @EqualsAndHashCode(callSuper = true) @NoArgsConstructor public class DocToscaServiceTemplate extends DocToscaEntity<ToscaServiceTemplate> { @SerializedName("data_types") private Map<String, DocToscaDataType> dataTypes; ------- public DocToscaServiceTemplate(ToscaServiceTemplate authorativeConcept) { this.fromAuthorative(authorativeConcept); } @Override public ToscaServiceTemplate toAuthorative() { final var toscaServiceTemplate = new ToscaServiceTemplate(); super.setToscaEntity(toscaServiceTemplate); super.toAuthorative(); if (dataTypes != null) { toscaServiceTemplate.setDataTypes(PfUtils.mapMap(dataTypes, DocToscaDataType::toAuthorative)); } ------- return toscaServiceTemplatethis.fromAuthorative(authorativeConcept); } @Override public voidToscaServiceTemplate fromAuthorativetoAuthorative(ToscaServiceTemplate toscaServiceTemplate) { final var toscaServiceTemplate = new ToscaServiceTemplate(); super.fromAuthorativesetToscaEntity(toscaServiceTemplate); super.toAuthorative(); if (toscaServiceTemplate.getDataTypes()dataTypes != null) { dataTypes = DocUtil.mapMap(toscaServiceTemplate.getDataTypes()toscaServiceTemplate.setDataTypes(PfUtils.mapMap(dataTypes, DocToscaDataType::newtoAuthorative)); } ------- } } |
In the example below the implementation of JpaToscaServiceTemplate for PostgreSQL/MariaDB
Code Block | ||||||
---|---|---|---|---|---|---|
| ||||||
@Entity @Table(name = "ToscaServiceTemplate") @Inheritance(strategy = InheritanceType.TABLE_PER_CLASS) @Data @EqualsAndHashCode(callSuper = false) public class JpaToscaServiceTemplate extends PfConcept implements PfAuthorative<ToscaServiceTemplate> { @EmbeddedId @VerifyKey @NotNull private PfConceptKey key; @Lob @Convert(converter = StringToServiceTemplateConverter.class) @NotNull private DocToscaServiceTemplate serviceTemplate; - return toscaServiceTemplate; } @Override public void fromAuthorative(ToscaServiceTemplate toscaServiceTemplate) { super.fromAuthorative(toscaServiceTemplate); if (toscaServiceTemplate.getDataTypes() != null) { dataTypes = DocUtil.mapMap(toscaServiceTemplate.getDataTypes(), DocToscaDataType::new); } ------- } } |
In the example below the implementation of JpaToscaServiceTemplate for PostgreSQL/MariaDB
Code Block | ||||||
---|---|---|---|---|---|---|
| ||||||
@Entity @Table(name = "ToscaServiceTemplate") @Inheritance(strategy = InheritanceType.TABLE_PER_CLASS) @Data @EqualsAndHashCode(callSuper = false) public class JpaToscaServiceTemplate extends PfConcept implements PfAuthorative<ToscaServiceTemplate> { @Override public ToscaServiceTemplate toAuthorative() { return serviceTemplate.toAuthorative(); } @Override@EmbeddedId public@VerifyKey void fromAuthorative(ToscaServiceTemplate authorativeConcept) {@NotNull private PfConceptKey key; serviceTemplate = new DocToscaServiceTemplate(authorativeConcept); @Lob @Convert(converter = setKey(serviceTemplate.getKey().asIdentifier().asConceptKey());StringToServiceTemplateConverter.class) }@NotNull private DocToscaServiceTemplate serviceTemplate; ------- } |
In the example below the implementation of JpaToscaServiceTemplate for MongoDB
Code Block | ||||||
---|---|---|---|---|---|---|
| ||||||
@Document(collection = "ToscaServiceTemplate") @Inheritance(strategy = InheritanceType.TABLE_PER_CLASS) @Data @EqualsAndHashCode(callSuper = false) public class JpaToscaServiceTemplate extends PfConcept implements PfAuthorative<ToscaServiceTemplate> { @Override public ToscaServiceTemplate toAuthorative() { return serviceTemplate.toAuthorative(); } @Override @Id public void fromAuthorative(ToscaServiceTemplate authorativeConcept) @VerifyKey{ @NonNull serviceTemplate private= PfConceptKey key; new DocToscaServiceTemplate(authorativeConcept); @NonNull private DocToscaServiceTemplate serviceTemplate; ------- @Override public ToscaServiceTemplate toAuthorative() { return serviceTemplate.toAuthorative(setKey(serviceTemplate.getKey().asIdentifier().asConceptKey()); } @Override ------- } |
In the example below the implementation of JpaToscaServiceTemplate for MongoDB
Code Block | ||||||
---|---|---|---|---|---|---|
| ||||||
@Document(collection = "ToscaServiceTemplate") @Inheritance(strategy = InheritanceType.TABLE_PER_CLASS) @Data @EqualsAndHashCode(callSuper = false) public class JpaToscaServiceTemplate extends PfConcept implements PfAuthorative<ToscaServiceTemplate> { public void fromAuthorative(ToscaServiceTemplate authorativeConcept) { serviceTemplate = new DocToscaServiceTemplate(authorativeConcept); @Id setKey(serviceTemplate.getKey().asIdentifier().asConceptKey());@VerifyKey }@NonNull private PfConceptKey key; @NonNull private DocToscaServiceTemplate serviceTemplate; ------- } |
Converters
Jakarta and Spring do no support json Type, but we can use Converters to convert DocToscaServiceTemplate to a Json String.
Code Block | ||||||
---|---|---|---|---|---|---|
| ||||||
@Converter(autoApply = true) public class StringToServiceTemplateConverter implements AttributeConverter<DocToscaServiceTemplate, String> { @Override public ToscaServiceTemplate toAuthorative() { return serviceTemplate.toAuthorative(); private} static final Coder coder = new StandardCoder(); @Override public Stringvoid convertToDatabaseColumnfromAuthorative(DocToscaServiceTemplateToscaServiceTemplate serviceTemplate) { tryauthorativeConcept) { return serviceTemplate == null ? null : coder.encode(serviceTemplatenew DocToscaServiceTemplate(authorativeConcept); } catch (CoderException e) { throw new PfModelRuntimeException(Response.Status.BAD_REQUEST, e.getMessage(), e); setKey(serviceTemplate.getKey().asIdentifier().asConceptKey()); } } ------- } |
Converters
Jakarta and Spring do no support json Type, but we can use Converters to convert DocToscaServiceTemplate to a Json String.
Code Block | ||||||
---|---|---|---|---|---|---|
| ||||||
@Converter(autoApply = true) public class StringToServiceTemplateConverter implements AttributeConverter<DocToscaServiceTemplate, String> { @Override public DocToscaServiceTemplate convertToEntityAttribute(String dbData) { if (dbData == null) { return new DocToscaServiceTemplate(); private static final }Coder coder = new StandardCoder(); @Override public String convertToDatabaseColumn(DocToscaServiceTemplate serviceTemplate) { try { return serviceTemplate == null ? null : coder.decode(dbData, DocToscaServiceTemplate.classencode(serviceTemplate); } catch (CoderException e) { throw new PfModelRuntimeException(Response.Status.BAD_REQUEST, e.getMessage(), e); } } } |
Note: Serialization and deserialization in Json is already used in policy-model (Un example could be found here [JpaToscaPolicy.java]). @Converter is just an elegant way to do the same thing.
ToscaServiceTemplate Table
...
|
...
|
...
|
...
|
...
@Override |
...
|
...
public |
...
DocToscaServiceTemplate convertToEntityAttribute(String dbData) { if (dbData == null) { return new DocToscaServiceTemplate(); } try { return coder.decode(dbData, DocToscaServiceTemplate.class); } catch (CoderException e) { throw new PfModelRuntimeException(Response.Status.BAD_REQUEST, e.getMessage(), e); } } } |
Note: Serialization and deserialization in Json is already used in policy-model (Un example could be found here [JpaToscaPolicy.java]). @Converter is just an elegant way to do the same thing.
ToscaServiceTemplate Table
clampacm=# \d ToscaServiceTemplate
Table "public.toscaservicetemplate"
Column | Type Table "public.toscaservicetemplate"
Column | Type | Collation | Nullable | Default
-----------------+------------------------+-----------+----------+---------
name | character varying(120) | | not null |
version | character varying(20) | | not null |
servicetemplate | text | | |
Indexes:
"toscaservicetemplate_pkey" PRIMARY KEY, btree (name, version)
clampacm=# select * from public.ToscaServiceTemplate;
name | Collation version| Nullable | servicetemplateDefault
-----------------+---+---------+------------+-----
PMSH_Test_Instance | 1.0.0 | 16505
(1 row)
select convert_from(lo_get(servicetemplate::oid), 'UTF8') from toscaservicetemplate;
clampacm=# select convert_from(lo_get(servicetemplate::oid), 'UTF8') from toscaservicetemplate;
{"tosca_definitions_version":"tosca_simple_yaml_1_3", ...
Note
- Using document storage, it involves only the ORM layer, it does not change the functionality of the application
- After migration to document storage as ORM layer, it will possible to adjust flexibility of Tosca Service Template Handling (POLICY-3236)
Example in Policy Admin models
Work in progress ...
JpaToscaWithTypeAndStringProperties class
Code Block | ||||||
---|---|---|---|---|---|---|
| ||||||
@MappedSuperclass
@Data
@EqualsAndHashCode(callSuper = true)
public abstract class JpaToscaWithTypeAndStringProperties<T extends ToscaWithTypeAndObjectProperties>
extends JpaToscaEntityType<T> implements PfAuthorative<T> {
.............................
@Convert(converter = StringToMapConverter.class)
@Column(columnDefinition = "json")
private Map<@NotNull String, @NotNull String> properties;
.............................
} |
Converters
Code Block | ||||||
---|---|---|---|---|---|---|
| ||||||
@Converter(autoApply = true)
public final class StringToMapConverter implements AttributeConverter<Map<String, ? extends Object>, String> {
private static final Coder coder = new StandardCoder();
@Override
public String convertToDatabaseColumn(Map<String, ? extends Object> map) {
try {
return map == null ? null : coder.encode(map);
} catch (CoderException e) {
throw new PfModelRuntimeException(Response.Status.BAD_REQUEST, e.getMessage(), e);
}
}
@Override
public Map<String, ? extends Object> convertToEntityAttribute(String dbData) {
if (dbData == null) {
return Map.of();
}
try {
return coder.decode(dbData, Map.class);
} catch (CoderException e) {
throw new PfModelRuntimeException(Response.Status.BAD_REQUEST, e.getMessage(), e);
}
}
}
|
Table of comparison
...
------+----------+---------
name | character varying(120) | | not null |
version | character varying(20) | | not null |
servicetemplate | text | | |
Indexes:
"toscaservicetemplate_pkey" PRIMARY KEY, btree (name, version)
clampacm=# select * from public.ToscaServiceTemplate;
name | version | servicetemplate
--------------------+---------+-----------------
PMSH_Test_Instance | 1.0.0 | 16505
(1 row)
select convert_from(lo_get(servicetemplate::oid), 'UTF8') from toscaservicetemplate;
clampacm=# select convert_from(lo_get(servicetemplate::oid), 'UTF8') from toscaservicetemplate;
{"tosca_definitions_version":"tosca_simple_yaml_1_3", ...
Proposals
The new ORM level should be additional and optional to the existence one. My propose options are shown below:
- Save ToscaServiceTemplace as Json String in a single Entity:
- ToscaServiceTemplace saved in a Text field as Json
- It is compatible with H2, MariaDB and PostgreSQL
- It is an additional code for policy-models and a medium impact for all applications that are using it
- That solution is compatible with not Spring Applications
- MongoDB/Cassandra
- Document oriented approach full supported by SpringBoot (not needs Converters)
- Compatible only with MongoDB/Cassandra (MongoDB and Cassandra are not compatible to each other)
- It is an additional code for policy-models and a huge impact for all applications that are using it (all repositories and
Proposals
Work in progress ...
My propose options are shown below:
- replace all ElementCollections with Converters:
- Is a fix and not a change of technologies
- Small Json instead of Java binary code
- compatible with H2, MariaDB and PostgreSQL
- it impacts only onap/policy/models, persistence classes used into ElementCollections have to change to a Document oriented classes
- 50% of tables (with postfix as _ATTRIBUTES, _PROPERTIES, _METADATA, _OCCURRENCES, _CONSTRAINTS, _TARGETS, _TRIGGERS and _INPUTS) will be removed
- Unit tests only onap/policy/models
- onap/policy/models still compatible with not Spring Applications
- Unique Json String:
- ToscaServiceTemplace saved as Json
- compatible with H2, MariaDB and PostgreSQL
- it impacts onap/policy/models and all applications related (runtime-acm, api and pap), persistence classes have to change to a Document oriented classes
- 90% of tables will be removed
- Unit tests for onap/policy/models and minor impact for all applications
- onap/policy/models still compatible with not Spring Applications
- MongoDB/Cassandra
- Document oriented approach full supported by SpringBoot (not needs Converters)
- it impacts onap/policy/models and all applications related (runtime-acm, api and pap), and all repositories and persistence classes have to change to a Document oriented classes
- compatible only with MongoDB/Cassandra (MongoDB and Cassandra are not compatible to each other)
- Unit tests need an Embedded Server (example for cassandra: EmbeddedCassandraServerHelper or CassandraContainer)
...
- )
- Unit tests need an Embedded Server (example for cassandra: EmbeddedCassandraServerHelper or CassandraContainer)
- Spring Boot has own annotations for Documents. Eventually for application not in Spring Boot, it needs additional dao style implementation
Note
- Using document storage, it involves only the ORM layer, it does not change the functionality of the application
- After migration to document storage as ORM layer, it will possible to adjust flexibility of Tosca Service Template Handling (POLICY-3236)
Benchmark Performance of runtime-acm
Work in progress ....
In order to generate the benchmark I have used:
...