Page History

...

Using PostgreSQL special type as "json" or "jsonb" rather than "text", all applications will lose any compatibility with H2 and MariaDB. They will be mentioned in this document but not recommended.

Requirements

• Policy types/Policies/Node Types/Node Templates are first order items
• Data Types have a scope of a first order item, so a data type definition only applies in the scope of a policy type or node type definition
• We should keep our current APIs, all changes should be internal
• We must provide an upgrade path to the new data structure and a rollback to the current structure

ORM Layer using Document Storage

ORM layer using document storage (PostgreSQL or MongoDB) could be organized in two layer:

• Document layer (Domain Model to be converted in Json) - implementation has no dependency from DB
• Persistence layer - (Domain Model depend of the DB used): Entities for PostgreSQL, Documents for MongoDB

An implementation on the Document layer could be found here: (https://gerrit.nordix.org/c/onap/policy/models/+/13633)

Example

In the example below DocToscaServiceTemplate should be serialized to Json.

Requirements

• Policy types/Policies/Node Types/Node Templates are first order items
• Data Types have a scope of a first order item, so a data type definition only applies in the scope of a policy type or node type definition
• We should keep our current APIs, all changes should be internal
• We must provide an upgrade path to the new data structure and a rollback to the current structure

ORM Layer using Document Storage

ORM layer using document storage (PostgreSQL or MongoDB) could be organized in two layer:

• Document layer (Domain Model to be converted in Json) - implementation has no dependency from DB
• Persistence layer - (Domain Model depend of the DB used): Entities for PostgreSQL, Documents for MongoDB

An implementation on the Document layer could be found here: (https://gerrit.nordix.org/c/onap/policy/models/+/13633)

Example

In the example below DocToscaServiceTemplate should be serialized to Json.

@Data
@EqualsAndHashCode(callSuper = true)
@NoArgsConstructor
public class DocToscaServiceTemplate extends DocToscaEntity<ToscaServiceTemplate> {

    @SerializedName("data_types")
    private Map<String, DocToscaDataType> dataTypes;

    -------

   public DocToscaServiceTemplate(ToscaServiceTemplate authorativeConcept) {

@Data
@EqualsAndHashCode(callSuper = true)
@NoArgsConstructor
public class DocToscaServiceTemplate extends DocToscaEntity<ToscaServiceTemplate> {

    @SerializedName("data_types")
    private Map<String, DocToscaDataType> dataTypes;

    -------

   public DocToscaServiceTemplate(ToscaServiceTemplate authorativeConcept) {
        this.fromAuthorative(authorativeConcept);
    }

    @Override
    public ToscaServiceTemplate toAuthorative() {
        final var toscaServiceTemplate = new ToscaServiceTemplate();
        super.setToscaEntity(toscaServiceTemplate);
        super.toAuthorative();

        if (dataTypes != null) {
            toscaServiceTemplate.setDataTypes(PfUtils.mapMap(dataTypes, DocToscaDataType::toAuthorative));
        }
     -------
 
        return toscaServiceTemplatethis.fromAuthorative(authorativeConcept);
    }

    @Override
    public voidToscaServiceTemplate fromAuthorativetoAuthorative(ToscaServiceTemplate toscaServiceTemplate) {
        final var toscaServiceTemplate = new ToscaServiceTemplate();
        super.fromAuthorativesetToscaEntity(toscaServiceTemplate);
        super.toAuthorative();

        if (toscaServiceTemplate.getDataTypes()dataTypes != null) {
            dataTypes = DocUtil.mapMap(toscaServiceTemplate.getDataTypes()toscaServiceTemplate.setDataTypes(PfUtils.mapMap(dataTypes, DocToscaDataType::newtoAuthorative));
        }
 
        -------
 
      }
}

In the example below the implementation of JpaToscaServiceTemplate for PostgreSQL/MariaDB

@Entity
@Table(name = "ToscaServiceTemplate")
@Inheritance(strategy = InheritanceType.TABLE_PER_CLASS)
@Data
@EqualsAndHashCode(callSuper = false)
public class JpaToscaServiceTemplate extends PfConcept implements PfAuthorative<ToscaServiceTemplate> {

    @EmbeddedId
    @VerifyKey
    @NotNull
    private PfConceptKey key;

    @Lob
    @Convert(converter = StringToServiceTemplateConverter.class)
    @NotNull
    private DocToscaServiceTemplate serviceTemplate;

     -  return toscaServiceTemplate;
    }

    @Override
    public void fromAuthorative(ToscaServiceTemplate toscaServiceTemplate) {
        super.fromAuthorative(toscaServiceTemplate);
 
        if (toscaServiceTemplate.getDataTypes() != null) {
            dataTypes = DocUtil.mapMap(toscaServiceTemplate.getDataTypes(), DocToscaDataType::new);
        }
 
        ------- 
 
   }
}

In the example below the implementation of JpaToscaServiceTemplate for PostgreSQL/MariaDB

@Entity
@Table(name = "ToscaServiceTemplate")
@Inheritance(strategy = InheritanceType.TABLE_PER_CLASS)
@Data
@EqualsAndHashCode(callSuper = false)
public class JpaToscaServiceTemplate extends PfConcept implements PfAuthorative<ToscaServiceTemplate> { @Override
    public ToscaServiceTemplate toAuthorative() {
        return serviceTemplate.toAuthorative();
    }

    @Override@EmbeddedId
    public@VerifyKey
 void fromAuthorative(ToscaServiceTemplate authorativeConcept) {@NotNull
    private PfConceptKey key;

  serviceTemplate = new DocToscaServiceTemplate(authorativeConcept); @Lob
    @Convert(converter =   setKey(serviceTemplate.getKey().asIdentifier().asConceptKey());StringToServiceTemplateConverter.class)
    }@NotNull
    private DocToscaServiceTemplate serviceTemplate;

     ------- 
 }

In the example below the implementation of JpaToscaServiceTemplate for MongoDB

@Document(collection = "ToscaServiceTemplate")
@Inheritance(strategy = InheritanceType.TABLE_PER_CLASS)
@Data
@EqualsAndHashCode(callSuper = false)
public class JpaToscaServiceTemplate extends PfConcept implements PfAuthorative<ToscaServiceTemplate> {
    @Override
    public ToscaServiceTemplate toAuthorative() {
        return serviceTemplate.toAuthorative();
    }

    @Override
    @Id
public void fromAuthorative(ToscaServiceTemplate authorativeConcept) @VerifyKey{
     @NonNull
   serviceTemplate private= PfConceptKey key;
new DocToscaServiceTemplate(authorativeConcept);
    @NonNull
    private DocToscaServiceTemplate serviceTemplate;

     ------- 
 
    @Override
    public ToscaServiceTemplate toAuthorative() {
        return serviceTemplate.toAuthorative(setKey(serviceTemplate.getKey().asIdentifier().asConceptKey());
    }

    @Override
     ------- 
 }

In the example below the implementation of JpaToscaServiceTemplate for MongoDB

@Document(collection = "ToscaServiceTemplate")
@Inheritance(strategy = InheritanceType.TABLE_PER_CLASS)
@Data
@EqualsAndHashCode(callSuper = false)
public class JpaToscaServiceTemplate extends PfConcept implements PfAuthorative<ToscaServiceTemplate> {
public void fromAuthorative(ToscaServiceTemplate authorativeConcept) {
        serviceTemplate = new DocToscaServiceTemplate(authorativeConcept);
    @Id
    setKey(serviceTemplate.getKey().asIdentifier().asConceptKey());@VerifyKey
    }@NonNull
    private PfConceptKey key;

    @NonNull
    private DocToscaServiceTemplate serviceTemplate;

     ------- 
 
    }

Converters

Jakarta and Spring do no support json Type, but we can use Converters to convert DocToscaServiceTemplate  to a Json String.

@Converter(autoApply = true)
public class StringToServiceTemplateConverter implements AttributeConverter<DocToscaServiceTemplate, String> {
@Override
    public ToscaServiceTemplate toAuthorative() {
        return serviceTemplate.toAuthorative();
    private}

 static final Coder coder = new StandardCoder();

    @Override
    public Stringvoid convertToDatabaseColumnfromAuthorative(DocToscaServiceTemplateToscaServiceTemplate serviceTemplate) {
        tryauthorativeConcept) {
            return serviceTemplate == null ? null : coder.encode(serviceTemplatenew DocToscaServiceTemplate(authorativeConcept);
        } catch (CoderException e) {
            throw new PfModelRuntimeException(Response.Status.BAD_REQUEST, e.getMessage(), e);
    setKey(serviceTemplate.getKey().asIdentifier().asConceptKey());
    }
    }

     ------- 
 }

Converters

Jakarta and Spring do no support json Type, but we can use Converters to convert DocToscaServiceTemplate  to a Json String.

@Converter(autoApply = true)
public class StringToServiceTemplateConverter implements AttributeConverter<DocToscaServiceTemplate, String> {
@Override
    public DocToscaServiceTemplate convertToEntityAttribute(String dbData) {
        if (dbData == null) {
            return new DocToscaServiceTemplate();
    private static final  }Coder coder = new StandardCoder();

    @Override
    public String convertToDatabaseColumn(DocToscaServiceTemplate serviceTemplate) {
        try {
            return serviceTemplate == null ? null : coder.decode(dbData, DocToscaServiceTemplate.classencode(serviceTemplate);
        } catch (CoderException e) {
            throw new PfModelRuntimeException(Response.Status.BAD_REQUEST, e.getMessage(), e);
        }
    }
}

Note: Serialization and deserialization in Json is already used in policy-model (Un example could be found here [JpaToscaPolicy.java]). @Converter is just an elegant way to do the same thing.

ToscaServiceTemplate Table

...

...

...

...

...

@Override
   

...

...

public 

...

DocToscaServiceTemplate convertToEntityAttribute(String dbData) {
        if (dbData == null) {
            return new DocToscaServiceTemplate();
        }
        try {
            return coder.decode(dbData, DocToscaServiceTemplate.class);
        } catch (CoderException e) {
            throw new PfModelRuntimeException(Response.Status.BAD_REQUEST, e.getMessage(), e);
        }
    }
}

Note: Serialization and deserialization in Json is already used in policy-model (Un example could be found here [JpaToscaPolicy.java]). @Converter is just an elegant way to do the same thing.

ToscaServiceTemplate Table

clampacm=# \d ToscaServiceTemplate
                    Table "public.toscaservicetemplate"
     Column      |          Type          Table "public.toscaservicetemplate"
     Column      |          Type          | Collation | Nullable | Default
-----------------+------------------------+-----------+----------+---------
 name            | character varying(120) |           | not null |
 version         | character varying(20)  |           | not null |
 servicetemplate | text                   |           |          |
Indexes:
    "toscaservicetemplate_pkey" PRIMARY KEY, btree (name, version)
clampacm=# select * from public.ToscaServiceTemplate;
        name        | Collation version| Nullable | servicetemplateDefault
-----------------+---+---------+------------+-----
 PMSH_Test_Instance | 1.0.0   | 16505
(1 row)
 select convert_from(lo_get(servicetemplate::oid), 'UTF8') from toscaservicetemplate;
clampacm=# select convert_from(lo_get(servicetemplate::oid), 'UTF8') from toscaservicetemplate;
 {"tosca_definitions_version":"tosca_simple_yaml_1_3", ...

Note

• Using document storage, it involves only the ORM layer, it does not change the functionality of the application
• After migration to document storage as ORM layer,  it will possible to adjust flexibility of Tosca Service Template Handling (POLICY-3236)

Example in Policy Admin models

Work in progress ...

JpaToscaWithTypeAndStringProperties class

@MappedSuperclass
@Data
@EqualsAndHashCode(callSuper = true)
public abstract class JpaToscaWithTypeAndStringProperties<T extends ToscaWithTypeAndObjectProperties>
        extends JpaToscaEntityType<T> implements PfAuthorative<T> {

    .............................

     @Convert(converter = StringToMapConverter.class)
     @Column(columnDefinition = "json")
     private Map<@NotNull String, @NotNull String> properties;

     .............................
 }

Converters

@Converter(autoApply = true)
public final class StringToMapConverter implements AttributeConverter<Map<String, ? extends Object>, String> {

    private static final Coder coder = new StandardCoder();

    @Override
    public String convertToDatabaseColumn(Map<String, ? extends Object> map) {
        try {
            return map == null ? null : coder.encode(map);
        } catch (CoderException e) {
            throw new PfModelRuntimeException(Response.Status.BAD_REQUEST, e.getMessage(), e);
        }
    }

    @Override
    public Map<String, ? extends Object> convertToEntityAttribute(String dbData) {
        if (dbData == null) {
            return Map.of();
        }
        try {
            return coder.decode(dbData, Map.class);
        } catch (CoderException e) {
            throw new PfModelRuntimeException(Response.Status.BAD_REQUEST, e.getMessage(), e);
        }
    }
}

Table of comparison

...

------+----------+---------
 name            | character varying(120) |           | not null |
 version         | character varying(20)  |           | not null |
 servicetemplate | text                   |           |          |
Indexes:
    "toscaservicetemplate_pkey" PRIMARY KEY, btree (name, version)


clampacm=# select * from public.ToscaServiceTemplate;
        name        | version | servicetemplate
--------------------+---------+-----------------
 PMSH_Test_Instance | 1.0.0   | 16505
(1 row)
 select convert_from(lo_get(servicetemplate::oid), 'UTF8') from toscaservicetemplate;


clampacm=# select convert_from(lo_get(servicetemplate::oid), 'UTF8') from toscaservicetemplate;
 {"tosca_definitions_version":"tosca_simple_yaml_1_3", ...

Proposals

The new ORM level should be additional and optional to the existence one. My propose options are shown below:

1. Save ToscaServiceTemplace as Json String in a single Entity:
   • ToscaServiceTemplace saved in a Text field as Json
   • It is compatible with H2, MariaDB and PostgreSQL
   • It is an additional code for policy-models and a medium impact for all applications that are using it
   • That solution is compatible with not Spring Applications
2. MongoDB/Cassandra
   • Document oriented approach full supported by SpringBoot (not needs Converters)
   • Compatible only with MongoDB/Cassandra (MongoDB and Cassandra are not compatible to each other)
   • It is an additional code for policy-models and a huge impact for all applications that are using it (all repositories and

Proposals

Work in progress ...

My propose options are shown below:

1. replace all ElementCollections with Converters:
   • Is a fix and not a change of technologies
   • Small Json instead of Java binary code
   • compatible with H2, MariaDB and PostgreSQL
   • it impacts only onap/policy/models, persistence classes used into ElementCollections have to change to a Document oriented classes
   • 50% of tables (with postfix as _ATTRIBUTES, _PROPERTIES, _METADATA, _OCCURRENCES, _CONSTRAINTS, _TARGETS, _TRIGGERS and _INPUTS) will be removed
   • Unit tests only onap/policy/models
   • onap/policy/models still compatible with not Spring Applications
2. Unique Json String:
   • ToscaServiceTemplace saved as Json
   • compatible with H2, MariaDB and PostgreSQL
   • it impacts onap/policy/models and all applications related (runtime-acm, api and pap), persistence classes have to change to a Document oriented classes
   • 90% of tables will be removed
   • Unit tests for onap/policy/models and minor impact for all applications
   • onap/policy/models still compatible with not Spring Applications
3. MongoDB/Cassandra
   • Document oriented approach full supported by SpringBoot (not needs Converters)
   • it impacts onap/policy/models and all applications related (runtime-acm, api and pap), and all repositories and persistence classes have to change to a Document oriented classes
   • compatible only with MongoDB/Cassandra (MongoDB and Cassandra are not compatible to each other)
   • Unit tests need an Embedded Server (example for cassandra: EmbeddedCassandraServerHelper or CassandraContainer)

...

1. • )
   • Unit tests need an Embedded Server (example for cassandra: EmbeddedCassandraServerHelper or CassandraContainer)
   • Spring Boot has own annotations for Documents. Eventually for application not in Spring Boot, it needs additional dao style implementation

Note

• Using document storage, it involves only the ORM layer, it does not change the functionality of the application
• After migration to document storage as ORM layer,  it will possible to adjust flexibility of Tosca Service Template Handling (POLICY-3236)

Benchmark Performance of runtime-acm

Work in progress ....

In order to generate the benchmark I have used:

...