...
- Convert all applications to document storage storage
- performance improvements
- Keep our current APIs, all changes will be internal
- Provide an upgrade path to the new data structure and a rollback to the current structure
- Old ORM layer will be not removed during this step
- namespace and imports could be defined but not implemented
- create more the one service template (Multi-service templates) is initially disabled in ORM layer
- Old ORM layer will be not removed during this step
- Add multi-service templates support (related to POLICY-3236 - adjust flexibility of Tosca Service Template Handling)
- Remove old ORM layer
- Enable namespace and Multi-service templates in ORM layer
- Adapt the current APIs to handle Multi-service templates where it needs
- move common code in policy-model
- Define and support complex solutionsany other complex features not implemented yet
- Enable import support support update of a common service template (imported by other service template)
Investigation about PostgreSQL and json types
...
Code Block | ||||||
---|---|---|---|---|---|---|
| ||||||
@Data @EqualsAndHashCode(callSuper = true) public class DocToscaServiceTemplate extends DocToscaEntity<ToscaServiceTemplate> { @SerializedName("data_types") private Map<String, @Valid DocToscaDataType> dataTypes; ------- public DocToscaServiceTemplate(ToscaServiceTemplate authorativeConcept) { this.fromAuthorative(authorativeConceptsuper(); } @Override setName(DEFAULT_NAME); public ToscaServiceTemplate toAuthorativesetVersion(DEFAULT_VERSION); { ------- final var} toscaServiceTemplate = new public DocToscaServiceTemplate(ToscaServiceTemplate( authorativeConcept); { superthis.setToscaEntityfromAuthorative(toscaServiceTemplateauthorativeConcept); } public super.toAuthorative(); DocToscaServiceTemplate(final DocToscaServiceTemplate copyConcept) { if (dataTypes != null) { super(copyConcept); this.toscaDefinitionsVersion = copyConcept.toscaDefinitionsVersion; toscaServiceTemplate.setDataTypes(this.dataTypes = DocUtil.docMapMapdocMapToMap(copyConcept.dataTypes, DocToscaDataType::toAuthorativenew, new LinkedHashMap<>()); } ------- return toscaServiceTemplate; } } @Override public voidToscaServiceTemplate fromAuthorativetoAuthorative(ToscaServiceTemplate toscaServiceTemplate) { super.fromAuthorative(toscaServiceTemplatefinal var toscaServiceTemplate = new ToscaServiceTemplate(); if super.setToscaEntity(toscaServiceTemplate.getDataTypes(); != null) { super.toAuthorative(); dataTypes = DocUtiltoscaServiceTemplate.mapDocMapsetDataTypes(toscaServiceTemplateDocUtil.getDataTypesdocMapToMap()dataTypes, DocToscaDataType::newtoAuthorative)); } ------- ------- return toscaServiceTemplate; } } |
In the example below the implementation of JpaToscaServiceTemplate for PostgreSQL/MariaDB (full implementation could be found here: https://gerrit.nordix.org/c/onap/policy/clamp/+/13642)
Code Block | ||||||
---|---|---|---|---|---|---|
| ||||||
@Entity @Table(name = "ToscaServiceTemplate") @Inheritance(strategy = InheritanceType.TABLE_PER_CLASS) @Data @EqualsAndHashCode(callSuper = false) public class JpaToscaServiceTemplate extends PfConcept implements PfAuthorative<ToscaServiceTemplate> { @EmbeddedId @VerifyKey @NotNull private PfConceptKey key; @Lob @Convert(converter = StringToServiceTemplateConverter.class) @NotNull @Valid private DocToscaServiceTemplate serviceTemplate; @Override public void fromAuthorative(ToscaServiceTemplate toscaServiceTemplate) { super.fromAuthorative(toscaServiceTemplate); if (toscaServiceTemplate.getDataTypes() != null) { dataTypes = DocUtil.mapDocMap(toscaServiceTemplate.getDataTypes(), DocToscaDataType::new); } ------- } @Override public ToscaServiceTemplateint toAuthorativecompareTo(DocToscaEntity<ToscaServiceTemplate> otherConcept) { return serviceTemplate.toAuthorative(int result = compareToWithoutEntities(otherConcept); } @Override if (result != 0) public{ void fromAuthorative(ToscaServiceTemplate authorativeConcept) { serviceTemplatereturn =result; new DocToscaServiceTemplate(authorativeConcept); } final DocToscaServiceTemplate other = setKey(serviceTemplate.getKey().asIdentifier().asConceptKey()); }DocToscaServiceTemplate) otherConcept; ------- return ObjectUtils.compare(toscaTopologyTemplate, other.toscaTopologyTemplate); } } |
In the example below the implementation of JpaToscaServiceTemplate for MongoDB PostgreSQL/MariaDB (full implementation could be found here: https://gerrit.nordix.org/c/onap/policy/clamp/+/1361513642)
Code Block | ||||||
---|---|---|---|---|---|---|
| ||||||
@Document@Entity @Table(collectionname = "ToscaServiceTemplate") @Inheritance(strategy = InheritanceType.TABLE_PER_CLASS) @Data @EqualsAndHashCode(callSuper = false) public class JpaToscaServiceTemplate extends PfConcept implements PfAuthorative<ToscaServiceTemplate> { @Id@EmbeddedId @VerifyKey @NonNull@NotNull private PfConceptKey key; @NonNull@Lob @Convert(converter = StringToServiceTemplateConverter.class) @NotNull @Valid private DocToscaServiceTemplate serviceTemplate; ------- @Override public ToscaServiceTemplate toAuthorative() { return serviceTemplate.toAuthorative(); } @Override public void fromAuthorative(ToscaServiceTemplate authorativeConcept) { serviceTemplate = new DocToscaServiceTemplate(authorativeConcept); setKey(serviceTemplate.getKey().asIdentifier().asConceptKey()); } ------- } |
Converters
Jakarta and Spring do no support json Type, but we can use Converters to convert DocToscaServiceTemplate to a Json String.
In the example below the implementation of JpaToscaServiceTemplate for MongoDB (full implementation could be found here: https://gerrit.nordix.org/c/onap/policy/clamp/+/13615)
Code Block | ||||||
---|---|---|---|---|---|---|
| ||||||
@Converter(autoApply@Document(collection = "ToscaServiceTemplate") @Inheritance(strategy = InheritanceType.TABLE_PER_CLASS) @Data @EqualsAndHashCode(callSuper = truefalse) public class JpaToscaServiceTemplate extends StringToServiceTemplateConverterPfConcept implements AttributeConverter<DocToscaServiceTemplate,PfAuthorative<ToscaServiceTemplate> String> { private@Id static final Coder coder = new StandardCoder(); @VerifyKey @Override@NonNull public String convertToDatabaseColumn(DocToscaServiceTemplate serviceTemplate) { private PfConceptKey key; @NonNull @Valid private tryDocToscaServiceTemplate {serviceTemplate; ------- @Override return serviceTemplate == nullpublic ? null : coder.encode(serviceTemplate);ToscaServiceTemplate toAuthorative() { } catch (CoderException e) {return serviceTemplate.toAuthorative(); } @Override throwpublic newvoid PfModelRuntimeException(Response.Status.BAD_REQUEST, e.getMessage(), e);fromAuthorative(ToscaServiceTemplate authorativeConcept) { } serviceTemplate } @Override public DocToscaServiceTemplate convertToEntityAttribute(String dbData) {= new DocToscaServiceTemplate(authorativeConcept); if (dbData == null) {setKey(serviceTemplate.getKey().asIdentifier().asConceptKey()); } return ------- } |
Converters
Jakarta and Spring do no support json Type, but we can use Converters to convert DocToscaServiceTemplate to a Json String.
Code Block | ||||||
---|---|---|---|---|---|---|
| ||||||
@Converter(autoApply = true) public class StringToServiceTemplateConverter implements AttributeConverter<DocToscaServiceTemplate, String> { private static final Coder coder = new StandardCoder(); @Override public String convertToDatabaseColumn(DocToscaServiceTemplate serviceTemplate) { try { return serviceTemplate == null ? null : coder.encode(serviceTemplatenew DocToscaServiceTemplate(); } try { return coder.decode(dbData, DocToscaServiceTemplate.class); } catch (CoderException e) { throw new PfModelRuntimeException(Response.Status.BAD_REQUEST, e.getMessage(), e); } } } |
Note: Serialization and deserialization in Json is already used in policy-model (Un example could be found here [JpaToscaPolicy.java]). @Converter is just an elegant way to do the same thing.
ToscaServiceTemplate Table
@Override
public DocToscaServiceTemplate convertToEntityAttribute(String dbData) {
if (dbData == null) {
return new DocToscaServiceTemplate();
}
try {
return coder.decode(dbData, DocToscaServiceTemplate.class);
} catch (CoderException e) {
throw new PfModelRuntimeException(Response.Status.BAD_REQUEST, e.getMessage(), e);
}
}
}
|
Note: Serialization and deserialization in Json is already used in policy-model (Un example could be found here [JpaToscaPolicy.java]). @Converter is just an elegant way to do the same thing.
ToscaServiceTemplate Table
clampacm=# \d ToscaServiceTemplate
clampacm=# \d ToscaServiceTemplate
Table "public.toscaservicetemplate"
Column | Type | Collation | Nullable | Default
-----------------+------------------------+-----------+----------+---------
name | character varying(120) | | not null |
version | character varying(20) | | not null |
servicetemplate | text | | |
Indexes:
"toscaservicetemplate_pkey" PRIMARY KEY, btree (name, version)
clampacm=# select * from public.ToscaServiceTemplate;
Table "public.toscaservicetemplate"
Column | Type name | Collation | versionNullable | servicetemplateDefault
-----------------+------------------------+-----------+----------+---------
PMSH_Test_Instance | 1.0.0 name | 16505
(1 row)
clampacm=# select convert_from(lo_get(servicetemplate::oid), 'UTF8') from toscaservicetemplate;
{"tosca_definitions_version":"tosca_simple_yaml_1_3", ...
Proposals
The new ORM layer will be implemented into an additional package and the old layer will be not changed. My propose options are shown below:
character varying(120) | | not null |
version | character varying(20) | | not null |
servicetemplate | text | | |
Indexes:
"toscaservicetemplate_pkey" PRIMARY KEY, btree (name, version)
clampacm=# select * from public.ToscaServiceTemplate;
name | version | servicetemplate
--------------------+---------+-----------------
PMSH_Test_Instance | 1.0.0 | 16505
(1 row)
clampacm=# select convert_from(lo_get(servicetemplate::oid), 'UTF8') from toscaservicetemplate;
{"tosca_definitions_version":"tosca_simple_yaml_1_3", ...
Proposals
The new ORM layer will be implemented into an additional package and the old layer will be not changed. My propose options are shown below:
- Save ToscaServiceTemplace as Json String in a single Entity:
- Each Service Template is stored as a JSON "LOB"
- It is compatible with H2, MariaDB and PostgreSQL
- It is an additional code for policy-models and a medium impact for all applications that are using it
- That solution is compatible with Applications not implemented in Spring
- MongoDB/Cassandra
- Document oriented approach full supported by SpringBoot (not needs Converters)
- Compatible only with MongoDB/Cassandra (MongoDB and Cassandra are not compatible to each other)
- It is an additional code for policy-models and a huge impact for all applications that are using it (all repositories and persistence classes have to change to a Document oriented classes)
- Unit tests need
- Save ToscaServiceTemplace as Json String in a single Entity:
- Each Service Template is stored as a JSON "LOB"
- It is compatible with H2, MariaDB and PostgreSQL
- It is an additional code for policy-models and a medium impact for all applications that are using it
- That solution is compatible with not Spring Applications
- MongoDB/Cassandra
- Document oriented approach full supported by SpringBoot (not needs Converters)
- Compatible only with MongoDB/Cassandra (MongoDB and Cassandra are not compatible to each other)
- It is an additional code for policy-models and a huge impact for all applications that are using it (all repositories and persistence classes have to change to a Document oriented classes)
- Unit tests need an Embedded Server (example for cassandra: EmbeddedCassandraServerHelper or CassandraContainer)
- Spring Boot has own annotations for Documents. Eventually for application not in Spring Boot, it needs additional dao style implementation
...
id cannot have dot '.' in MongoDB : solved with minimal configuration
Implementation (First step of
...
Roadmap)
Current ORM layer for ToscaServiceTemplate
...
ORM layer for ToscaServiceTemplate Proposed (during first step in Roadmap)
Simple-concepts entities will be maintained but not used.
Namespace and Import service templates
...
(second and third step in Roadmap)
In in TOSCA language namespace is supposed to be a URI. Name and version of service template, are not defined in TOSCA language, not used in yaml files, and not present in all examples neither Unit Tests, but they are used as primary key of the ToscaServiceTemplate table and they are used in REST endpoints as id of a resource.
...
Validation in current ORM layer
As the first step of Roadmap is backward compatible, the current validation will be maintained as it is, but it should be adapted to new DocToscaEntities. In the second step do not need other changes.
"type" and "type_version" | Referenced to |
---|---|
ToscaProperty | ToscaDataType |
ToscaPolicy | ToscaPolicyType |
ToscaNodeTemplate | ToscaNodeType |
"type_version" is optional and "0.0.0" is the default value. The "Key" is used for the validation to find if the ToscaEntity exists.
Examples:
Example | Key |
---|---|
type: onap.datatypes.ToscaConceptIdentifier | onap.datatypes.ToscaConceptIdentifier:0.0.0 |
type: org.onap.policy.clamp.acm.PolicyAutomationCompositionElement | org.onap.policy.clamp.acm.PolicyAutomationCompositionElement:1.0.0 |
...
Example | Key |
---|---|
derivedFrom: onap.datatypes.ToscaConceptIdentifier | onap.datatypes.ToscaConceptIdentifier (any version):{last_version} |
derivedFrom: org.onap.policy.clamp.acm.PolicyAutomationCompositionElement:1.0.0 | org.onap.policy.clamp.acm.PolicyAutomationCompositionElement:1.0.0 |
Validation
...
"type", "type_version" and "namespace"
...
in third step
In the last step, due the introduction of imports and namespace support, it needs to extend the validation.
"type_version" is optional and "0.0.0" is the default value, "host_namespace" is optional and "tosca" is the default value, type could be used as formatted string: {host_namespace}:{name}:{version}. The "Key" is used for the validation to find if the ToscaEntity exists in same ServiceTemplate or in other one.
...
Example | Key (if defined in same service template) | Key (for external service template) |
---|---|---|
type: onap.datatypes.ToscaConceptIdentifier | "onap.datatypes.ToscaConceptIdentifier:0.0.0" | "tosca:onap.datatypes.ToscaConceptIdentifier:0.0.0" |
type: org.onap.policy.clamp.acm.PolicyAutomationCompositionElement | "org.onap.policy.clamp.acm.PolicyAutomationCompositionElement:1.0.0" | "tosca:org.onap.policy.clamp.acm.PolicyAutomationCompositionElement:1.0.0" |
type: onap.datatype.acm.Target:1.2.3 | "onap.datatype.acm.Target:1.2.3" | "tosca:onap.datatype.acm.Target:1.2.3" |
type: CustomNamespace:onap.datatype.acm.Operation:1.0.1 | "onap.datatype.acm.Operation:1.0.1" | "CustomNamespace:onap.datatype.acm.Operation:1.0.1" |
"derivedFrom" should have same logic as before and it could be used as formatted string: {host_namespace}:{name}:{version}.
Example | Key (if defined in same service template) | Key (for external service template) |
---|---|---|
derivedFrom: onap.datatypes.ToscaConceptIdentifier | "onap.datatypes.ToscaConceptIdentifier:0.0.0"{last_version} | "tosca:onap.datatypes.ToscaConceptIdentifier:0.0.0"{last_version} |
derivedFrom: onap.datatype.acm.Target:1.2.3 | "onap.datatype.acm.Target:1.2.3" | "tosca:onap.datatype.acm.Target:1.2.3" |
derivedFrom: CustomNamespace:onap.datatype.acm.Operation:1.0.1 | "onap.datatype.acm.Operation:1.0.1" | "CustomNamespace:onap.datatype.acm.Operation:1.0.1" |
Update functionality (second and third step of Roadmap)
There are two kind of update:
- values (example "description", "metadata", ...)
- data structure (example Create or Delete of a ToscaEntity, or Update a reference as "type", "type_version" or "host_namespace")
Open questions about participants
Policy-Api
Enabling namespace and multi-service templates
Current Api in policy-api | Api in policy-api with namespace support |
---|---|
POST /nodetemplates GET /policytypes | POST /servicetemplates/{st_name}/versions/{st_version}/nodetemplates GET /servicetemplates/{st_name}/versions/{st_version}/policytypes |
Update functionality (second and third step of Roadmap)
There are two kind of update:
- values (example "description", "metadata", ...)
- data structure (example Create or Delete of a ToscaEntity, or Update a reference as "type", "type_version" or "host_namespace"
- Case scenario: we have two custom service template, a common service template and two automation composition. participant-policy creates and deletes policies and policy types connecting to policy-api.
- Could we have that scenario?
- Should participant-policy receive a full service template (custom and common service template)?
- Should participant policy collect all policies and policy types by GET policy-api, to know if them are already created?
- How participant policy know if can delete a policy and policy type if it could be used to other automation composition?
- Case scenario: we have two custom service template. A participant is defined in only one of them.
- Could we have that scenario? Should participant receive only the custom service template related? (A table to save relations between service templates and participants)
Proposals
- We could consider Service templates references to each other as a DAG ( directed acyclic graph), where Service templates are nodes, and a references as edges. Cyclic references are not allowed.
- Should be available a functionality to load a full version of the service template that contains own tosca entities and also all tosca entities referenced from other service templates (this functionality in for validation or business logic purpose and will be not visible to the end user)
- A service template cannot be deleted if there are references to it
- Any Tosca Entity in a common service template cannot be deleted if there are references to that service template
- Any Tosca Entity in a common service template that contains own tosca entities and also all tosca entities referenced from other service templates (this functionality in for validation or business logic purpose and will be not visible to the end user)
- A service template cannot be deleted if there are references to it
- cannot change "host_namespace", "type" and "type_version" if there are references to that service template
- Any Tosca Entity in a common service template cannot change values if there are automation compositions referenced to that service template (and references to that service template?)
- A Tosca Entity cannot be add Any Tosca Entity in a common service template cannot be deleted if there are references automation compositions referenced to that service template
- Any Tosca Entity in a common service template cannot change "host_namespace", "type" and "type_version" if there are references to that service template
- Any Tosca Entity in a common service template cannot change values if there are automation compositions referenced to that service template (and references to that service template?)
- A Tosca Entity cannot be add in a common service template if there are automation compositions referenced to that service template
- Functionality as create, update and delete of a ToscaEntity in a service template should be validated and restricted to the service template itself. (Example: update property values of a service template should never update property values from other service templates referenced)
- To reduce the complexity, could be useful to save additional information about the service template:
- A full version of the service template that contains own tosca entities and also all tosca entities referenced from other service templates. it will be used in read-only for business logic purpose
- A table to save relations between service templates
- A table to save relations between service templates and automation compositions (with all service templates referenced)
Conclusion
- Functionality as create, update and delete of a ToscaEntity in a service template should be validated and restricted to the service template itself. (Example: update property values of a service template should never update property values from other service templates referenced)
- To reduce the complexity, could be useful to save additional information about the service template:
- A full version of the service template that contains own tosca entities and also all tosca entities referenced from other service templates. it will be used in read-only for business logic purpose
- A table to save relations between service templates
- A table to save relations between service templates and automation compositions (with all service templates referenced)
Conclusion
- Move to document storage does not change the functionality of the application. Example if clampacm move to document storage, and policyadmin not yet, those applications will continue to work fine.
- Enable namespace and enable to handle more than one service template, impacts functionalities and applications dependency. Example if clampacm has namespace enabled, and policyadmin not yet, due the dependency, they could have some issues.
Document storage Optional?: using SpringBoot profiles and interfaces it could be possible to have a double implementation and run only one, (based on parameter in properties file), but it does not make sense due the second point.
Work in progress