The goal goals of this document is to investigate about the PostgreSQL capability to handle Json documentsdocument storage, what is the impact in the performance point of view, how it could be used to adjust flexibility of Tosca Service Template Handling, and define a clear roadmap how to reach all those goals.
Table of Contents maxLevel 2
Introduction
PostgreSQL offers two types for storing JSON data: json
and jsonb. T
he json
data type stores an exact copy of the input text, which processing functions must reparse on each execution; while jsonb
data is stored in a decomposed binary format that makes it slightly slower to input due to added conversion overhead, but significantly faster to process, since no reparsing is needed. jsonb
also supports indexing, which can be a significant advantage.
Because the json
type stores an exact copy of the input text, it will preserve semantically-insignificant white space between tokens, as well as the order of keys within JSON objects. Also, if a JSON object within the value contains the same key more than once, all the key/value pairs are kept. (The processing functions consider the last value as the operative one.) By contrast, jsonb
does not preserve white space, does not preserve the order of object keys, and does not keep duplicate object keys. If duplicate keys are specified in the input, only the last value is kept.
In general, most applications should prefer to store JSON data as jsonb
, unless there are quite specialized needs, such as legacy assumptions about ordering of object keys. (https://www.postgresql.org/docs/current/datatype-json.html)
Demo Application
To better understand these functionality, it has been created a Demo Application. That it is available here (https://gerrit.nordix.org/c/local/onap/clamp-pilot/+/11966).
Data Transfer Object
The Java code shown below, is the data transfer Object used by Rest Api Controller. (Validations and Swagger annotations have been removed for simplicity).
Code Block | ||||||
---|---|---|---|---|---|---|
| ||||||
public class ElementDto {
private String name;
private List<String> tags;
..........
}
public class DocumentDto {
private String name;
private Map<String, ElementDto> elements;
..........
}
public class ItemDto {
private String key;
private String text;
private Map<String, String> properties;
private DocumentDto document;
..........
} |
The Json shown below is an example of the model shown before.
There are different approaches that could be used to Implement a document storage: convert a the concept model to Json text or using a database document oriented. A service template stored as document, it will make simple to have more than one service template into a db.
Also it will be possible to implement a more complex solution using namespace and imports.
Roadmap
- Convert all applications to document storage
- performance improvements
- Keep our current APIs, all changes will be internal
- Provide an upgrade path to the new data structure and a rollback to the current structure
- Old ORM layer will be not removed during this step
- namespace and imports could be defined but not implemented
- create more the one service template (Multi-service templates) is initially disabled in ORM layer
- Add multi-service templates support (related to POLICY-3236 - adjust flexibility of Tosca Service Template Handling)
- Remove old ORM layer
- Enable namespace and Multi-service templates in ORM layer
- Adapt the current APIs to handle Multi-service templates where it needs
- move common code in policy-model
- Define and support any other complex features not implemented yet
- Enable import support
Investigation about PostgreSQL and json types
PostgreSQL offers two types for storing JSON data: json
and jsonb. T
he json
data type stores an exact copy of the input text, which processing functions must reparse on each execution; while jsonb
data is stored in a decomposed binary format that makes it slightly slower to input due to added conversion overhead, but significantly faster to process, since no reparsing is needed. jsonb
also supports indexing, which can be a significant advantage.
Because the json
type stores an exact copy of the input text, it will preserve semantically-insignificant white space between tokens, as well as the order of keys within JSON objects. Also, if a JSON object within the value contains the same key more than once, all the key/value pairs are kept. (The processing functions consider the last value as the operative one.) By contrast, jsonb
does not preserve white space, does not preserve the order of object keys, and does not keep duplicate object keys. If duplicate keys are specified in the input, only the last value is kept.
Note
Using PostgreSQL special type as "json" or "jsonb" rather than "text", has as consequence that all applications will lose any compatibility with H2 and MariaDB, so that solution is not recommended. The table below shows the compatibility by type.
Type | DataStore | Support Query | Index | Preserve Order |
---|---|---|---|---|
longtext | MariaDB | Yes | No | Yes |
PostgreSQL | No | No | Yes | |
H2 | No | No | Yes | |
json | PostgreSQL | Yes | No | Yes |
jsonb | PostgreSQL | Yes | Yes | No |
ORM Layer using Document Storage
ORM layer using document storage (PostgreSQL or MongoDB) could be organized in two layer:
- Document layer (Domain Model to be converted in Json) - implementation has no dependency from DB
- Persistence layer - (Domain Model depend of the DB used): Entities for PostgreSQL, Documents for MongoDB
An implementation on the Document layer could be found here: https://gerrit.nordix.org/c/onap/policy/models/+/13633
Example
In the example below DocToscaServiceTemplate should be serialized to Json.
Code Block | ||||||
---|---|---|---|---|---|---|
| ||||||
@Data
@EqualsAndHashCode(callSuper = true)
public class DocToscaServiceTemplate extends DocToscaEntity<ToscaServiceTemplate> {
@SerializedName("data_types")
private Map<String, @Valid DocToscaDataType> dataTypes;
-------
public DocToscaServiceTemplate() {
super();
setName(DEFAULT_NAME);
setVersion(DEFAULT_VERSION);
-------
}
public DocToscaServiceTemplate(ToscaServiceTemplate authorativeConcept) {
this.fromAuthorative(authorativeConcept);
}
public DocToscaServiceTemplate(final DocToscaServiceTemplate copyConcept) | ||||||
Code Block | ||||||
| ||||||
{ "key": "1", "text": "This is an example", "properties": { "additionalProp1": "Val 10", "additionalProp2": "11", "additionalProp3": "12" }, "document": { "name": "DocumentExample", "elements": { "additionalProp1": { "name": "ElementExample 1", "tags": [ "tag1" ] }, "additionalProp2": { "name": "ElementExample 2", super(copyConcept); this.toscaDefinitionsVersion = "tags": [ copyConcept.toscaDefinitionsVersion; this.dataTypes = DocUtil.docMapToMap(copyConcept.dataTypes, "tag2"DocToscaDataType::new, new LinkedHashMap<>()); ]------- }, "additionalProp3": { @Override public ToscaServiceTemplate toAuthorative() "name": "ElementExample 3", { final var toscaServiceTemplate = "tags": [new ToscaServiceTemplate(); "tag3"super.setToscaEntity(toscaServiceTemplate); ]super.toAuthorative(); } toscaServiceTemplate.setDataTypes(DocUtil.docMapToMap(dataTypes, } } } |
Domain model
To store data on PostgreSQL we still have to use Entities, but part of Dto classes could be stored as jsonb type or at least we can create new domain classes that are equal to those DTO classes.
In this example we can use just one class as persistence model as shown below.
Code Block | ||||||
---|---|---|---|---|---|---|
| ||||||
@Entity
@Data
public class Item implements Serializable {
private static final long serialVersionUID = 1L;
@Id
private String key;
@Column(name = "text", nullable = false)
private String text;
@Convert(converter = StringToMapConverter.class)
@Column(columnDefinition = "jsonb", nullable = false)
private Map<String, String> properties;
@Convert(converter = StringToDocumentConverter.class)
@Column(columnDefinition = "jsonb", nullable = false)
@NotNull
private DocumentDto document;
} |
Converters
Jakarta and Spring do no support json Type, but we can use Converters to convert an Object to a String. The more generic is a converter, more we can reuse it.
Code Block | ||||||
---|---|---|---|---|---|---|
| ||||||
@Slf4j @Converter(autoApply = true) public final class StringToMapConverter implements AttributeConverter<Map<String, ? extends Object>, String> { private static final ObjectMapper MAPPER = new ObjectMapper(); @Override public String convertToDatabaseColumn(Map<String, ? extends Object> map) { try {DocToscaDataType::toAuthorative)); ------- return toscaServiceTemplate; } @Override public void fromAuthorative(ToscaServiceTemplate toscaServiceTemplate) { super.fromAuthorative(toscaServiceTemplate); if (toscaServiceTemplate.getDataTypes() != null) { dataTypes = DocUtil.mapDocMap(toscaServiceTemplate.getDataTypes(), DocToscaDataType::new); } ------- } @Override public int compareTo(DocToscaEntity<ToscaServiceTemplate> otherConcept) { int result = compareToWithoutEntities(otherConcept); if (result != 0) { return result; } final DocToscaServiceTemplate other = (DocToscaServiceTemplate) otherConcept; ------- return map == null ? null : MAPPER.writeValueAsString(map); } catch (JsonProcessingException ex) { throw new IllegalStateException(ex); } } @Override public Map<String, ? extends Object> convertToEntityAttribute(String dbData) { if (dbData == nullObjectUtils.compare(toscaTopologyTemplate, other.toscaTopologyTemplate); } } |
In the example below the implementation of JpaToscaServiceTemplate for PostgreSQL/MariaDB (full implementation could be found here: https://gerrit.nordix.org/c/onap/policy/clamp/+/13642)
Code Block | ||||||
---|---|---|---|---|---|---|
| ||||||
@Entity @Table(name = "ToscaServiceTemplate") @Inheritance(strategy = InheritanceType.TABLE_PER_CLASS) @Data @EqualsAndHashCode(callSuper = false) public class JpaToscaServiceTemplate extends PfConcept implements PfAuthorative<ToscaServiceTemplate> { @EmbeddedId @VerifyKey @NotNull private PfConceptKey key; @Lob @Convert(converter = StringToServiceTemplateConverter.class) @NotNull @Valid private DocToscaServiceTemplate serviceTemplate; ------- @Override public ToscaServiceTemplate toAuthorative() { return MapserviceTemplate.oftoAuthorative(); } }@Override public void fromAuthorative(ToscaServiceTemplate authorativeConcept) try { serviceTemplate = new DocToscaServiceTemplate(authorativeConcept); return MAPPER.readValue(dbData, new TypeReference<Map<String, Object>>() {}); } catch (IOException ex) { log.error("Cannot convert String to Map." setKey(serviceTemplate.getKey().asIdentifier().asConceptKey()); } return Map.of(); } } } @Slf4j @Converter(autoApply = true) public final class StringToDocumentConverter implements AttributeConverter<DocumentDto, String> ------- } |
In the example below the implementation of JpaToscaServiceTemplate for MongoDB (full implementation could be found here: https://gerrit.nordix.org/c/onap/policy/clamp/+/13615)
Code Block | ||||||
---|---|---|---|---|---|---|
| ||||||
@Document(collection = "ToscaServiceTemplate") @Inheritance(strategy = InheritanceType.TABLE_PER_CLASS) @Data @EqualsAndHashCode(callSuper = false) public class JpaToscaServiceTemplate extends PfConcept implements PfAuthorative<ToscaServiceTemplate> { private static@Id final ObjectMapper MAPPER = new ObjectMapper(); @VerifyKey @Override@NonNull publicprivate String convertToDatabaseColumn(DocumentDto document) {PfConceptKey key; @NonNull try {@Valid private DocToscaServiceTemplate serviceTemplate; return document == null ? null : MAPPER.writeValueAsString(document);------- @Override public } catch (JsonProcessingException exToscaServiceTemplate toAuthorative() { throw new IllegalStateException(exreturn serviceTemplate.toAuthorative(); } } @Override public DocumentDtovoid convertToEntityAttributefromAuthorative(StringToscaServiceTemplate dbDataauthorativeConcept) { ifserviceTemplate (dbData == null) {new DocToscaServiceTemplate(authorativeConcept); return new DocumentDto(); setKey(serviceTemplate.getKey().asIdentifier().asConceptKey()); } ------- } |
Converters
Jakarta and Spring do no support json Type, but we can use Converters to convert DocToscaServiceTemplate to a Json String.
Code Block | ||||||
---|---|---|---|---|---|---|
| ||||||
@Converter(autoApply = true) public class StringToServiceTemplateConverter implements AttributeConverter<DocToscaServiceTemplate, String> { private static final Coder coder = new StandardCoder(); @Override public String convertToDatabaseColumn(DocToscaServiceTemplate serviceTemplate) { try {try { return MAPPER.readValue(dbData, DocumentDto.class); } catch (IOException ex) { log.error("Cannot convert String to Map."); return new DocumentDto(serviceTemplate == null ? null : coder.encode(serviceTemplate); } catch (CoderException e) { } } |
Repository
The java code below shows some examples how to use the functionalities offered by PosgreSLQ about Json Type. We can apply where clauses, extract and manipulate data into the Json data. In order to use these functionalities, all queries have to be native (nativeQuery = true).
Code Block | ||||||
---|---|---|---|---|---|---|
| ||||||
@Repository public interface ItemRepository extends JpaRepository<Item, String> { @Query(value = " SELECT * FROM item where document['name'] @> :value ", nativeQuery = true) Stream<Item> findByName(@Param("value") String value); @Modifying @Query(value = " UPDATE item SET document['name']=:value where key=:key ", nativeQuery = true) void updateNameByKey(@Param("key") String key, @Param("value") String value); @Query( throw new PfModelRuntimeException(Response.Status.BAD_REQUEST, e.getMessage(), e); } } @Override public DocToscaServiceTemplate convertToEntityAttribute(String dbData) { if (dbData == null) { return new DocToscaServiceTemplate(); } try { value = " SELECTreturn Cast(jsonb_path_query_array(document['elements'], '$[*].keyvalue().value.tags') as varchar) FROM item ", coder.decode(dbData, DocToscaServiceTemplate.class); } catch (CoderException e) { nativeQuery = true) throw Stream<String> findTags(); } |
Unit Tests
In order to make unit tests, we can not use H2 due of the incompatibility. Spring supports Testcontainers where we can configure a PostgreSQL Container that will up and running during tests.
Code Block | ||||||
---|---|---|---|---|---|---|
| ||||||
@Testcontainers
@ExtendWith(SpringExtension.class)
@SpringBootTest(webEnvironment = WebEnvironment.RANDOM_PORT)
@ContextConfiguration(
initializers = DemoApplicationTests.Initializer.class,
classes = DemoApplication.class)
@TestPropertySource(locations = {"classpath:application_test.properties"})
class DemoApplicationTests {
@Test
void contextLoads() {
}
@Container
public static PostgreSQLContainer postgreSQLContainer = new PostgreSQLContainer("postgres:latest")
.withDatabaseName("db_demo").withUsername("sa").withPassword("sa");
@Configuration
static class Initializer implements ApplicationContextInitializer<ConfigurableApplicationContext> {
@Override
public void initialize(ConfigurableApplicationContext configurableApplicationContext) {
TestPropertyValues
.of("spring.datasource.url=" + postgreSQLContainer.getJdbcUrl(),
"spring.datasource.username=" + postgreSQLContainer.getUsername(),
"spring.datasource.password=" + postgreSQLContainer.getPassword())
.applyTo(configurableApplicationContext.getEnvironment());
}
}
}
|
Item Table
db_demo=# \d item
Table "public.item"
Column | Type | Collation | Nullable | Default
------------+------------------------+-----------+----------+---------
key | character varying(255) | | not null |
document | jsonb | | not null |
properties | jsonb | | not null |
text | character varying(255) | | not null |
Indexes:
"item_pkey" PRIMARY KEY, btree (key)
Code Block | ||
---|---|---|
| ||
db_demo=# select * from public.item;
key | document | properties | text
-----+----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+---------------------------------------------------------------------------------+--------------------
1 | {"name": "DocumentExample", "elements": {"additionalProp1": {"name": "ElementExample 1", "tags": ["tag1"]}, "additionalProp2": {"name": "ElementExample 2", "tags": ["tag2"]}, "additionalProp3": {"name": "ElementExample 3", "tags": ["tag3"]}}} | {"additionalProp1": "Val 10", "additionalProp2": "11", "additionalProp3": "12"} | This is an example
(1 row)
|
Code Block | ||
---|---|---|
| ||
db_demo=# SELECT Cast(jsonb_path_query_array(document['elements'], '$[*].keyvalue().value.tags') as varchar) FROM item;
jsonb_path_query_array
--------------------------------
[["tag1"], ["tag2"], ["tag3"]]
(1 row)
|
Example in Policy Admin models
JpaToscaWithTypeAndStringProperties class
Code Block | ||||||
---|---|---|---|---|---|---|
| ||||||
@MappedSuperclass
@Data
@EqualsAndHashCode(callSuper = true)
public abstract class JpaToscaWithTypeAndStringProperties<T extends ToscaWithTypeAndObjectProperties>
extends JpaToscaEntityType<T> implements PfAuthorative<T> {
.............................
@Convert(converter = StringToMapConverter.class)
@Column(columnDefinition = "json")
private Map<@NotNull String, @NotNull String> properties;
.............................
} |
Converters
Code Block | ||||||
---|---|---|---|---|---|---|
| ||||||
@Converter(autoApply = true)
public final class StringToMapConverter implements AttributeConverter<Map<String, ? extends Object>, String> {
private static final Coder coder = new StandardCoder();
@Override
public String convertToDatabaseColumn(Map<String, ? extends Object> map) {
try {
return map == null ? null : coder.encode(map);
} catch (CoderException e) {
throw new PfModelRuntimeException(Response.Status.BAD_REQUEST, e.getMessage(), e);
}
}
@Override
public Map<String, ? extends Object> convertToEntityAttribute(String dbData) {
if (dbData == null) {
return Map.of();
}
try {
return coder.decode(dbData, Map.class);
} catch (CoderException e) {
throw new PfModelRuntimeException(Response.Status.BAD_REQUEST, e.getMessage(), e);
}
}
}
|
Table of comparison
...
new PfModelRuntimeException(Response.Status.BAD_REQUEST, e.getMessage(), e);
}
}
}
|
Note: Serialization and deserialization in Json is already used in policy-model (Un example could be found here [JpaToscaPolicy.java]). @Converter is just an elegant way to do the same thing.
ToscaServiceTemplate Table
clampacm=# \d ToscaServiceTemplate
Table "public.toscaservicetemplate"
Column | Type | Collation | Nullable | Default
-----------------+------------------------+-----------+----------+---------
name | character varying(120) | | not null |
version | character varying(20) | | not null |
servicetemplate | text | | |
Indexes:
"toscaservicetemplate_pkey" PRIMARY KEY, btree (name, version)
clampacm=# select * from public.ToscaServiceTemplate;
name | version | servicetemplate
--------------------+---------+-----------------
PMSH_Test_Instance | 1.0.0 | 16505
(1 row)
clampacm=# select convert_from(lo_get(servicetemplate::oid), 'UTF8') from toscaservicetemplate;
{"tosca_definitions_version":"tosca_simple_yaml_1_3", ...
Proposals
The new ORM layer will be implemented into an additional package and the old layer will be not changed. My propose options are shown below:
- Save ToscaServiceTemplace as Json String in a single Entity:
- Each Service Template is stored as a JSON "LOB"
- It is compatible with H2, MariaDB and PostgreSQL
- It is an additional code for policy-models and a medium impact for all applications that are using it
- That solution is compatible with Applications not implemented in Spring
- MongoDB/Cassandra
- Document oriented approach full supported by SpringBoot (not needs Converters)
- Compatible only with MongoDB/Cassandra (MongoDB and Cassandra are not compatible to each other)
- It is an additional code for policy-models and a huge impact for all applications that are using it (all repositories and persistence classes have to change to a Document oriented classes)
- Unit tests need an Embedded Server (example for cassandra: EmbeddedCassandraServerHelper or CassandraContainer)
- Spring Boot has own annotations for Documents. Eventually for application not in Spring Boot, it needs additional dao style implementation
Note
- Using document storage, it involves only the ORM layer, it does not change the functionality of the application (first step of Roadmap)
- Business logic could be optimized reducing the number of access to DB: read one time a service template and using it during the elaboration rather then fetch data from data from DB for each single search (load a property, load a list of ToscaNodeTemplate ecc..)
Benchmark Performance of runtime-acm
In order to generate the benchmark I have used (into a laptop) a Virtual Machine whit the follow configuration:
- 8192 Mb
- 2 CPU
For the tests:
- Jmeter to generate requests (same used by performance tests)
- Prometheus for monitoring
- DMaap simulator
- Participant simulator
- MariaDB/PostgreSQL/MongoDB
The existing system
Hibernate/Mariadb. Tosca Service template is saved as a schema entity relation.
Using Json in MariaDB
Hibernate/Mariadb. Tosca Service Template is saved into a longtext as Json.
Using Json in Postgres
Hibernate/PostgreSQL. Tosca Service Template is saved into a text type as Json.
MongoDB
MongoDB. Tosca Service Template and all other entities (Participants and AutomationComposition) are saved as MongoDB Document.
id cannot have dot '.' in MongoDB : solved with minimal configuration
Implementation (First step of Roadmap)
Current ORM layer for ToscaServiceTemplate
ORM layer for ToscaServiceTemplate Proposed (during first step in Roadmap)
Simple-concepts entities will be maintained but not used.
Namespace and Import service templates (second and third step in Roadmap)
In TOSCA language namespace is supposed to be a URI. Name and version of service template, are not defined in TOSCA language, not used in yaml files, and not present in all examples neither Unit Tests, but they are used as primary key of the ToscaServiceTemplate table and they are used in REST endpoints as id of a resource.
- TOSCA language could be found here: https://docs.oasis-open.org/tosca/TOSCA-Simple-Profile-YAML/v1.3/os/TOSCA-Simple-Profile-YAML-v1.3-os.pdf
- A yaml file contains only one service template
- TOSCA Service Templates MUST always have, as the first line of YAML, the keyword “tosca_definitions_version” with an associated TOSCA Namespace Alias value.
- Each service template has a unique name space
- A service template could import other service templates
- namespace as URI could be an issue if used as id of a resource
- name could be extracted as host of the namespace, and version could be extracted from the postfix of the namespace.
- A pattern could be like a http://{host_namespace}/service/template/{x}/{y}/{z}. As example "http://tosca/service/template/0/0/1" we can extract name: "tosca" and version: "0.0.1"
- Version could be omitted: http://{host_namespace}/service/template. As example "http://tosca/service/template/onap/org" we can extract name: "tosca" and default version: "0.0.1"
- http://tosca/service/template/onap/org could be used as default namespace
- name and version will be not present into the yaml file (service template) as now, but we can continue to use them as id of a resource
- host will be used as key to reference to external service template, and with "namespace_prefix" will be used to avoid collisions with same host but different version
- about type and type_version backward compatibility still valid
- type_version will be deprecated, type will be used as formatted string as follow:
- {type}
- {type}:{type_version}
- {host_namespace}:{type}:{type_version}
tosca001.yaml
Code Block | ||||
---|---|---|---|---|
| ||||
tosca_definitions_version: tosca_simple_yaml_1_1_0
node_types:
org.onap.nodeTypeA:
derived_from: tosca.nodetypes.Root
org.onap.nodeTypeB:
derived_from: org.onap.nodeTypeA
version: 1.2.3
topology_template:
node_templates:
nodeTemplate01:
version: 4.5.6
type: org.onap.nodeTypeB
type_version: 1.2.3
|
ServiceTemplate002.yaml
Code Block | ||||
---|---|---|---|---|
| ||||
tosca_definitions_version: tosca_simple_yaml_1_1_0
namespace: http://ServiceTemplate/service/template/onap/org/0/0/2
imports:
- repository: http://tosca/service/template/onap/org
node_types:
org.onap.nodeTypeC:
derived_from: tosca:org.onap.nodeTypeA
version: 2.3.4
topology_template:
node_templates:
nodeTemplate02:
version: 5.6.7
type: tosca:org.onap.nodeTypeB
type_version: 1.2.3
nodeTemplate03:
version: 6.7.8
type: tosca:org.onap.nodeTypeB:1.2.3
nodeTemplate04:
version: 7.8.9
type: org.onap.nodeTypeC
type_version: 2.3.4
nodeTemplate05:
version: 8.9.10
type: org.onap.nodeTypeC:2.3.4
|
ServiceTemplate003.yaml
Code Block | ||||
---|---|---|---|---|
| ||||
tosca_definitions_version: tosca_simple_yaml_1_1_0
namespace: http://ServiceTemplate/onap.org/service/template/onap/org/0/0/3
imports:
- repository: http://tosca/service/template/onap/org
namespace_prefix: defaultST
- repository: http://ServiceTemplate/service/template/onap/org/0/0/2
namespace_prefix: st02
node_types:
org.onap.nodeTypeZ:
derived_from: defaultST:org.onap.nodeTypeA
version: 9.3.4
org.onap.nodeTypeY:
derived_from: st02:org.onap.nodeTypeA
version: 9.3.4
topology_template:
node_templates:
nodeTemplate10:
version: 9.6.7
type: defaultST:org.onap.nodeTypeB:1.2.3
nodeTemplate11:
version: 9.8.9
type: org.onap.nodeTypeZ:2.3.4
|
Tosca Service Template Handling
The current work in progress is available here: https://gerrit.nordix.org/c/onap/policy/clamp/+/13755
Validation
Validation in current ORM layer
As the first step of Roadmap is backward compatible, the current validation will be maintained as it is, but it should be adapted to new DocToscaEntities. In the second step do not need other changes.
"type" and "type_version" | Referenced to |
---|---|
ToscaProperty | ToscaDataType |
ToscaPolicy | ToscaPolicyType |
ToscaNodeTemplate | ToscaNodeType |
"type_version" is optional and "0.0.0" is the default value. The "Key" is used for the validation to find if the ToscaEntity exists.
Examples:
Example | Key |
---|---|
type: onap.datatypes.ToscaConceptIdentifier | onap.datatypes.ToscaConceptIdentifier:0.0.0 |
type: org.onap.policy.clamp.acm.PolicyAutomationCompositionElement | org.onap.policy.clamp.acm.PolicyAutomationCompositionElement:1.0.0 |
"derivedFrom" |
---|
ToscaCapabilityAssignment |
ToscaCapabilityType |
ToscaDataType |
ToscaNodeTemplate |
ToscaNodeType |
ToscaPolicy |
ToscaPolicyType |
ToscaRelationshipType |
ToscaRequirement |
"derivedFrom" is referenced to a ToscaEntity of the same type and placed in same collection.
Example | Key |
---|---|
derivedFrom: onap.datatypes.ToscaConceptIdentifier | onap.datatypes.ToscaConceptIdentifier:{last_version} |
derivedFrom: org.onap.policy.clamp.acm.PolicyAutomationCompositionElement:1.0.0 | org.onap.policy.clamp.acm.PolicyAutomationCompositionElement:1.0.0 |
Validation in third step
In the last step, due the introduction of imports and namespace support, it needs to extend the validation.
"type_version" is optional and "0.0.0" is the default value, "host_namespace" is optional and "tosca" is the default value, type could be used as formatted string: {host_namespace}:{name}:{version}. The "Key" is used for the validation to find if the ToscaEntity exists in same ServiceTemplate or in other one.
Examples:
Example | Key (if defined in same service template) | Key (for external service template) |
---|---|---|
type: onap.datatypes.ToscaConceptIdentifier | onap.datatypes.ToscaConceptIdentifier:0.0.0 | tosca:onap.datatypes.ToscaConceptIdentifier:0.0.0 |
type: org.onap.policy.clamp.acm.PolicyAutomationCompositionElement | org.onap.policy.clamp.acm.PolicyAutomationCompositionElement:1.0.0 | tosca:org.onap.policy.clamp.acm.PolicyAutomationCompositionElement:1.0.0 |
type: onap.datatype.acm.Target:1.2.3 | onap.datatype.acm.Target:1.2.3 | tosca:onap.datatype.acm.Target:1.2.3 |
type: CustomNamespace:onap.datatype.acm.Operation:1.0.1 | onap.datatype.acm.Operation:1.0.1 | CustomNamespace:onap.datatype.acm.Operation:1.0.1 |
"derivedFrom" should have same logic as before and it could be used as formatted string: {host_namespace}:{name}:{version}.
Example | Key (if defined in same service template) | Key (for external service template) |
---|---|---|
derivedFrom: onap.datatypes.ToscaConceptIdentifier | onap.datatypes.ToscaConceptIdentifier:{last_version} | tosca:onap.datatypes.ToscaConceptIdentifier:{last_version} |
derivedFrom: onap.datatype.acm.Target:1.2.3 | onap.datatype.acm.Target:1.2.3 | tosca:onap.datatype.acm.Target:1.2.3 |
derivedFrom: CustomNamespace:onap.datatype.acm.Operation:1.0.1 | onap.datatype.acm.Operation:1.0.1 | CustomNamespace:onap.datatype.acm.Operation:1.0.1 |
Policy-Api
Enabling namespace and multi-service templates
Current Api in policy-api | Api in policy-api with namespace support |
---|---|
POST /nodetemplates GET /policytypes | POST /servicetemplates/{st_name}/versions/{st_version}/nodetemplates GET /servicetemplates/{st_name}/versions/{st_version}/policytypes |
Update functionality (second and third step of Roadmap)
There are two kind of update:
- values (example "description", "metadata", ...)
- data structure (example Create or Delete of a ToscaEntity, or Update a reference as "type", "type_version" or "host_namespace")
Proposals
- We could consider Service templates references to each other as a DAG ( directed acyclic graph), where Service templates are nodes, and a references as edges. Cyclic references are not allowed.
- Should be available a functionality to load a full version of the service template that contains own tosca entities and also all tosca entities referenced from other service templates (this functionality in for validation or business logic purpose and will be not visible to the end user)
- A service template cannot be deleted if there are references to it
- Any Tosca Entity in a common service template cannot be deleted if there are references to that service template
- Any Tosca Entity in a common service template cannot change "host_namespace", "type" and "type_version" if there are references to that service template
- Any Tosca Entity in a common service template cannot change values if there are automation compositions referenced to that service template (and references to that service template?)
- A Tosca Entity cannot be add in a common service template if there are automation compositions referenced to that service template
- Functionality as create, update and delete of a ToscaEntity in a service template should be validated and restricted to the service template itself. (Example: update property values of a service template should never update property values from other service templates referenced)
- To reduce the complexity, could be useful to save additional information about the service template:
- A full version of the service template that contains own tosca entities and also all tosca entities referenced from other service templates. it will be used in read-only for business logic purpose
- A table to save relations between service templates
- A table to save relations between service templates and automation compositions (with all service templates referenced)
Conclusion
- Move to document storage does not change the functionality of the application. Example if clampacm move to document storage, and policyadmin not yet, those applications will continue to work fine.
- Enable namespace and enable to handle more than one service template, impacts functionalities and applications dependency. Example if clampacm has namespace enabled, and policyadmin not yet, due the dependency, they could have some issues.
Document storage Optional?: using SpringBoot profiles and interfaces it could be possible to have a double implementation and run only one, (based on parameter in properties file), but it does not make sense due the second point.
Conclusion
My propose options are shown below:
- replace all ElementCollections with Converters (Minimum change with Minimum improvement):
- A lot of small Jsons
- Using longtext, we are be able to maintain compatibility with MariaDB
- Minimum Java code impact
- 50% of tables (with postfix as _ATTRIBUTES, _PROPERTIES, _METADATA, _OCCURRENCES, _CONSTRAINTS, _TARGETS, _TRIGGERS and _INPUTS) will be removed
- Unit tests not need to change
- Unique Json String:
- One big Json for each model (e.g. saving the whole ToscaServiceTemplace as Json)
- Using jsonb, only PostgreSQL will be supported
- Medium Java code impact: all persistence classes could be removed or have to be as Json Document
- 90% of tables will be removed
- Unit tests need PostgreSQLContainer
- Cassandra
- Document oriented approach full supported by SpringBoot (not needs Converters)
- Huge Java code impact: All repositories and all persistence classes have to change to a Json Document
- No support for MariaDB and PostgreSQL anymore
- Unit tests need an Embedded Cassandra Server (EmbeddedCassandraServerHelper or CassandraContainer)
Benchmark Performance
In order to generate the benchmark I have used:
- Jmeter to generate requests (same used by performance tests): Requests/sec: 287
- Prometheus for monitoring
- DMaap simulator
- Participant simulator
- MariaDB
The existing system
Eclipse-Link/Mariadb. Tosca Service template is saved as a schema entity relation.
Postgres with the JSON store
Hibernate/PostgreSQL. Tosca Service Template is saved into a jsonb type.
The equivalent in MariaDB
Hibernate/Mariadb. Tosca Service Template is saved into a longtext as Json.
...