The purpose of the page is to have content for review with the ARCHCOM before the end of July 2020
PoC Overview Diagram
Topic Areas
Topic Area | Description |
---|---|
MODEL DRIVEN SCHEMA: Relates to Design time activities. SDC operation. Distribution of the CSAR and the Model-driven Schema coming from Design time | |
UPDATE: Takes incoming Standards Defined VES event from DCAE (PNF) and is published on DMAAP | |
SYNCHRONIZATION: C&PS synchronizes to A&AI, which allows it to get the latest view of what is in A&AI and align it with the C&PS data records. | |
SERVICE PROVIDER INTERFACE: The C&PS interface to the DBMS | |
C&PS API: The API that allows other micro-services and other components to access C&PS DB with CRUD operations. |
Scope
TODO
- Select a YANG model for PoC. Needs to be simple, yet drive validation and upgrade. Ideally will have multiple objects/attributes to associate different CPS behavior
Base scope
- Read/write persisted Configuration Management data:
- defined by xNF (simple YANG model), published as YANG
- Models deployed @ runtime with no ONAP platform impacts or LCM events
- Show the benefits in terms of constraint validation, access and upgrade
Stretch target
Adapt behavior of CPS on read/write based on information in the model:
- Change notification emissions driven by the model
- Interaction with temporal data store, synergize with State Management PoC
- Volatile/prolific xNF data read-through (e.g. state data) – may be used to support histogram
- Model made available to CPS (Configuration and Persistency Service) using existing onboarding via SDC
- Provide some indication of dimension and user access characteristics that will be provided by CPS
Proof points
- Demonstrate Create/read operations using YANG fragments against a CPS backed by very simple schema / schema-less repository
- Demonstrate ability to deploy / upgrade YANG fragments at run-time
- Demonstrate CPS behavior driven by YANG model
- Provide architecture vision and roadmap for a target architecture, supported use cases, non-functional requirements towards an ONAP Project
Out of scope
- Automating the ingestion of models from SDC or controller (unless we get to it in the stretch target). The model LCM interface will be exercised manually during the PoC.
- Syncing with A&AI. The PoC will work on a fixed topology
Design
Context of CPS
Interface Id | Purpose | ||
---|---|---|---|
AAIE-1 | Read and maintain relevant parts of the topology | ||
SDCE-6 | React to CSAR packages (on-boarded in SDC) that contain models of interest | ||
CPS-E-01 | Model lifecycle (add and remove models) | ||
CPS-E-02 | Data access interface (CRUD) | ||
CPS-E-03 | Notification of data change (configured by model) |
C4 diagram of main components
Logical view of the CPS as a project
Participant Name | Functional purpose | PoC handling |
---|---|---|
YANG Model parser | Converts YANG models into java objects | Required |
YANG Data parser | Converts data compliant with a YANG model into java primitives, assists the CPS core | Required |
Model store | Keeps a local cache of the structures and relationships (generated from the YANG models) required by the CPS | Required |
SDC Model loader | Subscribes to DMaaP for CSAR publication. Parses and extracts the models from the CSAR and gives them to the mirror service | Not required |
Topology sync | Does initial sync and maintains an up to date view of the active topology state. Informs the mirror service of any changes | Not required |
Mirror service | Encapsulation of all model and data access that relates to xNF | Not required |
Mirror Data access | REST interface that provides access to xNF data | Not required |
CPS Core | Provides validation of and access to data. Is agnostic to the DB technology and schema | Required |
CPS Data access | Provides REST CRUD access to the data | Required |
CPS notification | Provides a DMaaP notification in the event that (1) data is changed and (2) it has been tagged/marked for notifcation | Required for stretch |
PostgreSQL plugin | Maps the java primitive representation of the YANG data objects to their DB technology (PostgreSQL) and schema specifics (SQL) | Required |
PostgreSQL | Database Management System for current data | Required |
Temporal [Stack] | Placeholder for logical components representing the temporal handling of data | Not required |
Temporal DB | Database Management System for temporal data | Not required |
The blue software encapsulates the handling of models and data related to xNF (mirrored data). This implementation is not required to prove the model driven aspects of the CPS.
The pink software encapsulates the integration with other data stores. This is being taken by the temporal DB PoC. This PoC needs to provide a feed of change notifications for any joint demonstrations. Configuring the parts of the model that will emit change notifications is part of the stretch goal.
Model handling
Below a simplified view of model handling artifacts and their relationships.
The run-time artifacts are shaded (orange). A typical simplified flow:
- A document representing xNF data is given to the CPS.
- It will parse (and validate) the document,
- Map data to the generic DB schema and store data.
In order for this to work:
- The document must be compliant to an xNF schema model. This schema model describes the structure of the data and any constraints that need to be checked to ensure integrity.
- The xNF schema model must be written in YANG. This language describes syntax and capabilities (basic concepts).
- A Yang parser is required to interpret documents compliant with the YANG language. The parser is dependent on the language.
- To store data in a DBMS, typically a schema is required. This is denoted above as, 'Generic schema model'.
- Obviously the xNF schema model and the Generic schema model are not the same. Therefor a mapping is required. Depending on how simple/generic/abstract the generic schema model, the mapping may be coupled with either the language or the xNF schema model. One of the aims of the PoC is to be agnostic to the xNF model at run-time, thus the mapping is made to the YANG language.
- To support multiple container run-times (Java, Python, Go, Rust, ...) for the CPS, the mapping is described in a 'Language Capability Map'.
- The model mapper depends on the language capability map and the Yang parser (in particular the java representation of the language features).
Key sequences/flows
Interfaces
Initial draft specification REST & Java
Behavior
Initial data structure specification, REST & Java
Introduction
This page summarizes issues (documented in child pages) and the decisions/direction taken.
Decisions/direction is discussed in the child pages, and ratified during the weekly meeting.
Assumptions
# | Assumption | Reasoning/Justification |
---|---|---|
1 | The CPS stand-alone access interface technology will be REST | This is common practice in ONAP. There is a drive to harmonize publication and documentation of component interfaces. |
2 | The CPS internal core implementation will be Java | Alternative language bindings will be possible (explicitly enabled by the architecture). The level of resources available indicates that a single implementation should be pursued first. Where other language run-times need access to data they may use the REST data access interfaces. |
3 | The native model language of the CPS will be YANG | Industry momentum towards YANG IETF RFC, widely supported Superset of most language capabilities |
4 | The PoC will use PostgreSQL as the DBMS | None, other than one is required |
Topic Areas
Issues & Decisions PoC & Honolulu
(closed issue will be marked in green)
# | Topic Area | Priority | Issue slogan/link | Brief description | Ready to discuss | Decision |
---|---|---|---|---|---|---|
1 | 4 | AGREED | Integration pattern for current, historical, temporal, timeseries, etc. | The CPS project proposal covers multiple types of storage. How will it integrate several DBMS technologies to support these different needs? | -- | Option C. Loose coupling. The focus of the PoC will be on current storage and model driven notifications to drive data towards other data stores. |
2 | 5 | AGREED | Backward Compatibility with Config DB | The current API is a basic hard coded REST interface with specific methods supporting 2 object types and a few operations. Although extensible it will be hard to keep the new proposed interface backward compatible with the original API. | -- | No Compatibility. Either a adapter will be provided later of some resource will be set reserved to migrate the existing client to the new service when available. |
3 | 4,5 | AGREED | Interface style | The access interface needs to suit the users of the data. It must also acknowledge the underlying data structure. There are two main options: Providing behavior interfaces on a logical representation of the data; Providing behavior interfaces on the CPS and explicitly providing reference to the data | -- | Option B, Access methods on the CPS; Data object provides context to those methods. |
4 | 4,5 | AGREED | Data Representation | The data that is read from and written to the CPS must comply with a known format. This is relevant to both Java and REST interfaces. The choices here are to provide generic objects/documents, or highly typed (using model info). | -- | Option A. Data will be represented as documents (REST) and or generic objects (glorified maps in Java) |
5 | 1,3 | AGREED | Flow of models into the system | The CPS will be model driven, which means that new models can be added to the CPS. How will models be introduced to the CPS from other components in the ONAP system or the network? | -- | There will be a single way to add a YANG model to CPS. This is done through the CPS model life-cycle management interface. There are three ways in which the ONAP platform can add models. See details in Flow of models into the system |
6 | 1,3 | POSTPONED | Upgrade of models | Postponed to Decisions for future releases, see #1 table below | ||
7 | 1 | POSTPONED | Specifiy and configure CPS behavior | Postponed to Decisions for future releases, see #2 table below | ||
8 | 1 | AGREED | Existing Yang Parser | Is there an existing Yang Parser in ONAP an/or OpenDayLight that can be used for C&PS: yes; ODL's Yang Tools. This is already widely used in OMAP Java Projects (for more details follow link in prev. column) PYang - a standard python tool for processing Yang model; benefit over ODL; conver to XML to perform own processing. Yang is a simple tree data structure. difficulty is extending/augmentation. PYang will come up with representation of data. Locked into Java vs a table. A Yang model can be served in V1/V2. Version management of Yang models. Augmentation to deal with modification of V1 of a model; upgrade from V1 > V2. maintaining schemas (Yang model) can ingest data in either V1 or V2 & serve it in V1/V2. | Yes | For the Java-based service ODL's Yang Tools will be used. For specific use-cases it might be more suitable yo use Python and PYang e.g. Model Visualization. |
9 | N/A | AGREED | Location of PoC Code | Dan Timony suggested to use and existing CCSDK repo, he mentioned ccsdk/features. As long as the PoC remains completely independent and doesn't affect delivery of existing artifacts in the same repo. | -- | ccsdk/features, see https://gerrit.onap.org/r/c/ccsdk/features/+/110385 (awaiting approval) |
10 | N/A | AGREED | Common information model, Data lake and Access control | How will the CPS help with managing coupling between ONAP components that make use of data lake and common information model | -- | We will start with Architectural Approach A in the PoC with the aim of fully supporting Architectural Approach C. I.e. access to the data lake will be conditional on permission granted by the data owner. In the PoC we will not implement the permission granting mechanism |
11 | 4,5 | AGREED | Transactional behavior | It needs to be clear to users the level of atomic operations supported by the CPS. Transactions are performed sequentially and atomically (one at a time). As an external user you have a Java interface (with a dedicated mSvc) & REST interface. It is up to the user of C&PS to specify the order of changes in their input document. | -- | The CPS will not use an external transaction manager. Where the underlying DBMS provides transactions, these will be exposed by the SPI and used by the CPS core library. The CPS core library will expose a simple session concept to co-deployed modules. This will enable fine grained control of the order of changes. This will help ensure that complex validation (e.g. WHEN clause) succeeds. For REST users, the session concept will not be exposed. Where documents representing several operations are input, the CPS will follow the order of the input document, and fail fast. Any failures will be abandoned (if the underlying DBMS supports transactions) and a failure response will be returned to the REST user. (Rest of the Page is accepted) |
12 | 5 | POSTPONED | Postponed to Decisions for future releases, see #1 table below | No | ||
13 | 5 | AGREED | CCSDK-2870 DP: CPS REST API Documentation | Discuss Rest API | Yes | |
14 | AGREED | CPS & DCAE DES concept sharing | Discussion about usage of DCAE Data Exposure Service (DES) concept. krishna moorthy presented on the MAPPER SERVICE at CPS meeting and | API Mapp/DES like transformer: TemplateBasedDataModelTransformer will be developed for this purpose. Initially a PoC in CPS but it might become a separate component | ||
15 | 5 | AGREED | Discuss the Java API and SPI wiki. Both pages include javadoc and a link to the gerrit repo. | Yes | ||
16 | 1,5 | AGREED | Yang modules to XPath format adaptation | Discuss the XPath adaptation to address data fragments in a context of Yang Modules revisions variety | For now there is no need to store full module identifier in the xpath field. The namespace, revision, parent module context will be taken into account using references through dedicated fields (module_id, parent_id). Limitation determined: only keyed collections are supported. The attribute order for composite keys to be handled as dedicated task. | |
17 | 1,5 | AGREED | Operating with Module Sets (groups of modules) | The module-set to module relationship, module-set immutability, taking into account cross-impact of modules on module-set persistence | Module-set is immutable collection of modules. None of modules can be added or removed independently, only as part of module-set insert or delete. Also module-set itself does not assume updates for now. The module-set uniqueness within a dataspace to be identified by string identifier now, also by content later The modules are stored as source definition, no processing on module entity persistence. On data validation it's expected the schema context (representing effective models) to be compiled with taking into account all the modules from the module-set. The persistence of pre-compiled context (as module-set content) or caching it or other performance optimizations to be discussed later. | |
18 | AGREED | ModuleSet content persistence proposal | The moduleset persistence approach, artifact/entity naming | Module Set sources (yang files) to be stored on a unique content basis. To rename: | ||
19 | AGREED | CPS Logging Config | Discuss issues/decisions in relation to logging in CPS. | |||
20 | AGREED | Decouple Anchor from Fragment proposal | Discuss the necessity of persisting anchors and data fragments at same table | |||
21 | AGREED | xNF Proxy MVP Delivery clarification / Deployment OPtions | Discuss and clarify the expectations and implementation approach on subject. Especially re. deployment options with SpringBoot application(s) |
| ||
22 | MEDIUM | Data update by xpath options | Discuss and clarify:
| Yes | ||
23 | AGREED |
Docker image deployment
DB integration
Certificates
Application configuration file location
| Yes | Agreed on Docker image deployment Tony Finnerty asked to ensure that the DB Integration is clearly documented in source and deployment documentation to ensure it stays de-coupled and can easily be substituted Application configuration file location Was contentious but the deciding factors where user-configurability and most common use pattern in ONAP. We still have the option to add a 'default' configuration option in CPS Application Module | ||
24 | MEDIUM | Update requirements for E2E NetworkSlicing UseCase | The preferred methods for updating are:
Any Interface impacts of this should be clearly document in Open API Yaml | Yes |
Issues & Decisions Istanbul & Later
# | Topic Area | Priority | Issue slogan/link | Brief description | Ready to discuss | Decision |
---|---|---|---|---|---|---|
1 | 1,3 | MEDIUM | Upgrade of models | As the network functions evolve, new versions of models will be introduced that will replace existing models for new versions of the NF. These new versions will need to co-exist alongside existing versions, such that different NF versions can be modelled using the appropriate model version Discuss with Christophe Closset (SDC) | No | |
2 | 1 | LOW | Specifiy and configure CPS behavior | The behavior of the CPS needs to be driven by a model. The native model language of CPS is YANG. There are several mechanisms available to achieve this need. One needs to be selected | No | |
3 | 5 | MEDIUM | 5G Service Modeling Use Case and interactions with C&PS. (1) Registration system - Creating a system for registration discovery & management mechanism to allow for coordination and inter-operation of micro-services who want access to common data. Models are also stored in the registry. | No | The scope of the CPS will include a 'Registry Service' More good discussion . Needs separate meeting to clarify |