...
This page documents the existing Yang Parser used in ONAP and OpenDayLight and will investigate if they can be used for it fulfill the needs of the C&PS.
Resources
- https://docs.opendaylight.org/en/latest/developer-guide/yang-tools.html*
- https://git.opendaylight.org/gerrit/admin/repos/yangtools
- https://gerrit.onap.org/r/admin/repos/ccsdk/sli/plugins
- https://javadoc.io/doc/org.opendaylight.yangtools
*Although this documentation link is to the latest ODL doc revision, it is very outdated and the code examples need significant updates, see findings in Mini-PoC below (bug reported: https://jira.opendaylight.org/browse/DOCS-126)
Overview
The Yang parser used in ONAP (CCSDK / SDNC) was developed (and still is) a OpenDayLight Library.
There are 2 different usage patterns within CCSDK, though.
Most of CCSDK/SDNC is using Yang primarily to define their northbound interfaces. In that context, there’s a maven plugin (org.opendaylight.yangtools:yang-maven-plugin) that is used to generate source code from the Yang model. Our application code in that case doesn’t really do anything directly with the Yang, since all of the that is handled for us by the generated code.
In ccsdk/sli/plugins, there is a plugin called restconf-client which was contributed by Huawei. That code uses the yangtool parser more directly so that it can interpret the results being returned when it calls a restconf interface.
Mini-PoC
Overview
The Yang parser used in ONAP (CCSDK / SDNC) was developed (and still is) a OpenDayLight Library.
There are 2 different usage patterns within CCSDK, though.
Most of CCSDK/SDNC is using Yang primarily to define their northbound interfaces. In that context, there’s a maven plugin (org.opendaylight.yangtools:yang-maven-plugin) that is used to generate source code from the Yang model. Our application code in that case doesn’t really do anything directly with the Yang, since all of the that is handled for us by the generated code.
In ccsdk/sli/plugins, there is a plugin called restconf-client which was contributed by Huawei. That code uses the yangtool parser more directly so that it can interpret the results being returned when it calls a restconf interface.
Resources
- https://docs.opendaylight.org/en/latest/developer-guide/yang-tools.html*
- https://git.opendaylight.org/gerrit/admin/repos/yangtools
- https://gerrit.onap.org/r/admin/repos/ccsdk/sli/plugins
- https://javadoc.io/doc/org.opendaylight.yangtools
Additional Resources (still to be examined)
*Although this documentation link is to the latest ODL doc revision, it is very outdated and the code examples need significant updates, see findings in Mini-PoC below (bug reported: https://jira.opendaylight.org/browse/DOCS-126)
Mini-PoC
To help this evaluation To help this evaluation I will create a small sample project with the goal to pare a yang file using the ODL Yang Tools. I will report my findings here
...
The documentation mentioned above lists many modules but the code examples do not clarify which exactly are needed to parse a Yang model and Yang file data files in java code.
- To be able to use yangtools teh project needs to use the mdsal 'binding-parent' pom
- The module yang-parser-impl contains all code required to parse yang files
- SLF4J has been added as the implementation requires a logger enabled
Code Block | ||||||||
---|---|---|---|---|---|---|---|---|
| ||||||||
Code Block | ||||||||
| ||||||||
<properties> <maven.compiler.version>3.8.1</maven.compiler.version> <maven.compiler.release>11</maven.compiler.release> <org.opendaylight. <odl.yangtools.version>5.0.3<5</org.opendaylightodl.yangtools.version> </properties> <build> <plugins> <plugin> <groupId>org.apache.maven.plugins</groupId> <plugins> <plugin> <groupId>org.apache.maven.plugins</groupId> <artifactId>maven-compiler-plugin</artifactId> <version>${maven.compiler.version}</version> <configuration> <configuration> <release>${maven.compiler.release}</release> </configuration> </plugin> </plugins> </build> <dependencies> <dependency> <dependency> <groupId>org.opendaylight.yangtools</groupId> <artifactId>yang <artifactId>yang-parser-api</artifactId> <version>${org.opendaylight <version>${odl.yangtools.version}</version> </dependency> <dependency> <dependency> <groupId>org.opendaylight.yangtools</groupId> <artifactId>yang-parser-impl</artifactId> <version>${orgodl.opendaylight.yangtools.version}</version> </dependency> <dependency> <groupId>org <dependency> <groupId>org.opendaylight.yangtools</groupId> <artifactId>yang-model-util</artifactId> <version>${org.opendaylight <version>${odl.yangtools.version}</version> </dependency> <dependency> <dependency> <groupId>org.opendaylight.yangtools</groupId> <artifactId>yang-data-codec-xml</artifactId> <version>${orgodl.opendaylight.yangtools.version}</version> </dependency> <dependency> <groupId>org.slf4j</groupId> <artifactId>slf4j-api</artifactId> <version>1.6.1</version> </dependency> <dependency> <groupId>org.slf4j</groupId> <artifactId>slf4j-log4j12</artifactId> <dependency> <groupId>org.opendaylight.yangtools</groupId> <artifactId>yang-data-codec-gson</artifactId> <version>${odl.yangtools.version}</version> </dependency> <!-- SLF4J API --> <dependency> <groupId>org.slf4j</groupId> <artifactId>slf4j-api</artifactId> <version>1.6.1</version> </dependency> </dependencies> |
Documentation Code Updates
The sample code provided in the documentation is faulty (using == for assigning?!) and is using some long deprecated and even removed classes and methods.
Bug reported: https://jira.opendaylight.org/browse/DOCS-126
Code Block | ||||||||||
---|---|---|---|---|---|---|---|---|---|---|
| ||||||||||
<!-- LOG4J --> static YangParserFactory PARSER_FACTORY; static {<dependency> final Iterator<YangParserFactory> it = ServiceLoader.load(YangParserFactory.class).iterator(); <groupId>org.slf4j</groupId> if (!it.hasNext()) { <artifactId>slf4j-log4j12</artifactId> throw new IllegalStateException("No YangParserFactory found");<version>1.6.1</version> }</dependency> </dependencies> |
Note. SLF4J has also been added as the implementation requires an enabled logger
Documentation Code Updates
The sample code provided in the documentation is faulty (using == for assigning?!) and is using some long deprecated and even removed classes and methods.
Bug reported: https://jira.opendaylight.org/browse/DOCS-126
Code Block | ||||||||||
---|---|---|---|---|---|---|---|---|---|---|
| ||||||||||
static YangParserFactory PARSER_FACTORY; static { PARSER_FACTORY = it.next(); } ... File file = new File(ClassLoader.getSystemClassLoader().getResource("example.yang").getFile()); YangTextSchemaSource source = YangTextSchemaSource.forFile(file); final YangParserIterator<YangParserFactory> yangParserit = PARSER_FACTORYServiceLoader.createParser(StatementParserMode.DEFAULT_MODEload(YangParserFactory.class).iterator(); yangParser.addSource(source); if (!it.hasNext()) { SchemaContext schemaContextthrow =new yangParser.buildEffectiveModel(IllegalStateException("No YangParserFactory found"); } PARSER_FACTORY = schemaContextit.getModules() |
This is the kind of object (module) that gets created:
Generated Java Object Structures per Yang concept
Object Structure
The SchemaContext object generated by the Yang Parser in java has the following possible structures (java collections)
Gliffy Diagram macroId 8eafa348-6609-4eae-a733-beb5a2766488 name Yang Parser Java Object View pagePin 4
ChildNodes are implemented as a Map <QName, DataSchemaNode>
The Class DataSchemaNode can represent a Yang Leaf, List or Container
Main data types
The package org.opendaylight.yangtools.yang.model.util.type contains classes for all the possible data types including:
- BaseBinaryType
- BaseBitsType
- BaseBooleanType
- BaseDecimalType
- BaseEnumerationType
- BaseInt8Type
- BaseInt16Type
- BaseInt32Type
- BaseInt64Type
- BaseStringType
- BaseUint8Type
- BaseUint16Type
- BaseUint32Type
- BaseUint64Type
There are also some special data types such as:
- BaseEmptyType
- BaseIdentityrefType
- BaseInstanceIdentifierType
- BaseLeafrefType
- BaseUnionType
next();
}
...
File file = new File(ClassLoader.getSystemClassLoader().getResource("example.yang").getFile());
YangTextSchemaSource source = YangTextSchemaSource.forFile(file);
final YangParser yangParser = PARSER_FACTORY.createParser(StatementParserMode.DEFAULT_MODE);
yangParser.addSource(source);
SchemaContext schemaContext = yangParser.buildEffectiveModel();
schemaContext.getModules() |
This is the kind of object (module) that gets created:
Generated Java Object Structures per Yang concept
Object Structure
The SchemaContext
object generated by the Yang Parser in java has the following possible structures (java collections)
Gliffy Diagram macroId 8eafa348-6609-4eae-a733-beb5a2766488 name Yang Parser Java Object View pagePin 5
SchemaTree
is implemented as a ImmutableMap<QName, SchemaTreeEffectiveStatement<?>>
The SchemaTree
can represent a tree of any Yang Leaf, List or Container. All leafs implement the LeafStatement
interface which provides methods for many Yang Language features including
getType()
getUnits()
getMandatory()
getWhenStatement()
getMustStatments()
Main data types
The package org.opendaylight.yangtools.yang.model.util.type
contains classes for all the possible data types including:
- BaseBinaryType
- BaseBitsType
- BaseBooleanType
- BaseDecimalType
- BaseEnumerationType
- BaseInt8Type
- BaseInt16Type
- BaseInt32Type
- BaseInt64Type
- BaseStringType
- BaseUint8Type
- BaseUint16Type
- BaseUint32Type
- BaseUint64Type
There are also some special data types such as:
- BaseEmptyType
- BaseIdentityrefType
- BaseInstanceIdentifierType
- BaseLeafrefType
- BaseUnionType
And also 'restricted' versions of the base types such as:
- RestrictedStringType
- RestrictedUint64Type
Description | Yang | Java Object View | Notes | XML Validation | JSON | ||
---|---|---|---|---|---|---|---|
Datatypes and basic constraints | |||||||
Basic String |
| TypeStatement TypeAwareDeclaredStatement.getType() | Yes | Yes | |||
Mandatory Basic String |
|
| No | No | |||
Limited String | leaf pnf-name { type string { length "0..256"; } |
| Yes | Yes | |||
typedef (String) with pattern |
| List<PatternConstraint> RestrictedStringType.getPatternConstraints() | Yes | Yes | |||
Limited uint64 | leaf cid { type uint64 { range "0..503"; } } | org.opendaylight.yangtools.yang.model.util.type.RestrictedUint64Type | Yes | Yes | |||
boolean with default value |
| org.opendaylight.yangtools.yang.model.util.type.DerivedBooleanType | N/A | N/A | |||
Unique | |||||||
Unique | list server { key "name"; unique "ip port"; leaf name { type string; } leaf ip { type dotted-quad; } leaf port { type uint32; } } | org.opendaylight.yangtools.yang.model.api.stmt.UniqueStatement | No | No | |||
Choice | |||||||
Choice | choice transfer-method { leaf transfer-interval { type uint64 { range "15..2880"; } units minutes; } leaf transfer-on-commit { type empty; } } | org.opendaylight.yangtools.yang.model.api.stmt.ChoiceStatement | N/A | N/A | |||
Must | |||||||
Must | leaf ifType { type enumeration { enum ethernet; enum atm; } } leaf ifMTU { type uint32; } must "ifType != 'ethernet' or " + "(ifType = 'ethernet' and ifMTU = 1500)" error-message 466px"An ethernet MTU must be 1500"; } | org.opendaylight.yangtools.yang.model.api.stmt.ErrorMessageStatement | No | No | |||
When | |||||||
When | leaf a {
type boolean;
}
leaf b {
type string;
when "../a = 'true'";
} | org.opendaylight.yangtools.yang.model.api.stmt.WhenStatement | No | No | |||
Extension | |||||||
Extension declaration |
| N/A | |||||
Extension usage | leaf attribute-with-temporal-storage { type string; cm-notify-api:store-state-ext "3d"; } |
which extends public interface UnknownStatement<A> extends DeclaredStatement<A> { | |||||
Description | Yang | Java Object View | Notes | XML Validation | JSON | ||
Datatypes and basic constraints | Basic String |
| Yes | Yes | Mandatory Basic String |
| No | No |
Limited String | leaf pnf-name { type string { length "0..256"; } | Specialized class to hold length limitation | Yes | Yes | |||
typedef (String) with pattern |
| Checked by XmlParser | Yes | Yes | Limited uint64 | leaf cid {325px400 type uint64 { range "0..503"; } } | Yes | Yes | boolean with default value |
| N/A | N/A |
Unique | Unique | list server { key "name"; unique "ip port"; leaf name { type string; } leaf ip { type dotted-quad; } leaf port { type uint32; } } | No | No | |||
Choice | Choice | choice transfer-method { leaf transfer-interval { type uint64 { range "15..2880"; } units minutes; } leaf transfer-on-commit { type empty; } } | N/A | N/A | |||
Must | Must | leaf ifType { type enumeration { enum ethernet; enum atm; } } leaf ifMTU { type uint32; } must "ifType != 'ethernet' or " + "(ifType = 'ethernet' and ifMTU = 1500)" error-message 466px"An ethernet MTU must be 1500"; } | No | No | |||
When | When | leaf a {
type boolean;
}
leaf b {
type string;
when "../a = 'true'";
} | No | No | |||
Extension | Extension declaration |
| N/A | Extension usage | leaf attribute-with-temporal-storage { type string; cm-notify-api:store-state-ext "3 d"; } | extension is stored as 'UnknownNode' and refers back to the extension declaration | N/A |
Augmentation | augment "server" { when "port = '8787'"; leaf enable-debug { type boolean; } } | N/A | |||||
RPC | |||||||
rpc | rpc nbrlist-change-notification { | N/A | rpc input | input { | N/A | ||
Augmentation | |||||||
augment "server" { when "port = '8787'"; leaf enable-debug { type boolean; } } | The additional leaf just appears in the SchemaTree (without any reference that it is an augmentation) | N/A | |||||
RPC | |||||||
rpc | rpc nbrlist-change-notification { | N/A | |||||
rpc input | input { | N/A | rpc output | outputleaf alias { | N/A |
Data Parsing and validation
XML Parsing
...
language | java |
---|---|
theme | Midnight |
title | XML Parsing Example |
linenumbers | true |
collapse | true |
...
type string { |
...
length "1..64"; |
...
...
|
...
} |
...
cid |
...
{ |
...
|
...
|
...
type string |
...
{ |
...
|
...
|
...
|
...
|
...
} |
...
|
...
|
...
|
...
uses x-0005b9-lte-g; |
...
|
...
|
...
|
...
|
...
type |
...
uint64; |
...
|
...
|
...
|
...
|
...
|
...
"Number of cells in a neighbor list that |
...
was changed"; |
...
key "plmnid cid"; |
...
|
...
uses lte-ran-neighbor-list-in-use-lte-cell-g; |
...
description in a neighbor list for this fap service"; } | N/A | ||||
rpc output | output { | N/A |
Yang Data Parsing and Validation
XML
*Note: the DataSchemaNode being used when creating the XmlParserStream HAS to be the root node of the xml data!
XML Validation Findings
- The XML Parser is found to do basic data type checks including range checks and (regex) pattern validation. If the dat input doesn't conform those a clear exception detailing the problem is thrown
- Features such as 'mandatory' and 'unique' are to be validated
- More advanced features such as 'must', 'when', 'choice' etc have not yet been tested
The table in the sections above has a column with the XML validation findings.
...
Parsing
Code Block | ||||||||||
---|---|---|---|---|---|---|---|---|---|---|
| ||||||||||
SchemaContext schemaContext = ... (see previous snippets) final JSONCodecFactoryModule jsonCodecFactorymodule = JSONCodecFactorySupplier.DRAFT_LHOTKA_NETMOD_YANG_JSON_02.getShared(schemaContextschemaContext.findModules("ultraconfig-interfaces").iterator().next(); QName qName = QName.create(module.getQNameModule(),"interfaces"); final NormalizedNodeResultOptional<DataSchemaNode> resultnode = new NormalizedNodeResult(); final NormalizedNodeStreamWriter streamWriter = ImmutableNormalizedNodeStreamWriter.from(result); final JsonParserStream jsonParser = JsonParserStream.create(streamWriter, jsonCodecFactory); final InputStream resourceAsStream = ClassLoader.getSystemClassLoader().getResourceAsStream("example2data.json"); final JsonReader jsonReader = new JsonReader(new InputStreamReader(resourceAsStream)); jsonParser.parse(jsonReader); final NormalizedNode<?, ?> transformedInput = result.getResult(); |
...
module.findDataChildByName(qName);
if (node.isPresent()) {
final InputStream resourceAsStream = ClassLoader.getSystemClassLoader().getResourceAsStream("example2data.xml");
final XMLStreamReader reader = UntrustedXML.createXMLStreamReader(resourceAsStream);
final NormalizedNodeResult result = new NormalizedNodeResult();
final NormalizedNodeStreamWriter streamWriter = ImmutableNormalizedNodeStreamWriter.from(result);
final XmlParserStream xmlParser = XmlParserStream.create(streamWriter, schemaContext, node.get() );
xmlParser.parse(reader);
final NormalizedNode<?, ?> transformedInput = result.getResult();
} |
*Note: the DataSchemaNode being used when creating the XmlParserStream HAS to be the root node of the xml data!
XML Validation Findings
- The XML Parser is found to do basic data type checks including range checks and (regex) pattern validation. If the dat input doesn't conform those a clear exception detailing the problem is thrown
- Features such as 'mandatory' and 'unique' are to be validated
- More advanced features such as 'must', 'when', 'choice' etc have not yet been tested
The table in the sections above has a column with the XML validation findings.
JSON Parsing
Code Block | ||||||||||
---|---|---|---|---|---|---|---|---|---|---|
| ||||||||||
SchemaContext schemaContext = ... (see previous snippets)
JSONCodecFactory jsonCodecFactory = JSONCodecFactorySupplier.DRAFT_LHOTKA_NETMOD_YANG_JSON_02.getShared(schemaContext);
final NormalizedNodeResult result = new NormalizedNodeResult();
final NormalizedNodeStreamWriter streamWriter = ImmutableNormalizedNodeStreamWriter.from(result);
final JsonParserStream jsonParser = JsonParserStream.create(streamWriter, jsonCodecFactory);
final InputStream resourceAsStream = ClassLoader.getSystemClassLoader().getResourceAsStream("example2data.json");
final JsonReader jsonReader = new JsonReader(new InputStreamReader(resourceAsStream));
jsonParser.parse(jsonReader);
final NormalizedNode<?, ?> transformedInput = result.getResult(); |
JSON Validation Findings
- As expected the parsing of string, originating form XML or JSON is done by the same code and the results are identical to those for XML Data Parsing
Conclusion
Pros
- EXTENSIVE - The YangTools model parser is comprehensive and covers all possible Yang Language elements we might require
- AVAILABLE - Extensively used throughout ONAP and ODL projects (able to convert to Java objects, code & API). These parsers are already used in many ONAP platform projects.
- MATURE CODE - Mature code, dates back to 2013 with contributions from many companies including Cisco and Redhat and Pantheon Tech. (No need for licenses, these are also open source Yang parser). ODL is open source.
- VALIDATION - Testing with parsing of JSON and XML data with validated for a (parsed) model included. Files to objects and vice versa is possible. Model violation & compilation validation is available.
- OBJECTIVES - Meets our two high-level requirements & objectives. Parse models from SDC into Java objects relate to persistence of data. Parsing of documents compliant to those schemas.
Cons
- LEARNING CURVE - Due to its completeness it also is a complicated piece of software which will take some time to get familiar with.
- DOCUMENTATION - Documentation out of date, this page hopes to address that somewhat