...
The Validation Scripts have historically produced a number of reports geared toward human consumption, but no report has existed that provided a full accounting of results in a machine readable format. A file named failures was produced in the output directory, but it contained only a subset of information about the execution run. The following proposal is to create a new file ( report.json ) that contains a full account of all available information. It would be produced when the --report-format json option is passed to the pytest command, and would be suitable for consumption for tools such as VNFSDK or Dovetail as part of the certification program.
File Specification
File Header/Top Level
| Anchor | ||||
|---|---|---|---|---|
|
The top level will include a summary of available execution metadata.
...
| Field Name | Required/ Optional/ Conditional | Datatype | Valid Values | Description |
|---|---|---|---|---|
version | Required | string | Version of the validation scripts that produced the report. For now, this is set to the ONAP release name, but it may change to a semantic version number in the future. | |
template_directory | Required | string | Absolute path of the directory containing the Heat templates that were validated | |
timestamp | Required | string | ISO 8601 Timestamp in UTC | |
checksum | Required | string | MD5 hash of all file contents in the template_directory | |
profile | Required | string | Validation profile used to validate the templates. In the default ONAP code base, this is always the empty string | |
outcome | Required | string |
| One of the valid values.
|
tests | Required | list of Test Result | List of Test Result | |
| requirements | Required | list of Requirement Result | List of Requirement Result |
Test Result
For each test result a JSON object will be provided that informs the consumer what tests was run, its result, and the requirements it validated.
...
| Field Name | Required/ Optional/ Conditional | Datatype | Valid Values | Description |
|---|---|---|---|---|
files | Required | list of string | List of files that were passed to the test case. NOTE: If | |
test_module | Required | string | Name of the module/file name that contains the test case | |
test_case | Required | string | Name of the test case | |
result | Required | string |
| One of the valid values.
|
error | Required | string | If the test failed or encountered an error, then this will be an message summarizing the error. If the test passed or was skipped, then this will be an empty string | |
requirements | Required | List of Requirement Metadata | A list of Requirement Metadata for each requirement this test case validates. While uncommon, this can be an empty list in some circumstances. |
...
| Field Name | Required/ Optional/ Conditional | Datatype | Valid Values | Description |
|---|---|---|---|---|
id | Required | string | Requirement ID from the VNFRQTS project NOTE: a requirement ID of "Unmapped" may be included if one or more tests are not mapped to a requirement. In this case, all test cases not mapped to are requirement will be rolled up to a single "Unmapped" requirement. | |
text | Required | string | Full text of the requirement. This text may be multi-line, and it will | |
keyword | Required | string |
| RFC 2119 keyword associated with the requirement |
result | Required | string |
| One of the valid values.
|
errors | Required | list of string | Error messages associated with this requirement. This will be an empty string if the result is PASS or SKIP |
...