Page History

Wiki to track the design requirements for Helm generator to support

Jira

server	ONAP JIRA
serverId	425b2b0a-557c-3c0c-b515-579789cceedb
key	DCAEGEN2-2694

Table of Contents

USECASES

Build helm chart generator taking following inputs (values.yaml) and templates to generate chart package and perform lint for consolidated charts.

Helm Chart directory structure

templates:

(Include dependent charts along with main chart so that the validation can be done)

Helm lint (checks for syntax) – Initial validation can be just done with lint.

Validation needs to be configured (default – enabled and can be disabled) - DONE - 20 May 2021

a. Run it as standalone tool - so that it can be run with the given input and generate helm chart package.

Note: This is for testing purpose only until the integration with Catalog Service is done.

tool parameters are Input template directory and output as tar - DONE 04 Jun 2021

2. Separate Values.yaml into separate templates and verify dynamic values.yaml generation (and parameter substitution

Configuration/parameters required common across MS ( highlighted in yellow in ppt)

Configuration/parameters to be templatized and values sourced from ComponentSpec ( highlighed in Red )

~~Note : Optional configuration/parameters to be templatized and included based on flags/properties from ComponentSpec~~ (Covered under 6 to 10)

The generator must consolidate these separate base values.template and created required values.yaml

TEST: Generated charts must be validated in ONAP lab K8S environment

Refer Sample Chart Yaml mapping from component spec and Sample Values yaml mapping from component spec in the requirements document.

Outstanding - 06/01/2021

stream_publish/stream_subscribe handle as string; change VES spec and test
readiness mapping Include PORT/initialdelaySeconds into auxillary.healthcheck on component spec schema as optional parameter for mapping into readiness check

Image Removed

globalNodeportPrefix - make default from base template. (It should be already supported)

...

3. Identify Component-spec schema changes for ENV setting mapping (refer REQ #1 below) DONE - 08 Jun 2021

04 Jun 2021 - Spec schema sample to be added under REQ1 section and also updated component spec schema file - Nick
04 Jun 2021 - Verify if the service configuration can be supported as list under common/service - Vijay Venkatesh Kumar

List is not supported in current common/service template

4. Identify Component-spec schema changes for Service mapping (and nodeport) (refer REQ #7 below) DONE - 11 Jun 2021

Spec schema sample to be added under REQ1 section and also updated component spec schema file; submit to gerrit with new updates (including #2, #3, #4)

~~5. Build helm chart generator taking as inputs template directory and template list file to be used for chart generation and perform lint for consolidated charts.~~ - No longer applicable; based on design 17 Jun 2021

~~Note: Use base/default template if corresponding template not found on specified template directory~~

6. Support MAPPING requirement – ENV SETTING (refer REQ #1 for details) - DONE - 11 Jun 2021 1.0.0

Schema Validation to be integrated - tracked separately under task #24

7. Support MAPPING requirement – CMPv2 Certificates (refer REQ #3 for details) - Target 06/24 completion

Include support for base + addon chart structure (certificate.yaml will be optional)

Add validation if enabled in spec and certificate.yaml not provided - tool must error

8. Support MAPPING requirement – Postgres (refer REQ #5 for details)

9. Support MAPPING requirement – Policy Sidecar (refer REQ #4 for details) Target 06/17 completion

10. Support SERVICE MAPPING based on spec file (refer REQ #7 for details) DONE - 11 Jun 2021 1.0.0

Schema Validation to be integrated - tracked separately under task #24

11. Support MAPPING requirement – ConfigMap support (refer REQ #2 for details)

12. Support MAPPING requirement – DMAAP Secure Topic/Feed (refer REQ #6 for details)

13. Create user guide for the tool detailing all command-line options/override

14. Submit code to ONAP; ensure compliance to ONAP coding standard and test coverage requirement (atleast 80%) and verify the library build/pushed to ONAP:nexus

15. Integrate tool into MOD/Runtime or MOD2/CatalogService

16. Verify E2E for ONAP DCAE MS spec file (TCA w/policy, PM-Mapper, VES, and validate corresponding charts generated in ONAP lab if components can be successfully deployed

17. Add distribution support in tool. Additional configuration support needed for below parameters either in tool property file or CMD line options

DistributionEnabled

DistributionURL

DistributionUsername

DistributionPwd

DistributionFormat - tgz or as directory

18. Provide REST interface to support HELM generation

19. Input Spec validation

20. Comments should be cascaded from template and when new yaml is generator and also preserve order

21. Secret mapping requirement TBD and pv mapping - Vijay Venkatesh Kumar

22. Support chart generation as directory (instead of tgz) via configuration (added on 01 Jun 2021 )

23. Support dependency download (helm dep up) from repo/access based on deployment configration (added on 01 Jun 2021 )

24. Add schema validator module; keep the schema definiton overriddable (either through CI or api parameter); this should be configurable - default enabled (added on 11 Jun 2021 based on svc mapping test)

25. Testing with ONAP spec files on ONAP lab with componen-spec (ves, pm-mapper, dfc, tcagen2, hv-ves, prh) - Sivakumar Santharam Target 06/17 completion

Modify spec to align with new schema
Generate charts and deploy and check if component comes up clean
Environment for test to be setup/checked (onap5) - Vijay Venkatesh Kumar - DONE (06/17/2021)

Issues found during Testing

...

logDirectory

certDirectory

tlsServer - optional

auxillary / helm section - Mandatory

policy-info - optional

...

Error: unable to build kubernetes objects from release manifest: error validating "": error validating data: [ValidationError(Deployment.spec.template.spec.containers[0].readinessProbe.httpGet): missing required field "port" in io.k8s.api.core.v1.HTTPGetAction, ValidationError(Deployment.spec.template.spec.containers[0].readinessProbe.periodSeconds): invalid type for io.k8s.api.core.v1.Probe.periodSeconds: got "string", expected "integer", ValidationError(Deployment.spec.template.spec.containers[0].readinessProbe.timeoutSeconds): invalid type for io.k8s.api.core.v1.Probe.timeoutSeconds: got "string", expected "integer"]

...

REQUIREMENTS

1. ENV SETTING SUPPORT

Component Spec

Need spec schema update to include list of parameters (key/value for applicationEnv) -->https://git.onap.org/dcaegen2/platform/tree/mod/component-json-schemas/component-specification/dcae-cli-v2/component-spec-schema.json

Code Block

language	js
theme	Midnight

"auxilary": {
.
.
"helm": {
 	"applicationEnv": {
    "PMSH_PG_URL": "dcae-pmsh-pg-primary",
    "PMSH_PG_USERNAME": {
    	"secretUid": "pgUserCredsSecretUid"

REQUIREMENTS

1. ENV SETTING SUPPORT

Component Spec

Need spec schema update to include list of parameters (key/value for applicationEnv) -->https://git.onap.org/dcaegen2/platform/tree/mod/component-json-schemas/component-specification/dcae-cli-v2/component-spec-schema.json

Code Block

language	js
theme	Midnight

"auxilary": {
.
.
"helm": {
 	"applicationEnv": {
    "PMSH_PG_URL": "dcae-pmsh-pg-primary",
    "PMSH_PG_USERNAME": {
    	"secretUid": "pgUserCredsSecretUid",
    	"key": "login"
    },
    "PMSH_PG_PASSWORD": {
         "secretUid": "pgUserCredsSecretUid",
         "key": "password"
    }
    }
}
.
.
}

...

Code Block

language	yml
theme	Midnight

externalVolumes:
  - name: dcae-external-repo-configmap-schema-map
    type: configmap
    mountPath: /opt/app/VESCollector/etc/externalRepo/
    optional: true (default)
  - name: '{{ include "common.release" . }}-another-example'   //dcae-external-repo-configmap-sa88-rel16
    type: configmap
    mountPath: /opt/app/VESCollector/etc/externalRepo/3gpp/rep/sa5/MnS/blob/SA88-Rel16/OpenAPI
    optional: false  //If set to false, the configMap must be present in order for the microservice's pod to start. Defaults to true.

...

Code Block

language	java
theme	Midnight		Midnight

	"policy_info":{
		"policy":[
			{
			    "node_label":"tca_policy_00",
			    "policy_model_id":"onap.policies.monitoring.cdap.tca.hi.lo.app"
				"policy_id":"tca_policy_id_10",
			},
			{
				"node_label":"tca_policy_11",
				"policy_id	"policy_info":{
		"policy":[
			{
			    "node_label":"tca_policy_id_0011",
			    	"policy_model_id":"onap.policies.monitoring.cdap.tca.hi.lo.app"
			}
		]
	}

Values.yaml specification

#dcaePolicySyncImage: onap/org.onap.dcaegen2.deployments.dcae-services-policy-sync:1.0.1 → From base template
policies:
duration: 300 → default

policyRelease: onap
policyID: |
'["tca_policy_id_11"

...

,"tca_policy_id_10"

...

]' → coming from spec file

5. POSTGRES SUPPORT

Component Spec

Code Block

language	java
theme	Midnight

        "databases": {
          "description": "The databases the application is connecting to using the pgaas",
          "type": "object",
          "additionalProperties": {
            "type": "string",

Values.yaml specification

...

5. POSTGRES SUPPORT

Component Spec

Code Block

language	java
theme	Midnight

        "databases": {
enum": [
              "descriptionpostgres":
   "The databases the application is connecting to using the pgaas",]
          "type": "object",}
        },

Need secret suffix or retrieve from spec-name?

Values.yaml specification

Code Block

language	yml
theme	Midnight

#################################################################
#  "additionalProperties": {Secrets Configuration.
#################################################################
secrets:
  - uid: pg-user-creds
    name: '{{  include "type": "string",common.release" . }}-pmsh-pg-user-creds'
    type: basicAuth
    externalSecret: '{{ ternary "enum": [
(tpl (default             "postgres"
            ]"" .Values.postgres.config.pgUserExternalSecret) .) (hasSuffix "pmsh-pg-user-creds" .Values.postgres.config.pgUserExternalSecret) }}'
    login: '{{     }.Values.postgres.config.pgUserName }}'
        },

Need secret suffix or retrieve from spec-name?

Values.yaml specification

Code Block

language	yml
theme	Midnight

#################################################################
# Secrets Configuration.
#################################################################
secrets:
  - uid: &pgUserCredsSecretUid pg-user-credspassword: '{{ .Values.postgres.config.pgUserPassword }}'
    passwordPolicy: generate


postgres:
  nameOverride: dcae-pmsh-postgres
  service:
    name: &pgUserCredsSecretName '{{ include "common.release" . }}dcae-pmsh-postgres
    name2: dcae-pmsh-pg-user-creds'primary
    typename3: basicAuthdcae-pmsh-pg-replica
  container:
  externalSecret: '{{ ternary "" (tpl (default "" .Values.postgres.config.pgUserExternalSecret) .) (hasSuffix "pmsh-pg-user-creds" .Values.postgres.config.pgUserExternalSecret) }}'
    login: '{{ .Values.postgres.config.pgUserName }}' name:
      primary: dcae-pmsh-pg-primary
      replica: dcae-pmsh-pg-replica
  persistence:
    password: '{{ .Values.postgres.config.pgUserPassword }}'mountSubPath: pmsh/data
    passwordPolicymountInitPath: generate


postgrespmsh
  config:
  nameOverride: dcae-pmsh-postgres
  service: pgUserName: pmsh
    namepgDatabase: dcae-pmsh-postgres
    name2: dcae-pmsh-pg-primary
    name3: dcae-pmsh-pg-replica
  container:
    name:
      primary: dcae-pmsh-pg-primary
      replica: dcae-pmsh-pg-replica
  persistence:
    mountSubPath: pmsh/data
    mountInitPath: pmsh
  config:
    pgUserName: pmsh
    pgDatabase: pmsh
    pgUserExternalSecret: *pgUserCredsSecretName

Note: applicationEnv setting if required should be mapped from spec as-is (req#1). Example above contains <pmsh> part of secret name and PG name which should be mapped to component-name from spec file

6. DMAAP – Secure Topic/Feed (WIP)

Component Spec

TBD

Values.yaml specification

Code Block

language	yml
theme	Midnight

#################################################################
# Secrets Configuration.
#################################################################
secrets:
  - uid: &aafCredsUID aafcreds
    type: basicAuth
    login: '{{ .Values.aafCreds.identity }}'
    password: '{{ .Values.aafCreds.password }}'
    passwordPolicy: required


# AAF Credentials
aafCreds:
  identity: dcae@dcae.onap.org
  password: demo123456!

credentials:
- name: AAF_USER
  uid: *aafCredsUID
  key: login
- name: AAF_PASSWORD
  uid: *aafCredsUID
  key: password

Note: applicationConfig should use same names as defined under credentials

Example:

enable_tls: true
aaf_identity: ${AAF_USER}
aaf_password: ${AAF_PASSWORD}
streams_publishes:
      ves-3gpp-fault-supervision:
      type: kafka
      aaf_credentials:
        username: ${AAF_USER}
        password: ${AAF_PASSWORD}
      kafka_info:
        bootstrap_servers: message-router-kafka:9092
        topic_name: SEC_3GPP_FAULTSUPERVISION_OUTPUT

7. SERVICE MAPPING

Component Spec

Code Block

language	js
theme	Midnight

"auxilary": {
.
.
"helm": {
      "services": [
        {
          "type": "NodePort",
          "name": "dcae-ves-collector",
          "ports": [
             {
                "name": "http",
                "port": 8443,
                "plain_port": 8080,
                "port_protocol": "http",
                "nodePort": 17,
                "useNodePortExt": true
             }
          ]
       }
      ]
    }
  }
.
.
}

Schema change required need to determine if nodeport vs clusterip
- Require type/name/ports
  - type - Nodeport or ClusterIPO
  - ports - list of objects mapped from spec as-is
  - constraints for ports can be added later

https://git.onap.org/dcaegen2/platform/tree/mod/component-json-schemas/component-specification/dcae-cli-v2/component-spec-schema.json

...

pgUserExternalSecret: '{{ include "common.release" . }}-pmsh-pg-user-creds'

Note: applicationEnv setting if required should be mapped from spec as-is (req#1). Example above contains <pmsh> part of secret name and PG name which should be mapped to component-name from spec file

Requirement.yaml

Code Block

theme	Midnight
title	requirement yaml content

  - name: postgres
    version: ~8.x-0
    repository: '@local'
    condition: postgres.enabled

6. DMAAP – Secure Topic/Feed (WIP)

Component Spec

TBD

Values.yaml specification

Code Block

language	yml
theme	Midnight

#################################################################
# Secrets Configuration.
#################################################################
secrets:
  - uid: &aafCredsUID aafcreds
    type: basicAuth
    login: '{{ .Values.aafCreds.identity }}'
    password: '{{ .Values.aafCreds.password }}'
    passwordPolicy: required


# AAF Credentials
aafCreds:
  identity: dcae@dcae.onap.org
  password: demo123456!

credentials:
- name: AAF_USER
  uid: *aafCredsUID
  key: login
- name: AAF_PASSWORD
  uid: *aafCredsUID
  key: password

Note: applicationConfig should use same names as defined under credentials

Example:

enable_tls: true
aaf_identity: ${AAF_USER}
aaf_password: ${AAF_PASSWORD}
streams_publishes:
      ves-3gpp-fault-supervision:
      type: kafka
      aaf_credentials:
        username: ${AAF_USER}
        password: ${AAF_PASSWORD}
      kafka_info:
        bootstrap_servers: message-router-kafka:9092
        topic_name: SEC_3GPP_FAULTSUPERVISION_OUTPUT

7. SERVICE MAPPING

Component Spec

Code Block

language	js
theme	Midnight

"auxilary": {
.
.
"helm": {
      "services": [
        {
          "type": "NodePort",
          "name": "dcae-ves-collector",
          "ports": [
             {
                "name": "http",
                "port": 8443,
                "plain_port": 8080,
                "port_protocol": "http",
                "nodePort": 17,
                "useNodePortExt": true
             }
          ]
       }
      ]
    }
  }
.
.
}

Schema change required need to determine if nodeport vs clusterip
- Require type/name/ports
  - type - Nodeport or ClusterIPO
  - ports - list of objects mapped from spec as-is
  - constraints for ports can be added later

https://git.onap.org/dcaegen2/platform/tree/mod/component-json-schemas/component-specification/dcae-cli-v2/component-spec-schema.json

Values.yaml specification

Code Block

language	yml
theme	Midnight

service:
  type: ClusterIP
  name: dcae-tcagen2
  ports:
    - port: 9091
      name: http

OR

Code Block

language	yml
theme	Midnight

global:
  nodePortPrefix: 302
  nodePortPrefixExt: 304

	# service configuration
service:
  type: NodePort
  name: dcae-ves-collector
  ports:
    - name: http
      port: 8443
      plain_port: 8080
      port_protocol: http
      nodePort: 17
      useNodePortExt: true

OR

Based on https://gerrit.onap.org/r/c/oom/+/121390

Code Block

language	yml
theme	Midnight

service:
  type: NodePort
  name: ClusterIP
  name: dcae-tcagen2dcae-ves-collector
  has_internal_only_ports: true
  ports:
    - portname: 9091http
      nameport: http

OR

Code Block

language	yml
theme	Midnight

global:8443
  nodePortPrefix:  302
  nodePortPrefixExtplain_port: 3048080

	# service configuration
service:
  type: NodePort
  nameport_protocol: dcae-ves-collectorhttp
  ports:
    - namenodePort: http17
      portuseNodePortExt: 8443true
    -  plain_portname: 8080metrics
      port_protocol: http4444
       nodePort: 17
      useNodePortExt: true

OR

Based on https://gerrit.onap.org/r/c/oom/+/121390

...

language	yml
theme	Midnight

...

internal_only: true

REVISED V3 SPEC

Component	V3 Schema	V2 Schema	With CMPV2	With Postgres	With Policy
VESCollector	vescollector-componentspec-v3-helm	vescollector-componentspec	vescollector-componentspec-cmpv2-v3-helm	vescollector-componentspec-postgres-v3-helm
TCAgen2	tcagen2_spec-v3-helm	tcagen2_spec			tcagen2_spec-policy-v3-helm
PRH	prh-componentspec-v3-helm (pending test)	prh-componentspec
hv_vescollector	hv-ves-collector-componentspec-v3-helm (pending test)	hv-ves-collector.componentspec
PM-Mapper	pmmapper-component-spec-v3-helm (need to update publisher and subscriber and pending test)	pmmapper-component-spec
DataFileCollector (DFC)	datafile-component-spec-v3-helm (need to update publisher and subscriber and pending test)	datafile-component-spec

REFERENCE

Discussed ppt slides Helm_deployment.pptx

...

Space shortcuts

Page tree

Versions Compared

Old Version 38

New Version Current

Key

USECASES

REQUIREMENTS

1. ENV SETTING SUPPORT

Component Spec

REQUIREMENTS

1. ENV SETTING SUPPORT

Component Spec

Values.yaml specification

5. POSTGRES SUPPORT

Component Spec

Values.yaml specification

5. POSTGRES SUPPORT

Component Spec

Values.yaml specification

Values.yaml specification

6. DMAAP – Secure Topic/Feed (WIP)

Component Spec

Values.yaml specification

7. SERVICE MAPPING

Component Spec

6. DMAAP – Secure Topic/Feed (WIP)

Component Spec

Values.yaml specification

7. SERVICE MAPPING

Component Spec

Values.yaml specification

REVISED V3 SPEC

REFERENCE