Wiki to track the design requirements for Helm generator to support
Jira | ||||||
---|---|---|---|---|---|---|
|
Table of Contents
USECASES
- Build helm chart generator taking following inputs (values.yaml) and templates to generate chart package and perform lint for consolidated charts.
Helm Chart directory structure
templates:
(Include dependent charts along with main chart so that the validation can be done)
Helm lint (checks for syntax) – Initial validation can be just done with lint.
Validation needs to be configured (default – enabled and can be disabled) - DONE -
a. Run it as standalone tool - so that it can be run with the given input and generate helm chart package.
Note: This is for testing purpose only until the integration with Catalog Service is done.
tool parameters are Input template directory and output as tar - DONE
2. Separate Values.yaml into separate templates and verify dynamic values.yaml generation (and parameter substitution
Configuration/parameters required common across MS ( highlighted in yellow in ppt)
Configuration/parameters to be templatized and values sourced from ComponentSpec ( highlighed in Red )
Note : Optional configuration/parameters to be templatized and included based on flags/properties from ComponentSpec (Covered under 6 to 10)
The generator must consolidate these separate base values.template and created required values.yaml
<Need to add from ppt>
TEST: Generated charts must be validated in ONAP lab K8S environment
Refer Sample Chart Yaml mapping from component spec and Sample Values yaml mapping from component spec in the requirements document.
Outstanding - 06/01/2021
- stream_publish/stream_subscribe handle as string; change VES spec and test
- readiness mapping Include PORT/initialdelaySeconds into auxillary.healthcheck on component spec schema as optional parameter for mapping into readiness check
- globalNodeportPrefix - make default from base template. (It should be already supported)
...
3. Identify Component-spec schema changes for ENV setting mapping (refer REQ #1 below) DONE -
- - Spec schema sample to be added under REQ1 section and also updated component spec schema file - Nick
- - Verify if the service configuration can be supported as list under common/service - Vijay Venkatesh Kumar
List is not supported in current common/service template
4. Identify Component-spec schema changes for Service mapping (and nodeport) (refer REQ #7 below) DONE -
- Spec schema sample to be added under REQ1 section and also updated component spec schema file; submit to gerrit with new updates (including #2, #3, #4)
5. Build helm chart generator taking as inputs template directory and template list file to be used for chart generation and perform lint for consolidated charts. - No longer applicable; based on design
Note: Use base/default template if corresponding template not found on specified template directory
6. Support MAPPING requirement – ENV SETTING (refer REQ #1 for details) - DONE - 1.0.0
Schema Validation to be integrated - tracked separately under task #24
7. Support MAPPING requirement – CMPv2 Certificates (refer REQ #3 for details) - Target 06/24
Include support for base + addon chart structure (certificate.yaml will be optional) - DONE -1.0.1?
Add validation if enabled in spec and certificate.yaml not provided - tool must error - switch to use use_external_tls/flag - DONE 1.3.2
8. Support MAPPING requirement – Postgres (refer REQ #5 for details) - Target 06/30 DONE 1.3.2
9. Support MAPPING requirement – Policy Sidecar (refer REQ #4 for details) Target 06/17 DONE 1.3.2
Format written out into values.yaml not as expected
10. Support SERVICE MAPPING based on spec file (refer REQ #7 for details) - DONE - 1.0.0
Schema Validation to be integrated - tracked separately under task #24
11. Support MAPPING requirement – ConfigMap support (refer REQ #2 for details) Target 06/30 DONE 1.3.2
12. Support MAPPING requirement – DMAAP Secure Topic/Feed (refer REQ #6 for details)
13. Create user guide for the tool detailing all command-line options/override
14. Submit code to ONAP; ensure compliance to ONAP coding standard and test coverage requirement (atleast 80%) and verify the library build/pushed to ONAP:nexus
15. Integrate tool into MOD/Runtime or MOD2/CatalogService
16. Verify E2E for ONAP DCAE MS spec file (TCA w/policy, PM-Mapper, VES, and validate corresponding charts generated in ONAP lab if components can be successfully deployed
17. Add distribution support in tool. Additional configuration support needed for below parameters either in tool property file or CMD line options
DistributionEnabled
DistributionURL
DistributionUsername
DistributionPwd
DistributionFormat - tgz or as directory
18. Provide REST interface to support HELM generation
19. Input Spec validation
20. Comments should be cascaded from template and when new yaml is generator and also preserve order
21. Secret mapping requirement TBD and pv mapping - Vijay Venkatesh Kumar - Target 06/30 DONE 1.3.2
- CMPv2 + application_env
Option 1: Change spec schema to include secret name as additional input and use that for helm values.yaml setting
Option 2: Have tool set the secret name as <component-name>-cmpv2-keystore-password - Current chosen option - Target
22. Support chart generation as directory (instead of tgz) via configuration (added on )
23. Support dependency download (helm dep up) from repo/access based on deployment configration (added on )
REQUIREMENTS
1. ENV SETTING SUPPORT
Component Spec
- Need spec schema update to include list of parameters (key/value for applicationEnv) -->https://git.onap.org/dcaegen2/platform/tree/mod/component-json-schemas/component-specification/dcae-cli-v2/component-spec-schema.json
Code Block | ||||
---|---|---|---|---|
| ||||
"auxilary": {
.
.
"helm": {
"applicationEnv": {
"PMSH_PG_URL": "dcae-pmsh-pg-primary",
"PMSH_PG_USERNAME": {
"secretUid": "pgUserCredsSecretUid" |
24. Add schema validator module; keep the schema definiton overriddable (either through CI or api parameter); this should be configurable - default enabled (added on based on svc mapping test) - Target
25. Testing with ONAP spec files on ONAP lab with componen-spec (ves, pm-mapper, dfc, tcagen2, hv-ves, prh) - Sivakumar Santharam ONGOING
- Modify spec to align with new schema
- https://git.onap.org/dcaegen2/collectors/hv-ves/tree/dpo/spec/hv-ves-collector.componentspec.json
- https://git.onap.org/dcaegen2/collectors/ves/tree/dpo/spec/vescollector-componentspec.json
- https://git.onap.org/dcaegen2/analytics/tca-gen2/tree/dcae-analytics/dpo/tcagen2_spec.json
- https://git.onap.org/dcaegen2/services/pm-mapper/tree/dpo/spec/pmmapper-component-spec.json
- https://git.onap.org/dcaegen2/collectors/datafile/tree/datafile-app-server/dpo/spec/datafile-component-spec.json
- https://git.onap.org/dcaegen2/services/prh/tree/dpo/spec/prh.componentspec.json
- Generate charts and deploy and check if component comes up clean
- Environment for test to be setup/checked (onap5) - Vijay Venkatesh Kumar - DONE (06/17/2021)
- Verify spec can be run through bp-gen and deployed (backward compatibility)
26. Submit updates on each spec json update to gerrit (to be verified if updated spec breaks bp-gen creation - incase new fields introduced are not ignored)
27. Support for readinesscheck for script based execution (e.g PMMapper); port/endpoint flagged as mandatory fields which are required only for HTTP - added
28. Dynamic secret creation and mapping to env variable - support from Helm tool - added
TEST ISSUES TRACKED UNDER Jira server ONAP Jira serverId 425b2b0a-557c-3c0c-b515-579789cceedb key DCAEGEN2-2839
REQUIREMENTS
1. ENV SETTING SUPPORT
Component Spec
- Need spec schema update to include list of parameters (key/value for applicationEnv) -->https://git.onap.org/dcaegen2/platform/tree/mod/component-json-schemas/component-specification/dcae-cli-v2/component-spec-schema.json
Code Block | ||||
---|---|---|---|---|
| ||||
"auxilary": {
.
.
"helm": {
"applicationEnv": {
"PMSH_PG_URL": "dcae-pmsh-pg-primary",
"PMSH_PG_USERNAME": {
"secretUid": "pgUserCredsSecretUid",
"key": "login"
},
"PMSH_PG_PASSWORD": {
"secretUid": "pgUserCredsSecretUid",
"key": "password"
}
}
}
.
.
} |
...
#dcaePolicySyncImage: onap/org.onap.dcaegen2.deployments.dcae-services-policy-sync:1.0.1 → From base template
policies:
duration: 300 → default
policyRelease: onap
policyID: |
'["tca_policy_id_11","tca_policy_id_10"]' → coming from spec file
5. POSTGRES SUPPORT
Component Spec
...
Code Block | ||||
---|---|---|---|---|
| ||||
################################################################# # Secrets Configuration. ################################################################# secrets: - uid: pg-user-creds name: '{{ include "common.release" . }}-pmsh-pg-user-creds' type: basicAuth externalSecret: '{{ ternary "" (tpl (default "" .Values.postgres.config.pgUserExternalSecret) .) (hasSuffix "pmsh-pg-user-creds" .Values.postgres.config.pgUserExternalSecret) }}' login: '{{ .Values.postgres.config.pgUserName }}' password: '{{ .Values.postgres.config.pgUserPassword }}' passwordPolicy: generate postgres: nameOverride: dcae-pmsh-postgres service: name: dcae-pmsh-postgres name2: dcae-pmsh-pg-primary name3: dcae-pmsh-pg-replica container: name: primary: dcae-pmsh-pg-primary replica: dcae-pmsh-pg-replica persistence: mountSubPath: pmsh/data mountInitPath: pmsh config: pgUserName: pmsh replicapgDatabase: dcae-pmsh-pg-replica persistence: mountSubPath: pmsh/data mountInitPath: pmsh config: pgUserName: pmsh pgUserExternalSecret: '{{ include "common.release" . }}-pmsh-pg-user-creds' |
Note: applicationEnv setting if required should be mapped from spec as-is (req#1). Example above contains <pmsh> part of secret name and PG name which should be mapped to component-name from spec file
Requirement.yaml
Code Block | ||||
---|---|---|---|---|
| ||||
- name: postgres version: ~8.x-0 pgDatabaserepository: pmsh pgUserExternalSecret: '{{ include "common.release" . }}-pmsh-pg-user-creds' |
Note: applicationEnv setting if required should be mapped from spec as-is (req#1). Example above contains <pmsh> part of secret name and PG name which should be mapped to component-name from spec file
Requirement.yaml
Code Block | ||||
---|---|---|---|---|
| ||||
- name: postgres
version: ~8.x-0
repository: '@local'
condition: postgres.enabled |
6. DMAAP – Secure Topic/Feed (WIP)
Component Spec
TBD
Values.yaml specification
Code Block | ||||
---|---|---|---|---|
| ||||
#################################################################
# Secrets Configuration.
#################################################################
secrets:
- uid: &aafCredsUID aafcreds
type: basicAuth
login: '{{ .Values.aafCreds.identity }}'
password: '{{ .Values.aafCreds.password }}'
passwordPolicy: required
# AAF Credentials
aafCreds:
identity: dcae@dcae.onap.org
password: demo123456!
credentials:
- name: AAF_USER
uid: *aafCredsUID
key: login
- name: AAF_PASSWORD
uid: *aafCredsUID
key: password |
Note: applicationConfig should use same names as defined under credentials
Example:
enable_tls: true
aaf_identity: ${AAF_USER}
aaf_password: ${AAF_PASSWORD}
streams_publishes:
ves-3gpp-fault-supervision:
type: kafka
aaf_credentials:
username: ${AAF_USER}
password: ${AAF_PASSWORD}
kafka_info:
bootstrap_servers: message-router-kafka:9092
topic_name: SEC_3GPP_FAULTSUPERVISION_OUTPUT
7. SERVICE MAPPING
Component Spec
Code Block | ||||
---|---|---|---|---|
| ||||
"auxilary": {
.
.
"helm": {
"services": [
{
"type": "NodePort",
"name": "dcae-ves-collector",
"ports": [
{
"name": "http",
"port": 8443,
"plain_port": 8080,
"port_protocol": "http",
"nodePort": 17,
"useNodePortExt": true
}
]
}
]
}
}
.
.
} |
- Schema change required need to determine if nodeport vs clusterip
- Require type/name/ports
- type - Nodeport or ClusterIPO
- ports - list of objects mapped from spec as-is
- constraints for ports can be added later
- Require type/name/ports
...
'@local'
condition: postgres.enabled |
6. DMAAP – Secure Topic/Feed (WIP)
Component Spec
TBD
Values.yaml specification
Code Block | ||||
---|---|---|---|---|
| ||||
#################################################################
# Secrets Configuration.
#################################################################
secrets:
- uid: &aafCredsUID aafcreds
type: basicAuth
login: '{{ .Values.aafCreds.identity }}'
password: '{{ .Values.aafCreds.password }}'
passwordPolicy: required
# AAF Credentials
aafCreds:
identity: dcae@dcae.onap.org
password: demo123456!
credentials:
- name: AAF_USER
uid: *aafCredsUID
key: login
- name: AAF_PASSWORD
uid: *aafCredsUID
key: password |
Note: applicationConfig should use same names as defined under credentials
Example:
enable_tls: true
aaf_identity: ${AAF_USER}
aaf_password: ${AAF_PASSWORD}
streams_publishes:
ves-3gpp-fault-supervision:
type: kafka
aaf_credentials:
username: ${AAF_USER}
password: ${AAF_PASSWORD}
kafka_info:
bootstrap_servers: message-router-kafka:9092
topic_name: SEC_3GPP_FAULTSUPERVISION_OUTPUT
7. SERVICE MAPPING
Component Spec
Code Block | ||||
---|---|---|---|---|
| ||||
"auxilary": {
.
.
"helm": {
"services": [
{
"type": "NodePort",
"name": "dcae-ves-collector",
"ports": [
{
"name": "http",
"port": 8443,
"plain_port": 8080,
"port_protocol": "http",
"nodePort": 17,
"useNodePortExt": true
}
]
}
]
}
}
.
.
} |
- Schema change required need to determine if nodeport vs clusterip
- Require type/name/ports
- type - Nodeport or ClusterIPO
- ports - list of objects mapped from spec as-is
- constraints for ports can be added later
- Require type/name/ports
Values.yaml specification
Code Block | ||||
---|---|---|---|---|
| ||||
service:
type: ClusterIP
name: dcae-tcagen2
ports:
- port: 9091
name: http |
OR
Code Block | ||||
---|---|---|---|---|
| ||||
global:
nodePortPrefix: 302
nodePortPrefixExt: 304
# service configuration
service:
type: NodePort
name: dcae-ves-collector
ports:
- name: http
port: 8443
plain_port: 8080
port_protocol: http
nodePort: 17
useNodePortExt: true |
OR
Based on https://gerrit.onap.org/r/c/oom/+/121390
Code Block | ||||
---|---|---|---|---|
| ||||
service: type: NodePort name: ClusterIP name: dcae-tcagen2dcae-ves-collector has_internal_only_ports: true ports: - portname: 9091http nameport: http |
OR
Code Block | ||||
---|---|---|---|---|
| ||||
global:8443 nodePortPrefix: 302 nodePortPrefixExtplain_port: 3048080 # service configuration service: type: NodePort nameport_protocol: dcae-ves-collectorhttp ports: - namenodePort: http17 portuseNodePortExt: 8443true - plain_portname: 8080metrics port_protocol: http4444 nodePort: 17 useNodePortExt: true |
OR
Based on https://gerrit.onap.org/r/c/oom/+/121390
...
language | yml |
---|---|
theme | Midnight |
...
internal_only: true |
REVISED V3 SPEC
Component | V3 Schema | V2 Schema | With CMPV2 | With Postgres | With Policy |
---|---|---|---|---|---|
VESCollector | vescollector-componentspec-v3-helm | vescollector-componentspec | vescollector-componentspec-cmpv2-v3-helm | vescollector-componentspec-postgres-v3-helm | |
TCAgen2 | tcagen2_spec-v3-helm | tcagen2_spec | tcagen2_spec-policy-v3-helm | ||
PRH | prh-componentspec-v3-helm (pending test) | prh-componentspec | |||
hv_vescollector | hv-ves-collector-componentspec-v3-helm (pending test) | hv-ves-collector.componentspec | |||
PM-Mapper | pmmapper-component-spec-v3-helm (need to update publisher and subscriber and pending test) | pmmapper-component-spec | |||
DataFileCollector (DFC) | datafile-component-spec-v3-helm (need to update publisher and subscriber and pending test) | datafile-component-spec |
REFERENCE
Discussed ppt slides Helm_deployment.pptx
...