The policy-sync sidecar is a simple utility that abstracts the ONAP policy interface. It is designed to function well as a Kubernetes sidecar container that is injected into a pod.
Table of Contents |
---|
Overview
The sync utility will:
- Periodically check policy (e.g. every 20 minutes) to see if there have been any changes that were not received over the dmaap
- If either a matching dmaap notification arrives or the periodic check reveals updates, it will pull the latest policies from the PDP using the rest api and dump them to a file for consumption by the microservice.
draw.io Diagram | ||||||||||||||||||
---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
|
Configuration
configuration is done via environment variables to support varied environments easily.
ENV_VARIABLE | flag | Description | Example |
---|---|---|---|
POLICY_SYNC_PDP_URL | --pdp | PDP URL to query | https://policy-conexus-ist-02.ecomp.cci.att.com:30281 |
POLICY_SYNC_FILTER | --filter | A regular expression or full policy name that the puller should track and query the PDP | DCAE.Config_MS_AGING_UVERSE_.* |
POLICY_SYNC_ID | --id | This just brings back a specific policy name for those who don't need or want a regular expression. | XYZ |
POLICY_SYNC_OUTFILE | --outfile | The output file to write the policy inventory to | /opt/etc/policies.json |
POLICY_SYNC_CHECKINTERVAL | --checkinterval | How often to check the PDP periodically for resiliency purposes | 20m |
POLICY_SYNC_USER | --user | Optional Username to use as part of basic auth for the PDP | userxyz |
POLICY_SYNC_PASS | --password | Optional Password to use as part of basic auth for the PDP | password123 |
Interface with Policy
Sidecar to use ONAP's policy API to interface and retrieve policies as described: Policy Life Cycle API — onap master documentation.
GET /policy/api/v1/policies (For Policy Filter based queries).
Note: Current policy API does not offer regex based querying...so the only way to implement this is to pull all policies and filter accordingly on client side.
Retrieve all versions of available policies
- Description: Returns all version of available policies
- Consumes: [‘application/json’, ‘application/yaml’]
- Produces: [‘application/json’, ‘application/yaml’]
Parameters
Name | Position | Description | Type |
---|---|---|---|
X-ONAP-RequestID | header | RequestID for http transaction | string |
mode | query | Fetch mode for policies, BARE for bare policies (default), REFERENCED for fully referenced policies | string |
Responses
200 - successful operation
401 - Authentication Error
403 - Authorization Error
404 - Resource Not Found
500 - Internal Server Error
GET /policy/api/v1/policytypes/{policyTypeId}/versions/{policyTypeVersion}/policies/{policyId}/versions/latest (For policy id based queries).
Retrieve the latest version of a particular policy
- Description: Returns the latest version of specified policy
- Produces: [‘application/json’, ‘application/yaml’]
Parameters
Name | Position | Description | Type |
---|---|---|---|
policyTypeId | path | ID of policy type | string |
policyTypeVersion | path | Version of policy type | string |
policyId | path | ID of policy | string |
X-ONAP-RequestID | header | RequestID for http transaction | string |
Responses
200 - successful operation; Latest version of specified policy matching specified policy type will be returned.
401 - Authentication Error
403 - Authorization Error
404 - Resource Not Found
500 - Internal Server Error
DMaaP Notifications
Utilize the PAP's DMaaP Notification URL to identify changes to policy and act accordingly: https://docs.onap.org/projects/onap-policy-parent/en/latest/pap/pap.html#dmaap-api
Interface with a microservice
Communication with the main application
A shared filesystem will serve as a method of inter-process communication between the policy-sync utility and the main application. The sync utility will update a json file which contains the current output of the /getConfig REST API call whenever a policy is created, updated, or removed from the system. The primary application should watch the json file that the policy-sync utility is creating and trigger app reconfiguration as needed.
To accomplish this applications could use:
- Shell utilities based on the inotify kernel subsystem that allow you to run commands whenever a file is changed. Busybox's inotifyd can be used for this (included in the alpine based images here)
- Java's built in watch service
- Python modules such as watchdog.
Sync Utility Output
In order to make it easier to integrate, formatting and structure are designed to mimic the policies key returned by consul and/or the config binding service. Microservices can then trade calls to the config binding service for reads of this file.
Code Block | ||
---|---|---|
| ||
{
"policies": {
"items": [
{
"config": {
"service": "DCAE_HighlandPark_AgingConfig",
"location": " Edge",
"uuid": "TestUUID",
"policyName": "DCAE.AGING_UVERS_PROD_Tosca_HP_GOC_Model_cl55973_IT64_testAging",
"configName": "DCAE_HighlandPark_AgingConfig",
"templateVersion": "1607",
"priority": "4",
"version": 11.0,
"policyScope": "resource=Test1,service=vSCP,type=configuration,closedLoopControlName=vSCP_F5_Firewall_d925ed73_8231_4d02_9545_db4e101f88f8",
"riskType": "test",
"riskLevel": "2",
"guard": "False",
"content": {
"signature": {
"filter_clause": "event.faultFields.alarmCondition LIKE('%chrisluckenbaugh%')"
},
"name": "testAging",
"context": [
"PROD"
],
"priority": 1,
"prePublishAging": 40,
"preCorrelationAging": 20
},
"policyNameWithPrefix": "DCAE.AGING_UVERSE_PSL_Tosca_HP_GOC_Model_cl55973_IT64_testAging"
},
"matchingConditions": {
"ECOMPName": "DCAE",
"ONAPName": "DCAE",
"ConfigName": "DCAE_HighlandPark_AgingConfig",
"service": "DCAE_HighlandPark_AgingConfig",
"uuid": "TestUUID",
"Location": " Edge"
},
"policyConfigMessage": "Config Retrieved! ",
"policyConfigStatus": "CONFIG_RETRIEVED",
"policyName": "DCAE.Config_MS_AGING_UVERSE_PROD_Tosca_HP_AGING_Model_cl55973_IT64_testAging.70.xml",
"policyType": "MicroService",
"policyVersion": "70",
"property": null,
"responseAttributes": {},
"type": "JSON"
}
]
}
} |
Running the sync utility as a sidecar container (For use with HELM deployments, etc.).
In this configuration, the utility should be injected as a sidecar container that is part of your pod but isolated from the main container. The policy-sync container and main container should communicate using a Kubernetes shared volume
Example K8s pod manifest (which would produce something like the diagram above):
Code Block | ||
---|---|---|
| ||
# policy-config-map
apiVersion: v1
kind: policy-config-map
metadata:
name: special-config
namespace: default
data:
POLICY_SYNC_USER: myusername
POLICY_SYNC_PASS: mypassword
POLICY_SYNC_PDP_URL: https://policy-conexus-ist-02.ecomp.cci.att.com:30281
POLICY_SYNC_FILTER: '["DCAE.Config_MS_AGING_UVERSE_PROD.*"]'
---
apiVersion: v1
kind: Pod
metadata:
name: Sidecar sample app
spec:
restartPolicy: Never
# The shared volume that the two containers use to communicate...empty dir for simplicity
volumes:
- name: policy-shared
emptyDir: {}
containers:
# Sample app that uses inotifyd (part of busybox/alpine). For demonstration purposes only...
- name: main
image: dockercentral.it.att.com:5100/com.att.ecompcntr.public/baseimages/dcae-alpine-java8:1.1.0
volumeMounts:
- name: policy-shared
mountPath: /etc/policies.json
subPath: policies.json
# For details on what this does see: https://wiki.alpinelinux.org/wiki/Inotifyd
# you can replace '-' arg below with a shell script to do more interesting
cmd: [ "inotifyd", "-", "/etc/policies.json:c" ]
# The sidecar app which keeps the policies in sync
- name: policy-sync
image: dockercentral.it.att.com:5100/com.att.dcae.hp/policy-sync:0.0.1
envFrom:
- configMapRef:
name: special-config
volumeMounts:
- name: policy-shared
mountPath: /etc/policies
|
Monitoring failures
Most issues should be with the sync container's connectivity to PDP (e.g. connection errors, credential issues, etc..). When these errors arise, the impact may be that a mS no longer receives new policies, receives old policies, receives policies slowly, etc.
For monitoring these kind of errors we can:
- Include a prometheus endpoint that can be automatically discovered and scraped by Prometheus to track numeric metrics (e.g. # of failures, # of collected policies, etc.).
- Log errors and other info to the pod logs. Basic troubleshooting could then be done using kubectl logs. You could then use log forwarding tools to route logs to a logging index (Azure monitor, Elasticsearch, loki, splunk).
- Log to a file in the /opt/logs filesystem on the worker node...then pick it up via splunk for alerting (probably necessary for EOM).
Attributions
Inspired by the open source git-sync utility which essentially does the same thing for git repositories.
A similar sidecar pattern is also utilized by the AAF-Certificate renewal container (already widely deployed in DCAE).
- The existing policy handler microservice: https://github.com/onap/dcaegen2-platform-policy-handler/tree/master/policyhandler
...