Versions Compared

Key

  • This line was added.
  • This line was removed.
  • Formatting was changed.

Table of Contents

Overview

The PM Subscription Handler (PMSH) is a micro service written in Python, which allows for the definition and creation of PM subscriptions on one or more network function (NF) instances.

Architecture Diagram

Image Added

Artifacts

Βlueprint (deployment artifact) :

Input file (deployment input)    :

Docker image                            :  nexus3.onap.org:10001/onap/org.onap.dcaegen2.services.pmsh

Deployment Prerequisite/dependencies


Image result for cloudifyImage Added

PMSH blueprint


Deployment Prerequisite/dependencies

  1. The following components must be up and running to successfully deploy PMSH

    1. DCAE Platform
    2. SDC
    3. CLAMP 
    4. POLICY
    5. A&AI (Should have at least 1 PNF registered)

Deployment Steps

Deployment of pmsh can be done using Dashboard UI or CloudifyUI or via CLI. Below steps are based on CLI.

...


Due to a bug (DCAEGEN2-2136). In the Frankfurt release, there is a workaround required in order to deploy the PM Subscription Handler. The monitoring policy for the microservice needs to be manually inserted into CONSUL. This requires two steps:


1. The CONSUL service is no longer exposed in the Frankfurt release, this needs to be exposed in order to be accessible outside the k8s cluster.

Validate blueprint

...

Code Block
languagebash
themeMidnight
title

...

cfy blueprints validate /blueprints/k8s-pmsh.yaml

Verify Plugin versions in target Cloudify instance match to blueprint imports

Expose CONSUL service
kubectl expose svc -n onap consul-server-ui --name=x-consul-server-ui --type=NodePort

2. Insert the monitoring policy into CONSUL

Code Block
languagebash
themeMidnight
title

...

cfy plugins list

If the version of plugin used are different, update the blueprint import to match.

PUT policy into CONSUL
curl -X PUT \
  http://<k8s-node-ip>:<consul-port>/v1/kv/dcae-pmsh:policy \
  -H 'Content-Type: application/json' \
  -d '{
    "subscription": {
        "subscriptionName": "test-subscription",
        "administrativeState": "LOCKED",
        "fileBasedGP": 15,
        "fileLocation": "/pm/pm.xml",
        "nfFilter": {
            "swVersions": [
                "1.0.0"
            ],
            "nfNames": [
                "^pnf.*"
            ]
        },
        "measurementGroups": [{
                "measurementGroup": {
                    "measurementTypes": [{
                            "measurementType": "countera"
                        },
                        {
                            "measurementType": "counterb"
                        }
                    ],
                    "managedObjectDNsBasic": [{
                            "DN": "dna"
                        },
                        {
                            "DN": "dnb"
                        }
                    ]
                }
            },
            {
                "measurementGroup": {
                    "measurementTypes": [{
                            "measurementType": "counterc"
                        },
                        {
                            "measurementType": "counterd"
                        }
                    ],
                    "managedObjectDNsBasic": [{
                            "DN": "dnc"
                        },
                        {
                            "DN": "dnd"
                        }
                    ]
                }
            }
        ]
    }
}'


Starting in Frankfurt, all DCAE service blueprints are bootstrapped, meaning they are all available in the inventory by default and do not need to be onboarded manually. Deployment of PMSH can be done using Dashboard UI/Cloudify UI or via CLI. Below steps are based on CLI using the deployment handler.

1. Get the typeId of PMSH using the inventory API

Code Block
languagebash
themeMidnight
titleGet serviceTypeId from DCAE inventory
curl https://<k8s-node-ip>:<inventory-port>/dcae-service-types


The PMSH typeID can then be taken from the response

Code Block
languagejs
themeMidnight
titleInventory response
...
{
        "owner": "admin",
        "typeName": "k8s-pmsh",
        "typeVersion": 100,
        "blueprintTemplate": "#\n# ============LICENSE_START=======================================================\n#  Copyright (C) 2020 Nordix Foundation.\n# ================================================================================\n# Licensed under the Apache License, Version 2.0 (the 'License');\n# you may not use this file except in compliance with the License.\n# You may obtain a copy of the License at\n#\n#      http://www.apache.org/licenses/LICENSE-2.0\n#\n# Unless required by applicable law or agreed to in writing, software\n# distributed under the License is distributed on an 'AS IS' BASIS,\n# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.\n# See the License for the specific language governing permissions and\n# limitations under the License.\n#\n# SPDX-License-Identifier: Apache-2.0\n# ============LICENSE_END=========================================================\n#\n\ntosca_definitions_version: cloudify_dsl_1_3\n\nimports:\n  - 'http://www.getcloudify.org/spec/cloudify/4.5.5/types.yaml'\n  - 'https://nexus.onap.org/service/local/repositories/raw/content/org.onap.dcaegen2.platform.plugins/R6/k8splugin/1.7.2/k8splugin_types.yaml'\n  - 'https://nexus.onap.org/service/local/repositories/raw/content/org.onap.ccsdk.platform.plugins/type_files/pgaas/1.1.0/pgaas_types.yaml'\n\ninputs:\n  tag_version:\n    type: string\n    description: Docker image to be used\n    default: 'nexus3.onap.org:10001/onap/org.onap.dcaegen2.services.pmsh:1.0.2'\n  replicas:\n    type: integer\n    description: Number of instances\n    default: 1\n  operational_policy_name:\n    type: string\n    default: 'pmsh-operational-policy'\n  control_loop_name:\n    type: string\n    default: 'pmsh-control-loop'\n  pmsh_publish_topic_name:\n    type: string\n    default: 'unauthenticated.DCAE_CL_OUTPUT'\n  policy_feedback_topic_name:\n    type: string\n    default: 'unauthenticated.PMSH_CL_INPUT'\n  aai_notification_topic_name:\n    type: string\n    default: 'AAI-EVENT'\n  publisher_client_role:\n    type: string\n    description: Client role to request secure access to topic\n    default: 'org.onap.dcae.pmPublisher'\n  subscriber_client_role:\n    type: string\n    description: Client role to request secure access to topic\n    default: 'org.onap.dcae.pmSubscriber'\n  dcae_location:\n    type: string\n    description: DCAE location for the subscriber, used to set up routing\n    default: 'san-francisco'\n  cpu_limit:\n    type: string\n    default: '1000m'\n  cpu_request:\n    type: string\n    default: '1000m'\n  memory_limit:\n    type: string\n    default: '1024Mi'\n  memory_request:\n    type: string\n    default: '1024Mi'\n  pgaas_cluster_name:\n    type: string\n    default: 'dcae-pg-primary.onap'\nnode_templates:\n  pgaasvm:\n    type: dcae.nodes.pgaas.database\n    properties:\n      writerfqdn: { get_input: pgaas_cluster_name }\n      name: 'pmsh'\n  pmsh:\n    type: dcae.nodes.ContainerizedServiceComponentUsingDmaap\n    interfaces:\n      cloudify.interfaces.lifecycle:\n        create:\n          inputs:\n            ports:\n              - '8443:0'\n            envs:\n              PMSH_PG_URL:\n                { get_attribute: [ pgaasvm, admin, host ] }\n              PMSH_PG_PASSWORD:\n                { get_attribute: [ pgaasvm, admin, password ] }\n              PMSH_PG_USERNAME:\n                { get_attribute: [ pgaasvm, admin, user ] }\n              PMSH_DB_NAME:\n                { get_attribute: [ pgaasvm, admin, database ] }\n\n    relationships:\n      - type: cloudify.relationships.depends_on\n        target: pgaasvm\n\n    properties:\n      service_component_type: 'dcae-pmsh'\n      service_component_name_override: 'dcae-pmsh'\n      application_config:\n        aaf_identity: 'dcae@dcae.onap.org'\n        aaf_password: 'demo123456!'\n        operational_policy_name: { get_input: operational_policy_name }\n        control_loop_name: { get_input: control_loop_name }\n        cert_path: '/opt/app/pmsh/etc/certs/cert.pem'\n        key_path: '/opt/app/pmsh/etc/certs/key.pem'\n        ca_cert_path: '/opt/app/pmsh/etc/certs/cacert.pem'\n        streams_publishes:\n          policy_pm_publisher:\n            type: message_router\n            dmaap_info:\n              topic_url: {concat: [\"https://message-router:3905/events/\", { get_input: pmsh_publish_topic_name }]}\n        streams_subscribes:\n          policy_pm_subscriber:\n            type: message_router\n            dmaap_info:\n              topic_url: {concat: [\"https://message-router:3905/events/\", { get_input: policy_feedback_topic_name }]}\n          aai_subscriber:\n            type: message_router\n            dmaap_info:\n              topic_url: {concat: [\"https://message-router:3905/events/\", { get_input: aai_notification_topic_name }]}\n      resource_config:\n        limits:\n          cpu: { get_input: cpu_limit }\n          memory: { get_input: memory_limit }\n        requests:\n          cpu: { get_input: cpu_request }\n          memory: { get_input: memory_request }\n      docker_config:\n        healthcheck:\n          endpoint: /healthcheck\n          interval: 15s\n          timeout: 1s\n          type: https\n      image: { get_input: tag_version }\n      replicas: { get_input: replicas }\n      log_info:\n        log_directory: '/var/log/ONAP/dcaegen2/services/pmsh'\n      tls_info:\n        cert_directory: '/opt/app/pmsh/etc/certs'\n        use_tls: true",
        "serviceIds": [],
        "vnfTypes": [],
        "serviceLocations": [],
        "asdcServiceId": "",
        "asdcResourceId": "",
        "asdcServiceURL": null,
        "application": "DCAE",
        "component": "dcae",
        "typeId": "92e9fcd2-17cb-4b87-b089-873e132c0415",
        "selfLink": {
            "rel": "self",
            "href": "https://10.10.10.36:31413/dcae-service-types/92e9fcd2-17cb-4b87-b089-873e132c0415"
        },
        "created": 1585228467040,
        "deactivated": null
}
...


2. Create a deployment using the deployment handler API

Code Block
languagebash
themeMidnight
titleCreate PMSH deployment
curl -X PUT \
  https://<k8s-node-ip>:<dep-handler-port>/dcae-deployments/dcae-pmsh \
  -H 'Content-Type: application/json' \
  -d '{
    "inputs": {},
	"serviceTypeId": "92e9fcd2-17cb-4b87-b089-873e132c0415"
}'


To un-deploy

Code Block
languagebash
themeMidnight
titleCreate PMSH deployment
curl -X DELETE https://<k8s-node-ip>:<dep-handler-port>/dcae-deployments/dcae-pmsh 


Initial Validation

After deployment, verify if PMSH pod is running correctly

Code Block
languagebash
themeMidnight
titleVerify Heartbeat is running
linenumberstrue
root@k8s-rancher:~# kubectl get pods -n onap | egrep "pmsh"

dep-dcae-pmsh-6b58cbc7f5-lm92g                                2/2     Running            0          149m


We can curl message router to verify that PMSH has published an event to it's configured topic. By default this is unauthenticated.DCAE_CL_OUTPUT

Code Block
languagebash
themeMidnight
titleGet from MR
curl -k https://message-router:3905/events/unauthenticated.DCAE_CL_OUTPUT/1/1


If PMSH is working correctly we should see an event published to message router

Code Block
languagebash
themeMidnight
titleResponse from MR
["{\"nfName\":\"pnf500\",\"policyName\":\"pmsh-operational-policy\",\"changeType\":\"CREATE\",\"subscription\":{\"administrativeState\":\"LOCKED\",\"subscriptionName\":\"test-subscription\",\"measurementGroups\":[{\"measurementGroup\":{\"measurementTypes\":[{\"measurementType\":\"countera\"},{\"measurementType\":\"counterb\"}],\"managedObjectDNsBasic\":[{\"DN\":\"dna\"},{\"DN\":\"dnb\"}]}},{\"measurementGroup\":{\"measurementTypes\":[{\"measurementType\":\"counterc\"},{\"measurementType\":\"counterd\"}],\"managedObjectDNsBasic\":[{\"DN\":\"dnc\"},{\"DN\":\"dnd\"}]}}],\"fileBasedGP\":15,\"fileLocation\":\"/pm/pm.xml\"},\"closedLoopControlName\":\"pmsh-control-loop\"}"]


This can also be verified by checking the logs to verify PMSH can connect to A&AI and DMaaP

Code Block
languagebash
themeMidnight
titleVerify PMSH logs
pmsh@dcae-pmsh:/opt/app/pmsh$ tail -f /var/log/ONAP/dcaegen2/services/pmsh/debug.log 
2020-03-11T08:40:12.363092|PMSH Configuration from Configbinding Service: {'config': {'aaf_identity': 'dcae@dcae.onap.org', 'control_loop_name': 'pmsh-control-loop', 'operational_policy_name': 'pmsh-operational-policy', 'key_path': '/opt/app/pmsh/etc/certs/key.pem', 'streams_publishes': {'policy_pm_publisher': {'type': 'message_router', 'dmaap_info': {'topic_url': 'https://message-router:3905/events/unauthenticated.DCAE_CL_OUTPUT'}}}, 'cert_path': '/opt/app/pmsh/etc/certs/cert.pem', 'streams_subscribes': {'policy_pm_subscriber': {'type': 'message_router', 'dmaap_info': {'topic_url': 'https://message-router:3905/events/unauthenticated.PMSH_CL_INPUT'}}, 'aai_subscriber': {'type': 'message_router', 'dmaap_info': {'topic_url': 'https://message-router:3905/events/AAI-EVENT'}}}, 'aaf_password': 'demo123456!', 'ca_cert_path': '/opt/app/pmsh/etc/certs/cacert.pem'}, 'policy': {'subscription': {'subscriptionName': 'test-subscription', 'administrativeState': 'LOCKED', 'fileBasedGP': 15, 'fileLocation': '/pm/pm.xml', 'nfFilter': {'swVersions': ['1.0.0', '1.0.1'], 'nfNames': ['^pnf.*', '^vnf.*']}, 'measurementGroups': [{'measurementGroup': {'measurementTypes': [{'measurementType': 'countera'}, {'measurementType': 'counterb'}], 'managedObjectDNsBasic': [{'DN': 'dna'}, {'DN': 'dnb'}]}}, {'measurementGroup': {'measurementTypes': [{'measurementType': 'counterc'}, {'measurementType': 'counterd'}], 'managedObjectDNsBasic': [{'DN': 'dnc'}, {'DN': 'dnd'}]}}]}}}
2020-03-11T08:40:12.363617|Administrative State did not change in the Config
2020-03-11T08:40:42.432892|PMSH Configuration from Configbinding Service: {'config': {'aaf_identity': 'dcae@dcae.onap.org', 'control_loop_name': 'pmsh-control-loop', 'operational_policy_name': 'pmsh-operational-policy', 'key_path': '/opt/app/pmsh/etc/certs/key.pem', 'streams_publishes': {'policy_pm_publisher': {'type': 'message_router', 'dmaap_info': {'topic_url': 'https://message-router:3905/events/unauthenticated.DCAE_CL_OUTPUT'}}}, 'cert_path': '/opt/app/pmsh/etc/certs/cert.pem', 'streams_subscribes': {'policy_pm_subscriber': {'type': 'message_router', 'dmaap_info': {'topic_url': 'https://message-router:3905/events/unauthenticated.PMSH_CL_INPUT'}}, 'aai_subscriber': {'type': 'message_router', 'dmaap_info': {'topic_url': 'https://message-router:3905/events/AAI-EVENT'}}}, 'aaf_password': 'demo123456!', 'ca_cert_path': '/opt/app/pmsh/etc/certs/cacert.pem'}, 'policy': {'subscription': {'subscriptionName': 'test-subscription', 'administrativeState': 'UNLOCKED', 'fileBasedGP': 15, 'fileLocation': '/pm/pm.xml', 'nfFilter': {'swVersions': ['1.0.0', '1.0.1'], 'nfNames': ['^pnf.*', '^vnf.*']}, 'measurementGroups': [{'measurementGroup': {'measurementTypes': [{'measurementType': 'countera'}, {'measurementType': 'counterb'}], 'managedObjectDNsBasic': [{'DN': 'dna'}, {'DN': 'dnb'}]}}, {'measurementGroup': {'measurementTypes': [{'measurementType': 'counterc'}, {'measurementType': 'counterd'}], 'managedObjectDNsBasic': [{'DN': 'dnc'}, {'DN': 'dnd'}]}}]}}}
2020-03-11T08:40:42.433357|Administrative State changed from "LOCKED" "to "UNLOCKED".
2020-03-11T08:40:42.941068|Publishing Event to Activate Sub: test-subscription for the nf: pnf500
2020-03-11T08:40:42.998828|Adding network functions to subscription test-subscription
2020-03-11T08:40:43.088201|NetworkFunctionModel: pnf500, Active
2020-03-11T08:40:43.180609|Listening to AAI-EVENT topic in MR.
2020-03-11T08:40:47.321604|Request sent to MR topic: https://message-router:3905/events/unauthenticated.PMSH_CL_INPUT



Functional tests

Given that the steps above were used to configure the PMSH monitoring policy with administrativeState = LOCKED. The subscription has not yet been "activated" on the network function. 

Activate an inactive subscription

If the subscription is in a LOCKED state (administrativeState=LOCKED) we can reconfigure PMSH to activate the subscription:

Code Block
languagebash
themeMidnight
titleActivate Subscription
curl -X PUT \
  http://<k8s-node-ip>:<consul-port>/v1/kv/dcae-pmsh:policy \
  -H 'Content-Type: application/json' \
  -d '{
    "subscription": {
        "subscriptionName": "test-subscription",
        "administrativeState": "UNLOCKED",
        "fileBasedGP": 15,
        "fileLocation": "/pm/pm.xml",
        "nfFilter": {
            "swVersions": [
                "1.0.0"
            ],
            "nfNames": [
                "^pnf.*"
            ]
        },
        "measurementGroups": [{
                "measurementGroup": {
                    "measurementTypes": [{
                            "measurementType": "countera"
                        },
                        {
                            "measurementType": "counterb"
                        }
                    ],
                    "managedObjectDNsBasic": [{
                            "DN": "dna"
                        },
                        {
                            "DN": "dnb"
                        }
                    ]
                }
            },
            {
                "measurementGroup": {
                    "measurementTypes": [{
                            "measurementType": "counterc"
                        },
                        {
                            "measurementType": "counterd"
                        }
                    ],
                    "managedObjectDNsBasic": [{
                            "DN": "dnc"
                        },
                        {
                            "DN": "dnd"
                        }
                    ]
                }
            }
        ]
    }
}'


We should then be able to see a CREATE event sent to message router:

Code Block
languagebash
themeMidnight
titleGet from MR
curl -k https://message-router:3905/events/unauthenticated.DCAE_CL_OUTPUT/1/1


If PMSH is working correctly we should see a CREATE event sent to message router:

Code Block
languagebash
themeMidnight
titleResponse from MR
["{\"nfName\":\"pnf500\",\"policyName\":\"pmsh-operational-policy\",\"changeType\":\"CREATE\",\"subscription\":{\"administrativeState\":\"LOCKED\",\"subscriptionName\":\"test-subscription\",\"measurementGroups\":[{\"measurementGroup\":{\"measurementTypes\":[{\"measurementType\":\"countera\"},{\"measurementType\":\"counterb\"}],\"managedObjectDNsBasic\":[{\"DN\":\"dna\"},{\"DN\":\"dnb\"}]}},{\"measurementGroup\":{\"measurementTypes\":[{\"measurementType\":\"counterc\"},{\"measurementType\":\"counterd\"}],\"managedObjectDNsBasic\":[{\"DN\":\"dnc\"},{\"DN\":\"dnd\"}]}}],\"fileBasedGP\":15,\"fileLocation\":\"/pm/pm.xml\"},\"closedLoopControlName\":\"pmsh-control-loop\"}"]


Deactivate an active subscription

If the subscription is in an UNLOCKED state (administrativeState=UNLOCKED) we can reconfigure PMSH to deactivate the subscription:

Code Block
languagebash
themeMidnight
titleDeactivate Subscription
curl -X PUT \
  http://<k8s-node-ip>:<consul-port>/v1/kv/dcae-pmsh:policy \
  -H 'Content-Type: application/json' \
  -d '{
    "subscription": {
        "subscriptionName": "test-subscription",
        "administrativeState": "LOCKED",
        "fileBasedGP": 15,
        "fileLocation": "/pm/pm.xml",
        "nfFilter": {
            "swVersions": [
                "1.0.0"
            ],
            "nfNames": [
                "^pnf.*"
            ]
        },
        "measurementGroups": [{
                "measurementGroup": {
                    "measurementTypes": [{
                            "measurementType": "countera"
                        },
                        {
                            "measurementType": "counterb"
                        }
                    ],
                    "managedObjectDNsBasic": [{
                            "DN": "dna"
                        },
                        {
                            "DN": "dnb"
                        }
                    ]
                }
            },
            {
                "measurementGroup": {
                    "measurementTypes": [{
                            "measurementType": "counterc"
                        },
                        {
                            "measurementType": "counterd"
                        }
                    ],
                    "managedObjectDNsBasic": [{
                            "DN": "dnc"
                        },
                        {
                            "DN": "dnd"
                        }
                    ]
                }
            }
        ]
    }
}'


We should then be able to see a DELETE event sent to message router:

Code Block
languagebash
themeMidnight
titleGet from MR
curl -k https://message-router:3905/events/unauthenticated.DCAE_CL_OUTPUT/1/1


If PMSH is working correctly we should see a DELETE event sent to message router:

Code Block
languagebash
themeMidnight
titleResponse from MR
["{\"nfName\":\"pnf500\",\"policyName\":\"pmsh-operational-policy\",\"changeType\":\"DELETE\",\"subscription\":{\"administrativeState\":\"LOCKED\",\"subscriptionName\":\"test-subscription\",\"measurementGroups\":[{\"measurementGroup\":{\"measurementTypes\":[{\"measurementType\":\"countera\"},{\"measurementType\":\"counterb\"}],\"managedObjectDNsBasic\":[{\"DN\":\"dna\"},{\"DN\":\"dnb\"}]}},{\"measurementGroup\":{\"measurementTypes\":[{\"measurementType\":\"counterc\"},{\"measurementType\":\"counterd\"}],\"managedObjectDNsBasic\":[{\"DN\":\"dnc\"},{\"DN\":\"dnd\"}]}}],\"fileBasedGP\":15,\"fileLocation\":\"/pm/pm.xml\"},\"closedLoopControlName\":\"pmsh-control-loop\"}"]


Dynamic Configuration Update

As the PMSH service periodically polls Consul KV using configbindingService API's - the run time configuration of PMSH

Deploy Service

Code Block
languagebash
themeMidnight
titleUpload and deploy blueprint
linenumberstrue
cfy install -b pmsh -d pmsh -i /k8s-pmsh-inputs.yaml /blueprints/k8s-pmsh.yaml

To un-deploy

Uninstall running component and delete deployment

Code Block
languagebash
themeMidnight
titleUninstall component
linenumberstrue
cfy uninstall pmsh

Delete blueprint

Code Block
languagebash
themeMidnight
titleDelete blueprint
linenumberstrue
cfy blueprints delete pmsh

Initial Validation

After deployment, verify if pmsh POD and mongoDB pod are running correctly

Code Block
languagebash
themeMidnight
titleVerify Heartbeat is running
linenumberstrue
root@k8s-rancher:~# kubectl get pods -n onap | egrep "pmsh"

And then check the logs to see if it can connect to DMaaP, polling for events.

...

languagebash
themeMidnight
titleVerify Logs for Dmaap poll
linenumberstrue

Functional tests

Following default configuration is loaded into pmsh (set in blueprint configuration)

...

languagebash
themeMidnight
titleConfiguration

To simulate the event flow and trigger missing pmsh event, we can simulate a VES event into pmsh subscription topic (using curl).

Generate pmsh CL Onset

...

languagebash
themeMidnight
titleTrigger a CL event via curl
linenumberstrue

...

languagebash
themeMidnight
titleLogs showing event being processed
linenumberstrue

We can check that a new DCAE_CL_OUTPUT event has been published (make sure you target the correct DMaaP IP address)

Code Block
languagebash
themeMidnight
titleFetch events from DCAE_CL_OUTPUT topic
linenumberstrue
curl http://10.12.5.8:30227/events/unauthenticated.DCAE_CL_OUTPUT/vv/1

...

languagebash
themeMidnight
titlePublished event
linenumberstrue

Dynamic Configuration Update

As the pmsh service periodically polls Consul KV using configbindingService api's - the run time configuration of pmsh service can be updated dynamically without having to redeploy/restart the service. The updates to the configuration can be triggered either from Policy (or CLAMP) or made directly in Consul.


Locate the servicename service name by executing into pmsh PMSH Service pod and getting env HOSTNAME value

...

Code Block
languagebash
themeMidnight
titleConsul URL
http://<k8snodeip>:30270<k8s-node-ip>:<consul-ip>/ui/#/dc1/kv/


Consul Snapshot <>

...