- Created by Ajay Deep Singh, last modified on Jul 21, 2021
This document highlights design considered for to Provision authenticated, authorized DMaaP topics on Message Router and feeds on Data Router .
Overview
Bus Controller is a RESTful web service used to provision DMaaP topics (on Message Router) and feeds (on Data Router), with associated authorization (on AAF).
Endpoints are used to provision :
- Authorized topic on MR, and to create and grant permission for Publishers and Subscribers.
- Feed on Data Router, with associated user authentication.
- Bus Controller API link : https://docs.onap.org/projects/onap-dmaap-buscontroller/en/latest/apis/api.html
Gerrit review links
Ticket No. | Gerrit link | Comments | 5gbulkpm Test Result | Status |
---|---|---|---|---|
DMAAP-1561 | https://gerrit.onap.org/r/c/dmaap/buscontroller/+/121543 | Updated dmaap-dbc script | Pass | Merged |
DCAEGEN2-2715 (DCAE Data File Collector) | https://gerrit.onap.org/r/c/oom/+/120492 | To be tested with 5gBulkpm gaiting testcase | Pass | Merged |
DCAEGEN2-2714 (DCAE PM-Mapper) | https://gerrit.onap.org/r/c/oom/+/120213 | To be tested with 5gBulkpm gaiting testcase | Pass | Merged |
Dmaap dbc-client docker Image (onap/dmaap/dbc-client:2.0.7)
This dbc-client image is utility (containing shell script) by making use of which we can initiate HTTP REST Request towards Dmaap Bus Controller app for creating dynamic Feeds, Topics.
Design details
- Existing design consist fo kubernetes Job Manifest which consist of onap/dmaap/dbc-client docker image for making http request toward dmaap-bc (Dmaap Bus Controller) pod.
- In DCAE Design-1 highlighted in below pic is considered where we need to create dmmap provisionning Init-Container which will be making use of same onap/dmaap/dbc-client docker image to make Http Rest Request towards dmap-db pod.
- Once the http request is made response will be written back to shared EmptyDir voulme.
- Second Init-container (Update Config) will be reading the response from share EmptyDir voulme and will merge it with application config.
Draw.io design-1 :
- Dmaap Provisioning init-container sends POST request to Dmaap Bus-Controller Pod for creation of Topics, Feeds.
- DCAE Merge init config container merges dcae application config and response received back from dmaap provisioning init-container.
Data Router Feed, Publisher, Subscriber creation input details. Old approach- ------------------------------------------ feedConfig: owner: dcaecm feedVersion: 0.0 feedName: bulk_pm_feed asprClassification: unclassified feedDescription: DFC Feed Creation pubs: - username: dcaepub userpwd: tpJN3gjaAzPAiAa4 dcaeLocationName: loc00 ------------------------------------------ New approach- ------------------------------------------ # DataRouter Feed Configuration drFeedConfig: - feedName: bulk_pm_feed owner: dcaecm feedVersion: 0.0 asprClassification: unclassified feedDescription: DFC Feed Creation # DataRouter Publisher Configuration drPubConfig: - feedName: bulk_pm_feed dcaeLocationName: loc00 # DataRouter Subscriber Configuration drSubConfig: - feedName: bulk_pm_feed decompress: true username: ${DR_USERNAME} userpwd: ${DR_PASSWORD} dcaeLocationName: loc00 privilegedSubscriber: true deliveryURL: https://dcae-pm-mapper:8443/delivery # MessageRouter Topic, Publisher Configuration mrTopicsConfig: - topicName: PERFORMANCE_MEASUREMENTS topicDescription: PM Mapper publishes perf3gpp VES PM Events to authenticated MR topic owner: dcaecm tnxEnabled: false clients: - dcaeLocationName: san-francisco clientRole: org.onap.dcae.pmPublisher action: - pub - view ------------------------------------------ ConfigMap Configuration for Feed, Dr_Publisher ------------------------------------------ volumes: - name: feeds-config path: /opt/app/config/feeds/ - name: drpub-config path: /opt/app/config/dr_pubs/ - name: drsub-config path: /opt/app/config/dr_subs/ - name: topics-config path: /opt/app/config/topics
Code snip for updating application config with dmaap DR Feed/Pub/Sub and MR Topics details.
- name: {{ include "common.name" $dot }}-init-merge-config image: {{ include "repositoryGenerator.image.envsubst" $dot }} imagePullPolicy: {{ $dot.Values.global.pullPolicy | default $dot.Values.pullPolicy }} command: - /bin/sh args: - -c - | if [ -d /opt/app/config/cache ]; then cd /opt/app/config/cache for file in $(ls feed*); do NUM=$(echo "$file" | sed 's/feedConfig-\([0-9]\+\)-resp.json/\1/') export DR_LOG_URL_"$NUM"="$(grep -o '"logURL":"[^"]*' "$file" | cut -d '"' -f4)" export DR_FILES_PUBLISHER_URL_"$NUM"="$(grep -o '"publishURL":"[^"]*' "$file" | cut -d '"' -f4)" done for file in $(ls drpub*); do NUM=$(echo "$file" | sed 's/drpubConfig-\([0-9]\+\)-resp.json/\1/') export DR_USERNAME_"$NUM"="$(grep -o '"username":"[^"]*' "$file" | cut -d '"' -f4)" export DR_PASSWORD_"$NUM"="$(grep -o '"userpwd":"[^"]*' "$file" | cut -d '"' -f4)" export DR_FILES_PUBLISHER_ID_"$NUM"="$(grep -o '"pubId":"[^"]*' "$file" | cut -d '"' -f4)" done for file in $(ls drsub*); do NUM=$(echo "$file" | sed 's/drsubConfig-\([0-9]\+\)-resp.json/\1/') export DR_FILES_SUBSCRIBER_ID_"$NUM"="$(grep -o '"subId":"[^"]*' "$file" | cut -d '"' -f4)" done for file in $(ls topics*); do NUM=$(echo "$file" | sed 's/topicsConfig-\([0-9]\+\)-resp.json/\1/') export MR_FILES_PUBLISHER_CLIENT_ID_"$NUM"="$(grep -o '"mrClientId":"[^"]*' "$file" | cut -d '"' -f4)" done else echo "No Response logged for Dmaap BusController Http POST Request..!" fi cd /config-input && for PFILE in `ls -1`; do envsubst <${PFILE} >/config/${PFILE}; done env: {{- range $cred := $dot.Values.credentials }} - name: {{ $cred.name }} {{- include "common.secret.envFromSecretFast" (dict "global" $dot "uid" $cred.uid "key" $cred.key) | nindent 4 }} {{- end }} volumeMounts: - mountPath: /opt/app/config/cache name: dbc-response-cache - mountPath: /config-input name: app-config-input - mountPath: /config name: app-config
Container Logs
1. Log snippet collected for dmaap-bc-dmaap-provisioning conatiner making sure it work with updated changes in dbc script.
$ kubectl logs -f -n onap dev-dmaap-bc-dmaap-provisioning-n9zfp + export 'PATH=/usr/local/bin:/bin:/usr/bin:/usr/local/sbin:/usr/sbin:/sbin' + RESP_CACHE= + RESP=/dev/null + APP_ROOT=/opt/app/dbc-client + CONFIGMAP_ROOT=/opt/app/config + PORT=8443 + DBC=dmaap-bc + PROTO=https + PARAM='useExisting=true' + REQUESTID=dmaap-bc-dmaap-provisioning + URL=https://dmaap-bc:8443/webapi/ + CA_PEM=ca.pem + KEY_PEM=key.pem + CLIENT_PEM=client.pem + PEM_DIR=/opt/app/osaaf/local + CERT_PWD='2U[iOZzMHI:.#tdCwlBqc;}S' + BA_PWD='demo123456!' + AUTH_METHOD=basicAuth + BA_IDENTITY=dmaap-bc@dmaap-bc.onap.org + init_config + '[' '!' -d /opt/app/dbc-client -a '!' -d /opt/app/config ] + cd /opt/app/osaaf/local + '[' basicAuth '=' basicAuth ] + echo '-u dmaap-bc@dmaap-bc.onap.org:demo123456!' + CURL_CRED='-K /opt/app/osaaf/local/curl.cred' + init_dbc_provisioning + cd /opt/app/config + '[' -d dmaap ] + ls dmaap/onap.json + do_http_post dmaap/onap.json dmaap + RETRY_TIME=60 + '[' -n ] + true + '[' dmaap '!=' feeds -a dmaap '!=' topics ] + cat dmaap/onap.json + envsubst + json_input='{ "dmaapName": "mr", "drProvUrl": "https://dmaap-dr-prov", "version": "1", "topicNsRoot": "org.onap.dmaap", "bridgeAdminTopic": "DCAE_MM_AGENT" }' + xcurl -o /dev/null -d '{ "dmaapName": "mr", "drProvUrl": "https://dmaap-dr-prov", "version": "1", "topicNsRoot": "org.onap.dmaap", "bridgeAdminTopic": "DCAE_MM_AGENT" }' https://dmaap-bc:8443/webapi/dmaap + curl -X POST -s '-K /opt/app/osaaf/local/curl.cred' -w '%{http_code}' -H 'X-ECOMP-RequestID: dmaap-bc-dmaap-provisioning' -H 'Content-Type: application/json' -o /dev/null -d '{ "dmaapName": "mr", "drProvUrl": "https://dmaap-dr-prov", "version": "1", "topicNsRoot": "org.onap.dmaap", "bridgeAdminTopic": "DCAE_MM_AGENT" }' https://dmaap-bc:8443/webapi/dmaap Http Post request is successful with response code=200 + rc=200 + '[' 200 '=' 200 -o 200 '=' 201 -o 200 '=' 409 ] + echo 'Http Post request is successful with response code=200' + break + '[' -d dcaeLocations ] + ls dcaeLocations/san-francisco.json + do_http_post dcaeLocations/san-francisco.json dcaeLocations + RETRY_TIME=60 + '[' -n ] + true + '[' dcaeLocations '!=' feeds -a dcaeLocations '!=' topics ] + cat dcaeLocations/san-francisco.json + envsubst + json_input='{ "dcaeLayer": "kubernetes-central", "dcaeLocationName": "san-francisco" }' + xcurl -o /dev/null -d '{ "dcaeLayer": "kubernetes-central", "dcaeLocationName": "san-francisco" }' https://dmaap-bc:8443/webapi/dcaeLocations + curl -X POST -s '-K /opt/app/osaaf/local/curl.cred' -w '%{http_code}' -H 'X-ECOMP-RequestID: dmaap-bc-dmaap-provisioning' -H 'Content-Type: application/json' -o /dev/null -d '{ "dcaeLayer": "kubernetes-central", "dcaeLocationName": "san-francisco" }' https://dmaap-bc:8443/webapi/dcaeLocations + rc=201 + '[' 201 '=' 200 -o 201 '=' 201 -o 201 '=' 409 ] + echo 'Http Post request is successful with response code=201' + break + '[' -d mr_clusters ] Http Post request is successful with response code=201 + ls mr_clusters/san-francisco.json + do_http_post mr_clusters/san-francisco.json mr_clusters + RETRY_TIME=60 + '[' -n ] + true + '[' mr_clusters '!=' feeds -a mr_clusters '!=' topics ] + cat mr_clusters/san-francisco.json + envsubst + json_input='{ "dcaeLocationName": "san-francisco", "fqdn": "message-router", "topicProtocol": "http", "topicPort": "3904" }' + xcurl -o /dev/null -d '{ "dcaeLocationName": "san-francisco", "fqdn": "message-router", "topicProtocol": "http", "topicPort": "3904" }' https://dmaap-bc:8443/webapi/mr_clusters + curl -X POST -s '-K /opt/app/osaaf/local/curl.cred' -w '%{http_code}' -H 'X-ECOMP-RequestID: dmaap-bc-dmaap-provisioning' -H 'Content-Type: application/json' -o /dev/null -d '{ "dcaeLocationName": "san-francisco", "fqdn": "message-router", "topicProtocol": "http", "topicPort": "3904" }' https://dmaap-bc:8443/webapi/mr_clusters + rc=201 + '[' 201 '=' 200 -o 201 '=' 201 -o 201 '=' 409 ] + echo 'Http Post request is successful with response code=201' + break + '[' -d topics ] Http Post request is successful with response code=201 + ls topics/PNF_READY.json topics/PNF_REGISTRATION.json topics/mirrormakeragent.json + do_http_post topics/PNF_READY.json topics + RETRY_TIME=60 + '[' -n ] + true + '[' topics '!=' feeds -a topics '!=' topics ] + xcurl -o /dev/null -d @topics/PNF_READY.json 'https://dmaap-bc:8443/webapi/topics/?useExisting=true' + curl -X POST -s '-K /opt/app/osaaf/local/curl.cred' -w '%{http_code}' -H 'X-ECOMP-RequestID: dmaap-bc-dmaap-provisioning' -H 'Content-Type: application/json' -o /dev/null -d @topics/PNF_READY.json 'https://dmaap-bc:8443/webapi/topics/?useExisting=true' Http Post request for feed creation is successful with response code=201 + rc=201 + '[' 201 '=' 200 -o 201 '=' 201 -o 201 '=' 409 ] + echo 'Http Post request for feed creation is successful with response code=201' + break + do_http_post topics/PNF_REGISTRATION.json topics + RETRY_TIME=60 + '[' -n ] + true + '[' topics '!=' feeds -a topics '!=' topics ] + xcurl -o /dev/null -d @topics/PNF_REGISTRATION.json 'https://dmaap-bc:8443/webapi/topics/?useExisting=true' + curl -X POST -s '-K /opt/app/osaaf/local/curl.cred' -w '%{http_code}' -H 'X-ECOMP-RequestID: dmaap-bc-dmaap-provisioning' -H 'Content-Type: application/json' -o /dev/null -d @topics/PNF_REGISTRATION.json 'https://dmaap-bc:8443/webapi/topics/?useExisting=true' + rc=201 + '[' 201 '=' 200 -o 201 '=' 201 -o 201 '=' 409 ] + echo 'Http Post request for feed creation is successful with response code=201' + break Http Post request for feed creation is successful with response code=201 + do_http_post topics/mirrormakeragent.json topics + RETRY_TIME=60 + '[' -n ] + true + '[' topics '!=' feeds -a topics '!=' topics ] + xcurl -o /dev/null -d @topics/mirrormakeragent.json 'https://dmaap-bc:8443/webapi/topics/?useExisting=true' + curl -X POST -s '-K /opt/app/osaaf/local/curl.cred' -w '%{http_code}' -H 'X-ECOMP-RequestID: dmaap-bc-dmaap-provisioning' -H 'Content-Type: application/json' -o /dev/null -d @topics/mirrormakeragent.json 'https://dmaap-bc:8443/webapi/topics/?useExisting=true' Http Post request for feed creation is successful with response code=201 + rc=201 + '[' 201 '=' 200 -o 201 '=' 201 -o 201 '=' 409 ] + echo 'Http Post request for feed creation is successful with response code=201' + break + '[' -d mr_clients ] + '[' -d dr_nodes ] + ls 'dr_nodes/*.json' ls: dr_nodes/*.json: No such file or directory + '[' -d feeds ] + ls 'feeds/*.json' ls: feeds/*.json: No such file or directory + '[' -d dr_pubs ] + '[' -d dr_subs ]
2. Log snippet from dcae-datafile-collector initContainer dcae-datafile-collector-init-dmaap-provisioning for creation of Feeds and Dr Publisher.
$ kubectl logs -f -n onap dev-dcae-datafile-collector-d996487db-6wdjb -c dcae-datafile-collector-init-dmaap-provisioning + export 'PATH=/usr/local/bin:/bin:/usr/bin:/usr/local/sbin:/usr/sbin:/sbin' + RESP_CACHE=/opt/app/config/cache + RESP=/dev/null + APP_ROOT=/opt/app/dbc-client + CONFIGMAP_ROOT=/opt/app/config + PORT=8443 + DBC=dmaap-bc + PROTO=https + PARAM='useExisting=true' + REQUESTID=dcae-datafile-collector-dmaap-provisioning + URL=https://dmaap-bc:8443/webapi/ + CA_PEM=ca.pem + KEY_PEM=key.pem + CLIENT_PEM=client.pem + PEM_DIR=/opt/app/osaaf/local + CERT_PWD='2U[iOZzMHI:.#tdCwlBqc;}S' + BA_PWD='demo123456!' + AUTH_METHOD=basicAuth + BA_IDENTITY=dmaap-bc@dmaap-bc.onap.org + init_config + '[' '!' -d /opt/app/dbc-client -a '!' -d /opt/app/config ] + cd /opt/app/osaaf/local + '[' basicAuth '=' basicAuth ] + echo '-u dmaap-bc@dmaap-bc.onap.org:demo123456!' + CURL_CRED='-K /opt/app/osaaf/local/curl.cred' + init_dbc_provisioning + cd /opt/app/config + '[' -d dmaap ] + '[' -d dcaeLocations ] + '[' -d mr_clusters ] + '[' -d topics ] + '[' -d mr_clients ] + '[' -d dr_nodes ] + '[' -d feeds ] + ls feeds/feedConfig-0.json + do_http_post feeds/feedConfig-0.json feeds + RETRY_TIME=60 + '[' -n /opt/app/config/cache ] + cut -d . -f1 + echo feedConfig-0.json + RESP=/opt/app/config/cache/feedConfig-0-resp.json + true + '[' feeds '!=' feeds -a feeds '!=' topics ] + xcurl -o /opt/app/config/cache/feedConfig-0-resp.json -d @feeds/feedConfig-0.json 'https://dmaap-bc:8443/webapi/feeds/?useExisting=true' + curl -X POST -s '-K /opt/app/osaaf/local/curl.cred' -w '%{http_code}' -H 'X-ECOMP-RequestID: dcae-datafile-collector-dmaap-provisioning' -H 'Content-Type: application/json' -o /opt/app/config/cache/feedConfig-0-resp.json -d @feeds/feedConfig-0.json 'https://dmaap-bc:8443/webapi/feeds/?useExisting=true' Http Post request for feed creation is successful with response code=200 + rc=200 + '[' 200 '=' 200 -o 200 '=' 201 -o 200 '=' 409 ] + echo 'Http Post request for feed creation is successful with response code=200' + break + '[' -d dr_pubs ] + ls dr_pubs/drpubConfig-0.json + do_http_post dr_pubs/drpubConfig-0.json dr_pubs + RETRY_TIME=60 + '[' -n /opt/app/config/cache ] + echo drpubConfig-0.json + cut -d . -f1 + RESP=/opt/app/config/cache/drpubConfig-0-resp.json + true + '[' dr_pubs '!=' feeds -a dr_pubs '!=' topics ] + cat dr_pubs/drpubConfig-0.json + envsubst + json_input='{"dcaeLocationName":"loc00","feedName":"bulk_pm_feed"}' + xcurl -o /opt/app/config/cache/drpubConfig-0-resp.json -d '{"dcaeLocationName":"loc00","feedName":"bulk_pm_feed"}' https://dmaap-bc:8443/webapi/dr_pubs + curl -X POST -s '-K /opt/app/osaaf/local/curl.cred' -w '%{http_code}' -H 'X-ECOMP-RequestID: dcae-datafile-collector-dmaap-provisioning' -H 'Content-Type: application/json' -o /opt/app/config/cache/drpubConfig-0-resp.json -d '{"dcaeLocationName":"loc00","feedName":"bulk_pm_feed"}' https://dmaap-bc:8443/webapi/dr_pubs + rc=201 + '[' 201 '=' 200 -o 201 '=' 201 -o 201 '=' 409 ] + echo 'Http Post request is successful with response code=201' + break + '[' -d dr_subs ] Http Post request is successful with response code=201
3. Log snippet from dcae-pm-mapper initContainer dcae-pm-mapper-init-dmaap-provisioning for creation of Feeds and Dr Subscriber and MessageRouter Topics.
$ kubectl logs -f -n onap dev-dcae-pm-mapper-6d84cb4757-mljn9 -c dcae-pm-mapper-init-dmaap-provisioning + export 'PATH=/usr/local/bin:/bin:/usr/bin:/usr/local/sbin:/usr/sbin:/sbin' + RESP_CACHE=/opt/app/config/cache + RESP=/dev/null + APP_ROOT=/opt/app/dbc-client + CONFIGMAP_ROOT=/opt/app/config + PORT=8443 + DBC=dmaap-bc + PROTO=https + PARAM='useExisting=true' + REQUESTID=dcae-pm-mapper-dmaap-provisioning + URL=https://dmaap-bc:8443/webapi/ + CA_PEM=ca.pem + KEY_PEM=key.pem + CLIENT_PEM=client.pem + PEM_DIR=/opt/app/osaaf/local + CERT_PWD='2U[iOZzMHI:.#tdCwlBqc;}S' + BA_PWD='demo123456!' + AUTH_METHOD=basicAuth + BA_IDENTITY=dmaap-bc@dmaap-bc.onap.org + init_config + '[' '!' -d /opt/app/dbc-client -a '!' -d /opt/app/config ] + cd /opt/app/osaaf/local + '[' basicAuth '=' basicAuth ] + echo '-u dmaap-bc@dmaap-bc.onap.org:demo123456!' + CURL_CRED='-K /opt/app/osaaf/local/curl.cred' + init_dbc_provisioning + cd /opt/app/config + '[' -d dmaap ] + '[' -d dcaeLocations ] + '[' -d mr_clusters ] + '[' -d topics ] + ls topics/topicsConfig-0.json + do_http_post topics/topicsConfig-0.json topics + RETRY_TIME=60 + '[' -n /opt/app/config/cache ] + echo+ topicsConfig-0.json cut -d . -f1 + RESP=/opt/app/config/cache/topicsConfig-0-resp.json + true + '[' topics '!=' feeds -a topics '!=' topics ] + xcurl -o /opt/app/config/cache/topicsConfig-0-resp.json -d @topics/topicsConfig-0.json 'https://dmaap-bc:8443/webapi/topics/?useExisting=true' + curl -X POST -s '-K /opt/app/osaaf/local/curl.cred' -w '%{http_code}' -H 'X-ECOMP-RequestID: dcae-pm-mapper-dmaap-provisioning' -H 'Content-Type: application/json' -o /opt/app/config/cache/topicsConfig-0-resp.json -d @topics/topicsConfig-0.json 'https://dmaap-bc:8443/webapi/topics/?useExisting=true' + rc=201 + '[' 201 '=' 200 -o 201 '=' 201 -o 201 '=' 409 ] + echo 'Http Post request for feed creation is successful with response code=201' + break + '[' -d mr_clients ] + '[' -d dr_nodes ] + '[' -d feeds ] Http Post request for feed creation is successful with response code=201 + ls feeds/feedConfig-0.json + do_http_post feeds/feedConfig-0.json feeds + RETRY_TIME=60 + '[' -n /opt/app/config/cache ] + echo feedConfig-0.json + cut -d . -f1 + RESP=/opt/app/config/cache/feedConfig-0-resp.json + true + '[' feeds '!=' feeds -a feeds '!=' topics ] + xcurl -o /opt/app/config/cache/feedConfig-0-resp.json -d @feeds/feedConfig-0.json 'https://dmaap-bc:8443/webapi/feeds/?useExisting=true' + curl -X POST -s '-K /opt/app/osaaf/local/curl.cred' -w '%{http_code}' -H 'X-ECOMP-RequestID: dcae-pm-mapper-dmaap-provisioning' -H 'Content-Type: application/json' -o /opt/app/config/cache/feedConfig-0-resp.json -d @feeds/feedConfig-0.json 'https://dmaap-bc:8443/webapi/feeds/?useExisting=true' + rc=200 + '[' 200 '=' 200 -o 200 '=' 201 -o 200 '=' 409 ] + echo 'Http Post request for feed creation is successful with response code=200' + break + '[' -d dr_pubs ] + '[' -d dr_subs ] Http Post request for feed creation is successful with response code=200 + ls dr_subs/drsubConfig-0.json + do_http_post dr_subs/drsubConfig-0.json dr_subs + RETRY_TIME=60 + '[' -n /opt/app/config/cache ] + echo drsubConfig-0.json + cut -d . -f1 + RESP=/opt/app/config/cache/drsubConfig-0-resp.json + true + '[' dr_subs '!=' feeds -a dr_subs '!=' topics ] + cat dr_subs/drsubConfig-0.json + envsubst + json_input='{"dcaeLocationName":"loc00","decompress":true,"deliveryURL":"https://dcae-pm-mapper:8443/delivery","feedName":"bulk_pm_feed","privilegedSubscriber":true,"username":"username","userpwd":"password"}' + xcurl -o /opt/app/config/cache/drsubConfig-0-resp.json -d '{"dcaeLocationName":"loc00","decompress":true,"deliveryURL":"https://dcae-pm-mapper:8443/delivery","feedName":"bulk_pm_feed","privilegedSubscriber":true,"username":"username","userpwd":"password"}' https://dmaap-bc:8443/webapi/dr_subs + curl -X POST -s '-K /opt/app/osaaf/local/curl.cred' -w '%{http_code}' -H 'X-ECOMP-RequestID: dcae-pm-mapper-dmaap-provisioning' -H 'Content-Type: application/json' -o /opt/app/config/cache/drsubConfig-0-resp.json -d '{"dcaeLocationName":"loc00","decompress":true,"deliveryURL":"https://dcae-pm-mapper:8443/delivery","feedName":"bulk_pm_feed","privilegedSubscriber":true,"username":"username","userpwd":"password"}' https://dmaap-bc:8443/webapi/dr_subs Http Post request is successful with response code=201 + rc=201 + '[' 201 '=' 200 -o 201 '=' 201 -o 201 '=' 409 ] + echo 'Http Post request is successful with response code=201' + break
4. GET Request for DR Feeds, MR Topics from Bus Controller API.
1. >> GET : https://172.16.0.70:30242/webapi/feeds/ [ { "lastMod": 1625906852119, "status": "VALID", "feedId": "1", "feedName": "bulk_pm_feed", "feedVersion": "0", "feedDescription": "DFC Feed Creation", "owner": "dcaecm", "asprClassification": "unclassified", "publishURL": "https://dmaap-dr-prov/publish/1", "subscribeURL": "https://dmaap-dr-prov/subscribe/1", "suspended": false, "logURL": "https://dmaap-dr-prov/feedlog/1", "formatUuid": null, "pubs": [ { "lastMod": 1625906852105, "status": "VALID", "dcaeLocationName": "loc00", "username": "tmp_edftprplxki85pw", "userpwd": "gi6nvqz4skzepa1", "feedId": "1", "pubId": "1.da72f", "feedName": null, "feedVersion": null, "statusValid": true }, { "lastMod": 1625906851630, "status": "VALID", "dcaeLocationName": "san-francisco", "username": "tmp_rmu408ke6c47cjd", "userpwd": "3ae2ahymjl6sndw", "feedId": "1", "pubId": "1.9hl81", "feedName": null, "feedVersion": null, "statusValid": true } ], "subs": [ { "lastMod": 1625906974644, "status": "VALID", "dcaeLocationName": "loc00", "username": "username", "userpwd": "password", "feedId": "1", "deliveryURL": "https://dcae-pm-mapper:8443/delivery", "logURL": "https://dmaap-dr-prov/sublog/1", "subId": "1", "use100": false, "suspended": false, "owner": "DGL", "guaranteedDelivery": false, "guaranteedSequence": false, "privilegedSubscriber": true, "decompress": true, "feedName": null, "feedVersion": null, "statusValid": true } ], "bytes": "eyJuYW1lIjogImJ1bGtfcG1fZmVlZCIsICJ2ZXJzaW9uIjogIjAiLCAiZGVzY3JpcHRpb24iOiAiREZDIEZlZWQgQ3JlYXRpb24iLCAic3VzcGVuZCI6IGZhbHNlLCAiYXV0aG9yaXphdGlvbiI6IHsgImNsYXNzaWZpY2F0aW9uIjogInVuY2xhc3NpZmllZCIsICJlbmRwb2ludF9hZGRycyI6IFtdLCJlbmRwb2ludF9pZHMiOiBbCXsiaWQiOiAidG1wX2VkZnRwcnBseGtpODVwdyIsInBhc3N3b3JkIjogImdpNm52cXo0c2t6ZXBhMSJ9CSx7ImlkIjogInRtcF9ybXU0MDhrZTZjNDdjamQiLCJwYXNzd29yZCI6ICIzYWUyYWh5bWpsNnNuZHcifV19fQ==", "statusValid": true } ] 2. >> GET Topics : https://172.16.0.70:30242/webapi/topics/ [ { "lastMod": 1625906819480, "status": "VALID", "fqtn": "org.onap.dmaap.mr.PNF_REGISTRATION", "topicName": "PNF_REGISTRATION", "topicDescription": "the VES collector will be publishing pnfRegistration events in this topic", "tnxEnabled": "false", "owner": "VEScollector", "formatUuid": null, "replicationCase": "REPLICATION_NONE", "globalMrURL": null, "fqtnStyle": null, "version": null, "partitionCount": "2", "replicationCount": "1", "publisherRole": "org.onap.dmaap.mr.PNF_REGISTRATION.publisher", "subscriberRole": "org.onap.dmaap.mr.PNF_REGISTRATION.subscriber", "clients": [ { "lastMod": 1625906820030, "status": "VALID", "dcaeLocationName": "san-francisco", "topicURL": "http://message-router:3904/events/org.onap.dmaap.mr.PNF_REGISTRATION", "fqtn": "org.onap.dmaap.mr.PNF_REGISTRATION", "clientRole": "org.onap.dmaap.mr.PNF_REGISTRATION.sub", "action": [ "sub", "view" ], "mrClientId": "1625906820030", "clientIdentity": null, "publisher": false, "subscriber": true, "statusValid": true } ], "numClients": 1, "bytes": "eyAidG9waWNOYW1lIjogIm9yZy5vbmFwLmRtYWFwLm1yLlBORl9SRUdJU1RSQVRJT04iLCAidG9waWNEZXNjcmlwdGlvbiI6ICJ0aGUgVkVTIGNvbGxlY3RvciB3aWxsIGJlIHB1Ymxpc2hpbmcgcG5mUmVnaXN0cmF0aW9uIGV2ZW50cyBpbiB0aGlzIHRvcGljIiwgInBhcnRpdGlvbkNvdW50IjogIjIiLCAicmVwbGljYXRpb25Db3VudCI6ICIxIiB9IA==", "statusValid": true }, { "lastMod": 1625906820603, "status": "VALID", "fqtn": "org.onap.dmaap.mr.mirrormakeragent", "topicName": "mirrormakeragent", "topicDescription": "the topic used to provision the MM agent whitelist", "tnxEnabled": "false", "owner": "dmaap", "formatUuid": null, "replicationCase": "REPLICATION_NONE", "globalMrURL": null, "fqtnStyle": null, "version": null, "partitionCount": "1", "replicationCount": "1", "publisherRole": "org.onap.dmaap.mr.mirrormakeragent.publisher", "subscriberRole": "org.onap.dmaap.mr.mirrormakeragent.subscriber", "clients": [ { "lastMod": 1625906821157, "status": "VALID", "dcaeLocationName": "san-francisco", "topicURL": "http://message-router:3904/events/org.onap.dmaap.mr.mirrormakeragent", "fqtn": "org.onap.dmaap.mr.mirrormakeragent", "clientRole": null, "action": [ "pub", "sub", "view" ], "mrClientId": "1625906821157", "clientIdentity": "dmaap-bc-mm-prov@dmaap-bc-mm-prov.onap.org", "publisher": true, "subscriber": true, "statusValid": true }, { "lastMod": 1625906827509, "status": "VALID", "dcaeLocationName": "san-francisco", "topicURL": "http://message-router:3904/events/org.onap.dmaap.mr.mirrormakeragent", "fqtn": "org.onap.dmaap.mr.mirrormakeragent", "clientRole": null, "action": [ "pub", "sub", "view" ], "mrClientId": "1625906827509", "clientIdentity": "demo@people.osaaf.org", "publisher": true, "subscriber": true, "statusValid": true }, { "lastMod": 1625906826491, "status": "VALID", "dcaeLocationName": "san-francisco", "topicURL": "http://message-router:3904/events/org.onap.dmaap.mr.mirrormakeragent", "fqtn": "org.onap.dmaap.mr.mirrormakeragent", "clientRole": null, "action": [ "pub", "sub", "view" ], "mrClientId": "1625906826491", "clientIdentity": "dmaap-bc-topic-mgr@dmaap-bc-topic-mgr.onap.org", "publisher": true, "subscriber": true, "statusValid": true } ], "numClients": 3, "bytes": "eyAidG9waWNOYW1lIjogIm9yZy5vbmFwLmRtYWFwLm1yLm1pcnJvcm1ha2VyYWdlbnQiLCAidG9waWNEZXNjcmlwdGlvbiI6ICJ0aGUgdG9waWMgdXNlZCB0byBwcm92aXNpb24gdGhlIE1NIGFnZW50IHdoaXRlbGlzdCIsICJwYXJ0aXRpb25Db3VudCI6ICIxIiwgInJlcGxpY2F0aW9uQ291bnQiOiAiMSIgfSA=", "statusValid": true }, { "lastMod": 1625906971092, "status": "VALID", "fqtn": "org.onap.dmaap.mr.PERFORMANCE_MEASUREMENTS", "topicName": "PERFORMANCE_MEASUREMENTS", "topicDescription": "PM Mapper publishes perf3gpp VES PM Events to authenticated MR topic", "tnxEnabled": "false", "owner": "dcaecm", "formatUuid": null, "replicationCase": "REPLICATION_NONE", "globalMrURL": null, "fqtnStyle": null, "version": null, "partitionCount": "2", "replicationCount": "1", "publisherRole": "org.onap.dmaap.mr.PERFORMANCE_MEASUREMENTS.publisher", "subscriberRole": "org.onap.dmaap.mr.PERFORMANCE_MEASUREMENTS.subscriber", "clients": [ { "lastMod": 1625906971956, "status": "VALID", "dcaeLocationName": "san-francisco", "topicURL": "http://message-router:3904/events/org.onap.dmaap.mr.PERFORMANCE_MEASUREMENTS", "fqtn": "org.onap.dmaap.mr.PERFORMANCE_MEASUREMENTS", "clientRole": "org.onap.dcae.pmPublisher", "action": [ "pub", "view" ], "mrClientId": "1625906971956", "clientIdentity": null, "publisher": true, "subscriber": false, "statusValid": true } ], "numClients": 1, "bytes": "eyAidG9waWNOYW1lIjogIm9yZy5vbmFwLmRtYWFwLm1yLlBFUkZPUk1BTkNFX01FQVNVUkVNRU5UUyIsICJ0b3BpY0Rlc2NyaXB0aW9uIjogIlBNIE1hcHBlciBwdWJsaXNoZXMgcGVyZjNncHAgVkVTIFBNIEV2ZW50cyB0byBhdXRoZW50aWNhdGVkIE1SIHRvcGljIiwgInBhcnRpdGlvbkNvdW50IjogIjIiLCAicmVwbGljYXRpb25Db3VudCI6ICIxIiB9IA==", "statusValid": true }, { "lastMod": 1625906816791, "status": "VALID", "fqtn": "org.onap.dmaap.mr.PNF_READY", "topicName": "PNF_READY", "topicDescription": "This topic will be used to publish the PNF_READY events generated by the PNF REgistration Handler service in the DCAE platform.", "tnxEnabled": "false", "owner": "PNFRegistrationHandler", "formatUuid": null, "replicationCase": "REPLICATION_NONE", "globalMrURL": null, "fqtnStyle": null, "version": null, "partitionCount": "2", "replicationCount": "1", "publisherRole": "org.onap.dmaap.mr.PNF_READY.publisher", "subscriberRole": "org.onap.dmaap.mr.PNF_READY.subscriber", "clients": [ { "lastMod": 1625906817506, "status": "VALID", "dcaeLocationName": "san-francisco", "topicURL": "http://message-router:3904/events/org.onap.dmaap.mr.PNF_READY", "fqtn": "org.onap.dmaap.mr.PNF_READY", "clientRole": "org.onap.dmaap.mr.PNF_READY.pub", "action": [ "pub", "view" ], "mrClientId": "1625906817506", "clientIdentity": null, "publisher": true, "subscriber": false, "statusValid": true } ], "numClients": 1, "bytes": "eyAidG9waWNOYW1lIjogIm9yZy5vbmFwLmRtYWFwLm1yLlBORl9SRUFEWSIsICJ0b3BpY0Rlc2NyaXB0aW9uIjogIlRoaXMgdG9waWMgd2lsbCBiZSB1c2VkIHRvIHB1Ymxpc2ggdGhlIFBORl9SRUFEWSBldmVudHMgZ2VuZXJhdGVkIGJ5IHRoZSBQTkYgUkVnaXN0cmF0aW9uIEhhbmRsZXIgc2VydmljZSBpbiB0aGUgRENBRSBwbGF0Zm9ybS4iLCAicGFydGl0aW9uQ291bnQiOiAiMiIsICJyZXBsaWNhdGlvbkNvdW50IjogIjEiIH0g", "statusValid": true } ]
Test details
- Create Sftp server.
- Upload PM Files to xNF SFTP Server.
- Send File Ready Event to VES Collector.
- Verify Ves Collector logs for Event send to unauthenticated.VES_NOTIFICATION_OUTPUT : Ves Collector log file.
- Verify Data File Collector Logs if it contains message > Publishing file A1625946870.67.xml.gz to DR successful! : DFC log file.
- Vefify PM-Mapper log if it contains > org.onap.dcaegen2.services.pmmapper.messagerouter.VESPublisher. Successfully published VES events to messagerouter : Pm-Mapper log file.
- Verify Message-Router log if it contains Publisher Log Details : [publisherId=dcae, topicId=org.onap.dmaap.mr.PERFORMANCE_MEASUREMENTS, messageTimestamp=null, publisherIp=10.42.6.27, messageBatchId=10072021195549, messageSequence=1, messageLengthInBytes=2530, transactionEnabled=true, transactionId=10-07-2021::07:55:44:917::10.42.6.27::10072021195549::1, publishTimestamp=10-07-2021::07:55:44:918, serverIp=10.42.3.28]
Improvements
- Datafile Collector not sending dr publisher credentails as secret.
- No labels
1 Comment
Vijay Venkatesh Kumar
Thanks Ajay Deep Singh for documentations. As we have now working design/solutions with E2E flow - suggest following updates to wiki (same can be used for RTD)