Problems/Issues encountered or steps required when using ONAP Casablanca after a fresh installation

Problem creating topics in DMaaP

1. Restart DMaaP

Delete DMaaP-MR POD
Delete DMaaP-Kafka POD
Delete DMaaP-Zookeeper POD
Recreate DMaaP-Zookeeper POD
Recreate DMaaP-Kafka POD
Recreate DMaaP-MR POD

2. Make sure there are no conflicting topics, e.g. unauthenticated_PNF_READY conflicts with unauthenticated.PNF_READY

3. Delete conflicting topics in dmaap-zookeeper database (no delete topic API in Casablanca release)

Create a complex in AAI

See How-To: Register a VIM/Cloud Instance to ONAP

PUT https://aai.api.simpledemo.openecomp.org:30233/aai/v14/cloud-infrastructure/complexes/complex/clli1 

Authorization:Basic QUFJOkFBSQ==
X-TransactionId:jimmy-postman
X-FromAppId:AAI
Content-Type:application/json
Accept:application/json
Cache-Control:no-cache

Body:
{
    "physical-location-id": "clli1",
    "data-center-code": "data-center-code-BINZ",
    "complex-name": "clli1",
    "identity-url": "identity-url-BINZ",
    "physical-location-type": "physical-location-type-val-28399",
    "street1": "example-street1-val-28399",
    "street2": "example-street2-val-28399",
    "city": "example-city-val-28399",
    "state": "example-state-val-28399",
    "postal-code": "example-postal-code-val-28399",
    "country": "example-country-val-28399",
    "region": "example-region-val-28399",
    "latitude": "1111",
    "longitude": "2222",
    "elevation": "example-elevation-val-28399",
    "lata": "example-lata-val-28399"
}

Register a VIM via ESR GUI: http://msb.api.discovery.simpledemo.onap.org:30280/iui/aai-esr-gui/extsys/vim/vimView.html

Policy

See https://onap.readthedocs.io/en/casablanca/submodules/policy/engine.git/docs/platform/swarch_srm.html#before-installing-policies

CLAMP certificate to access web portal

See Control Loop Flows and Models for Casablanca#Configure

CLDS URL: https://clamp.api.simpledemo.onap.org:30258/designer/index.html

VID Portal. Problem with browser certificate

Add security exception in browser for VID URL https://vid.api.simpledemo.onap.org:30200/vid/welcome.htm

Demo init script fails

We assume ONAP is installed using the Integration team script (Heat + OOM deployment): https://github.com/onap/integration/tree/master/deployment/heat/onap-oom/scripts

demo-k8s.sh onap init

root@oom-rancher:~/oom/kubernetes/robot# ./demo-k8s.sh onap init

Number of parameters:
2
KEY:
init
++ kubectl --namespace onap get pods
++ sed 's/ .*//'
++ grep robot
+ POD=dev-robot-robot-f97bcf797-gg788
+ ETEHOME=/var/opt/OpenECOMP_ETE
++ kubectl --namespace onap exec dev-robot-robot-f97bcf797-gg788 -- bash -c 'ls -1q /share/logs/ | wc -l'
+ export GLOBAL_BUILD_NUMBER=14
+ GLOBAL_BUILD_NUMBER=14
++ printf %04d 14
+ OUTPUT_FOLDER=0014_demo_init
+ DISPLAY_NUM=104
+ VARIABLEFILES='-V /share/config/vm_properties.py -V /share/config/integration_robot_properties.py -V /share/config/integration_preload_parameters.py'
+ kubectl --namespace onap exec dev-robot-robot-f97bcf797-gg788 -- /var/opt/OpenECOMP_ETE/runTags.sh -V /share/config/vm_properties.py -V /share/config/integration_robot_properties.py -V /share/config/integration_preload_parameters.py -d /share/logs/0014_demo_init -i InitDemo --display 104
Starting Xvfb on display :104 with res 1280x1024x24
Executing robot tests at log level TRACE
==============================================================================
Testsuites
==============================================================================
Testsuites.Demo :: Executes the VNF Orchestration Test cases including setu...
==============================================================================
Initialize Customer And Models                                        | FAIL |
'200 <= 401 < 300' should be true.
------------------------------------------------------------------------------
Testsuites.Demo :: Executes the VNF Orchestration Test cases inclu... | FAIL |
1 critical test, 0 passed, 1 failed
1 test total, 0 passed, 1 failed
==============================================================================
Testsuites                                                            | FAIL |
1 critical test, 0 passed, 1 failed
1 test total, 0 passed, 1 failed
==============================================================================
Output:  /share/logs/0014_demo_init/output.xml
Log:     /share/logs/0014_demo_init/log.html
Report:  /share/logs/0014_demo_init/report.html

Solution

check cloud related config (e.g. OpenStack) in ~/integration-override.yaml file and redeploy affected components by running:

helm deploy dev local/onap -f ~/oom/kubernetes/onap/resources/environments/public-cloud.yaml -f ~/integration-override.yaml --namespace onap --verbose

./demo-k8s.sh onap init_robot

root@oom-rancher:~/oom/kubernetes/robot# ./demo-k8s.sh onap init_robot
Number of parameters:
2
KEY:
init_robot
WEB Site Password for user 'test': ++ kubectl --namespace onap get pods
++ sed 's/ .*//'
++ grep robot
+ POD=dev-robot-robot-f97bcf797-hbwbv
+ ETEHOME=/var/opt/OpenECOMP_ETE
++ kubectl --namespace onap exec dev-robot-robot-f97bcf797-hbwbv -- bash -c 'ls -1q /share/logs/ | wc -l'
+ export GLOBAL_BUILD_NUMBER=35
+ GLOBAL_BUILD_NUMBER=35
++ printf %04d 35
+ OUTPUT_FOLDER=0035_demo_init_robot
+ DISPLAY_NUM=125
+ VARIABLEFILES='-V /share/config/vm_properties.py -V /share/config/integration_robot_properties.py -V /share/config/integration_preload_parameters.py'
+ kubectl --namespace onap exec dev-robot-robot-f97bcf797-hbwbv -- /var/opt/OpenECOMP_ETE/runTags.sh -V /share/config/vm_properties.py -V /share/config/integration_robot_properties.py -V /share/config/integration_preload_parameters.py -v WEB_PASSWORD:test -d /share/logs/0035_demo_init_robot -i UpdateWebPage --display 125
Starting Xvfb on display :125 with res 1280x1024x24
Executing robot tests at log level TRACE
==============================================================================
Testsuites
==============================================================================
Testsuites.Update Onap Page :: Initializes ONAP Test Web Page and Password
==============================================================================
Update ONAP Page                                                      | PASS |
------------------------------------------------------------------------------
Testsuites.Update Onap Page :: Initializes ONAP Test Web Page and ... | PASS |
1 critical test, 1 passed, 0 failed
1 test total, 1 passed, 0 failed
==============================================================================
Testsuites                                                            | PASS |
1 critical test, 1 passed, 0 failed
1 test total, 1 passed, 0 failed
==============================================================================
Output:  /share/logs/0035_demo_init_robot/output.xml
Log:     /share/logs/0035_demo_init_robot/log.html
Report:  /share/logs/0035_demo_init_robot/report.html

SDC distribution fails

./ete-k8s.sh onap health

SDC Healthcheck (DMaaP:None):PASS

Check zookeper image version (=3.4.9):

# kubectl exec -ti dev-dmaap-message-router-zookeeper-6f68c699d7-jn9pg -- bash
root@dev-dmaap-message-router-zookeeper-6f68c699d7-jn9pg:/opt/zookeeper-3.4.9# env | grep -i version
ZOOKEEPER_VERSION=3.4.9

kubectl -n onap edit configMap dev-dmaap-dmaap-dr-node-node-props-configmap
	ProvisioningURL=https://dmaap-dr-prov:8443/internal/prov -> ProvisioningURL=http://dmaap-dr-prov:8080/internal/prov

restart dmaap-bus-controller, dmaap-message-router
kubectl delete po dev-dmaap-dmaap-dr-node-77454c5f45-k7p4l

Helm Error: trying to send message larger than max

Error: trying to send message larger than max

# helm ls
Error: trying to send message larger than max (23353031 vs. 20971520)


# kubectl get configmap | wc -l
286
# kubectl --namespace=kube-system get cm | wc -l
708

Solution: delete old configmap versions and limit history

history max

helm init --history-max 10 --upgrade

> https://github.com/helm/helm/issues/2332#issuecomment-336565784

delete old configmap versions

#!/usr/bin/env bash

TARGET_NUM_REVISIONS=10
TARGET_NUM_REVISIONS=$(($TARGET_NUM_REVISIONS+0))

RELEASES=$(kubectl --namespace=kube-system get cm -l OWNER=TILLER -o go-template --template='{{range .items}}{{ .metadata.labels.NAME }}{{"\n"}}{{ end }}' | sort -u)

# create the directory to store backups
mkdir configmaps

for RELEASE in $RELEASES
do
  # get the revisions of this release
  REVISIONS=$(kubectl --namespace=kube-system get cm -l OWNER=TILLER -l NAME=$RELEASE | awk '{if(NR>1)print $1}' | sed 's/.*\.v//' | sort -n)
  NUM_REVISIONS=$(echo $REVISIONS | tr " " "\n" | wc -l)
  NUM_REVISIONS=$(($NUM_REVISIONS+0))

  echo "Release $RELEASE has $NUM_REVISIONS revisions. Target is $TARGET_NUM_REVISIONS."
  if [ $NUM_REVISIONS -gt $TARGET_NUM_REVISIONS ]; then
    NUM_TO_DELETE=$(($NUM_REVISIONS-$TARGET_NUM_REVISIONS))
    echo "Will delete $NUM_TO_DELETE revisions"

    TO_DELETE=$(echo $REVISIONS | tr " " "\n" | head -n $NUM_TO_DELETE)

    for DELETE_REVISION in $TO_DELETE
    do
      CMNAME=$RELEASE.v$DELETE_REVISION
      echo "Deleting $CMNAME"
      # Take a backup
      kubectl --namespace=kube-system get cm $CMNAME -o yaml > configmaps/$CMNAME.yaml
      # Do the delete
      kubectl --namespace=kube-system delete cm $CMNAME
    done
  fi
done

Update Edge Rules in running AAI deployment (OOM, Casablanca branch)

Note: this is not a recommended solution, as it is not persistent after PODs deletion. See https://lists.onap.org/g/onap-discuss/message/16171 and AAI-2154 - Getting issue details... STATUS

1) Add new edge rules to aai-traversal, aai-resources and aai-graphadmin containers. For instance:

kubectl exec -ti dev-aai-aai-resources-5b6c5f454c-kbmdh bash
root@aai-resources:/opt/app/aai-resources# vi /opt/app/aai-resources/resources/schema/onap/dbedgerules/v14/DbEdgeRules_newRules_v14.json

2) Run createDBSchema script in aai-graphadmin

kubectl exec -ti dev-aai-aai-graphadmin-75d6587db4-xpmt5 bash
root@aai-graphadmin:/opt/app/aai-graphadmin# su aaiadmin
aaiadmin@aai-graphadmin:/opt/app/aai-graphadmin$ /opt/app/aai-graphadmin/bin/createDBSchema.sh

3) Restart docker containers aai-traversal, aai-resources and aai-graphadmin using docker restart

Change PRH (or any DCAE platform mS) docker image version in a running DCAE deployment (OOM, Casablanca branch) using cloudify manager

[root@dev-dcaegen2-dcae-cloudify-manager-dc549746b-6km6k prh]# cfy deployments update -i tag_version=nexus3.onap.org:10001/onap/org.onap.dcaegen2.services.prh.prh-app-server:1.2.1-SNAPSHOT prh

Install Cloudify blueprint

cloudify blueprint installation

cfy blueprints validate /blueprints/k8s-prh.yaml
cfy blueprints upload -b prh /blueprints/k8s-prh.yaml
cfy deployments create -b prh -i /inputs/k8s-prh-inputs.yaml prh
cfy executions start -d prh install

Space shortcuts

Page tree

Problem creating topics in DMaaP

Create a complex in AAI

Policy

CLAMP certificate to access web portal

VID Portal. Problem with browser certificate

Demo init script fails

Solution

SDC distribution fails

Helm Error: trying to send message larger than max

Update Edge Rules in running AAI deployment (OOM, Casablanca branch)

Change PRH (or any DCAE platform mS) docker image version in a running DCAE deployment (OOM, Casablanca branch) using cloudify manager

Install Cloudify blueprint

Space shortcuts

Page tree

ONAP Casablanca Troubleshooting

Problem creating topics in DMaaP

Create a complex in AAI

Policy

CLAMP certificate to access web portal

VID Portal. Problem with browser certificate

Demo init script fails

Solution

SDC distribution fails

Helm Error: trying to send message larger than max

Update Edge Rules in running AAI deployment (OOM, Casablanca branch)

Change PRH (or any DCAE platform mS) docker image version in a running DCAE deployment (OOM, Casablanca branch) using cloudify manager

Install Cloudify blueprint