AAI has a single instance of ONAP in the "A & AI" project in the windriver lab.

Once you are connected to the openlab via openvpn, use ssh to connect to to the AAI rancher server as user 'ubuntu'

Rancher server IP address is currently 10.12.6.205

$ ssh -i ~/.ssh/onap_key ubuntu@10.12.6.205

Become root:

$ sudo /bin/su

If you want to test out a change to one of AAI's charts, make the change and submit to gerrit. 

Cherry pick your change (in this example we're pulling patch set 1 of 47121).  If you click "Download" from the gerrit change you want to cherry pick, it will give you a dialog box as below and you can copy it to your clipboard:

 

$ cd /root/oom
$ git fetch https://gerrit.onap.org/r/oom refs/changes/21/47121/1 && git cherry-pick FETCH_HEAD

Now I have my changes local to rancher, where I can give it a try.

Run the following to get your change applied:


$ cd /root/oom/kubernetes

$ make all

$ cd /root

$ helm upgrade -i dev local/onap --namespace onap -f integration-override.yaml --set aai.enabled=false

$ helm upgrade -i dev local/onap --namespace onap -f integration-override.yaml --set aai.enabled=true


Look at the AAI pods:

$ kubectl get pods -n onap | grep aai


You'll see something similar to this if everything came up correctly:

root@rancher:/home/ubuntu# kubectl get pods -n onap | grep aai

dev-aai-77d54bc645-kkmb8                        1/1       Running            0          22h

dev-aai-babel-7cf9689996-jt866                  2/2       Running            0          22h

dev-aai-cassandra-0                             1/1       Running            0          12h

dev-aai-cassandra-1                             1/1       Running            0          22h

dev-aai-cassandra-2                             1/1       Running            0          22h

dev-aai-champ-5dfdf9dfb5-7fc79                  1/1       Running            0          15h

dev-aai-data-router-7989bcfcb4-p5qwn            1/1       Running            0          22h

dev-aai-elasticsearch-7fc496d745-tf67m          1/1       Running            0          22h

dev-aai-gizmo-5c46b4fc4c-8d42w                  2/2       Running            0          22h

dev-aai-hbase-5d9f9b4595-fx82p                  1/1       Running            5          22h

dev-aai-modelloader-787b7c9c54-jnd9j            2/2       Running            0          22h

dev-aai-resources-6c6f56cd86-6clpb              2/2       Running            0          22h

dev-aai-search-data-5f8465445f-l7j8n            2/2       Running            0          22h

dev-aai-sparky-be-687fc7487d-7mqzg              2/2       Running            1          15h

dev-aai-traversal-7955b55464-tz98p              2/2       Running            0          22h

Run a healthdist to check the AAI services

$ cd /root/oom/kubernetes/robot; ./ete-k8s.sh onap healthdist

Log into the resources node:

$ kubectl exec -it $(kubectl get pods -n onap | grep dev-aai-resources | awk '{print $1}') /bin/bash

From the resources container you can look at logs to see application status, or send requests via curl to test the API.


  • No labels

13 Comments

  1. Zi Li

    Hi Jimmy,

    How many resources are need to setup a AAI with helm?  I tried to set up with a k8s work node with 12 VCPUs/64GB RAM and 160GB disk. But at last several pods status are Evicted

    1. James Forsyth

      Hi, Zi Li - that should work for you - which ones are exited?  Do you get any logs for the failed ones?

      1. Zi Li

        For my circumstance. Both data-router/hbase/search-data are evicted. And the status of sparky-be/traversal are kept "Init:0/1".

        the evicted pod details shows bellow: 

        ubuntu@k8s-master:~$ kubectl describe pod aai-aai-data-router-74b89f4449-ft6j5 -n onap
        Name: aai-aai-data-router-74b89f4449-ft6j5
        Namespace: onap
        Priority: 0
        PriorityClassName: <none>
        Node: k8s-worker1/
        Start Time: Thu, 16 Aug 2018 09:09:17 +0000
        Labels: app=aai-data-router
        pod-template-hash=3064590005
        release=aai
        Annotations: <none>
        Status: Failed
        Reason: Evicted
        Message: The node was low on resource: ephemeral-storage. Container aai-data-router was using 136048Ki, which exceeds its request of 0.
        IP:
        Controlled By: ReplicaSet/aai-aai-data-router-74b89f4449
        Init Containers:
        init-sysctl:
        Image: docker.io/busybox
        Port: <none>
        Host Port: <none>
        Command:
        /bin/sh
        -c
        mkdir -p /logroot/data-router/logs
        chmod -R 777 /logroot/data-router/logs
        chown -R root:root /logroot

        Environment:
        NAMESPACE: onap (v1:metadata.namespace)
        Mounts:
        /logroot/ from aai-aai-data-router-logs (rw)
        /var/run/secrets/kubernetes.io/serviceaccount from default-token-6wpx5 (ro)
        Containers:
        aai-data-router:
        Image: nexus3.onap.org:10001/onap/data-router:1.2.2
        Port: 9502/TCP
        Host Port: 0/TCP
        Liveness: tcp-socket :9502 delay=10s timeout=1s period=10s #success=1 #failure=3
        Readiness: tcp-socket :9502 delay=10s timeout=1s period=10s #success=1 #failure=3
        Environment:
        SERVICE_BEANS: /opt/app/data-router/dynamic/conf
        CONFIG_HOME: /opt/app/data-router/config/
        KEY_STORE_PASSWORD: OBF:1y0q1uvc1uum1uvg1pil1pjl1uuq1uvk1uuu1y10
        DYNAMIC_ROUTES: /opt/app/data-router/dynamic/routes
        KEY_MANAGER_PASSWORD: OBF:1y0q1uvc1uum1uvg1pil1pjl1uuq1uvk1uuu1y10
        PATH: /usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin
        JAVA_HOME: usr/lib/jvm/java-8-openjdk-amd64
        Mounts:
        /etc/localtime from localtime (ro)
        /logs/ from aai-aai-data-router-logs (rw)
        /opt/app/data-router/config/auth from aai-aai-data-router-auth (rw)
        /opt/app/data-router/config/data-router.properties from aai-aai-data-router-properties (rw)
        /opt/app/data-router/config/schemaIngest.properties from aai-aai-data-router-properties (rw)
        /opt/app/data-router/dynamic/conf/entity-event-policy.xml from aai-aai-data-router-dynamic-policy (rw)
        /opt/app/data-router/dynamic/routes/entity-event.route from aai-aai-data-router-dynamic-route (rw)
        /var/run/secrets/kubernetes.io/serviceaccount from default-token-6wpx5 (ro)
        Volumes:
        localtime:
        Type: HostPath (bare host directory volume)
        Path: /etc/localtime
        HostPathType:
        aai-aai-data-router-auth:
        Type: Secret (a volume populated by a Secret)
        SecretName: aai-aai-data-router
        Optional: false
        aai-aai-data-router-properties:
        Type: ConfigMap (a volume populated by a ConfigMap)
        Name: aai-aai-data-router-prop
        Optional: false
        aai-aai-data-router-dynamic-route:
        Type: ConfigMap (a volume populated by a ConfigMap)
        Name: aai-aai-data-router-dynamic
        Optional: false
        aai-aai-data-router-dynamic-policy:
        Type: ConfigMap (a volume populated by a ConfigMap)
        Name: aai-aai-data-router-dynamic
        Optional: false
        aai-aai-data-router-logs:
        Type: HostPath (bare host directory volume)
        Path: /dockerdata-nfs/aai/aai/data-router/logs
        HostPathType:
        default-token-6wpx5:
        Type: Secret (a volume populated by a Secret)
        SecretName: default-token-6wpx5
        Optional: false
        QoS Class: BestEffort
        Node-Selectors: <none>
        Tolerations: node.kubernetes.io/not-ready:NoExecute for 300s
        node.kubernetes.io/unreachable:NoExecute for 300s
        Events: <none>

      2. Zi Li

        Hi, James Forsyth, For the minimum function of AAI -to store/query/update data. I think I could remove some charts to decrease the resources consumption. May be the sparky/modelloader/elasticsearch could be remove ? Any suggestion for the minimum system of AAI?

        1. James Forsyth

          I think the bare minimum configuration to do store/query/update graph data would be resources, traversal, haproxy, Cassandra. Venkata Harish Kajur anything else?

  2. Zi Li

    Also, after I add a new work node to the k8s cluster, try to "helm del --purge aai" and helm install again. I got a error " persistent volumes aai-aai-cassandra-0 already exists". It seems that the persistent volumes can not be deleted with helm. 

    So, do you know how to deal with this issue about install aai again in the original k8s environment?

    1. James Forsyth

      Venkata Harish Kajur or Pavel Paroulek have you guys seen this before?

      1. Pavel Paroulek

        Hi James Forsyth, no I haven't.

        I don't know what storage provider is in openlab (by default AAI uses the default storage provider), if this not a single occurance of this error message (it might be - for example someone already installed AAI), it would be better to open a JIRA bug.

        1. Zi Li

          Hi James Forsyth, Pavel Paroulek, I tried "kubectl get pvc -n onap" and then delete all the pvc items. Then the aai could be re-installed.

    2. Pavel Paroulek

      Hi Zi Li, I am not sure where the error comes from but what you could do is:

      1. manually remove the persistent volumes with the following command - kubectl delete pv aai-aai-cassandra-0 -n onap (I assume the namespace is "onap")
      2. When you re-create aai with helm, pass values.yaml as the argument and try setting persistence: enabled: false in the values.yaml file


       Please before deleting the volumes in step 1 could you paste the output of the following command here: kubectl describe pv aai-aai-cassandra-0 -n onap

      Thanks!


  3. Brian Freeman

    What I tend to do on a re-install of a entire project is to not only do the kubectl -n onap get pv but also pvc, security, services and make sure the project is gone

    Then I go to /dockerdata-nfs/dev and delete the subtending directory for that project

    then I do the helm upgrade with  --set project.enabled=true


    K8S / helm basic commands for ONAP integration is a page with some of the commands


    1. Pavel Paroulek

      Maybe there should be a purging mechanism as these manual steps seem error prone. It even seems easier to spin up a new VM (smile) (j/k)

  4. Brian Freeman

    Feel free to submit a proposed script perhaps to the integration project ? A good topic for the integration meeting on Wednesdays