AAF 2.1.8 bootstrap is missing some required permissions. 

Symptoms: 

  • Authenticated topic provisioning via dmaap-bc does not succeed, and so mirrormaker pod is unable to start.
  • dmaap-bc pod log says "/opt/app/dmaapbc/ok_to_exit does not exist.  Sticking around..."   
  • If you login to a shell on dmaap-bc pod and examine /opt/app/dmaapbc/logs/ONAP/error.log, there will be an Error about service credentials not being valid for  AAF connection. 
  • Authenticated access to the dmaap-bc API will fail.  In particular, robot DMaaP Bus Controller Health Check With Basic Auth will fail.  (as reported in  DMAAP-1178 - Getting issue details... STATUS )


Resolution:

  1.  Deploy AAF separately first.
  2. In AAF GUI add:

role create org.onap.dmaap-bc.service
perm grant org.onap.dmaap-bc.api.access * read org.onap.dmaap-bc.service
perm grant org.onap.dmaap.mr.access * * org.onap.dmaap-bc.service
perm grant org.onap.dmaap.mr.topic * view org.onap.dmaap-bc.service
perm create org.onap.dmaap.mr.topic * * org.onap.dmaap-bc.service
perm create org.onap.dmaap-dr.feed * * org.onap.dmaap-bc.service
perm create org.onap.dmaap-dr.sub * * org.onap.dmaap-bc.service
perm create org.onap.dmaap.mr.topicFactory :org.onap.dmaap.mr.topic:org.onap.dmaap.mr create,destroy org.onap.dmaap-bc.service
role user add org.onap.dmaap-bc.service dmaap-bc@dmaap-bc.onap.org
role user add org.onap.dmaap-bc.api.Controller dmaap-bc@dmaap-bc.onap.org

       3. Deploy dmaap

NOTES:

  1. the message-router-mirrormaker pod is dependent on topic provisioning and AAF permissions being granted, which is done as a result of the message-router post-install job.  Sometimes this sequence takes a while and the message-router-mirrormaker pod status gets in a crashback loop.  Patience: if all the steps above are taken, it should eventually reach a ready state.  However, it will never succeed if the full topic provisioning wasn't successful.
  2. depending on your environment, the deployment of all the components takes a while and can easily exceed the default helm timeout.  Recommend adding --timeout 900 to your helm install command line.
  • No labels

3 Comments

  1. Brian Freeman

    Can you modify the dmaap helm charts to put a delay before the mirromaker pod - we do that for things like application waiting for the databsae to be configured

    1. Dominic Lunanuova

      A few thoughts on this suggestion:

      • we have put some delays in the post-install jobs (set in env vars so they can be tuned), but it's a bit of a guess as to what value to use.  Seems to be huge performance variations in different environments.
      • the DMaaP deployment already times out (when using the default), so I'm reluctant to add more (constant) delay.  
      • a delay seems to be counter to the kubernetes philosophy: retry until it works.

      Still, the crashback loop is disconcerting to a human - even if it eventually goes away.

  2. sunil unnava

    If the Helm hooks are executed without any issue, mirrormaker pod will not restart many a times, like what we are seeing now.

    We also asked the AAF to copy the permissions required to start the mirrormaker pod to Helm charts. Once ,i.e completed, this issue will never arise , as permissions are pre-loaded in to the AAF service