Types of Users and Usage Instructions:


Short Video Series available at :  https://www.youtube.com/playlist?list=PLj-oRfbkqkfnN_2vnfhivCesJ118SA_zG

Demo day demonstration  recording and slides available at :  https://wiki.onap.org/display/DW/2020-04-16+DCAE+Demo


Sr.No

User

Usage Instructions

1.

Developers who are looking to onboard their mS

·        Access the Nifi  Web UI url provided to you

·        Follow steps  2.b to 2.d

·        You should be able to see your microservices in the Nifi Web UI by clicking and dragging ‘Processor’ on the canvas, and searching for the name of the microservice/component/processor.

2.

Designers who are building the flows through UI and triggering distribution

·        Access the Nifi  Web UI url provided to you

·        Follow steps 3 to the end of the document

3.

Infrastructure/ Admins who want to stand up DCAE Mod and validate it

·        Follow start to the end



1.    Deployment of DCAE MOD components via Helm charts

The DCAE MOD components are deployed using the standard ONAP OOM deployment process.   When deploying ONAP using the helm deploy command, DCAE MOD components are deployed when the dcaemod.enabled flag is set to true, either via a --set option on the command line or by an entry in an overrides file.  In this respect, DCAE MOD is no different from any other ONAP subsystem.

The default DCAE MOD deployment relies on an nginx ingress controller being available in the Kubernetes cluster where DCAE MOD is being deployed.   The Rancher RKE installation process sets up a suitable ingress controller.   In order to enable the use of the ingress controller, it is necessary to override the OOM default global settings for ingress configuration.   Specifically, the installation needs to set the following configuration in an override file:


Global Ingress configuration
  #Global ingress configuration
  ingress:
    enabled: true
    virtualhost:
      baseurl: "simpledemo.onap.org"


When DCAE MOD is deployed with an ingress controller, several endpoints are exposed outside the cluster at the ingress controller's external IP address and port.   (In the case of a Rancher RKE installation, there is an ingress controller on every worker node, listening at the the standard HTTP port (80).)  These exposed endpoints are needed by users using machines outside the Kubernetes cluster.

Endpoint

 Routes to (cluster internal address)

Description

/nifi

http://dcaemod-designtool:8080/nifi

Design tool Web UI

/nifi-api

http://dcaemod-designtool:8080/nifi-api

Design tool API

/nifi-jars

http://dcaemod-nifi-registry:18080/nifi-jars

Flow registry listing of JAR files built from component specs

/onboarding

http://dcaemod-onboarding-api:8080/onboarding

Onboarding API

/distributor

http://dcaemod-distributor-api:8080/distributor

Distributor API

To access the design Web UI, for example, a user would use the URL : http://ingress_controller_address:ingress_controller_port/nifi.
ingress_controller_address is the the IP address or DNS FQDN of the ingress controller and
ingress_controller_port is the port on which the ingress controller is listening for HTTP requests.  (If the port is 80, the HTTP default, then there is no need to specify a port.)

There are two additional internal endpoints that users need to know, in order to configure a registry client and a distribution target in the design tool's controller settings.

Configuration Item

Endpoint URL

Registry client

http://dcaemod-nifi-registry:18080

Distribution target

http://dcaemod-runtime-api:9090


As OOM/ingress template has been updated in Guilin release to enable virtual host, MOD API's and UI access via ingress should use dcaemod.api.simpledemo.onap.org 

Add entry for dcaemod.simpledemo.onap.org in /etc/hosts with the correct IP (any of K8S node IP can be specified)


Using DCAE MOD without an Ingress Controller

Not currently supported.

2.    Configuring DCAE mod

Our demo is hosted on 10.12.7.116. The IP Address for the purpose of this demo will hence be 10.12.7.116. In case of other deployments, we would have used the IP Address, or the DNS FQDN, if there is one, for one of the Kubernetes nodes.

Now let’s access the Nifi (DCAE designer) UI - http://dcaemod.simpledemo.onap.org/nifi/


a) Get the artifacts to test and onboard.

 

Let's fetch the artifacts/ spec files 


A sample Component DCAE-VES-Collector : https://git.onap.org/dcaegen2/collectors/ves/tree/dpo/spec/vescollector-componentspec.json

A sample Data Format : https://git.onap.org/dcaegen2/collectors/ves/tree/dpo/data-formats/VES-5.28.4-dataformat.json


For the purpose of onboarding, a Sample Request body should be of the type -

{ "owner": "<some value>", "spec": <some json object> }

 where the json object inside the spec field can be a component spec json.

Request bodies of this type will be used in the onboarding requests you make using curl or the onboarding swagger interface.


The prepared Sample Request body for a component dcae-ves-collector looks like so


The prepared Sample request body for a sample data format  looks like so -


b) To onboard a data format and a component

Each component has a description that tells what it does.

These requests would be of the type-

curl -X POST http://<onboardingapi host>/onboarding/dataformats     -H "Content-Type: application/json" -d @<filepath to request>
curl -X POST http://<onboardingapi host>/onboarding/components     -H "Content-Type: application/json" -d @<filepath to request>   

In our case, 
curl -X POST http://dcaemod.simpledemo.onap.org/onboarding/dataformats     -H "Content-Type: application/json" -d @<filepath to request>
curl -X POST http://dcaemod.simpledemo.onap.org/onboarding/components     -H "Content-Type: application/json" -d @<filepath to request>  

You can download the Components and Data Formats for the demo from – 

Components: 

https://git.onap.org/dcaegen2/collectors/ves/tree/dpo/spec/vescollector-componentspec.json 

https://git.onap.org/dcaegen2/analytics/tca-gen2/tree/dcae-analytics/dpo/tcagen2_spec.json 

Corresponding Data Formats: 

https://git.onap.org/dcaegen2/collectors/ves/tree/dpo/data-formats
https://git.onap.org/dcaegen2/analytics/tca-gen2/tree/dcae-analytics/dpo/


c) Verify the resources were created using

curl -X GET http://dcaemod.simpledemo.onap.org/onboarding/dataformats

curl -X GET http://dcaemod.simpledemo.onap.org/onboarding/components


d) Verify the genprocessor (which polls onboarding periodically to convert component specs to nifi processor), converted the component

Open http://dcaemod.simpledemo.onap.org/nifi-jars/ in a browser.

These jars should now be available for you to use in the nifi UI as processors.


3.    Design & Distribution Flow


a) Configure Nifi Registry url

Next check Nifi settings by selecting the Hamburger button in the Nifi UI. It should lead you to the Nifi Settings screen

Add a registry client. The Registry client url will be http://dcaemod-nifi-registry:18080


b) Add distribution target which will be the runtime api url

Set the distribution target in the controller settings


Distribution target URL will be http://dcaemod-runtime-api:9090


c) To start creating flows, we need to create a process group first. The name of the process group will be the name of the flow. Drag and Drop on the canvas, the ‘Processor Group’ icon from the DCAE Designer bar on the top.



Now enter the process group by double clicking it,

You can now drag and drop on the canvas ‘Processor’ icon from the top DCAE Designer tab. You can search for a particular component in the search box that appears when you attempt to drag the ‘Processor’ icon to the canvas.


If the Nifi registry linking worked, you should see the “Import” button when you try to add a Processor or Process group to the Nifi canvas, like so-


By clicking on the import button, we can import already created saved and version controlled flows from the Nifi registry, if they are present.


We can save created flows by version controlling them like so starting with a 'right click' anywhere on the canvas-





Ideally you would name the flow and process group the same, because functionally they are similar.



When the flow is checked in, the bar at the bottom shows a green checkmark


Note: Even if you move a component around on the canvas, and its position on the canvas changes, it is recognized as a change, and it will have to recommitted.



d) Adding components and building the flow

You can add additional components in your flow and connect them.

DcaeVesCollector connects to DockerTcagen2.





Along the way you need to also provide topic names in the settings section. These can be arbitrary names.


To recap, see how DcaeVesCollector connects to DockerTcagen2. Look at the connection relationships. Currently there is no way to validate these relationships. Notice how it is required to name the topics by going to Settings.

The complete flow after joining our components looks like so




 

e) Submit/ Distribute the flow:

Once your flow is complete and saved in the Nifi registry, you can choose to submit it for distribution.


If the flow was submitted successfully to the runtime api, you should get a pop up a success message like so -


At this step, the design was packaged and sent to Runtime api.

The runtime is supposed to generate the blueprint out of the packaged design/flow and push it to the DCAE inventory and the DCAE Dasboard.


f) Checking the components in the DCAE Dashboard

You should see the generated artifact/ blueprint in the DCAE Dashboard dashboard at https://10.12.7.116:30418/ccsdk-app/login_external.htm in our deployment. The name for each component will be appended by the flow name followed by underscore followed by the component’s name.

The credentials to access the DCAE Dashboard are

Login: su1234

Password: fusion


The generated Blueprint can be viewed.



Finally, the generated Blueprint can be deployed.



You can use/import the attached input configurations files to deploy. Drag and Drop these sample JSON files to fill in the configuration values.

NOTE 1: Increase memory limit to 512Mi

NOTE 2: Verify image URL

tca-deploy.input.jsonves-deploy.input.json


  • No labels

14 Comments

  1. Damian Nowak

    Vijay Venkatesh Kumar  A kind request - Woudl You be able to use a Confleunce "Code" or "Fold" macro the JSONs?

    These plain JSONs consume currently 80% of the page space, honestly speaking, I have doubts, if people would be really scrolling down to DCAE MOD user-guide, which is in fact a nice doc.

    1. Deven Panchal

      Vijay Venkatesh Kumar Damian Nowak  Cleaned up. Hope this looks better.

      1. Damian Nowak

        Deven Panchal Oh yes, looks much cleaner now!!
        Something like a live demo, or a recorded demo would be a whoa! thing.

        1. Deven Panchal

          Damian Nowak Yes, we are preparing one. Should be available here sometime this month.

          1. Vijay Venkatesh Kumar

            Thanks Deven Panchal

            Damian Nowak  - Demo should be scheduled soon (possibly next week). Will send out the invite/notice to discuss list once finalized.

  2. Damian Nowak

    Vijay Venkatesh Kumar  One more question: Are You planning to interate DCAE Dashboard with ONAP Portal? This would be helpful.

    1. Vijay Venkatesh Kumar

      Damian Nowak - Indeed yes. This was stretch goal for Frankfurt (DCAEGEN2-1870) but had to be deferred.  Will assess this integration for Guilin.

  3. Hieu Nguyen

    Vijay Venkatesh Kumar I have a problem with DCAE MOD, actually cannot add "dcaemod-runtime-api", here is "http://dcaemod-runtime-api:9090" to "Distribution Target Environments" under "Controller Settings" via DCAE Designer Web UI. The error was shown as follows:

    " Parse Error home

    <!DOCTYPE html> <html> <head> <meta http-equiv="Content-Type" content="text/html; charset=utf-8"/> <link rel="shortcut icon" href="/nifi/images/nifi16.ico"/> <title>NiFi</title> <link rel="stylesheet" href="/nifi/assets/reset.css/reset.css" type="text/css"/> <link rel="stylesheet" href="/nifi/css/common-ui.css" type="text/css"/> <link rel="stylesheet" href="/nifi/fonts/flowfont/flowfont.css" type="text/css"/> <link rel="stylesheet" href="/nifi/assets/font-awesome/css/font-awesome.min.css" type="text/css"/> <link rel="stylesheet" href="/nifi/css/message-pane.css" type="text/css"/> <link rel="stylesheet" href="/nifi/css/message-page.css" type="text/css"/> <meta http-equiv="Refresh" content="5; url=/nifi/"> </head> <body class="message-pane"> <div class="message-pane-message-box"> <p class="message-pane-title"> Did you mean: <a href="/nifi/">/nifi</a> </p> <p class="message-pane-content">You may have mistyped... but we'll try to redirect you in 5 seconds.</p> </div> </body> </html>"

    I also tried to use "dcaemod-distributor-api" HTTP API to add http://dcaemod-runtime-api:9090 to "Distribution Target Environments", but after that, such added target environment did not appear on DCAE Designer Web UI, as well as still cannot assigned "Process Group" into such target environment via "dcaemod-distributor-api" HTTP API, please see the step-by-step below for the details:
    1. Add http://dcaemod-runtime-api:9090 by:   curl -X POST "http://dcaemod-distributor-api:8080/distributor/distribution-targets" -H "accept: application/json" -H "Content-Type: application/json" -d @distribution-target.json
    Response:

    {"selfUrl": "http://dcaemod-distributor-api:8080/distributor/distribution-targets/66563aec-3bac-4f79-a0b7-efa9ade94aec", "id": "66563aec-3bac-4f79-a0b7-efa9ade94aec", "name": "runtimedcae", "runtimeApiUrl": "http://dcaemod-runtime-api:9090", "description": "test_runtime_dcae", "nextDistributionTargetId": null, "created": "2020-05-07T09:08:10.239677", "modified": "2020-05-07T09:08:10.239677", "processGroups": []}


    Note: the distribution-target.json file content

    {
    "name": "runtimedcae",
    "runtimeApiUrl": "http://dcaemod-runtime-api:9090",
    "description": "test_runtime_dcae"
    }

    2. Verify added success by: curl -X GET "http://dcaemod-distributor-api:8080/distributor/distribution-targets" | python -mjson.tool

    Response:
    {
    "distributionTargets": [
    {
    "created": "2020-05-07T09:08:10.239677",
    "description": "test_runtime_dcae",
    "id": "66563aec-3bac-4f79-a0b7-efa9ade94aec",
    "modified": "2020-05-07T09:08:10.239677",
    "name": "runtimedcae",
    "nextDistributionTargetId": null,
    "processGroups": [],
    "runtimeApiUrl": "http://dcaemod-runtime-api:9090",
    "selfUrl": "http://dcaemod-distributor-api:8080/distributor/distribution-targets/66563aec-3bac-4f79-a0b7-efa9ade94aec"
    }
    ]
    }
    3. Add "Process Group" with id = edf21370-0171-1000-fbc9-b27a0badca35 (this Process Group Id exists in Nifi) by: curl -X POST "http://dcaemod-distributor-api:8080/distributor/distribution-targets/66563aec-3bac-4f79-a0b7-efa9ade94aec/process-groups" -H "accept: application/json" -H "Content-Type: application/json" -d @process-groups.json
    Response: {"message": "Internal Server Error"}


    Note: the process-groups.json file content
    {
    "processGroupId": "edf21370-0171-1000-fbc9-b27a0badca35",
    "dt_id": "66563aec-3bac-4f79-a0b7-efa9ade94aec"
    }


    Please help on this issues. Thank you!

    1. Shabnam Sultana

      Hi Hieu Nguyen,

      I am also facing the same issue with Guilin setup of ONAP where I get exactly the same error when I try to add the runtime API into the distribution target environment. I wanted to know if you could solve it and it would be great if you could let me know how this issue was solved.

      Thanks in advance. 

    2. Sylwia Jakubek

      I also had the same problem. Maybe this will be helpful for you. I was "fooled" by the key name in the json body of the POST request to API distribution: processGroupId - which suggests that its value should be the group's process id. After tracing the designtool-web frontend code, it turned out that processGroupId does not mean process group id (nifi api). Its value should be flow id (nifi registry api), which can be obtained by GET request: .../nifi-api/process-groups/<process group id> Then the flowId can be found in the JSON response:

      response ["component"] ["versionControlInformation"] ["flowId"]

      1. Hieu Nguyen

        Hello Shabnam Sultanaand Sylwia Jakubek:

        This is exactly Sylwia Jakubek mentioned, it is because of wrong "processGroupId", the correct request is: {"processGroupId": flow_id} and this flow_id will be used to distribute the app to dcae environment.

        We have at least 3 options to get the flow_id, as follows:
        1. curl -X GET http://nifi.simpledemo.onap.org/nifi-api/flow/process-groups/root   --> The flow id will be extracted by: flow_id = data["processGroupFlow"]["flow"]["processGroup"]["component"]["versionControlInformation"]["flowId"]

        2. curl -X GET http://nifi.simpledemo.onap.org/nifi-api/process-groups/root/process-groups  --> The flow id will be extracted by: flow_id = data["processGroup"]["component"]["versionControlInformation"]["flowId"]

        3. get the "flowId" via "Process Group ID", you can get such "Process Group ID" via API, or via NiFi Flow Web UI. Then sent request to:

        curl -X GET http://nifi-api.simpledemo.onap.org/nifi-api/process-groups/<Process Group ID>  --> The flow id will be extracted by: flow_id = data["component"]["versionControlInformation"]["flowId"]

        When you have flowId, assign process group id (represented by flow id) into distribution target environment by:

        1. curl -X POST "http://nifi.simpledemo.onap.org/distributor/distribution-targets/<distributionTargets Id>/process-groups" -H  "accept: application/json" -H  "Content-Type: application/json" -d "{"processGroupId": <flowId>}"

        or

        2. curl -X POST "http://dcaemod-distributor-api:8080/distributor/distribution-targets/<distributionTargets Id>/process-groups" -H  "accept: application/json" -H  "Content-Type: application/json" -d "{"processGroupId": <flowId>}"


  4. Kornel Janiak

    Do we have a possibility to remove onboarding components? Seems that adding two with same name blocks nifi to show in processor window of freshly added component spec. Does somebody observed something similar ?

    1. Deven Panchal

      No functionality to remove onboarding component from the GUI yet.

    2. Deven Panchal

      You could onboard a newer version of the component if you like.