Context:
Our goal is to reach VNF from APP-C through Ansible, but we're facing couple of issues. In reach I mean to be able to execute arbitrary playbook in Ansible server instance from APP-C. We have Ansible server installed and configured which is accessible on this endpoint: http://10.74.1.84:8000/Dispatch, we have verified following request works (playbook gets actually executed):
Q1: We are expecting whole VM to be restarted, is our assumption correct? If yes what could be the cause of failure above?
Then we tried to check the status by ActionStatus action which is new in Beijing
Q2: What is the definition of Beijing release? In the OOM project there are 2 branches: amsterdam and master. We assumed running OOM scripts from master branch would install Beijing. Is it correct?
Q3: Is our expectation correct? What could error from the logs mean? From the package name org.openecomp.appc.adapter in stacktrace I can presume Ansible adapter which is deployed currently is little bit old, in the current source code package is renamed to org.onap.appc.adapter, again this looks like an amsterdam realease, doesn't it?
Bonus Questions:
2. We assume Ansible server should be installed only once manually and it's not the part of the ONAP automation, is this correct?
3. Does vFW/vSink demo VNF support NETCONF interface?
Additional notes I have taken can be found at APPC ONAP Testing Notes. I'll be doing something similar for the Beijing testing of APPC.
Hi Scott, thank you for the answers.
We progressed a little bit since OP, we managed to solve problem #1 i.e. restart VM from the APP-C REST endpoint. We had to change this:
"payload": "{\"vm-id\": \"2f1b24ee-6382-4e2c-b405-e8fbaa3a2d64\"}"
to this:
"payload": "{\"vm-id\": \"http://<openstack_hostname>:8774/v2.1/<tenant_id>/servers/2f1b24ee-6382-4e2c-b405-e8fbaa3a2d64\"}"
Also we had to override default OpenStack credentials from Config and storage > Config Maps > appc-conf-configmap menu of kubernetes dashboard with ones specific to our environment.
We also managed to solve problem #2 i.e. execute Ansible DG from SLI. We were missing Version parameter and non-empty values for User/Password parameters. So all good so far.
The error you're seeing with DMaaP is due to the DMaaP topic not being created yet (I.e. APPC-LCM-READ topic does not exist on your DMaaP server).
When APPC-LCM-READ topic should be created? We believe after action completes (ex. restart) and result becomes available. Correct?
Main problem now is to find CDT, it is missing from our installation. There is neither frontend neither backend (/restconf/operations/design-services:dbservice) available. Version of our appc is as follows:
nexus3.onap.org:10001/openecomp/appc-image v1.2.0 399e222d320b 4 months ago 3.04 GB
What could be the reason? Can you confirm CDT should be available in v1.2.0 version?
P.S. why can't you access pastebin links? Do you receive 404 or it's just empty content? I verified with my colleagues and it works for us
DMaap Tpics:
APPC CDT
Logs via Paste Bin:
Hi Scott,
appc-cdt needs backend which should be accessible at /restconf/operations/design-services:dbservice. We assume this endpoint is part of the appc (by the looks of the source code). We can confirm there is no such endpoint in our deployed 1.2.0. We tried to update appc version to 1.3.0 from the kubernetes deployment, but it failed to start up, probably because newer version requires additional volume mounts. We are aware we're using slightly older OOM (March 27) and there were some changes since then. We would like to avoid reinstalling latest ONAP unless we're sure it really will contain design-services:dbservice endpoint. We tried to spin up only appc-1.3.0 container (with overridden entry-point command since it fails to startup due to reason mentioned above IMO) and then from the container shell checked contents of /opt/onap/appc/features/ directory, there is no such thing as design-services, which makes me think 1.3.0 does not have the required backend functionality neither.
Then we decided to build appc-inbound module of appc project (from latest master), obtained appc-design-services-installer-1.3.0.zip, placed in /opt/onap/appc/features/ directory of 1.2.0 appc container and tried to run install_feature.sh script, but it failed with:
Error executing command: Illegal character in opaque part at index 48: mvn:org.opendaylight.controller/features-mdsal/${odl.controller.mdsal.features.version}/xml/features
Any ideas what to try next?
Beka
Scott Seabolt any comment on this? We are totally blocked! I'm surprised we are the only one facing this issue? I was trying reverse engineering things, i.e.modifying APP-C DB manually to simulate CDT actions, but this process is very inefficient, there should be an easier way!
We even pulled latest master OOM and installed only APP-C on separate VM, but design-services is still missing!
I realised we haven't actually performed self-service VNF onboarding after the VNF instantiation (this kinda answers my question about how APP-C determines which adapter to use). In this video it is mentioned that CDT is a temporary solution and I can confirm we don't have it in our installation (which we believe is Beijing).
New question is: what is the current recommended way to perform self-service VNF onboarding. Anyone?
Hi Beka,
The CDT Tool is appropriate tool to utilize for onboarding VNFs into APPC for the near future. In various forums there has been discussion regarding the CDT Tool possibly moving to SDC or at least a tool utilized in combination with the other controllers.
However, the CDT Tool is and will be the design tool for onboarding into APPC. More information on the CDT Tool can be found at http://onap.readthedocs.io/en/latest/submodules/appc.git/docs/APPC%20CDT%20Guide/APPC%20CDT%20Guide.html.
Additionally, I will combine my thoughts for the remainder of your questions above into a separate response.
Hi Beka,
My apologies I missed this last question. Please pull the latest code the error you were seeing was due to a defect in CDT. Also, there is a video showing how to onboard a vLoadBalancer for ConfigScaleOut testing. The team was still busy trying to work out some bugs when you last pulled the code. Should be all clear now.
APPC ConfigScaleOut Testing with vDNS/vLoadBalancer
-Scott Seabolt
Hi Scott,
What you mean by latest? Can you tell me exact version(s) of artifacts used? Like OOM commit hash, docker image tags/ids? Additional info how you setup your environment would be helpful. It's important to be able to reproduce exact environment as images are built daily and there could be a regression introduced. I'm having trouble setting up latest ONAP. Here's what I did:
Setup Kubernetes as described here: ONAP on Kubernetes on Rancher - deprecated. As a result environment looks like this:
root@app-c:~/aai# helm version Client: &version.Version{SemVer:"v2.8.2", GitCommit:"a80231648a1473929271764b920a8e346f6de844", GitTreeState:"clean"} Server: &version.Version{SemVer:"v2.8.2", GitCommit:"a80231648a1473929271764b920a8e346f6de844", GitTreeState:"clean"} root@app-c:~/aai# kubectl version Client Version: version.Info{Major:"1", Minor:"8", GitVersion:"v1.8.10", GitCommit:"044cd262c40234014f01b40ed7b9d09adbafe9b1", GitTreeState:"clean", BuildDate:"2018-03-19T17:51:28Z", GoVersion:"go1.8.3", Compiler:"gc", Platform:"linux/amd64"} Server Version: version.Info{Major:"1", Minor:"8+", GitVersion:"v1.8.10-rancher1", GitCommit:"66aaf7681d4a74778ffae722d1f0f0f42c80a984", GitTreeState:"clean", BuildDate:"2018-03-20T16:02:56Z", GoVersion:"go1.8.3", Compiler:"gc", Platform:"linux/amd64"} root@app-c:~/aai# docker version Client: Version: 17.03.2-ce API version: 1.27 Go version: go1.7.5 Git commit: f5ec1e2 Built: Tue Jun 27 03:35:14 2017 OS/Arch: linux/amd64 Server: Version: 17.03.2-ce API version: 1.27 (minimum version 1.12) Go version: go1.7.5 Git commit: f5ec1e2 Built: Tue Jun 27 03:35:14 2017 OS/Arch: linux/amd64 Experimental: false root@app-c:~/aai# docker ps | grep rancher-server 5d47f7bc5a61 rancher/server:v1.6.14 "/usr/bin/entry /u..." 4 hours ago Up 2 hours 3306/tcp, 0.0.0.0:8880->8080/tcp rancher-server
I created onap-lite.yaml values file for helm chart which only has appc enabled:
root@app-c:~/oom# cat kubernetes/onap-lite.yaml # Copyright © 2017 Amdocs, Bell Canada # # Licensed under the Apache License, Version 2.0 (the "License"); # you may not use this file except in compliance with the License. # You may obtain a copy of the License at # # http://www.apache.org/licenses/LICENSE-2.0 # # Unless required by applicable law or agreed to in writing, software # distributed under the License is distributed on an "AS IS" BASIS, # WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. # See the License for the specific language governing permissions and # limitations under the License. ################################################################# # Global configuration overrides. # # These overrides will affect all helm charts (ie. applications) # that are listed below and are 'enabled'. ################################################################# global: # Change to an unused port prefix range to prevent port conflicts # with other instances running within the same k8s cluster nodePortPrefix: 302 # ONAP Repository # Uncomment the following to enable the use of a single docker # repository but ONLY if your repository mirrors all ONAP # docker images. This includes all images from dockerhub and # any other repository that hosts images for ONAP components. #repository: nexus3.onap.org:10001 repositoryCred: user: docker password: docker # readiness check - temporary repo until images migrated to nexus3 readinessRepository: oomk8s # logging agent - temporary repo until images migrated to nexus3 loggingRepository: docker.elastic.co # image pull policy pullPolicy: IfNotPresent # default mount path root directory referenced # by persistent volumes and log files persistence: mountPath: /dockerdata # flag to enable debugging - application support required debugEnabled: false # Repository for creation of nexus3.onap.org secret repository: nexus3.onap.org:10001 ################################################################# # Enable/disable and configure helm charts (ie. applications) # to customize the ONAP deployment. ################################################################# aaf: enabled: false aai: enabled: false appc: enabled: true config: openStackType: OpenStackProvider openStackName: OpenStack openStackKeyStoneUrl: <URL> openStackServiceTenantName: default openStackDomain: default openStackUserName: <USERNAME> openStackEncryptedPassword: <PASSWORD> clamp: enabled: false cli: enabled: false consul: enabled: false dcaegen2: enabled: false dmaap: enabled: false esr: enabled: false log: enabled: false sniro-emulator: enabled: false oof: enabled: false msb: enabled: false multicloud: enabled: false nbi: enabled: false config: # openstack configuration openStackUserName: "vnf_user" openStackRegion: "Yolo" openStackKeyStoneUrl: "http://1.2.3.4:5000" openStackServiceTenantName: "service" openStackEncryptedPasswordHere: "c124921a3a0efbe579782cde8227681e" policy: enabled: false portal: enabled: false robot: enabled: false sdc: enabled: false sdnc: enabled: false replicaCount: 1 config: enableClustering: false mysql: disableNfsProvisioner: true replicaCount: 1 so: enabled: false replicaCount: 1 liveness: # necessary to disable liveness probe when setting breakpoints # in debugger so K8s doesn't restart unresponsive container enabled: true # so server configuration config: # message router configuration dmaapTopic: "AUTO" # openstack configuration openStackUserName: "vnf_user" openStackRegion: "RegionOne" openStackKeyStoneUrl: "http://1.2.3.4:5000" openStackServiceTenantName: "service" openStackEncryptedPasswordHere: "c124921a3a0efbe579782cde8227681e" # configure embedded mariadb mariadb: config: mariadbRootPassword: password uui: enabled: false vfc: enabled: false vid: enabled: false vnfsdk: enabled: false
Built charts by running make all
Then executed command:
helm install local/onap -n onap-lite --namespace onap -f onap-lite.yaml
Apparently deployment was trying to pull dmaap-mr:1.1.3 which does not exist, so I changed it to 1.1.4 from the Kubernetes dashboard
Finally all went green and as a result following Docker images were pulled:
root@app-c:~/oom# docker images | grep onap nexus3.onap.org:10001/onap/appc-cdt-image 1.3.0-SNAPSHOT-latest 0fd42205579b 24 hours ago 205 MB nexus3.onap.org:10001/onap/appc-image 1.3.0-SNAPSHOT-latest 58ef0ee0c1a7 24 hours ago 2.8 GB nexus3.onap.org:10001/onap/ccsdk-dgbuilder-image 0.2-STAGING-latest 07fbe45737ca 25 hours ago 1.02 GB nexus3.onap.org:10001/onap/data-router 1.2-STAGING-latest 71eb8474e885 25 hours ago 672 MB nexus3.onap.org:10001/onap/msb/msb_apigateway 1.1.0-STAGING-latest 1f84e3ae2622 25 hours ago 222 MB nexus3.onap.org:10001/onap/search-data-service 1.2-STAGING-latest b469cac22d9f 25 hours ago 629 MB nexus3.onap.org:10001/onap/msb/msb_discovery 1.1.0-STAGING-latest 72567cfc8218 25 hours ago 205 MB nexus3.onap.org:10001/onap/aai-resources 1.2-STAGING-latest ca105340737b 25 hours ago 667 MB nexus3.onap.org:10001/onap/babel 1.2-STAGING-latest 964c9395bc40 25 hours ago 790 MB nexus3.onap.org:10001/onap/sparky-be 1.2-STAGING-latest bbfe5d400949 25 hours ago 697 MB nexus3.onap.org:10001/onap/model-loader 1.2-STAGING-latest 7cd6378192b0 25 hours ago 632 MB nexus3.onap.org:10001/onap/aai-traversal 1.2-STAGING-latest b19dcfe08bdb 25 hours ago 685 MB nexus3.onap.org:10001/onap/oom/kube2msb latest bcf13ddd527e 25 hours ago 41.3 MB nexus3.onap.org:10001/onap/dmaap/dmaap-mr 1.1.4 f0858b5c021c 8 days ago 863 MB nexus3.onap.org:10001/onap/sniroemulator latest e755cbf017f3 11 days ago 548 MB aaionap/haproxy 1.2.2 3e52a8117da7 12 days ago 139 MB nexus3.onap.org:10001/onap/gizmo 1.1-STAGING-latest bf2bfa4a0e54 6 weeks ago 702 MB aaionap/hbase 1.2.0 3568a85848ab 8 months ago 432 MB
However lcm-provider API is missing from ODL apidoc explorer. I noticed most of the appc features are uninstalled:
opendaylight-user@root>feature:list | grep appc appc-dg-util | 1.3.0.SNAPSHOT | | Uninstalled | appc-1.3.0-SNAPSHOT | application DG utillity appc-iaas-adapter | 1.3.0.SNAPSHOT | | Uninstalled | appc-iaas-adapter-1.3.0-SNAPSHOT | appc-iaas-adapter appc-config-params | 1.3.0.SNAPSHOT | | Uninstalled | appc-config-params-1.3.0-SNAPSHOT | Application Controller Config Params appc-event-listener | 1.3.0.SNAPSHOT | | Uninstalled | appc-event-listener-1.3.0-SNAPSHOT | event listener appc-license-manager | 1.3.0.SNAPSHOT | x | Started | appc-license-manager-1.3.0-SNAPSHOT | application license manager appc-dmaap-adapter | 1.3.0.SNAPSHOT | | Uninstalled | appc-dmaap-adapter-1.3.0-SNAPSHOT | appc-dmaap-adapter appc-artifact-handler | 1.3.0.SNAPSHOT | | Uninstalled | app-cntrl-artifact-handler-1.3.0-SNAPSHOT | Application Controller Artifact Handler appc-command-executor | 1.3.0.SNAPSHOT | | Uninstalled | appc-command-executor-1.3.0-SNAPSHOT | application executor appc-config-flow-controller | 1.3.0.SNAPSHOT | | Uninstalled | appc-config-flow-controller-1.3.0-SNAPSHOT | Application Controller Flow Executor appc-request-handler | 1.3.0.SNAPSHOT | | Uninstalled | appc-request-handler-1.3.0-SNAPSHOT | appc-request-handler appc-oam | 1.3.0.SNAPSHOT | | Uninstalled | appc-1.3.0-SNAPSHOT | appc oam module appc-dg-shared | 1.3.0.SNAPSHOT | | Uninstalled | appc-dg-shared-1.3.0-SNAPSHOT | appc-dg-shared appc-ssh-adapter | 1.3.0.SNAPSHOT | | Uninstalled | appc-ssh-adapter-1.3.0-SNAPSHOT | appc-ssh-adapter appc-dispatcher | 1.3.0.SNAPSHOT | | Uninstalled | appc-dispatcher-1.3.0-SNAPSHOT | appc-dispatcher appc-network-inventory-client | 1.3.0.SNAPSHOT | | Uninstalled | appc-network-inventory-client-1.3.0-SNAPSHOT | Application Controller Instar Client appc-interfaces-service | 1.3.0.SNAPSHOT | | Uninstalled | appc-interfaces-service-1.3.0-SNAPSHOT | Application Controller Interfaces Services appc-sdc-listener | 1.3.0.SNAPSHOT | | Uninstalled | appc-sdc-listener-1.3.0-SNAPSHOT | sdc listener appc-aai-client | 1.3.0.SNAPSHOT | | Uninstalled | appc-aai-client-1.3.0-SNAPSHOT | Application Controller AAI Client appc-metric | 1.3.0.SNAPSHOT | x | Started | appc-1.3.0-SNAPSHOT | application executor appc-config-generator | 1.3.0.SNAPSHOT | | Uninstalled | appc-config-generator-1.3.0-SNAPSHOT | Application Controller Config Generator appc-config-data-services | 1.3.0.SNAPSHOT | | Uninstalled | appc-config-data-services-1.3.0-SNAPSHOT | Application Controller Config Data Service appc-config-adaptor | 1.3.0.SNAPSHOT | | Uninstalled | appc-config-adaptor-1.3.0-SNAPSHOT | Application Controller Config Adaptor appc-design-services | 1.3.0.SNAPSHOT | | Uninstalled | appc-design-services-1.3.0-SNAPSHOT | Application Controller Design Services appc-netconf-adapter | 1.3.0.SNAPSHOT | | Uninstalled | appc-netconf-adapter-1.3.0-SNAPSHOT | appc-netconf-adapter appc-ansible-adapter | 1.3.0.SNAPSHOT | | Uninstalled | appc-ansible-adapter-1.3.0-SNAPSHOT | appc-ansible-adapter appc-sequence-generator | 1.3.0.SNAPSHOT | | Uninstalled | appc-1.3.0-SNAPSHOT | Appc Sequence Generator appc-lifecycle-management | 1.3.0.SNAPSHOT | x | Started | appc-lifecycle-management-1.3.0-SNAPSHOT | application executor appc-provider | 1.3.0.SNAPSHOT | | Uninstalled | appc-1.3.0-SNAPSHOT | application controller appc-config-audit | 1.3.0.SNAPSHOT | | Uninstalled | appc-config-audit-1.3.0-SNAPSHOT | Application Controller Config Audit appc-rest-adapter | 1.3.0.SNAPSHOT | | Uninstalled | appc-rest-adapter-1.3.0-SNAPSHOT | appc-rest-adapter appc-config-encryption-tool | 1.3.0.SNAPSHOT | | Uninstalled | appc-config-encryption-tool-1.3.0-SNAPSHOT | Application Controller Encription Tool appc-workflow-management | 1.3.0.SNAPSHOT | | Uninstalled | appc-workflow-management-1.3.0-SNAPSHOT | application executor appc-chef-adapter | 1.3.0.SNAPSHOT | | Uninstalled | appc-chef-adapter-1.3.0-SNAPSHOT | appc-chef-adapter
And there were bunch of errors produced by appc container (which does not make any sense for me): https://pastebin.com/raw/7ZsuJRZZ
I'm struggling with this for almost 3 days... Any ideas?
Thanks
Seems problem was inside nexus3.onap.org:10001/onap/appc-image:1.3.0-SNAPSHOT-latest, when I switched to nexus3.onap.org:10003/onap/appc-image:latest karaf errors were gone.
Context
Our goal is to reach VNF from APP-C through Ansible, but we're facing couple of issues. In reach I mean to be able to execute arbitrary playbook in Ansible server instance from APP-C. We have Ansible server installed and configured which is accessible on this endpoint: http://10.74.1.84:8000/Dispatch, we have verified following request works (playbook gets actually executed):
We performed demo VNF onboarding and instantiation with the help of materials available here: Running the ONAP Demos. Note we don't care about closed-loop yet, our sole goal is to interact with virtual network functions through APP-C.
Problem #1
For simplicity we tried to execute Restart action first, we believe it does not require any interaction with Ansible server, rather it uses OpenStack directly to restart whole VM.
We got ACCEPTED as a response, however nothing happened. Here are the debug logs from APP-C ODL console: https://pastebin.com/raw/5W6quZSt.
We also see similar messages in logs:
Q1: We are expecting whole VM to be restarted, is our assumption correct? If yes what could be the cause of failure above?
Then we tried to check the status by ActionStatus action which is new in Beijing (http://onap.readthedocs.io/en/latest/submodules/appc.git/docs/APPC%20LCM%20API%20Guide/APPC%20LCM%20API%20Guide.html#actionstatus):
but surprisingly request was rejected with following response:
Now we're starting to doubt whether we really have Beijing, because I was told elsewhere that our installation looks like an Amsterdam release.
Q2: What is the definition of Beijing release? In the OOM project there are 2 branches: amsterdam and master. We assumed running OOM scripts from master branch would install Beijing. Is it correct?
Problem #2
Next we attempted to bypass LCM API and hit directly SLI, we executed following Ansible DG:
We were expecting playbook to be executed, but it didn't. Seems there is some error in the logs: https://pastebin.com/raw/hrQihnEc
Q3: Is our expectation correct? What could error from the logs mean? From the package name org.openecomp.appc.adapter in stacktrace I can presume Ansible adapter which is deployed currently is little bit old, in the current source code package is renamed to org.onap.appc.adapter, again this looks like an amsterdam realease, doesn't it?
Bonus questions
Thanks!
Beka