This page describes steps taken and issues encountered in transitioning ONAP project to use small, multi-platform images
Issues encountered:
1) docker-maven-plugin. Some projects used spotify dockermaven-plugin, which is not supported on Arm. Changed to fabric8
2) Nexus docker registry. docker pull/push repos default nexus/onap does not support multi-platform images
a) fabric8 docker-maven-plugin allows setting from command line
b) or, build machine must set in ~/.m2/settings/xml
3) base image shell differences (alpine is not bash) impacts entrypoint and scripts
- for containers that rely heavily on scripts, e.g. dmaap/kafak11aaf add bash in image
4) docker-compose.yml must be configured for local dev
- refer to local images, not nexus
- includes dependencies that are not multi-platform
eg dmaap/buscontroller crunchydata/crunchy-postgres:centos7-10.4-2.0.0
5) Dockerfile sometimes define docker USER root. Not recommended by ONAP
6) image naming is inconsistent among projects, some append registry to name, e.g. buscontroller
- appends <onap.nexus.dockerregistry.daily$, other's simply name with onap/<project$<name$ e.g messageservice
- recommendation: use simple names and push registry definition to target push repo
7) testing methodology is inconsistent. Some projects provide docker-compose files, some not. Some projects provide guidance on specific testing, such as curl commands, some not. Ideally, projects could be launched in OOM (k8s.Helm) and tested with robot tests. However, we've found cases (dmaap-mr) where issues have been found that were not revealed by thin robot testing. Ideally. robust test scripts (e.g. Postman scripts) would be provided by teams. However, building and maintaining OOM on Intel and ARM infrastructure is difficult at best.
8) CIST. Some changes to image build pipeline, for instance changes to docker build command-line flags, require updates to CSIT. The CSIT pipeline is fragile in that changes must be verified offline (i.e. locally) and can cause issues in 'production'.
dmaap
datarouter
This section describes the migration of datarouter to small, multi-platform images.
The datarouter project generates 3 docker images:
datarouter-prov
datarouter-node
datarouter-subscriber
Each sub-project is structured similarly, therefore the changes to each to migrate to a small,
multi-platfrom image are similar and involve the following steps:
1. update pom to migrate from spotify docker-maven-plug to fabri8 docker-maven-plugin
2. create Dockerfile
#### Step 1: build and run on Intel (Mac Air)
$ git clone ssh://sandovalfrOAM@gerrit.onap.org:29418/dmaap/datarouter && scp -p -P 29418 sandovalfrOAM@gerrit.onap.org:hooks/commit-msg datarouter/.git/hooks/
$ cd datarouter
$ mvn clean install -DskipTests -Pdocker
$ docker images
nexus3.onap.org:10003/onap/dmaap/datarouter-subscriber latest b163a167f8c1 24 seconds ago 672MB
nexus3.onap.org:10003/onap/dmaap/datarouter-node latest eaadf6b93bcf 38 seconds ago 674MB
nexus3.onap.org:10003/onap/dmaap/datarouter-prov latest 8c9adc9a9851 About a minute ago 704MB
java 8 d23bdf5b1b1b 2 years ago 643MB
TEST
$ cd datarouter-docker-compose/src/main/resources
- edit docker-compose, change nexus 0001 (remote pull repo) to 0003 (local build)
$ docker-compose up
- terminal 2
$ docker container ls -a
CONTAINER ID IMAGE COMMAND CREATED STATUS PORTS NAMES
c193317ec860 nexus3.onap.org:10003/onap/dmaap/datarouter-node "sh startup.sh" About a minute ago Up About a minute 0.0.0.0:9090-$8080/tcp, 0.0.0.0:9443-$8443/tcp datarouter-node
e8dab741550e nexus3.onap.org:10003/onap/dmaap/datarouter-prov "sh startup.sh" About a minute ago Up About a minute (healthy) 0.0.0.0:8080-$8080/tcp, 0.0.0.0:8443-$8443/tcp, 0.0.0.0:443-$8443/tcp datarouter-prov
cf0e996f0f31 nexus3.onap.org:10003/onap/dmaap/datarouter-subscriber "sh startup.sh" About a minute ago Up About a minute 8080/tcp, 0.0.0.0:7070-$7070/tcp, 8443/tcp subscriber-node
73affb6364f9 mariadb:10.2.14 "docker-entrypoint.s…" About a minute ago Up About a minute (healthy) 0.0.0.0:3306-$3306/tcp mariadb
$ docker exec -it datarouter-node /bin/bash
# curl http://dmaap-dr-prov:8080/internal/prov
$ docker exec -it datarouter-prov /bin/bash
# curl -v -X POST -H "Content-Type : application/vnd.dmaap-dr.feed" -H "X-DMAAP-DR-ON-BEHALF-OF: rs873m" --data-ascii @/opt/app/datartr/addFeed3.txt --post301 --location-trusted -k https://dmaap-dr-prov:8443
#### Step 2: migrate to alpine
- edit datarouter-[prov, node, subscriber]/pom.xml
+ <baseImage$openjdk:8-jre-alpine</baseImage$
+ <entryPoint$["sh", "/startup.sh"]</entryPoint
- note shell for openjdk:8-jre-alpine is sh, therefore entrypoint = ["sh", "startup.sh"]
$ mvn clean install -DskipTests -Pdocker
$ docker images
REPOSITORY TAG IMAGE ID CREATED SIZE
nexus3.onap.org:10003/onap/dmaap/datarouter-subscriber latest 03ea672a6d73 28 seconds ago 102MB
nexus3.onap.org:10003/onap/dmaap/datarouter-node latest 76345553e1ec 43 seconds ago 118MB
nexus3.onap.org:10003/onap/dmaap/datarouter-prov latest db1d6afe5c3f About a minute ago 148MB
nexus3.onap.org:10001/openjdk 8-jre-alpine 1b46cc2ba839 3 weeks ago 85MB
- Test as above, Note change to "/bin/sh" per alpine
$ docker exec -it datarouter-node /bin/sh
# curl http://dmaap-dr-prov:8080/internal/prov
$ docker exec -it datarouter-prov /bin/sh
# curl -v -X POST -H "Content-Type : application/vnd.dmaap-dr.feed" -H "X-DMAAP-DR-ON-BEHALF-OF: rs873m" --data-ascii @/opt/app/datartr/addFeed3.txt --post301 --location-trusted -k https://dmaap-dr-prov:8443
#### Step 3: migrate pom to fabric8 docker-maven-plugin
- Note: spotify docker-maven-plugin does not run on Arm
[ERROR] Failed to execute goal com.spotify:docker-maven-plugin:1.0.0:build (default-cli) on project datarouter-prov: Exception caught: java.util.concurrent.ExecutionException: com.spotify.docker.client.shaded.javax.ws.rs.ProcessingException:
java.lang.UnsatisfiedLinkError: could not load FFI provider jnr.ffi.provider.jffi.Provider: ExceptionInInitializerError: Can't overwrite cause with java.lang.UnsatisfiedLinkError: cannot determine CPU -$ [Help 1]
- See: https://github.com/spotify/docker-maven-plugin/issues/233
- Note: spotify dockerfile-maven-plugin is alternate to fabric8, but other ONAP teams are using fabric8 so there
appears to be no compelling reason to stick w spotify
- Note: docker pull repo must be managed
- edit datarouter-[prov, node, subscriber]/pom.xml
Note that onap ~/.m2/settings.xml defines
<docker.pull.registry$nexus3.onap.org:10001</docker.pull.registry$
which forces an amd64 specific image.
to support other platforms, add to docker-maven-plugin configuration:
pullRegistry$${docker.pull.registry}</pullRegistry$
and invoke mvn with platfrom specific pull repo, e.g. on Arm
$ mvn install -DskipTests -Pdocker -Ddocker.pull.registry=docker.io
- create datarouter-[prov, node, subscriber]/src/main/resources/docker/Dockerfile
- Test, As per step 2
#### Step 4: build and run on Arm
- obtain access to server and install git, maven, docker (not described here)
$ git clone https://sandovalfrOAM@gerrit.onap.org/r/a/dmaap/datarouter
Cloning into 'datarouter'...
Password for 'https://sandovalfrOAM@gerrit.onap.org':
- go to gerrit profile/settings/http password - generate password
$ cd datarouter
$ mvn clean install -DskipTests -Pdocker -Ddocker.pull.registry=docker.io
- Test, as above step 2
- May encounter docker-compose incompatibilities, edit docker-compose
$ docker-compose up
ERROR: Version in "./docker-compose.yml" is unsupported. You might be seeing this error because you're using the wrong
Compose file version. Either specify a version of "2" (or "2.0") and place your service definitions under the `services`
key, or omit the `version` key and place your service definitions at the root of the file to use version 1.
For more on the Compose file format versions, see https://docs.docker.com/compose/compose-file/
ERROR: The Compose file './docker-compose.yml' is invalid because:
Unsupported config option for services.datarouter-prov: 'healthcheck'
Unsupported config option for services.mariadb_container: 'healthcheck'
services.datarouter-prov.depends_on contains an invalid type, it should be an array
## dmaap buscontroller
Issues encountered:
1) docker-maven-plugin. Original spotify plugin not supported on Arm
2) docker pull repo default nexus/onap does not support multi-platform images
a) fabric8 docker-maven-plugin allows setting from command line
b) or, build machine must set in ~/.m2/settings/xml
3) base image shell differences (alpine is not bash) impacts entrypoint and scripts
4) docker-compose.yml includes dependencies that are not multi-platform
- eg dmaap/buscontroller crunchydata/crunchy-postgres:centos7-10.4-2.0.0
5) docker USER root. Not recommended by ONAP
### This section describes how to build and test datarouter images on a host such as
a laptop or remote server.
- Install git, maven, docker
- See https://wiki.onap.org/display/DW/Setting+Up+Your+Development+Environment
- See https://docs.docker.com/install/
### Step 1: Build and run on local machine (Mac Air)
- in terminal
$ git clone ssh://sandovalfrOAM@gerrit.onap.org:29418/dmaap/buscontroller
$ cd buscontroller
$ mvn clean install -P docker docker:build
$ docker images
REPOSITORY TAG IMAGE ID CREATED SIZE
...
nexus3.onap.org:10003/onap/dmaap/dbc-client latest c4b7f26a93ba 59 seconds ago 5.94MB
nexus3.onap.org:10003/onap/dmaap/dmaap-bc latest ad5670a6c7b8 About a minute ago 335MB
alpine 3.8 491e0ff7a8d5 4 weeks ago 4.41MB
java openjdk-8-jre e44d62cf8862 2 years ago 311MB
$ cd dmaap-bc/src/main/resources
$ set PWD=$(pwd)
- edit docker-compose.yml
- use $PWD to mount volume
- change repo from nexus 00001 to 00003
- change image from busconroller to dmaap-bc
- edit Dockerfile
- mkdir /opt/app/dmaap/config
- to avoid failure in dmaapbc script
$ docker-compose up
- looks good(?)
- script does not 'enter init'
NOTE: running as root in container not recommended
### Step 2: migrate to alpine
- edit buscontroller/dmaapbc/pom.xml
- change java:openjdk-8-jre to openjdk:8-jre-alpine
- add "sh" to entrypoint
$ cd <$/buscontroller
$ mvn clean install -P docker docker:build
$ docker images
REPOSITORY TAG IMAGE ID CREATED SIZE
nexus3.onap.org:10003/onap/dmaap/dbc-client 1.0.3 ee726a04cf12 15 seconds ago 5.94MB
nexus3.onap.org:10003/onap/dmaap/dbc-client latest ee726a04cf12 15 seconds ago 5.94MB
nexus3.onap.org:10003/onap/dmaap/dmaap-bc 1.1.1 f70f2363d754 30 seconds ago 109MB
nexus3.onap.org:10003/onap/dmaap/dmaap-bc latest f70f2363d754 30 seconds ago 109MB
openjdk 8-jre-alpine 1b46cc2ba839 3 weeks ago 85MB
alpine 3.8 491e0ff7a8d5 4 weeks ago 4.41MB
$ cd dmaap-bc/src/main/resources
$ docker-compose up
- debug dmaapbc script (alpine sh v bash)
dmaap-bc_1 | ./bin/dmaapbc: source: line 61: can't open '/opt/app/config/conf/buscontroller.env': Not a directory
- move SOURCE to conditional above
dmaap-bc_1 | ./bin/dmaapbc: line 93: can't create config/PolicyEngineApi.properties: nonexistent directory
- mkdir conf in pom/dockerfile
$ docker-compose up
dmaap-bc_1 | ENTER start
...
dmaap-bc_1 | org.onap.dmaap.dbcapi.server.Main
dmaap-bc_1 | + set +x
dmaap-bc_1 | + PIDS=29
dmaap-bc_1 | + set +x
#### Step 3: migrate pom to fabric8 docker-maven-plugin
- Note: spotify docker-maven-plugin does not run on Arm
[ERROR] Failed to execute goal com.spotify:docker-maven-plugin:1.0.0:build (default-cli) on project datarouter-prov: Exception caught: java.util.concurrent.ExecutionException: com.spotify.docker.client.shaded.javax.ws.rs.ProcessingException:
java.lang.UnsatisfiedLinkError: could not load FFI provider jnr.ffi.provider.jffi.Provider: ExceptionInInitializerError: Can't overwrite cause with java.lang.UnsatisfiedLinkError: cannot determine CPU -$ [Help 1]
- See: https://github.com/spotify/docker-maven-plugin/issues/233
- Note: spotify dockerfile-maven-plugin is alternate to fabric8, but other ONAP teams are using fabric8 so there
appears to be no compelling reason to stick w spotify
- Note: docker pull repo must be managed
- edit datarouter-[prov, node, subscriber]/pom.xml
Note that onap ~/.m2/settings.xml defines
<docker.pull.registry$nexus3.onap.org:10001</docker.pull.registry$
which forces an amd64 specific image.
to support other platforms, add to docker-maven-plugin configuration:
pullRegistry$${docker.pull.registry}</pullRegistry$
and invoke mvn with platfrom specific pull repo, e.g. on Arm
$ mvn install -DskipTests -Pdocker -Ddocker.pull.registry=docker.io
- create datarouter-[prov, node, subscriber]/src/main/resources/docker/Dockerfile
- Test, As per step 2
#### Step 4: build and run on Arm
- terminal on Arm machine
$ git clone https://gerrit.onap.org/r/dmaap/buscontroller
- copy modified files from workstation to Arm server
$ scp misc/dmaapbc root@147.75.94.202:/root/onap/dmaap/buscontroller/dmaap-bc/misc/
$ scp pom.xml root@147.75.94.202:/root/onap/dmaap/buscontroller/dmaap-bc
$ scp src/main/resources/docker-compose.yml root@147.75.94.202:/root/onap/dmaap/buscontroller/dmaap-bc/src/main/resources
$ scp src/main/resources/Dockerfile root@147.75.94.202:/root/onap/dmaap/buscontroller/dmaap-bc/src/main/resources
$ mvn clean install -DskipTests -Pdocker -Ddocker.pull.registry=docker.io
$ cd src/main/resources
- edit docker-compose.yml
- crunchydata/crunchy-postgres:centos7-10.4-2.0.0 not multi-platform
- use postgres
$ docker-compose up
#### Step 5: build docker image from script, not maven
- create dbuild.sh
#### Step 6: test CSIT
Modify jenkins jobs to accommodate changes to docker-maven-plugin
- git clone ci-management
- git clone csit
$ cd csit
$ ./run-csit.sh plans/dmaap-buscontroller/with_mr "" ../ci-management
edit ci-management/jjb/dmaap/dmaap-buscontroller.yaml
- '{project-name}-{stream}-docker-java-shell-daily':
mvn-goals: 'clean package -P docker docker:build docker:push'
- git review
view jenkins configurations here: https://jenkins.onap.org/view/dmaap/job/dmaap-buscontroller-master-docker-java-shell-daily/configure
#### Notes: test spotify dockerfile-maven-plugin
- throws 'broken pipe' error which is amisdirection to registry naming issue
- See https://github.com/spotify/dockerfile-maven/issues/95
- maven install error
- See: https://github.com/spotify/dockerfile-maven/issues/252
- upgrade Arm server to maven 3.6.0
- see: https://maven.apache.org/users/index.html
- requires multiple execution blocks for multiple tags
- see https://github.com/spotify/dockerfile-maven/issues/10
DMAAP - messageservice
# CIA image refactoring for dmaap message router
Objective: Minimize size and support multi-platfrom execution of dmaap docker images
Jira: INT-910, INT-932
Gerrit Project: dmaap/messagerouter/messageservice
Note that dmaap-mr depends on kafka and zookeeper images specialized by the project
Sequence of steps:
1) build and test as-is on local machine
2) migrate base images to alpine on Intel
3) build and test alpine based images on Arm machine
4) test using ONAP OOM (K8s/Helm) on Intel
5) test using ONAP OOM (K8s/Helm) on Arm
6) patch repo with changes
## Step 1: build and test as-is on local machine
- Build local images
- Start containers
- Test
### build local images
- open terminal
$ git clone ssh://sandovalfrOAM@gerrit.onap.org:29418/dmaap/messagerouter/messageservice
$ cd messageservice
- per docs/installation/installation.rst
- edit bundleconfig-local/etc/appprops/MsgRtrApi.properties
config.zk.servers=zookeeper
kafka.metadata.broker.list=kafka:9092
$ cp bundleconfig-local/etc/appprops/MsgRtrApi.properties /var/tmp
- configure Docker engine: 'docker-$properties-$file sharing' to include /var/tmp
$ mvn clean install -DskipTests -Pdocker
$ docker images
REPOSITORY TAG IMAGE ID CREATED SIZE
...
onap/dmaap/dmaap-mr latest ec37060b0ef0 4 minutes ago 782MB
...
openjdk 8-jdk 5f8e49b0a018 3 weeks ago 624MB
- build kafka11aaf image (see separate guide)
- build zookeeper image (see separate guide)
### start containers
$ cd src/main/resources/docker-compose
$ cp docker-compose.yml docker-compose-local.yml
- edit docker-compose-local.yml
image: onap/dmaap/zookeeper:latest
image: onap/dmaap/kafka01101:latest
image: onap/dmaap/dmaap-mr:latest
$ docker-compose -f docker-compose-local.yml up -d
### Test
See https://docs.onap.org/en/latest/submodules/dmaap/messagerouter/messageservice.git/docs/offeredapis/offeredapis.html
- create sample file (any arbitrary content,e g "sample data")
e.g. $ echo "sample data" $ sample.txt
$ curl -H "Content-Type:text/plain" -X POST -d @sample.txt http://localhost:3904/events/TestTopic1
{"serverTimeMs": 645, ...
- give it some time....
$ curl -H "Content-Type:text/plain" -X GET http://localhost:3904/events/TestTopic1/CG1/C1?timeout=1000
["sample data"]
- view topics
$ curl -H "Content-Type:text/plain" -X GET http://10.233.38.121:3904/topics
## Step 2: migrate base images to alpine on Intel
### edit messageservice/src/main/resources/Dockerfile
- from openjdk:8-jdk to openjdk:8-alpine, etc (atp-get to apk, ...)
$ mvn clean install -DskipTests -Pdocker
$ docker images
REPOSITORY TAG IMAGE ID CREATED SIZE
...
onap/dmaap/dmaap-mr latest e8fbf3bebd9b 46 seconds ago 244MB
...
openjdk 8-jre-alpine 1b46cc2ba839 4 weeks ago 85MB
### Test as per step 1
## Step 3: build and test alpine based images on Arm machine
### edit messageservice/pom.xml
- migrate from spotify to fabric8 docker-maven-plugin
- add variables to control pull/push docker repos from cmd line
- Note: spotify docker-maven-plugin does not run on Arm
- See: https://github.com/spotify/docker-maven-plugin/issues/233
- spotify dockerfile-maven-plugin is alternate to fabric8, but other ONAP teams are using fabric8 so there
appears to be no compelling reason to stick w spotify
- Note: default docker registry nexus3.onap.org:10001 does not support manifest images, i.e. no multi-platform support
- invoke mvn with appropriate pull repo if building on non-Intel machine, e.g. on Arm
- push to local registry to test. To run local docker registry, see here: https://docs.docker.com/registry/deploying/
$ mvn install -DskipTests -Pdocker -Ddocker.pull.registry=docker.io
$ mvn install -DskipTests -Pdocker
### Local test of plugin, as per step 1
### Test on Arm
- access Arm server, i.e Platform bare-metal
- terminal on Arm machine
$ git clone https://gerrit.onap.org/r/dmaap/messagerouter/messageservice
- copy modified files from workstation to Arm server
$ scp pom.xml root@147.75.94.202:/root/onap/dmaap/messageservice
$ scp src/main/resources/docker/Dockerfile root@147.75.94.202:/root/onap/dmaap/messageservice/src/main/resources/docker
$ scp src/main/resources/docker-compose/docker-compose-local.yml root@147.75.94.202:/root/onap/dmaap/messageservice/src/main/resources/docker- compose
$ scp src/main/resources/docker-compose/MsgRtrApi.properties root@147.75.94.202:/root/onap/dmaap/messageservice/src/main/resources/docker-compose
$ mvn clean install -DskipTests -Pdocker -Ddocker.pull.registry=docker.io
- build kafka11aaf image (see separate guide)
- build zookeeper image (see separate guide)
### start containers
$ cd src/main/resources/docker-compose
$ docker-compose -f docker-compose-local.yml up -d
### Test as above
## Step 4: test using ONAP OOM (K8s/Helm) on Intel
### Intel - test as-is
- access Laas server https://labs.opnfv.org
Note: k8s and Helm operations can take some time (+several minutes)
patience is required to step through instructions and allow services to become operational
$ ssh-keygen
$ cat /home/opnfv/.ssh/id_rsa.pub $$ ~/.ssh/authorized_keys
$ git clone https://gerrit.opnfv.org/gerrit/auto
$ cd auto/
$ export ONAP_BRANCH=master
$ export ONAP_COMPONENT="dmaap robot"
$ ./ci/deploy-onap-kubespray.sh <ip of server$ |& tee ~/deploy.log
if opnfv@10.200.123.137's password: exit and ssh in again????
...
Nr. Datetime of check Err/Total PODs
1 Thu Apr 11 01:52:09 EDT 2019 0/0
...
Ctrl-C
$ sudo -i
# kubectl get pods --namespace onap
NAME READY STATUS RESTARTS AGE
dev-dbc-pg-0 1/1 Running 0 17m
dev-dbc-pg-1 1/1 Running 0 15m
dev-dbc-pgpool-7ff5894667-jlz9n 1/1 Running 0 17m
dev-dbc-pgpool-7ff5894667-v95gg 1/1 Running 0 17m
dev-dmaap-bc-547dc7594c-znhg6 1/1 Running 0 17m
dev-dmaap-bc-post-install-j9vgk 0/1 Completed 0 17m
dev-dmaap-dr-db-0 1/1 Running 0 17m
dev-dmaap-dr-db-1 1/1 Running 0 15m
dev-dmaap-dr-node-0 2/2 Running 0 17m
dev-dmaap-dr-prov-5f58fd778d-9vmdk 2/2 Running 0 17m
dev-message-router-0 1/1 Running 0 17m
dev-message-router-kafka-0 1/1 Running 3 17m
dev-message-router-kafka-1 1/1 Running 3 17m
dev-message-router-kafka-2 1/1 Running 3 17m
dev-message-router-mirrormaker-5dd964cdfc-pl5sr 0/1 CrashLoopBackOff 6 17m
dev-message-router-post-install-zdr4j 1/1 Running 0 14m
dev-message-router-zookeeper-0 1/1 Running 0 17m
dev-message-router-zookeeper-1 1/1 Running 0 17m
dev-message-router-zookeeper-2 1/1 Running 0 17m
dev-robot-8495d96c8-hx9hv 1/1 Running 0 17m
$ sudo -i
$ cd /home/opnfv/oom/kubernetes/robot/
$ ./ete-k8s.sh onap health | tee ~/onap-health.log
Starting Xvfb on display :92 with res 1280x1024x24
Executing robot tests at log level TRACE
==============================================================================
Testsuites
==============================================================================
Testsuites.Health-Check :: Testing ecomp components are available via calls.
==============================================================================
...
------------------------------------------------------------------------------
Basic DMAAP Data Router Health Check | PASS |
------------------------------------------------------------------------------
Basic DMAAP Message Router Health Check | PASS |
------------------------------------------------------------------------------
...
60 critical tests, 2 passed, 58 failed
60 tests total, 2 passed, 58 failed
==============================================================================
Testsuites | FAIL |
60 critical tests, 2 passed, 58 failed
60 tests total, 2 passed, 58 failed
==============================================================================
Output: /share/logs/0002_ete_health/output.xml
Log: /share/logs/0002_ete_health/log.html
Report: /share/logs/0002_ete_health/report.html
### clean-up
helm del --purge dev
kubectl delete --namespace onap pods,services,jobs,configmaps,deployments,secrets,pv,pvc --all
kubectl delete clusterrolebindings --namespace onap --all
# just to be sure do cleanup once more
kubectl delete --namespace onap pods,services,jobs,configmaps,deployments,secrets,pv,pvc --all
kubectl delete namespace onap
### Intel - test alpine images
- on local machine build alpine images, as in Step 2
- tag and push images to docker hub, so they can be pulled on lab machine
$ docker tag onap/dmaap/dmaap-mr:latest oamtech/dmaap-mr:latest
$ docker push oamtech/dmaap-mr:latest
- same for onap/dmaap/kafka and zookeeper
- on server
- NOTE: cleaning up k8s/Helm can be problematic. Sometimes its easiest to simply request a fresh server
split deploy-onap-kubespray.sh into two parts in order to edit helm charts
$ cp deploy-onap-kubespray.sh deploy-kubespray.sh
$ cp deploy-onap-kubespray.sh deploy-onap-helm.sh
-edit scripts to separate phases
* deploy-kubespray.sh: remove lines below 209. add exit 0
* deploy-onap-helm.sh: remove lines 164-179, 231-233, 240-243
$ ./deploy-kubespray.sh <ip of server$ |& tee ~/deploy.log
- if fresh server. If Helm charts have been installed and cleaned up as above, start below
git clone -b master http://gerrit.onap.org/r/oom
edit kubernetes/dmaap/message-router charts to point to alpine images on docker hub, e.g.
auto/oom/kubernetes/dmaap/components/message-router/charts/message-router-kafka$ vi values.yaml
- change image to oamtech/message-router:latest
$ ./deploy-onap-helm.sh <ip of server$ |& tee ~/deploy.log
### Tests
run robot tests
$ ./ete-k8s.sh onap health | tee ~/onap-health.log
then test via curl (from server) as in Step 1.
To find IP of message-router pod:
$ kubectl get service message-router -n onap
- message router service defined in oom/kubernetes/dmaap/components/message-router/vaues.yaml
DMAAP KAFKA11aaf
## dmaap kafka
INT-933, INT-912
### Step 1: Build and run on local machine (Mac Air)
- in terminal
$ git clone https://gerrit.onap.org/r/dmaap/kafka11aaf && (cd kafka11aaf && curl -Lo `git rev-parse --git-dir`/hooks/commit-msg https://gerrit.onap.org/r/tools/hooks/commit-msg; chmod +x `git rev-parse --git-dir`/hooks/commit-msg)
$ cd kafka11aaf
$ mvn clean install -P docker docker:build
$ docker images
...
onap/dmaap/kafka01101 latest 6c10dd9e9095 About a minute ago 380MB
anapsix/alpine-java latest c45785c254c5 2 weeks ago 126MB
- Test
- no directions to test, ie no docker-compose.yml, README, etc.
$ test via messageservice
### Step 2: migrate to alpine
- edit kafka11aaf/pom.xml
- change anapsix/alpine-java to openjdk:8-alpine
$ mvn clean install -P docker docker:build
$ docker images
...
onap/dmaap/kafka01101 latest 5bfd9b168e39 53 seconds ago 439MB
openjdk 8-alpine 792ff45a2a17 4 weeks ago 105MB
Note that images are LARGER! change to multi-platform normative base image openjdk:8-alpine required nevertheless
- test as in step 1
#### Step 3: migrate pom to fabric8 docker-maven-plugin
- Note: spotify docker-maven-plugin does not run on Arm
[ERROR] Failed to execute goal com.spotify:docker-maven-plugin:1.0.0:build ...jnr.ffi.provider.jffi.Provider: ExceptionInInitializerError: Can't overwrite cause with java.lang.UnsatisfiedLinkError: cannot determine CPU -$ [Help 1]
- See: https://github.com/spotify/docker-maven-plugin/issues/233
- Note: spotify dockerfile-maven-plugin is alternate to fabric8, but other ONAP teams are using fabric8 so there
appears to be no compelling reason to stick w spotify
- Note: docker pull/push repos must be managed
- edit kafka11aaf/pom.xml
- add pull repo defintion
Note that onap ~/.m2/settings.xml defines
<docker.pull.registry$nexus3.onap.org:10001</docker.pull.registry$
which forces an amd64 specific image.
to support other platforms, add to docker-maven-plugin configuration:
<pullRegistry$${docker.pull.registry}</pullRegistry$
and invoke mvn with appropriate pull repo if building on non-Intel machine, e.g. on Arm
$ mvn install -DskipTests -Pdocker -Ddocker.pull.registry=docker.io
- add push repo definition
add <pushRegistry$${docker.push.registry}</pushRegistry$
push to local registry to test
$ mvn install -DskipTests -Pdocker -Ddocker.push.registry=localhost:5000 docker:push
- create src/main/resources/docker/Dockerfile
$ mvn clean install -DskipTests -Ddocker.pull.registry=docker.io -Pdocker
- Test, As per step 2
#### Step 4: build and run on Arm
- terminal on Arm machine
$ git clone https://gerrit.onap.org/r/dmaap/kafka
- copy modified files from workstation to Arm server
$ scp pom.xml root@147.75.94.202:/root/onap/dmaap/buscontroller/dmaap-bc
$ scp src/main/resources/docker-compose.yml root@147.75.94.202:/root/onap/dmaap/buscontroller/dmaap-bc/src/main/resources
$ scp src/main/resources/Dockerfile root@147.75.94.202:/root/onap/dmaap/buscontroller/dmaap-bc/src/main/resources
$ mvn clean install -DskipTests -Pdocker -Ddocker.pull.registry=docker.io
$ cd src/main/resources
- edit docker-compose.yml
- crunchydata/crunchy-postgres:centos7-10.4-2.0.0 not multi-platform
- use postgres
$ docker-compose up
## dmaap zookeeper
INT-911, INT-934
### Step 1: Build and run on local machine (Mac Air)
- in terminal
$ git clone ssh://sandovalfrOAM@gerrit.onap.org:29418/dmaap/zookeeper
$ cd zookeeper
$ mvn clean install -DskipTests docker:build
$ docker images
REPOSITORY TAG IMAGE ID CREATED SIZE
onap/dmaap/zookeeper latest 2be3b0226e21 About a minute ago 272MB
ubuntu 16.04 9361ce633ff1 7 days ago 118MB
- Test
$ docker run -d onap/dmaap/zookeeper:latest
$ docker logs
Validating environment
...
2019-03-19 20:20:41,076 [myid:] - INFO [main:NIOServerCnxnFactory@89] - binding to port 0.0.0.0/0.0.0.0:2181
### Step 2: migrate to alpine
- edit src/main/docker/Dockerfile
- openjdk-8-jre-headless does not appear to be available for alpine
- change base to openjdk:8-alpine
- change JAVE-HOME
- change apt-get to apk
- add bash
- change entrypoint
- edit pom to use Arm compatible fabric8
$ mvn clean install -DskipTests docker:build
$ docker images
onap/dmaap/zookeeper 3.0.0-SNAPSHOT-20190320T180949Z d1285d9e0220 4 seconds ago 118MB
onap/dmaap/zookeeper 3.0.0-SNAPSHOT-latest d1285d9e0220 4 seconds ago 118MB
onap/dmaap/zookeeper latest d1285d9e0220 4 seconds ago 118MB
nexus3.onap.org:10001/openjdk 8-alpine e9ea51023687 12 days ago 105MB
openjdk 8-alpine e9ea51023687 12 days ago 105MB
Test
- as Step 1
### Step 2: test on Arm
$ git clone https://gerrit.onap.org/r/dmaap/zookeeper
- copy Dockerfile and pom from local to server, eg
$ scp pom.xml root@147.75.94.202:/root/onap/dmaap/zookeeper/
$ mvn clean install -DskipTests docker:build -Ddocker.pull.registry=docker.io
Test
- as Step 1
# AAF
INT-865
------------
Step 1: follow instruction here:
See: https://onap.readthedocs.io/en/latest/submodules/aaf/authz.git/docs/sections/installation/AAF_3.1_build.html
Step 2: multi-Platform
- AAF pulls from onap nexus cache, which does not support multi-arch
- edit auth/auth-cass/docker/Dockerfile.cass
- FROM nexus3.onap.org:10001/cassandra:3.11
+ FROM ${REGISTRY}cassandra:3.11
- edit auth/docker/Dockerfile.base
- FROM nexus3.onap.org:10001/openjdk:8-jre-alpine
+ FROM ${REGISTRY}openjdk:8-jre-alpine
- edit auth-cass/docker/dbuild.sh and docker/dbuild.sh to parameterize docker registry
- clean docker
- Note: must start Cassandra first
$ cd -authz/auth/auth-cass/docker
$ bash dbuild.sh
$ bash dinstall.sh
$ cd -authz/auth/docker
$ bash dbuild.sh
$ bash drun.sh
$ docker ps
see below **