DataLake Development Environment Setup

This page uses 2 VMs in China Mobile lab as an example to demonstrate how to setup a development environment. 

We have 2 VMs:

datalake01: 172.30.1.74
datalake02: 172.30.1.75

1. Setup host names in both VMs and your local PC
   In both VMs and your PC, sudo vi /etc/hosts, add these lines: (In windows, the file is C:\Windows\System32\drivers\etc\hosts.)
   

   172.30.1.74 message-router-zookeeper message-router-kafka 
   172.30.1.74 dl_druid dl_superset

   172.30.1.75 dl_es dl_couchbase
   
   
   

2. Install JDK 8 in both VMs
   sudo apt install openjdk-8-jdk-headless
   
   
3. Setup Kafka in datalake01
   (ref https://kafka.apache.org/quickstart)
   If you have DMaaP or an existing Kafka to use, you make skip this step.

   mkdir ~/kafka
   cd ~/kafka
   wget https://www-eu.apache.org/dist/kafka/2.0.0/kafka_2.11-2.0.0.tgz
   tar -xzf kafka_2.11-2.0.0.tgz
   cd ~/kafka/kafka_2.11-2.0.0
   
   sudo vi config/server.properties 
   add this line:

   listeners=PLAINTEXT://172.30.1.74:9092

   To start Zookeeper and Kafka:
   nohup bin/zookeeper-server-start.sh config/zookeeper.properties > zk.log &
   nohup bin/kafka-server-start.sh config/server.properties > kf.log &

   Here are commands to stop them:
   bin/zookeeper-server-stop.sh
   bin/kafka-server-stop.sh
   

   
   

4. Create test Kafka topics 
   In datalake01
   

   cd ~/kafka/kafka_2.11-2.0.0

   ./bin/kafka-topics.sh --create --zookeeper message-router-zookeeper:2181 --replication-factor 1 --partitions 1 --topic AAI-EVENT
   ./bin/kafka-topics.sh --create --zookeeper message-router-zookeeper:2181 --replication-factor 1 --partitions 1 --topic unauthenticated.DCAE_CL_OUTPUT
   ./bin/kafka-topics.sh --create --zookeeper message-router-zookeeper:2181 --replication-factor 1 --partitions 1 --topic unauthenticated.SEC_FAULT_OUTPUT
   ./bin/kafka-topics.sh --create --zookeeper message-router-zookeeper:2181 --replication-factor 1 --partitions 1 --topic msgrtr.apinode.metrics.dmaap
   
   

   In case you want to reset the topics, here are the scripts to delete them:
   

   ./bin/kafka-topics.sh --zookeeper message-router-zookeeper:2181 --delete --topic AAI-EVENT
   ./bin/kafka-topics.sh --zookeeper message-router-zookeeper:2181 --delete --topic unauthenticated.DCAE_CL_OUTPUT
   ./bin/kafka-topics.sh --zookeeper message-router-zookeeper:2181 --delete --topic unauthenticated.SEC_FAULT_OUTPUT
   ./bin/kafka-topics.sh --zookeeper message-router-zookeeper:2181 --delete --topic msgrtr.apinode.metrics.dmaap
   
   

5. Load test data to Kafka
   In datalake01
   
   See step 9 for how to get the source code and data files.
   
   cd ~/kafka/kafka_2.11-2.0.0
   

   ./bin/kafka-console-producer.sh --broker-list message-router-kafka:9092 --topic AAI-EVENT < ~/git/onap/datalake/storage/src/main/resources/druid/AAI-EVENT-full.json
   ./bin/kafka-console-producer.sh --broker-list message-router-kafka:9092 --topic unauthenticated.DCAE_CL_OUTPUT < ~/git/onap/datalake/storage/src/main/resources/druid/DCAE_CL_OUTPUT-full.json
   ./bin/kafka-console-producer.sh --broker-list message-router-kafka:9092 --topic unauthenticated.SEC_FAULT_OUTPUT < ~/git/onap/datalake/storage/src/main/resources/druid/SEC_FAULT_OUTPUT-full.json
   ./bin/kafka-console-producer.sh --broker-list message-router-kafka:9092 --topic msgrtr.apinode.metrics.dmaap < ~/git/onap/datalake/storage/src/main/resources/druid/msgrtr.apinode.metrics.dmaap-full.json

   
   To check if the data is successfully loaded, one can read the data: 

   bin/kafka-console-consumer.sh --bootstrap-server message-router-kafka:9092 --topic AAI-EVENT --from-beginning
   bin/kafka-console-consumer.sh --bootstrap-server message-router-kafka:9092 --topic unauthenticated.DCAE_CL_OUTPUT --from-beginning
   bin/kafka-console-consumer.sh --bootstrap-server message-router-kafka:9092 --topic unauthenticated.SEC_FAULT_OUTPUT --from-beginning
   bin/kafka-console-consumer.sh --bootstrap-server message-router-kafka:9092 --topic msgrtr.apinode.metrics.dmaap --from-beginning
   
   

6. Setup Couchbase in datalake02 
   • Start docker
     sudo docker run -d --name couchbase -p  8091-8094:8091-8094  -p 11210:11210 couchbase/server-sandbox:6.0.0
   • Create users and buckets
     

     Access http://dl_couchbase:8091/ , use login: "Administrator/password". 

     Create bucket "dl", with memory quota 100MB.
     Create user dl/dl1234 , with “Application access” to bucket "dl".

     Create bucket "dmaap", with memory quota 100MB.
     Create user dmaap/dmaap1234, with “Application Access” and "Views Admin" to bucket "dmaap".

   • Create a document for default topic in bucket dl 
     _DL_DEFAULT_
     {
     "enabled": true,
     "saveRaw": false,
     "dataFormat": "JSON",
     "ttl": 1000,
     "supportElasticsearch":true,
     "supportCouchbase":true
     }

   • Disable some topics
     
     __consumer_offsets
     {
     "enabled": false
     }
7. Setup ElasticSearch & Kibana in datalake02 
   (ref https://docs.swiftybeaver.com/article/33-install-elasticsearch-kibana-via-docker)
   

   sudo docker pull docker.elastic.co/elasticsearch/elasticsearch:6.6.1
   sudo docker run -d --rm -p 9200:9200 -p 9300:9300 -e "discovery.type=single-node" --name elastic docker.elastic.co/elasticsearch/elasticsearch:6.6.1
   sudo docker pull docker.elastic.co/kibana/kibana:6.6.1
   sudo docker run -d --rm --link elastic:dl_es -e "ELASTICSEARCH_URL=http://dl_es:9200" -p 5601:5601 --name kibana docker.elastic.co/kibana/kibana:6.6.1
   
   

8. Create test Indices in ElasticSearch 
   Indices should be auto created, which will be addressed in JIRA xxx. For now we need to manually create them.

   curl -X PUT "dl_es:9200/aai-event?pretty"

   curl -X PUT "dl_es:9200/unauthenticated.dcae_cl_output?pretty"

   curl -X PUT "dl_es:9200/unauthenticated.sec_fault_output?pretty"

   curl -X PUT "dl_es:9200/msgrtr.apinode.metrics.dmaap?pretty"

   
   To access Kibana: http://dl_es:5601/ .
   In case you want to reset the Indices, here are the scripts to delete them:
   

   curl -X DELETE "dl_es:9200/aai-event?pretty"

   curl -X DELETE "dl_es:9200/unauthenticated.dcae_cl_output?pretty"

   curl -X DELETE "dl_es:9200/unauthenticated.sec_fault_output?pretty"

   curl -X DELETE "dl_es:9200/msgrtr.apinode.metrics.dmaap?pretty"

   
   
9. Run DataLake feeder in Eclipse
   Check out DataLake source code from http://172.30.3.87/SOTN/datalake.git to ~/git/onap/datalake or C:\git\onap\datalake. If you don't have access, please apply one from Guobiao Mo.

   The feeder is a Spring boot application. The entry point is org.onap.datalake.storage.Application. After started, the app reads the topic list from Zookeeper, and starts pulling data from these Kafka topics, and insert the data to Couchbase and Elasticsearch. 
   
   

10. More to come on Docker, Druid, Superset etc. ...