These instructions have been verified on OSX High Sierra 10.13.4 using Docker for Mac Version 18.05.0-ce-rc1-mac63 (24246), which is latest edge version, stable channel will work as well.

Requirements:

  1. Recent Docker for Mac (tested with latest edge and stable release)
  2. Docker for Mac has to be sized to allow running all SDC containers, see instructions.
  3. Basic development utils for building the source code (Git, java, maven etc..)


Instructions:

Make sure your docker is up and running, and is fully updated. Once docker is up and running (you can pull images and run them, ie docker pull ubuntu && docker run -it ubuntu /bin/bash), set resources for docker to something like this(these work on my MacBook Pro 15 Retina):

Save settings and restart docker.

Building the source code

Change directory to SDC project and run:

Building source code
mvn clean install -Pdocker

This will build entire project and all docker containers. Note that this might take a while.

Preparing the configuration files

These instructions are taken from: Deploying SDC on a Linux VM for Development and slightly modified to facilitate for OSX specifics.

  1. Inside SDC project root directory, create new folder called data
  2. Add above created data directory to gitignore, as we don't want this to be included with any commits.
  3. Inside data directory, do following:
    1. create opt/config/ 
    2. place the following files in this data/opt/config/ folder:
    3. Configuration files
      nexus_username.txtenv_name.txtnexus_docker_repo.txtnexus_password.
    4. Inside data directory, create environments directory 
    5. Copy the file from here: https://git.onap.org/sdc/tree/sdc-os-chef/environments/Template.json into data/environments/AUTO.json
    6. Edit AUTO.json file, replace yyy with your en0 IP address (your laptop's IP address), and update name and description xxx with AUTO
      1. #brew install iproute2mac # uncomment if needed for following commands
         
        IP=`ip route get 8.8.8.8 | awk '/src/{ print $7 }'`
        sed -i '' -e"s/yyy/${IP}/g" ./AUTO.json
         
        sed -i '' -e"s/xxx/AUTO/g" ./AUTO.json

Example file looks like this after editing:

AUTO.json
{
  "name": "AUTO",
  "description": "OpenSource-AUTO",
  "cookbook_versions": {
    "Deploy-SDandC": "= 1.0.0"
  },
  "json_class": "Chef::Environment",
  "chef_type": "environment",

  "default_attributes": {
    "disableHttp": false,
    "CS_VIP": "159.107.152.117",
    "BE_VIP": "159.107.152.117",
    "ONBOARDING_BE_VIP": "159.107.152.117",
    "FE_VIP": "159.107.152.117",
    "ES_VIP": "159.107.152.117",
    "KB_VIP": "159.107.152.117",
    "DCAE_BE_VIP": "159.107.152.117",
    "DCAE_FE_VIP": "159.107.152.117",
    "interfaces": {
      "application": "eth0",
      "private": "eth1"
    },
    "ECompP": {
      "ecomp_rest_url": "http://portal.api.simpledemo.onap.org:8989/ONAPPORTAL/auxapi",
      "ueb_url_list": "10.0.11.1,10.0.11.1",
      "app_secret": "XftIATw9Jr3VzAcPqt3NnJOu",
      "app_key": "x9UfO7JsDn8BESVX",
      "inbox_name": "ECOMP-PORTAL-INBOX",
      "ecomp_redirect_url": "http://portal.api.simpledemo.openecomp.org:8989/ECOMPPORTAL/login.htm",
      "app_topic_name": "ECOMP-PORTAL-OUTBOX-SDC1",
      "decryption_key": "AGLDdG4D04BKm2IxIWEr8o=="
    },
    "UEB": {
      "PublicKey": "iPIxkpAMI8qTcQj8",
      "SecretKey": "Ehq3WyT4bkif4zwgEbvshGal",
      "fqdn": ["10.0.11.1", "10.0.11.1"]
    },
    "Nodes": {
      "CS": [
        "159.107.152.117"
      ],
      "BE":            "159.107.152.117",
      "ONBOARDING_BE": "159.107.152.117",
      "FE":            "159.107.152.117",
      "ES": [
        "159.107.152.117"
      ],
      "KB":            "159.107.152.117"
    },
    "Plugins": {
      "DCAE": {
        "dcae_discovery_url": "159.107.152.117",
        "dcae_source_url": "159.107.152.117"
      },
      "WORKFLOW": {
        "workflow_discovery_url": "159.107.152.117",
        "workflow_source_url": "159.107.152.117"
      }
    },
    "VnfRepo": {
      "vnfRepoPort": "8702",
      "vnfRepoHost": "192.168.50.5"
    }
  },
  "override_attributes": {
    "FE": {
      "http_port": "8181",
      "https_port": "9443"
    },
    "BE": {
      "http_port": "8080",
      "https_port": "8443"
    },
    "ONBOARDING_BE": {
      "http_port": "8081",
      "https_port": "8445"
    },
    "elasticsearch": {
      "cluster_name": "SDC-ES-",
      "ES_path_home": "/usr/share/elasticsearch",
      "ES_path_data": "/usr/share/elasticsearch/data",
      "num_of_replicas": "0",
      "num_of_shards": "1"
    },

    "cassandra": {
      "concurrent_reads": "32",
      "num_tokens": "256",
      "data_dir": "/var/lib/cassandra/data",
      "hinted_handoff_enabled": "true",
      "cassandra_user": "asdc_user",
      "cassandra_password": "Aa1234%^!",
      "concurrent_writes": "32",
      "cluster_name": "SDC-CS-",
      "multithreaded_compaction": "false",
      "cache_dir": "/var/lib/cassandra/saved_caches",
      "log_file": "/var/lib/cassandra/log/system.log",
      "phi_convict_threshold": "8",
      "commitlog_dir": "/var/lib/cassandra/commitlog",
      "socket_read_timeout": "20000",
      "socket_connect_timeout": "20000",
      "titan_connection_timeout": "10000"
    }
  }
}


Running containers:

Change directory to SDC project and export WORKSPACE env variable:

WORKSPACE ENV VAR
export WORKSPACE=$(pwd)

and you can start locally built containers now with:

Running SDC containers
sdc-os-chef/scripts/docker_run.sh -e AUTO -l

It takes a while to start them all and init cassandra, elastic and import normatives, but after sometime you should have:


All containers will be exposed (as per Docker for OSX) on localhost and your machine IP (en0 interface).

Troubleshooting:

  1. All scripts are located here: sdc-os-chef/scripts/, in case something isnt going right, modify first line /bin/bash to /bin/bash -xv for additional debugging.
  2. Make sure docker_run.sh command is executed only after WORKSPACE env var is defined, otherwise scripts will use / as path for data directory (look for files in /data and /opt)


  • No labels

11 Comments

  1. Dejan Kitic, this is quite helpful, thank you!


    One question, for testing distribution of network services, did you setup DMaaP too ? I couldn't find latest/proper documentation to setup DMaaP.

  2. No, I didn't, sorry.

  3. Hi Dejan Kitic,

    The instructions 99% worked for me. However I had to setup CS_HOST_IP manually in docker_run.sh inorder to get it through. Else it was trying to contact 127.0.0.1 port 9160 and was failing.

  4. Dejan Kitic,

    BTW I'm not able to bring up SDC simulator docker in mac. Docker4Mac doesn't mount /data properly. Any thoughts on this one?

    Hitting with "docker: Error response from daemon: Mounts denied: EOF." error. I tried exposing /data & /etc/localtime in preferences, but no luck.

  5. I was able to run the simulator natively and documented the steps here : Building & Running SDC Simulator natively in OSX 

  6. newbie fail ;-(

    Following instructions from Building the Source Code above.

    Is there something else that needs to be installed first?

    1. install Docker CE for MAC OSX
    2. git clone http://gerrit.onap.org/r/sdc
    3. cd sdc
    4. mvn clean install -Pdocker

    5. initial warning regarding the catalog file
    6. build failure at 164/171 on the catalog file

    5. initial WARNINGs

    [WARNING] Some problems were encountered while building the effective model for org.openecomp.sdc:catalog-be:war:1.3.0-SNAPSHOT
    [WARNING] 'dependencies.dependency.(groupId:artifactId:type:classifier)' must be unique: javax.servlet:javax.servlet-api:jar -> version ${servlet-api.version} vs 3.1.0 @ org.openecomp.sdc:catalog-be:[unknown-version], /Users/mstaffor/dev/onap/sdc/catalog-be/pom.xml, line 569, column 15
    [WARNING]
    [WARNING] It is highly recommended to fix these problems because they threaten the stability of your build.
    [WARNING]
    [WARNING] For this reason, future Maven versions might no longer support building such malformed projects.
    [WARNING]


    6. Build error on the catalog.be

    [INFO] catalog-be ......................................... FAILURE [17:58 min]
    [INFO] asdctool ........................................... SKIPPED
    [INFO] catalog-ui ......................................... SKIPPED
    [INFO] catalog-fe ......................................... SKIPPED
    [INFO] test-apis-ci ....................................... SKIPPED
    [INFO] ui-ci .............................................. SKIPPED
    [INFO] sdc-os-chef ........................................ SKIPPED
    [INFO] webseal-simulator 1.3.0-SNAPSHOT ................... SKIPPED
    [INFO] ------------------------------------------------------------------------
    [INFO] BUILD FAILURE
    [INFO] ------------------------------------------------------------------------
    [INFO] Total time: 50:56 min
    [INFO] Finished at: 2018-07-20T14:43:27+10:00
    [INFO] ------------------------------------------------------------------------
    [ERROR] Failed to execute goal org.apache.maven.plugins:maven-assembly-plugin:3.1.0:single (default-cli) on project catalog-be: Error reading assemblies: No assembly descriptors found. -> [Help 1]
    org.apache.maven.lifecycle.LifecycleExecutionException: Failed to execute goal org.apache.maven.plugins:maven-assembly-plugin:3.1.0:single (default-cli) on project catalog-be: Error reading assemblies: No assembly descriptors found.
  7. Make sure you have right settings.xml on your ~/.m2 repo

    1. thanks.  Actually the main problem that was encountered was the file too big message

      [ERROR] Failed to execute goal org.apache.maven.plugins:maven-assembly-plugin:3.1.0:single (assemble-file) on project asdctool: Execution assemble-file of goal org.apache.maven.plugins:maven-assembly-plugin:3.1.0:single failed: group id '2078647107' is too big ( > 2097151 ). Use STAR or POSIX extensions to overcome this limit -> [Help 1]
      
      org.apache.maven.lifecycle.LifecycleExecutionException: Failed to execute goal org.apache.maven.plugins:maven-assembly-plugin:3.1.0:single (assemble-file) on project asdctool: Execution assemble-file of goal org.apache.maven.plugins:maven-assembly-plugin:3.1.0:single failed: group id '2078647107' is too big ( > 2097151 ). Use STAR or POSIX extensions to overcome this limit

      For reference, I modified the pom.xml file in each of the following directories, and then finally the build was successful.

      catalog-be / asdctool / ui-ci / test-apis-ci with the following line in the <configuration></configuration> section

      <tarLongFileMode>posix</tarLongFileMode>
  8. To SDC experts, I am trying to run SDC locally on MacOS. I have followed all the instructions from this page but somehow I am getting this error after running this script 

    sdc-os-chef/scripts/docker_run.sh -e AUTO -l

     It looks like it fails to start  sdc-cassandra-init due to which other containers like (sdc-BE, sdc-FE etc are not able to start). This is the error I am getting 

    docker run sdc-cassandra-init...

    [2019-03-07T21:40:51+00:00] INFO: Started chef-zero at chefzero://localhost:1 with repository at /root/chef-solo

      One version per cookbook


    [2019-03-07T21:40:51+00:00] INFO: Started chef-zero at chefzero://localhost:1 with repository at /root/chef-solo

      One version per cookbook


    [2019-03-07T21:40:51+00:00] INFO: Forking chef instance to converge...

    [2019-03-07T21:40:51+00:00] INFO: Forking chef instance to converge...

    Starting Chef Client, version 13.8.5

    [2019-03-07T21:40:51+00:00] INFO: *** Chef 13.8.5 ***

    [2019-03-07T21:40:51+00:00] INFO: *** Chef 13.8.5 ***

    [2019-03-07T21:40:51+00:00] INFO: Platform: x86_64-linux-musl

    [2019-03-07T21:40:51+00:00] INFO: Platform: x86_64-linux-musl

    [2019-03-07T21:40:51+00:00] INFO: Chef-client pid: 14

    [2019-03-07T21:40:51+00:00] INFO: Chef-client pid: 14

    [2019-03-07T21:40:51+00:00] INFO: The plugin path /etc/chef/ohai/plugins does not exist. Skipping...

    [2019-03-07T21:40:51+00:00] INFO: The plugin path /etc/chef/ohai/plugins does not exist. Skipping...

    [2019-03-07T21:40:52+00:00] INFO: Setting the run_list to ["role[cassandra-actions]"] from CLI options

    [2019-03-07T21:40:52+00:00] INFO: Setting the run_list to ["role[cassandra-actions]"] from CLI options

    [2019-03-07T21:40:52+00:00] INFO: Run List is [role[cassandra-actions]]

    [2019-03-07T21:40:52+00:00] INFO: Run List is [role[cassandra-actions]]

    [2019-03-07T21:40:52+00:00] INFO: Run List expands to [cassandra-actions::01-createCsUser, cassandra-actions::03-schemaCreation, cassandra-actions::04-importComformance]

    [2019-03-07T21:40:52+00:00] INFO: Run List expands to [cassandra-actions::01-createCsUser, cassandra-actions::03-schemaCreation, cassandra-actions::04-importComformance]

    [2019-03-07T21:40:52+00:00] INFO: Starting Chef Run for 1eef96bea083

    [2019-03-07T21:40:52+00:00] INFO: Starting Chef Run for 1eef96bea083

    [2019-03-07T21:40:52+00:00] INFO: Running start handlers

    [2019-03-07T21:40:52+00:00] INFO: Running start handlers

    [2019-03-07T21:40:52+00:00] INFO: Start handlers complete.

    [2019-03-07T21:40:52+00:00] INFO: Start handlers complete.

    resolving cookbooks for run list: ["cassandra-actions::01-createCsUser", "cassandra-actions::03-schemaCreation", "cassandra-actions::04-importComformance"]

    [2019-03-07T21:40:54+00:00] INFO: Loading cookbooks [cassandra-actions@0.0.0]

    [2019-03-07T21:40:54+00:00] INFO: Loading cookbooks [cassandra-actions@0.0.0]

    Synchronizing Cookbooks:

      - cassandra-actions (0.0.0)

    Installing Cookbook Gems:

    Compiling Cookbooks...


    ================================================================================

    Recipe Compile Error in /root/chef-solo/cache/cookbooks/cassandra-actions/recipes/03-schemaCreation.rb

    ================================================================================


    NoMethodError

    -------------

    undefined method `+' for nil:NilClass


    Cookbook Trace:

    ---------------

      /root/chef-solo/cache/cookbooks/cassandra-actions/recipes/03-schemaCreation.rb:21:in `block in from_file'

      /root/chef-solo/cache/cookbooks/cassandra-actions/recipes/03-schemaCreation.rb:15:in `from_file'


    Relevant File Content:

    ----------------------

    /root/chef-solo/cache/cookbooks/cassandra-actions/recipes/03-schemaCreation.rb:


     14:  

     15:  template "titan.properties" do

     16:    sensitive true

     17:    path "/tmp/sdctool/config/titan.properties"

     18:    source "titan.properties.erb"

     19:    mode "0755"

     20:    variables({

     21>>      :DC_NAME      => node['cassandra']['datacenter_name']+node.chef_environment,

     22:       :cassandra_ip  => node['Nodes']['CS'].first,

     23:       :cassandra_pwd => node['cassandra'][:cassandra_password],

     24:       :cassandra_usr => node['cassandra'][:cassandra_user],

     25:       :titan_connection_timeout => node['cassandra']['titan_connection_timeout']

     26:    })

     27:  end

     28:  

     29:  

     30:  template "/tmp/sdctool/config/configuration.yaml" do


    System Info:

    ------------

    chef_version=13.8.5

    platform=alpine

    platform_version=3.8.1

    ruby=ruby 2.5.2p104 (2018-10-18 revision 65133) [x86_64-linux-musl]

    program_name=chef-solo worker: ppid=8;start=21:40:51;

    executable=/usr/bin/chef-solo



    Running handlers:

    [2019-03-07T21:40:55+00:00] ERROR: Running exception handlers

    [2019-03-07T21:40:55+00:00] ERROR: Running exception handlers

    Running handlers complete

    [2019-03-07T21:40:55+00:00] ERROR: Exception handlers complete

    [2019-03-07T21:40:55+00:00] ERROR: Exception handlers complete

    Chef Client failed. 0 resources updated in 04 seconds

    [2019-03-07T21:40:55+00:00] FATAL: Stacktrace dumped to /root/chef-solo/cache/chef-stacktrace.out

    [2019-03-07T21:40:55+00:00] FATAL: Stacktrace dumped to /root/chef-solo/cache/chef-stacktrace.out

    [2019-03-07T21:40:55+00:00] FATAL: Please provide the contents of the stacktrace.out file if you file a bug report

    [2019-03-07T21:40:55+00:00] FATAL: Please provide the contents of the stacktrace.out file if you file a bug report

    [2019-03-07T21:40:55+00:00] ERROR: undefined method `+' for nil:NilClass

    [2019-03-07T21:40:55+00:00] ERROR: undefined method `+' for nil:NilClass

    [2019-03-07T21:40:55+00:00] FATAL: Chef::Exceptions::ChildConvergeError: Chef run process exited unsuccessfully (exit code 1)

    [2019-03-07T21:40:55+00:00] FATAL: Chef::Exceptions::ChildConvergeError: Chef run process exited unsuccessfully (exit code 1)


    Anyone aware of this ?


    1. I got past this error by adding datacenter_name: "SDC-CS-" into the AUTO.json.  Found the change in https://git.onap.org/sdc/tree/sdc-os-chef/environments/Template.json

  9. Rather than using the IP address of my host laptop, I substitute "host.docker.internal" for "yyy".   Docker will automatically create a DNS (or hosts file - not sure) where this address resolves to the IP of the host machine.   This allows me to run both on the corporate network and at home without constantly updating the AUTO.json file.