Valet Devstack Plugin

This commit contains the scripts for the Valet Devstack
plugin. The plugin.sh file is the entry point into the
plugin. These scripts provide the "stack", "unstack" and
"clean" capabilities.

Task: #4591
Story: #2001035

Change-Id: Ib7ec301802bb05969ebc83d060823c0bb8146f53

devstack/README.md

@@ -0,0 +1,57 @@
+# All-in-1 Valet on Devstack Setup
+
+### Pre-setup notes and disclaimers:
+* Valet as a plugin to Devstack has been tested with Ubuntu 16.04, according to the instructions on the [Devstack website](https://docs.openstack.org/developer/devstack/). The stable Newton version of Devstack was used for testing.
+
+### Pre-requisites:
+* If you require proxies in order to upload/download files off of the internet, make sure those proxies are set.
+
+* Git clones should use https by default. To force this (which you will need when you stack), use the following command:
+		git config --global url."https://".insteadOf git://
+
+* Put the following line in the /etc/sudoers file:
+		ubuntu ALL=(ALL) NOPASSWD: ALL
+		   (replace "ubuntu" with whatever user name you're stacking with)
+
+* Make sure you have SSH'd into your devstack VM directly as the stack user (whether it's "stack", "ubuntu" or whatever), otherwise you will not be able to run the screen command to restart services.
+
+### Stacking Instructions
+
+* Follow the quick setup instructions at the [Devstack website](https://docs.openstack.org/developer/devstack/) to clone and setup Devstack. Make sure to use the correct Devstack as specified in the pre-setup notes above. You may choose to create a "stack" user or just use the "ubuntu" (or default user) on your VM.
+
+* When the stack.sh script is called, Valet will be cloned as part of the stacking process. Copy the following line into your local.conf file, underneath the "local|localrc" section:
+	enable_plugin valet https://git.openstack.org/openstack/valet
+
+* In your local.conf file, set your HOST_IP variable to the IP address assigned to the management interface (192.168.56.xxx). If you use 127.0.0.1, the services will come up but you won't be able to access the services from a browser running external to your VM.
+
+* In your local.conf file, enable heat: (add this line to the end of the file)
+	ENABLED_SERVICES+=,heat,h-api,h-api-cfn,h-api-cw,h-eng
+
+* Check the settings in the "settings" file. If you want to override them you can export the variables in the shell prior to calling "stack.sh". The MUSIC related variables below are mandatory to set; the others can remain at their defaults. The variables are listed below, along with their default values:
+
+	CASSANDRA_DEBIAN_MIRROR - this is the debian mirror entry that will be placed in the /etc/apt/sources.list file in order to download the desired version of Cassandra. Default: "deb http://www.apache.org/dist/cassandra/debian 310x main".
+
+	OPENSTACK_PLUGIN_PATH - the path to the "valet_plugins" directory within the valet repo. Default: /opt/stack/valet/plugins/valet_plugins.
+
+	MUSIC_URL - (mandatory) the mirror URL that the plugin will use to retrieve the MUSIC package. Default: None. Example value: "http://github.com/att/music/". (don't forget the trailing slash)
+
+	MUSIC_FILE - (mandatory) the name of the MUSIC package. Default: None. Example value: music_3.5.0-0_amd64.deb
+
+* Start stackin!
+	cd <devstack-repo>
+	./stack.sh
+
+* Currently, the Valet Devstack plugin does not 100% automate the setup of Valet. Until 100% automation is achieved, you will manually need to re-configure Heat and Nova according to the stack-valet-plugins-configure-manual.sh script.
+
+### Unstacking Script
+
+* Unstacking Devstack using the unstack.sh script will trigger the Valet Devstack plugin to stop all Valet/Music related services and processes.
+
+### Clean Script
+
+* When Devstack's clean.sh script is called, the Valet Devstack plugin will be triggered to uninstall/remove all Valet/Music related components. Note, however, that this "clean" operation will leave the valet-related configuration files in place, as it is very annoying to unstack+clean and find that your customized .conf files have been removed.
+
+### Known Issues and Workarounds
+* Just after the plugin has deployed Valet (as part of stack.sh), the valet-engine process sometimes does not come up properly (tracebacks are seen in the startup sequence - /var/log/valet/engine.log). Workaround is to restart the valet-engine process (sudo -u valet python /usr/bin/valet-engine -c stop; sudo -u valet python /usr/bin/valet-engine -c start).
+
+* Just before Valet services are started, the pecan_populate.sh script is called to populate the database and setup the WSGI app in Apache2. A "usage" warning is printed at this time: "invalid choice: 'populate'". This issue does not appear to prevent Valet from working properly within the Devstack environment.

devstack/clean-valet-cleanup.sh

@@ -0,0 +1,15 @@
+#!/usr/bin/env bash
+
+echo "Removing Valet service/endpoint/user from OpenStack"
+cd ${HOME}/devstack
+source openrc admin admin
+openstack endpoint delete valet
+openstack service delete valet
+openstack user delete valet
+
+echo "Removing Valet executables and data files..."
+sudo rm -rf /opt/app/aic-valet-tools
+sudo rm /usr/bin/valet-engine
+sudo rm -rf /var/run/valet
+sudo rm -rf /var/lib/cassandra
+sudo rm -rf /var/lib/tomcat7/webapps/MUSIC*

devstack/clean-valet-uninstall.sh

@@ -0,0 +1,11 @@
+#!/usr/bin/env bash
+
+echo "Uninstalling services required by valet..."
+apt -y remove cassandra
+apt -y remove zookeeper
+apt -y remove zookeeperd
+apt -y remove tomcat7
+dpkg -r music_3.5.0-0_amd64
+pip uninstall --yes notario
+pip uninstall --yes pecan-notario
+

devstack/plugin.sh

@@ -0,0 +1,73 @@
+#!/usr/bin/env bash
+# plugin.sh - DevStack plugin.sh dispatch script template
+
+# check for service enabled
+if is_service_enabled valet; then
+    CWD=`pwd`
+
+    if [ -z ${HOST_IP} ]; then
+        echo "HOST_IP is not set.  It must be set to install/configure Valet!"
+        return 1
+    fi
+    if [ -z ${HOME} ]; then
+        echo "HOME is not set.  It must be set to install/configure Valet!"
+        return 1
+    fi
+
+    echo "HOME = ${HOME}"
+    echo "HOST_IP = ${HOST_IP}"
+
+    cd ${HOME}
+
+    if [[ "$1" == "stack" && "$2" == "pre-install" ]]; then
+
+        # set hostname to standard region/rack/node format
+        # so that the topology will be understood properly
+        # by Valet. Put that hostname, and valet into /etc/hosts.
+        sudo hostname Region1r001c001
+        if [[ ! `grep "Region1r001c001" /etc/hosts` ]]; then
+            export HOST_ENTRY="${HOST_IP}   Region1r001c001 valet"
+            export CMD="echo ${HOST_ENTRY} >> /etc/hosts"
+            sudo -E bash -c "${CMD}"
+        fi
+    fi
+
+    if [[ "$1" == "stack" && "$2" == "extra" ]]; then
+        if [ ! -d ${HOME}/.valet_venv ]; then
+            echo "Creating new virtual environment for Valet..."
+            virtualenv .valet_venv
+        fi
+        source .valet_venv/bin/activate
+
+        export HOST_IP=${HOST_IP}
+        /opt/stack/valet/devstack/stack-valet-music-install.sh
+        sudo -E bash -c /opt/stack/valet/devstack/stack-valet-music-configure.sh
+        /opt/stack/valet/devstack/stack-valet-python-install.sh
+        /opt/stack/valet/devstack/stack-valet-valet-install.sh
+        sudo -E bash -c /opt/stack/valet/devstack/stack-valet-valet-configure.sh
+        /opt/stack/valet/devstack/stack-valet-services-start.sh
+        /opt/stack/valet/devstack/stack-valet-openstack-configure.sh
+
+        # Then the user needs to run the manual steps laid out in
+        # the following file:
+        #   stack-valet-plugins-configure-manual.sh
+        deactivate
+    fi
+
+    if [[ "$1" == "unstack" ]]; then
+        source .valet_venv/bin/activate
+
+        /opt/stack/valet/devstack/unstack-valet-services-stop.sh
+        deactivate
+    fi
+
+    if [[ "$1" == "clean" ]]; then
+        source .valet_venv/bin/activate
+
+        sudo bash -c /opt/stack/valet/devstack/clean-valet-uninstall.sh
+        /opt/stack/valet/devstack/clean-valet-cleanup.sh
+        deactivate
+        rm -rf ${HOME}/.valet_venv
+    fi
+    cd ${CWD}
+fi

devstack/settings

@@ -0,0 +1,6 @@
+enable_service valet
+export OPENSTACK_PLUGIN_PATH=${OPENSTACK_PLUGIN_PATH:-/opt/stack/valet/plugins/valet_plugins}
+export MUSIC_URL=${MUSIC_URL}
+export MUSIC_FILE=${MUSIC_FILE}
+export CASSANDRA_DEBIAN_MIRROR=${CASSANDRA_DEBIAN_MIRROR:-"deb http://www.apache.org/dist/cassandra/debian 310x main"}
+

devstack/stack-valet-music-configure.sh

@@ -0,0 +1,11 @@
+#!/usr/bin/env bash
+
+# Configure MUSIC components
+#===========================
+
+sed -i 's/Test Cluster/Valet Cluster/g' /etc/cassandra/cassandra.yaml
+sed -i 's/snitch: SimpleSnitch/snitch: GossipingPropertyFileSnitch/g' /etc/cassandra/cassandra.yaml
+echo "quorumListenOnAllIPs=true" >> /etc/zookeeper/conf/zoo.cfg
+echo "server.1=${HOST_IP}:2888:3888" >> /etc/zookeeper/conf/zoo.cfg
+sed -i "s/replace this text with the cluster-unique zookeeper's instance id (1-255)/1/g" /var/lib/zookeeper/myid
+

devstack/stack-valet-music-install.sh

@@ -0,0 +1,31 @@
+#!/usr/bin/env bash
+
+# Install all MUSIC components and MUSIC itself
+#==============================================
+
+# Install Cassandra
+if [[ ! `sudo grep cassandra /etc/apt/sources.list` ]]; then
+    export SRC_CMD="echo ${CASSANDRA_DEBIAN_MIRROR} >> /etc/apt/sources.list"
+    sudo -E bash -c "${SRC_CMD}"
+    sudo curl https://www.apache.org/dist/cassandra/KEYS | sudo apt-key add -
+fi
+sudo apt-get -y update
+sudo apt-get -y install cassandra
+
+sudo service cassandra stop
+sudo rm -rf /var/lib/cassandra/data/*
+sudo rm -rf /var/lib/cassandra/commitlog/*
+sudo chown -R cassandra:cassandra /var/lib/cassandra
+
+# Install latest Zookeeper
+sudo apt-get -y install zookeeper
+sudo apt-get -y install zookeeperd
+
+# Install latest Tomcat
+sudo apt-get -y install tomcat7
+
+# Install Music
+echo "Installing music from ${MUSIC_URL}..."
+sudo wget ${MUSIC_URL}/${MUSIC_FILE}
+sudo dpkg -i ${MUSIC_FILE}
+

devstack/stack-valet-openstack-configure.sh

@@ -0,0 +1,24 @@
+#!/usr/bin/env bash
+
+# CONFIGURE OPENSTACK with VALET and RESTART NOVA/HEAT
+#====================================================
+
+cd ${HOME}/devstack
+source openrc admin admin
+if [[ ! `openstack service list | grep valet` ]]; then
+    echo "Creating Valet (placement) service..."
+    openstack service create --name valet placement
+fi
+if [[ ! `openstack endpoint list | grep valet` ]]; then
+    echo "Creating Valet service endpoint..."
+    openstack endpoint create --publicurl http://${HOST_IP}:8090/v1 --adminurl http://${HOST_IP}:8090/v1 --internalurl http://${HOST_IP}:8090/v1 --region RegionOne valet
+fi
+if [[ ! `openstack user list | grep valet` ]]; then
+    echo "Creating Valet user and adding appropriate roles..."
+    openstack user create --project service --enable --password valet valet
+    openstack role add --project service --user valet admin
+fi
+
+echo "Copying ${OPENSTACK_PLUGIN_PATH} into /usr/local/lib/python2.7/dist-packages..."
+sudo cp -R ${OPENSTACK_PLUGIN_PATH} /usr/local/lib/python2.7/dist-packages
+

devstack/stack-valet-plugins-configure-manual.sh

@@ -0,0 +1,40 @@
+#!/usr/bin/env bash
+
+# Modify Heat conf file to include Valet configuration
+sudo vi /etc/heat/heat.conf
+  [DEFAULT]
+  ...
+  plugin_dirs=/usr/local/lib/python2.7/dist-packages/valet_plugins/heat
+
+  [valet]
+  read_timeout=5
+  url=http://${HOST_IP}:8090/v1
+  connect_timeout=1
+
+# Modify Nova conf file to include Valet configuration
+sudo vi /etc/nova/nova.conf
+  Comment out the existing scheduler_default_filters line (or delete it) and then add:
+
+  [DEFAULT]
+  ...
+  scheduler_available_filters=nova.scheduler.filters.all_filters
+  scheduler_available_filters=valet_plugins.plugins.nova.valet_filter.ValetFilter
+  scheduler_default_filters = RetryFilter,AvailabilityZoneFilter,RamFilter,DiskFilter,ComputeFilter,ComputeCapabilitiesFilter,ImagePropertiesFilter,ServerGroupAntiAffinityFilter,ServerGroupAffinityFilter,SameHostFilter,DifferentHostFilter,ValetFilter
+  ...
+
+  [valet]
+  url=http://${HOST_IP}:8090/v1
+  admin_username=valet
+  connect_timeout=5
+  admin_tenant_name=service
+  admin_auth_url=http://${HOST_IP}:35357/v2.0
+  read_timeout=600
+  failure_mode=reject
+  admin_password=valet
+
+# Restart nova scheduler (n-sch) and heat processes (h-eng, h-api, h-api-cfn, h-api-cw) using screen
+screen -dr
+   (ctrl-A N to move forward through the processes)
+   (ctrl-A P to move backward through the processes)
+   (ctrl-C to kill a process; then press up arrow key to bring up the service start command and ENTER)
+   (ctrl-A D to detach/exit from screen)

devstack/stack-valet-python-install.sh

@@ -0,0 +1,19 @@
+#!/usr/bin/env bash
+
+#Python Package installation in venv
+#===================================
+pip install daemon
+pip install pyparsing
+pip install requests
+pip install python-daemon
+pip install simplejson
+pip install pika
+pip install python-keystoneclient
+pip install python-novaclient==6.0.0
+pip install python-heatclient
+pip install oslo.db
+pip install oslo.messaging
+pip install pecan==1.1.1
+pip install notario==0.0.11
+pip install pecan-notario==0.0.3
+

devstack/stack-valet-services-start.sh

@@ -0,0 +1,35 @@
+#!/usr/bin/env bash
+
+# START SERVICES and VALET
+#=========================
+
+# Start MUSIC Services required for Valet
+echo "Starting Cassandra service..."
+sudo service cassandra restart
+echo "Starting Zookeeper service..."
+sudo service zookeeper restart
+echo "Starting Tomcat service..."
+sudo service tomcat7 restart
+
+echo "Started MUSIC services...wait 30 seconds before proceeding..."
+sleep 30
+
+# Start Valet API
+echo "Starting Valet API..."
+sudo apachectl graceful
+
+# Populate the database and setup Apache wsgi app
+# NOTE: this must come after apache is restarted because this
+#       next step helps set up the WSGI App running in apache.
+sleep 10
+echo "Populating Valet Database and Configuring WSGI app..."
+cd /opt/app/aic-valet-tools
+./pecan_populate.sh
+
+# Start Valet Engine
+# NOTE: this must come after apache is restarted because keystone
+#       also runs within apache!
+sleep 30
+echo "Starting Valet Engine..."
+sudo -u valet python /usr/bin/valet-engine -c start
+

devstack/stack-valet-valet-configure.sh

@@ -0,0 +1,51 @@
+#!/usr/bin/env bash
+
+# Configure Valet Core Components
+#================================
+
+# Generate valet.conf
+echo -e "[DEFAULT]" > /etc/valet/valet.conf
+echo -e "default_log_levels=\"api=DEBUG,valet=DEBUG,ostro_daemon=DEBUG,ostro_listener=INFO,music=INFO,requests=ERROR,pika=ERROR,pecan=ERROR,urllib3=ERROR\"" >> /etc/valet/valet.conf
+echo -e "logging_default_format_string='%(asctime)s.%(msecs)03d [%(levelname)-5.5s] [%(name)s] - %(message)s'" >> /etc/valet/valet.conf
+echo -e "use_stderr=False" >> /etc/valet/valet.conf
+echo -e "log_dir=/var/log/valet\n" >> /etc/valet/valet.conf
+echo -e "[messaging]" >> /etc/valet/valet.conf
+echo -e "username=stackrabbit" >> /etc/valet/valet.conf
+echo -e "password=stackqueue" >> /etc/valet/valet.conf
+echo -e "host=${HOST_IP}" >> /etc/valet/valet.conf
+echo -e "port=5672\n" >> /etc/valet/valet.conf
+echo -e "[identity]" >> /etc/valet/valet.conf
+echo -e "project_name=service" >> /etc/valet/valet.conf
+echo -e "username=valet" >> /etc/valet/valet.conf
+echo -e "password=valet" >> /etc/valet/valet.conf
+echo -e "auth_url=http://${HOST_IP}:35357/v2.0\n" >> /etc/valet/valet.conf
+echo -e "[music]" >> /etc/valet/valet.conf
+echo -e "hosts=${HOST_IP}" >> /etc/valet/valet.conf
+echo -e "port=8080" >> /etc/valet/valet.conf
+echo -e "keyspace=valet_keyspace" >> /etc/valet/valet.conf
+echo -e "music_server_retries=3\n" >> /etc/valet/valet.conf
+echo -e "[engine]" >> /etc/valet/valet.conf
+echo -e "datacenter_name=Region1" >> /etc/valet/valet.conf
+echo -e "priority=1" >> /etc/valet/valet.conf
+echo -e "compute_trigger_frequency=1800" >> /etc/valet/valet.conf
+echo -e "topology_trigger_frequency=3600" >> /etc/valet/valet.conf
+echo -e "update_batch_wait=600" >> /etc/valet/valet.conf
+echo -e "default_cpu_allocation_ratio=8" >> /etc/valet/valet.conf
+echo -e "default_ram_allocation_ratio=1" >> /etc/valet/valet.conf
+echo -e "default_disk_allocation_ratio=1" >> /etc/valet/valet.conf
+echo -e "static_cpu_standby_ratio=0" >> /etc/valet/valet.conf
+echo -e "static_mem_standby_ratio=0" >> /etc/valet/valet.conf
+echo -e "static_local_disk_standby_ratio=0" >> /etc/valet/valet.conf
+echo -e "num_of_region_chars=6" >> /etc/valet/valet.conf
+
+# Modify valet_apache.conf and envvars
+sed -i 's/valet_user/valet/g' /etc/apache2/sites-available/valet_apache.conf
+sed -i 's/www-data/valet/g' /etc/apache2/envvars
+
+# WORKAROUND FOR CQLSH
+if [[ `grep -i dist-packages /usr/bin/cqlsh.py` ]]; then
+    echo "cqlsh workaround already installed"
+else
+    sed -i "s/from uuid import UUID/from uuid import UUID\n\nsys.path.append('\/usr\/lib\/python2.7\/dist-packages')/g" /usr/bin/cqlsh.py
+fi
+

devstack/stack-valet-valet-install.sh

@@ -0,0 +1,57 @@
+#!/usr/bin/env bash
+
+# Install Valet Core (still in venv)
+#==================================
+cd /opt/stack/valet
+if [ ! `id -u valet` ]; then
+    sudo adduser --system --group valet
+fi
+sudo python setup.py install
+
+if [ ! -d /etc/valet ]; then
+    sudo mkdir /etc/valet
+    sudo chmod 0777 /etc/valet
+    sudo chown -R valet:valet /etc/valet
+fi
+if [ ! -d /var/log/valet ]; then
+    sudo mkdir /var/log/valet
+    sudo chown valet:valet /var/log/valet
+    sudo chmod 0777 /var/log/valet
+fi
+if [ ! -d /var/run/valet ]; then
+    sudo mkdir /var/run/valet
+    sudo chown valet:valet /var/run/valet
+    sudo chmod 0750 /var/run/valet
+fi
+sudo cp /opt/stack/valet/bin/valet-engine /usr/bin/valet-engine
+sudo chown valet:valet /usr/bin/valet-engine
+
+# Following are needed to be able to run cleandb.sh
+# and pecan_populate.sh
+sudo mkdir /opt/app
+sudo mkdir /opt/app/aic-valet-tools
+sudo cp /opt/stack/valet/tools/utils/* /opt/app/aic-valet-tools
+sudo chmod -R 0777 /opt/app/aic-valet-tools
+sudo chown -R valet:valet /opt/app/aic-valet-tools
+sudo pip install notario==0.0.11
+sudo pip install pecan-notario==0.0.3
+
+# Install Apache configuration files/directories
+if [ ! -f /etc/apache2/sites-available/valet_apache.conf ]; then
+    sudo cp /usr/local/etc/valet/valet/api/valet_apache.conf /etc/apache2/sites-available/valet_apache.conf
+    sudo ln -s /etc/apache2/sites-available/valet_apache.conf /etc/apache2/sites-enabled/valet_apache.conf
+fi
+sudo mkdir /var/www/valet
+sudo cp /usr/local/etc/valet/valet/api/config.py /var/www/valet/config.py
+sudo cp /usr/local/etc/valet/valet/api/app.wsgi /var/www/valet/app.wsgi
+sudo chown -R valet:valet /var/www/valet
+sudo chmod 0777 /var/www/valet
+if [ ! -d /var/log/apache2/valet ]; then
+    sudo mkdir /var/log/apache2/valet
+    sudo chmod 0777 /var/log/apache2/valet
+fi
+
+# Remove this dir as it is no longer needed, and would be
+# confusing to leave it there.
+sudo rm -R /usr/local/etc/valet
+

devstack/unstack-valet-services-stop.sh

@@ -0,0 +1,13 @@
+#!/usr/bin/env bash
+
+echo "Stopping valet-engine process..."
+sudo -u valet python /usr/bin/valet-engine -c stop
+echo "Stopping Apache2 service..."
+sudo service apache2 stop
+echo "Stopping Tomcat service..."
+sudo service tomcat7 stop
+echo "Stopping Zookeeper service..."
+sudo service zookeeper stop
+echo "Stopping Cassandra service..."
+sudo service cassandra stop
+echo "Done shutting down Valet services."