Browse Source

Include VM image in contaimer

add script to build VM image and container, to extract VM image from container
update doc
remove version check with agent
update vm image upload
auto upload VM image from pkg root dir
show simplified version string

Change-Id: I9dc44c52656d81f271c91d6826dc0e128fed980d
changes/93/486793/1 7.0.0
ahothan 5 years ago
parent
commit
8f892eaeeb
  1. 6
      .dockerignore
  2. 1
      .gitignore
  3. 26
      Dockerfile
  4. 24
      doc/source/adv_features.rst
  5. 94
      doc/source/development.rst
  6. 15
      doc/source/faq.rst
  7. 56
      doc/source/quickstart_docker.rst
  8. 16
      doc/source/quickstart_git.rst
  9. 6
      doc/source/quickstart_pip.rst
  10. 17
      doc/source/readme.rst
  11. 61
      doc/source/usage.rst
  12. 96
      kb_build.sh
  13. 5
      kb_dib/README.rst
  14. 37
      kb_dib/build-image.sh
  15. 21
      kb_dib/elements/kloudbuster/static/kb_test/kb_vm_agent.py
  16. 86
      kb_extract_img_from_docker.sh
  17. 16
      kloudbuster/cfg.scale.yaml
  18. 22
      kloudbuster/kb_config.py
  19. 13
      kloudbuster/kb_runner_base.py
  20. 4
      kloudbuster/kb_runner_http.py
  21. 4
      kloudbuster/kb_runner_multicast.py
  22. 4
      kloudbuster/kb_runner_storage.py
  23. 28
      kloudbuster/kloudbuster.py
  24. 23
      kloudbuster/tests/base.py
  25. 8
      kloudbuster/tests/test_kloudbuster.py
  26. 1
      setup.py
  27. 2
      tox.ini

6
.dockerignore

@ -1,3 +1,7 @@
.git
.gitignore
.gitreview
.tox
.mailmap
.coveragerc
.test*
*.egg-info/

1
.gitignore vendored

@ -6,6 +6,7 @@
# Packages
*.egg
*.egg-info
.eggs/
dist
build
eggs

26
Dockerfile

@ -1,15 +1,29 @@
# docker file for creating a container that has kloudbuster installed and ready to use
FROM ubuntu:14.04
# this will build from uptreams master latest
FROM ubuntu:16.04
MAINTAINER kloudbuster-core <kloudbuster-core@lists.launchpad.net>
# Simpler would be to clone direct from upstream (latest)
# but the content might differ from the curent repo
# So we'd rather copy the current kloudbuster directory
# along with the pre-built qcow2 image
COPY ./ /kloudbuster/
# Install KloudBuster script and dependencies
# Note the dot_git directory must be renamed to .git
# in order for pip install -e . to work properly
RUN apt-get update && apt-get install -y \
git \
libyaml-dev \
python \
python-dev \
python-pip \
python-virtualenv \
&& rm -rf /var/lib/apt/lists/*
RUN pip install pytz
RUN pip install kloudbuster
&& pip install -U -q pip \
&& pip install -U -q setuptools \
&& cd /kloudbuster \
&& pip install -q -e . \
&& rm -rf .git \
&& rm -rf /var/lib/apt/lists/* \
&& apt-get autoremove -y && apt-get clean && rm -rf /var/lib/apt/lists/*

24
doc/source/adv_features.rst

@ -89,39 +89,39 @@ are given to help get started quickly.
make sure KloudBuster is talking to the clouds well.
Example 1: Single-cloud Mode
""""""""""""""""""""""""""""
Example 1: HTTP Scale, Single-cloud Mode
""""""""""""""""""""""""""""""""""""""""
Kloudbuster will create both server VMs and client VMs in the same cloud if
only one RC file is provided::
$ kloudbuster --tested-rc ~/admin_openrc.sh --tested-passwd admin
$ kloudbuster --rc ~/admin_openrc.sh --passwd admin
Example 2: Dual-cloud Mode, Save results
""""""""""""""""""""""""""""""""""""""""
Example 2: HTTP Scale, Dual-cloud Mode, Save results
""""""""""""""""""""""""""""""""""""""""""""""""""""
Assume the cloud for server VMs is ~/admin_openrc1.sh, and the cloud for
client VMs is ~/admin_openrc2.sh. The password for both clouds is "admin".
Also save the results to a JSON file once the run is finished::
$ kloudbuster --tested-rc ~/admin_openrc1.sh --tested-passwd admin --testing-rc ~/admin_openrc2.sh --testing-passwd admin --json result.json
$ kloudbuster --rc ~/admin_openrc1.sh --passwd admin --testing-rc ~/admin_openrc2.sh --testing-passwd admin --json result.json
Example 3: Single-cloud Mode, Customized VM placements
""""""""""""""""""""""""""""""""""""""""""""""""""""""
Example 3: HTTP Scale, Single-cloud Mode, Customized VM placements
""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""
.. code::
$ kloudbuster --tested-rc ~/admin_openrc.sh --tested-passwd admin -t cfg.topo.yaml
$ kloudbuster --rc ~/admin_openrc.sh --passwd admin -t cfg.topo.yaml
Example 4: Single-cloud Mode, Running storage test, Save results to JSON
""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""
Example 4: Storage benchmark, Save results to JSON
""""""""""""""""""""""""""""""""""""""""""""""""""
.. code::
$ kloudbuster --tested-rc ~/aio-openrc.sh --tested-passwd lab --storage --json aio.json
$ kloudbuster --rc ~/aio-openrc.sh --passwd lab --storage --json aio.json

94
doc/source/development.rst

@ -4,111 +4,67 @@ Development
.. _build_vm_image:
Build the KloudBuster VM Image
------------------------------
Building the KloudBuster VM and Docker images
---------------------------------------------
This section describes how to rebuild:
- the KloudButer VM Image from the git repository source code
- the KloudBuster Docker image
A Linux server with python, git and qemu utilities installed is required.
Build on Linux
^^^^^^^^^^^^^^
Your Linux server must have python, git and qemu utilities installed.
Ubuntu/Debian based:
Create a virtual environment (if not done already):
.. code-block:: bash
$ sudo apt-get install python-dev git qemu-utils
$ # Source the virtual environment if you have one
$ pip install PyYAML
virtualenv vkb
source vkb/bin/activate
Redhat/Fedora/CentOS based:
Ubuntu/Debian based:
.. code-block:: bash
$ sudo yum install python-devel git qemu-img
$ # Source the virtual environment if you have one
$ pip install PyYAML
sudo apt-get install python-dev git qemu-utils
pip install PyYAML
Build the image with below commands:
Redhat/Fedora/CentOS based:
.. code-block:: bash
$ # Clone the kloudbuster repository if you have not done so
$ git clone https://github.com/openstack/kloudbuster.git
$ # Go to the dib directory
$ cd kloudbuster/kb_dib
$ # Run the build image script, which will install DIB and start the build
$ ./build-image.sh
After a few minutes, the qcow2 image will be built and available in the same
directory. You can then upload it to OpenStack using the glance CLI.
If you get an error message saying that import yaml fails (seems to happen
only on Ubuntu)::
dib-run-parts Thu Jul 2 09:27:50 PDT 2015 Running /tmp/image.ewtpa5DW/hooks/extra-data.d/99-squash-package-install
"/tmp/image.ewtpa5DW/hooks/extra-data.d/../bin/package-installs-squash",
line 26, in <module>
import yaml
ImportError: No module named yaml
sudo yum install python-devel git qemu-img
pip install PyYAML
You need to comment out the secure_path option in your /etc/sudoers file (use
"sudo visudo" to edit that file)::
#Defaults secure_path="/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin"
Build on MacOSX
^^^^^^^^^^^^^^^
You need to install first:
* `Virtualbox <https://www.virtualbox.org/wiki/Downloads>`_
* `Vagrant <https://www.vagrantup.com/downloads.html>`_
And build the image with below commands:
Build the image with below commands:
.. code-block:: bash
$ # Clone the kloudbuster repository if you have not done so
$ git clone https://github.com/openstack/kloudbuster.git
$ # Go to the dib directory
$ cd kloudbuster/kb_dib
$ # Run vagrant and start building the image
$ vagrant up
# Clone the kloudbuster repository if you have not done so
git clone https://github.com/openstack/kloudbuster.git
cd kloudbuster
# Install kloudbuster
pip install -e .
# Run the build image script
$ ./kb_build.sh
After a few minutes (depending on virtualbox overhead), the qcow2 image will
be built and available in the same directory. You can then upload it to
OpenStack using the glance CLI, destroy the vagrant VM ("vagrant destroy") and
dispose of the kloudbuster directory (if no longer needed).
After a few minutes, the qcow2 and container images will be built and available in the same
directory. The qcow2 and container images will be named after the version (e.g. kloudbuster-7.0.0.qcow2 and berrypatch/kloudbuster:7.0.0).
Build the KloudBuster Docker Container Image
--------------------------------------------
Pushing the Docker container to DockerHub
-----------------------------------------
The KloudBuster Docker images are published in the DockerHub berrypatch repository:
`<https://hub.docker.com/r/berrypatch/kloudbuster/>`_
The Dockerfile at the root of the git repository can be used to build a new container based on Ubuntu 14.04.
To build for tag 6.0.3 (replace as needed with the real tag), go to the root of the repository then execute the docker build command:
.. code-block:: bash
sudo docker build --tag=berrypatch/kloudbuster:6.0.3 .
To publish you need to be a member of the berrypatch kloudbuster team. After the login (requires your DockerHub username and password), push the appropriate version to berrypatch:
.. code-block:: bash
sudo docker login
sudo docker push berrypatch/kloudbuster:6.0.3
sudo docker push berrypatch/kloudbuster:7.0.0

15
doc/source/faq.rst

@ -96,8 +96,19 @@ How are KloudBuster VM images managed?
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
KloudBuster VM images are built using OpenStack diskimage-builder (or DIB) and
have a version (single number). The default name of an image is
"kloudbuster_v<version>" (e.g. "kludbuster_v6"). Normally each KloudBuster
have a version (single number).
Starting from version 7.0.0
^^^^^^^^^^^^^^^^^^^^^^^^^^^
The name of the VM image is "kloudbuster-<version>" (e.g. "kloudbuster-7.0.0").
That image is now always included in the container image (which is why it is big).
When running KloudBuster from the container, it will automatically upload
that VM image from the local copy in the container.
Prior to version 7.0.0
^^^^^^^^^^^^^^^^^^^^^^
The default name of an image is
"kloudbuster_v<version>" (e.g. "kloudbuster_v6"). Normally each KloudBuster
application is associated to a recommended KloudBuster VM image version.
This is indicated in the output of --version::

56
doc/source/quickstart_docker.rst

@ -2,8 +2,7 @@
KloudBuster Docker Container Quick Start Guide
==============================================
The KloudBuster Docker container provides a quick way to use KloudBuster if you
are already familiar with Docker.
The recommended way to install KloudBuster is using Docker as it is the simplest.
Prerequisites
-------------
@ -32,40 +31,11 @@ admin user to run KloudBuster as much as possible (otherwise there are
restrictions on what you can do). Instructions below assume a copy of that
file is saved under the local directory with the name "admin-openrc.sh"
We assume in the below example that you have an openrc file available called
"admin-openrc.sh" in the local directory and that the corresponding OpenStack
password is "admin".
3. Upload the KloudBuster VM image to the cloud under test
----------------------------------------------------------
If your OpenStack cloud has full access to the Internet, you can skip this step
as KloudBuster will instruct Glance to download the KloudBuster VM image
directly from the OpenStack (skip to next step).
Otherwise, :ref:`download the latest kloudbuster image <upload_kb_image>` from
the OpenStack App Catalog.
In addition to the method described to upload the image using the Horizon
dashboard or the glance CLI, you can also use the glance CLI that is already
available in the KloudBuster container. Start a bash shell in the container
and map the local directory to '/opt/kb' in the container so that you have
access to the image and the RC file:
.. code-block:: bash
docker run -v $PWD:/opt/kb --rm -it berrypatch/kloudbuster bash
Then from inside the container bash prompt, source the openrc file, invoke the
glance CLI to upload the VM image (should take a few minutes) then exit and
terminate the container:
.. code-block:: bash
source /opt/kb/admin-openrc.sh
glance image-create --name "kloudbuster_v6" --visibility public --disk-format qcow2 --container-format bare --file /opt/kb/kloudbuster_v6.qcow2
Now you should be back to the host and should see the kloudbuster image in the
current directory.
4. Running the KloudBuster CLI
3. Running the KloudBuster CLI
------------------------------
If you do not really need a Web UI or REST interface, you can simply run
@ -73,12 +43,8 @@ KloudBuster scale test straight from CLI in the container.
.. code-block:: bash
docker run -v $PWD:/opt/kb --rm -t berrypatch/kloudbuster kloudbuster -h
We assume in the below example that you have an openrc file available called
"admin-openrc.sh" in the local directory and that the corresponding OpenStack
password is "admin".
docker run -v $PWD:/opt/kb --rm berrypatch/kloudbuster kloudbuster --version
docker run -v $PWD:/opt/kb --rm berrypatch/kloudbuster kloudbuster -h
Run the default HTTP data plane scale test
@ -88,7 +54,7 @@ The default HTTP scale test is described :ref:`here <default_http_scale>`.
.. code-block:: bash
docker run --rm -t -v $PWD:/opt/kb berrypatch/kloudbuster kloudbuster --tested-rc /opt/kb/admin-openrc.sh --tested-passwd admin
docker run --rm -v $PWD:/opt/kb berrypatch/kloudbuster kloudbuster --rc /opt/kb/admin-openrc.sh --passwd admin
Run the default storage scale test
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
@ -97,7 +63,7 @@ The default storage scale test is described :ref:`here <default_storage_scale>`.
.. code-block:: bash
docker run --rm -t -v $PWD:/opt/kb berrypatch/kloudbuster kloudbuster --tested-rc /opt/kb/admin-openrc.sh --tested-passwd admin --storage
docker run --rm -v $PWD:/opt/kb berrypatch/kloudbuster kloudbuster --rc /opt/kb/admin-openrc.sh --passwd admin --storage
Run KloudBuster with a custom configuration
@ -117,7 +83,7 @@ local directory to "/opt/kb" (for example):
.. code-block:: bash
docker run --rm -t -v $PWD:/opt/kb berrypatch/kloudbuster kloudbuster --tested-rc /opt/kb/admin-openrc.sh --tested-passwd admin --config /opt/kb/kb.cfg
docker run --rm -t -v $PWD:/opt/kb berrypatch/kloudbuster kloudbuster --rc /opt/kb/admin-openrc.sh --passwd admin --config /opt/kb/kb.cfg
5. Running KloudBuster as a WebUI/REST Server
---------------------------------------------
@ -128,7 +94,7 @@ the same port number at the host level:
.. code-block:: bash
docker run -p 8080:8080 --rm berrypatch/kloudbuster kb_start_server&
docker run -d -p 8080:8080 --rm berrypatch/kloudbuster kb_start_server
The first port number is the host listen port (any port of your choice) while
the second one after the column is the container listen port (always 8080 for

16
doc/source/quickstart_git.rst

@ -6,8 +6,7 @@ KloudBuster Git Quick Start Guide
This is the default installation method for code development.
It is recommended to run KloudBuster inside a virtual environment. However,
it can be skipped if installed in a dedicated VM.
It is recommended to run KloudBuster inside a virtual environment.
1. Install Dependencies and Clone Git Repository
------------------------------------------------
@ -23,6 +22,7 @@ Quick installation on Ubuntu/Debian
$ source ./vkb/bin/activate
$ git clone https://github.com/openstack/kloudbuster.git
$ cd kloudbuster
$ pip install -e .
$ pip install -r requirements-dev.txt
Quick installation on RHEL/Fedora/CentOS
@ -36,6 +36,7 @@ Quick installation on RHEL/Fedora/CentOS
$ source ./vkb/bin/activate
$ git clone https://github.com/openstack/kloudbuster.git
$ cd kloudbuster
$ pip install -e .
$ pip install -r requirements-dev.txt
Quick installation on MacOSX
@ -58,6 +59,7 @@ First, download XCode from App Store, then execute below commands:
$ source ./vkb/bin/activate
$ git clone https://github.com/openstack/kloudbuster.git
$ cd kloudbuster
$ pip install -e .
$ pip install -r requirements-dev.txt
If you need to run the KloudBuster Web UI you need to install coreutils
@ -77,7 +79,7 @@ To verify kloudbuster is installed, from the root of the kloudbuster repository
.. code-block:: bash
$ python kloudbuster/kloudbuster.py --help
kloudbuster --version
2. Upload the KloudBuster VM image
----------------------------------
@ -105,7 +107,7 @@ The default HTTP scale test is described :ref:`here <default_http_scale>`.
.. code-block:: bash
python kloudbuster/kloudbuster.py --tested-rc admin-openrc.sh --tested-passwd admin
python kloudbuster/kloudbuster.py --rc admin-openrc.sh --passwd admin
Run the default storage scale test
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
@ -114,7 +116,7 @@ The default storage scale test is described :ref:`here <default_storage_scale>`.
.. code-block:: bash
python kloudbuster/kloudbuster.py --tested-rc admin-openrc.sh --tested-passwd admin --storage
kloudbuster --rc admin-openrc.sh --passwd admin --storage
Run KloudBuster with a custom configuration
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
@ -131,14 +133,14 @@ the custom configuration:
.. code-block:: bash
python kloudbuster/kloudbuster.py --tested-rc admin-openrc.sh --tested-passwd admin --config kb.cfg
kloudbuster --rc admin-openrc.sh -passwd admin --config kb.cfg
5. Running KloudBuster as a WebUI/REST Server
---------------------------------------------
.. code-block:: bash
python kloudbuster/start_server.py&
kb_start_server&
You should see a message similar to the one below, which indicates the server
is up running::

6
doc/source/quickstart_pip.rst

@ -97,7 +97,7 @@ The default HTTP scale test is described :ref:`here <default_http_scale>`.
.. code-block:: bash
kloudbuster --tested-rc admin-openrc.sh --tested-passwd admin
kloudbuster --rc admin-openrc.sh --passwd admin
Run the default storage scale test
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
@ -106,7 +106,7 @@ The default storage scale test is described :ref:`here <default_storage_scale>`.
.. code-block:: bash
kloudbuster --tested-rc admin-openrc.sh --tested-passwd admin --storage
kloudbuster --rc admin-openrc.sh --passwd admin --storage
Run KloudBuster with a custom configuration
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
@ -124,7 +124,7 @@ the custom configuration:
.. code-block:: bash
kloudbuster --tested-rc admin-openrc.sh --tested-passwd admin --config kb.cfg
kloudbuster --rc admin-openrc.sh --passwd admin --config kb.cfg
6. Running KloudBuster as a WebUI/REST Server
---------------------------------------------

17
doc/source/readme.rst

@ -1,5 +1,5 @@
=====================
KloudBuster version 6
KloudBuster version 7
=====================
How good is your OpenStack **data plane** or **storage service** under real
@ -61,6 +61,9 @@ Feature List
* Supports any mix of read/write
* Supports fixed load (e.g. 1000 IOPs/VM) or highest load measurement
(KloudBuster will increase the load until latency spikes)
* IOPs, bandwitdh and loss-less millisecond-precision latency aggregation
for every IO operation (typically millions per run)
@ -95,6 +98,14 @@ graphical charts generated straight off the tool.
**Examples of results** are available in :ref:`gallery`.
New in Release 7
----------------
* The KloudBuster Docker container now includes the KloudBuster VM image for easier
setup (no more need to install/uplaod the VM image separately)
* Supports more recent OpenStack releases with newer API versions (Newton, Ocata)
Limitations and Non-Goals
-------------------------
@ -125,8 +136,8 @@ Contributions and Feedbacks
If you are interested in OpenStack Performance and Scale, contributions and
feedbacks are welcome!
If you have any feedbacks or would like to make small or large contributions,
simply send an email to openstack-dev@lists.openstack.org with a '[kloudbuster]'
If you have any feedbacks or would like to contribute,
send an email to openstack-dev@lists.openstack.org with a '[kloudbuster]'
tag in the subject.

61
doc/source/usage.rst

@ -27,7 +27,7 @@ KloudBuster Installation Options
There are 4 different ways to install KloudBuster:
- use a pre-built Docker container (recommended if you already use Docker)
- use a pre-built Docker container (recommended)
- use a pre-built VM image (if you prefer to run the KloudBuster application in a VM and do not need CLI)
- install from PyPI (if you prefer to use pip install)
- install directly from GitHub (git clone, for code development or if you want to browse the code)
@ -90,61 +90,50 @@ Examples of REST requests
.. _upload_kb_image:
KloudBuster VM Image Upload
---------------------------
Get the KloudBuster VM Image
----------------------------
Before you can use KloudBuster you must upload the KloudBuster VM image to your
OpenStack cloud under test. KloudBuster needs one "universal" test VM image
KloudBuster needs one "universal" test VM image
(referred to as "KloudBuster image") that contains the necessary test software.
The KloudBuster image is then instantiated by the KloudBuster application in
potentially large number of VMs using the appropriate role (HTTP server, HTTP
traffic generator...).
Pre-built VM images are available for download from the
`OpenStack App Catalog <http://apps.openstack.org/#tab=glance-images>`_.
Upload of the VM image to OpenStack is automatic with the KloudBuster container (as the VM image is
included in the container itself). For non container usages, it requires building a VM image or obtaining
it from the Internet (see below).
.. note::
The same KloudBuster VM image can be instantiated for running the test functions
(HTTP servers, HTTP traffic generators, file access tools) and for running KloudBuster as a web service.
.. note::
If your OpenStack Glance is able to access the Internet and you only use
the CLI to launch KloudBuster, you can skip this section (KloudBuster CLI
will request Glance to download the image from the OpenStack App Catalog when
it is not present in Glance).
Download the KloudBuster VM image to the local directory
Extract the KloudBuster VM image to the local directory
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
You must download a local copy if your OpenStack cloud does not have dirct
access to the Internet. Download the latest image directly from
`OpenStack App Catalog <http://apps.openstack.org/#tab=glance-images>`_ using
your favorite browser (search for "kloudbuster") or using wget. KloudBuster VM
images are qcow2 images named "kloudbuster_v<version>.qcow2" (e.g.
"kloudbuster_v6.qcow2"). Look for an image named with the "kloudbuster_v"
prefix and download the latest version from the list.
This requires Docker to be installed and requires Internet access to DockerHub.
Example for downloading the v6 image using wget:
Use the kb_extract_img_from_socker.sh script to download a copy of the VM image from DockerHub.
By default the script will download the VM image with the same version as the installed
KloudBuster package.
.. code-block:: bash
wget http://storage.apps.openstack.org/images/kloudbuster_v6.qcow2
kb_extract_img_from_socker.sh
Once extracted, you can let KloudBuster upload the VM image for you on a subsequent run (simplest and recommended) or the VM image can be manually uploaded to OpenStack using Horizon or the glance API.
KloudBuster by default will look into the root of the KloudBuster package or into the current directory to
check if the VM image file is present, and automatically upload it if it is not already in OpenStack.
Upload the KloudBuster VM image using the Horizon Dashboard
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
Upload the KloudBuster VM image using the Horizon Dashboard (optional)
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
From the dashboard, create a new image and select either "Image File" if you
want to uplaod from the local copy of the image or "Image Location" if you want
to upload directly from the OpenStack App Catalog (you will need the complete
URL of the image).
From the dashboard, create a new image and select "Image File" and select the VM image file.
The name of the image in Glance *must* match exactly the image name in the App
Catalog (without the .qcow2 extension, e.g. "kloudbuster_v6").
The name of the image in Glance *must* match exactly the image name (without the .qcow2 extension, e.g. "kloudbuster-7.0.0").
Upload the KloudBuster VM image using the Glance CLI
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
Upload the KloudBuster VM image using the Glance CLI (optional)
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
This assumes that you have installed the OpenStack Glance API and have sourced
the appropriate openrc file.
@ -153,6 +142,6 @@ To upload the image from a local copy of that image using the Glance CLI:
.. code-block:: bash
glance image-create --file kloudbuster_v6.qcow2 --disk-format qcow2 --container-format bare --visibility public --name kloudbuster_v6
glance image-create --file kloudbuster-7.0.0.qcow2 --disk-format qcow2 --container-format bare --visibility public --name kloudbuster-7.0.0

96
kb_build.sh

@ -0,0 +1,96 @@
#!/bin/bash
# This script will build the kloudbuster VM image and the container image under the ./build directory
# Check we are in a virtual environment
function check_in_venv {
IN_VENV=$(python -c 'import sys; print hasattr(sys, "real_prefix")')
echo $IN_VENV
}
# build the VM image first
function build_vm {
kb_image_name=kloudbuster-$KB_TAG
if [ ! -f $kb_image_name.qcow2 ]; then
echo "Building $kb_image_name.qcow2..."
pip install diskimage-builder
cd ./kb_dib
# Add the kloudbuster elements directory to the DIB elements path
export ELEMENTS_PATH=./elements
# Install Ubuntu 16.04
export DIB_RELEASE=xenial
time disk-image-create -o $kb_image_name ubuntu kloudbuster
rm -rf venv $kb_image_name.d
mv $kb_image_name.qcow2 ..
cd ..
else
echo "Reusing $kb_image_name.qcow2"
fi
ls -l $kb_image_name.qcow2
}
# Build container
function build_container {
sudo docker build --tag=berrypatch/kloudbuster:$KB_TAG .
}
function help {
echo
echo "Usage: bash build.sh <options>"
echo " --vm-only to only build the KloudBuster VM qcow2 image"
echo
echo "Builds the KloudBuster VM and Docker container images"
echo "The Docker container image will include the VM image for easier upload"
echo
echo "Must run in a virtual environment and must be called from the root of the repository"
exit 1
}
build_vm_only=0
while [[ $# -gt 0 ]]; do
key="$1"
case "$key" in
--vm-only)
build_vm_only=1
;;
-h|--help|*)
help
;;
esac
# Shift after checking all the cases to get the next option
shift
done
in_venv=$(check_in_venv)
if [ $in_venv != "True" ]; then
echo "Error: Must be in a virtual environment to run!"
exit 2
fi
# check we're at the root of the kloudbuster repo
if [ ! -d kloudbuster -o ! -f Dockerfile ]; then
echo "Error: Must be called from the root of the kloudbuster repository to run!"
exit 2
fi
# Install kloudbuster in the virtual env
pip install -q -U setuptools
pip install -q -e .
# Get the kloudbuster version (must be retrieved from stderr)
KB_TAG=$(kloudbuster --version 2>&1)
if [ $? != 0 ]; then
echo "Error retrieving kloudbuster version:"
echo
kloudbuster --version
exit 2
fi
echo
echo "Building KloudBuster with tag $KB_TAG"
build_vm
if [ $build_vm_only = 0 ]; then
build_container
fi

5
kb_dib/README.rst

@ -1,5 +0,0 @@
====================================
KLOUDBUSTER IMAGE BUILD INSTRUCTIONS
====================================
Refer to `here <http://kloudbuster.readthedocs.org/en/latest/installation.html#build-vm-image>`_ for the instructions of building KloudBuster image.

37
kb_dib/build-image.sh

@ -1,37 +0,0 @@
#!/bin/bash
#
# A shell script to build the kloudbuster image using diskinage-builder
#
# The following packages must be installed prior to using this script:
# sudo apt-get -y install git
# sudo apt-get -y install qemu-utils
# install diskimage-builder
git clone git://github.com/openstack/diskimage-builder.git
git clone git://github.com/openstack/dib-utils.git
# Add diskimage-builder and dib-utils bin to the path
export PATH=$PATH:`pwd`/diskimage-builder/bin:`pwd`/dib-utils/bin
# Add the kloudbuster elements directory to the DIB elements path
export ELEMENTS_PATH=`pwd`/elements
# Install Ubuntu 16.04
export DIB_RELEASE=xenial
# Extract image version number '__version__ = 2.0' becomes '__version__=2_0'
ver=`grep '^__version__' ../kloudbuster/kb_vm_agent.py | tr -d ' ' | tr '.' '_'`
eval $ver
kb_image_name=kloudbuster_v$__version__
echo "Building $kb_image_name.qcow2..."
time disk-image-create -o $kb_image_name ubuntu kloudbuster
ls -l $kb_image_name.qcow2
# cleanup
rm -rf diskimage-builder dib-utils

21
kb_dib/elements/kloudbuster/static/kb_test/kb_vm_agent.py

@ -29,27 +29,14 @@ import traceback
# Define the version of the KloudBuster agent and VM image
#
# When VM is up running, the agent will send the READY message to the
# KloudBuster main program, along with its version. The main program
# will check the version to see whether the image meets the minimum
# requirements to run, and stopped with an error if not.
# KloudBuster main program, along with its version.
#
# This version must be incremented if the interface changes or if new features
# are added to the agent VM
__version__ = '6'
# This version is no longer checked starting from release 7
# and can be left constant moving forward.
__version__ = '7'
# TODO(Logging on Agent)
def get_image_name():
'''Return the versioned VM image name that corresponds to this
agent code. This string must match the way DIB names the kloudbuster image.
Return:
the versioned image name without the extension ('.qcow2' is implicit)
'''
return 'kloudbuster_v' + __version__
def get_image_version():
return __version__
def exec_command(cmd, cwd=None):
p = subprocess.Popen(cmd, cwd=cwd, stdout=subprocess.PIPE, stderr=subprocess.PIPE)
(stdout, stderr) = p.communicate()

86
kb_extract_img_from_docker.sh

@ -0,0 +1,86 @@
#!/bin/bash
# Scriot to extract the same version kloudbuster VM image from docker hub
# and store it on the current directory
# Should be used when running kloudbuster using PyPI or git clone and
# rebuilding the VM is not an option
# Requires kloudbuster to be installed (pip install -e .)
# Requires docker to be installed!
function usage {
echo "
Usage: kb_extract_img_from_docker.sh [--version <version>]
<version> optional KloudBuster version to extract (e.g. 7.0.0)
Extracts a KloudBuster VM image from DockerHub and stores it on the local directory
If version is provided, the image for that version is retrieved
If version is not provided, the image for the current KloudBuster installation is
retrieved (requires KloudBuster to be installed using 'pip install -e .')
This script should only be used when not running KloudBuster from Docker
(e.g. running it from pip or from a git clone) and when rebuilding the VM image
is not an option.
"
exit 1
}
# Get the exact version
# Get the kloudbuster version (must be retrieved from stderr)
function get_kloudbuster_version {
KB_VER=$(kloudbuster --version 2>&1)
if [ $? != 0 ]; then
echo "Error running kloudbuster, make sure it is installed with 'pip install -e .'"
echo
exit 2
fi
}
function extract_vm_image {
VM_IMG_NAME=kloudbuster-$KB_VER.qcow2
if [ -f $VM_IMG_NAME ]; then
echo "$VM_IMG_NAME is already available"
else
# Pull the right version container from docker hub
IMG_NAME=berrypatch/kloudbuster:$KB_VER
docker pull $IMG_NAME
if [ $? != 0 ]; then
echo "Error pulling $IMG_NAME, please check it is available"
exit 1
fi
docker run --rm -v $PWD:/tmp/local $IMG_NAME cp /kloudbuster/$VM_IMG_NAME /tmp/local
if [ $? != 0 ]; then
echo "Error extracting $VM_IMG_NAME from the container"
exit 3
else
echo "Successfully extracted $VM_IMG_NAME from the container"
fi
fi
ls -l $VM_IMG_NAME
}
KB_VER=0
while [[ $# -gt 0 ]]; do
key="$1"
case "$key" in
--version|-v)
KB_VER=$2
shift
;;
-h|--help|*)
usage
;;
esac
# Shift after checking all the cases to get the next option
shift
done
if [ $KB_VER = 0 ]; then
get_kloudbuster_version
fi
extract_vm_image

16
kloudbuster/cfg.scale.yaml

@ -15,19 +15,25 @@ openrc_file:
# Name of the image to use for all test VMs (client, server and proxy)
# without the qcow2 extension
# The image name must exist in OpenStack and built with appropriate packages.
# The default test VM image is named "kloudbuster_v<version>" where
# <version> is the KloudBuster test VM image version (e.g. "kloudbuster_v6")
#
# Leave empty to use the default test VM image (recommended).
# If non empty use quotes if there are space characters in the name (e.g. 'my image')
# The default test VM image is named "kloudbuster-<version>" where
# <version> is the KloudBuster version (e.g. "kloudbuster-7.0.0")
image_name:
# KloudBuster can automatically upload a VM image if the image named by
# image_name is missing, for that you need to specify a file location where
# the image can be retrieved
#
# To upload the image as a file, download it to preferred location
# file://<full path of the image with qcow2 extension>
# To upload the image, download it first to a local file system before running kloudbuster
# Fill with the full pathname of the image with qcow2 extension
# e.g.
# vm_image_file: /kloudbuster/kloudbuster-7.0.0.qcow2
# If empty, KloudBuster will attempt to locate that file (with the default name)
# under the following directories:
# - root of the kloudbuster package
# - current directory
vm_image_file:
# Keystone admin role name (default should work in most deployments)

22
kloudbuster/kb_config.py

@ -22,7 +22,7 @@ from oslo_config import cfg
from pkg_resources import resource_string
import credentials
import kb_vm_agent
import kloudbuster
CONF = cfg.CONF
LOG = logging.getLogger(__name__)
@ -113,9 +113,25 @@ class KBConfig(object):
self.config_scale = self.config_scale + AttrDict(self.alt_cfg)
# Use the default image name for Glance
# defaults to something like "kloudbuster_v3"
# defaults to something like "kloudbuster-7.0.0"
default_image_name = 'kloudbuster-' + kloudbuster.__version__
if not self.config_scale['image_name']:
self.config_scale['image_name'] = kb_vm_agent.get_image_name()
self.config_scale['image_name'] = default_image_name
# Check if the default image is located at the default locations
# if vm_image_file is empty
if not self.config_scale['vm_image_file']:
# check current directory
default_image_file = default_image_name + '.qcow2'
if os.path.isfile(default_image_file):
self.config_scale['vm_image_file'] = default_image_file
else:
# check at the root of the package
# root is up one level where this module resides
pkg_root = os.path.dirname(os.path.dirname(os.path.abspath(__file__)))
default_image_file = pkg_root + '/' + default_image_file
if os.path.isfile(default_image_file):
self.config_scale['vm_image_file'] = default_image_file
# A bit of config dict surgery, extract out the client and server side
# and transplant the remaining (common part) into the client and server dict

13
kloudbuster/kb_runner_base.py

@ -15,7 +15,6 @@
from __future__ import division
import abc
from collections import deque
from distutils.version import LooseVersion
import json
import log as logging
import redis
@ -42,7 +41,7 @@ class KBRunner(object):
Control the testing VMs on the testing cloud
"""
def __init__(self, client_list, config, expected_agent_version, single_cloud=True):
def __init__(self, client_list, config, single_cloud=True):
self.full_client_dict = dict(zip([x.vm_name for x in client_list], client_list))
self.client_dict = self.full_client_dict
self.config = config
@ -50,7 +49,6 @@ class KBRunner(object):
self.result = {}
self.host_stats = {}
self.tool_result = {}
self.expected_agent_version = expected_agent_version
self.agent_version = None
self.report = None
@ -206,15 +204,6 @@ class KBRunner(object):
if cnt_succ != len(self.client_dict):
raise KBVMUpException("Some VMs failed to start.")
self.send_cmd('ACK', None, None)
if not self.agent_version:
self.agent_version = "0"
if (LooseVersion(self.agent_version) != LooseVersion(self.expected_agent_version))\
and (self.expected_agent_version not in vm_version_mismatches):
# only warn once for each unexpected VM version
vm_version_mismatches.add(self.expected_agent_version)
LOG.warning("The VM image you are running (%s) is not the expected version (%s) "
"this may cause some incompatibilities" %
(self.agent_version, self.expected_agent_version))
def gen_host_stats(self):
self.host_stats = {}

4
kloudbuster/kb_runner_http.py

@ -34,8 +34,8 @@ class KBRunner_HTTP(KBRunner):
Control the testing VMs on the testing cloud
"""
def __init__(self, client_list, config, expected_agent_version, single_cloud=True):
KBRunner.__init__(self, client_list, config, expected_agent_version, single_cloud)
def __init__(self, client_list, config, single_cloud=True):
KBRunner.__init__(self, client_list, config, single_cloud)
def header_formatter(self, stage, vm_count):
conns = vm_count * self.config.http_tool_configs.connections

4
kloudbuster/kb_runner_multicast.py

@ -30,8 +30,8 @@ class KBRunner_Multicast(KBRunner):
Control the testing VMs on the testing cloud
"""
def __init__(self, client_list, config, expected_agent_version, single_cloud=True):
KBRunner.__init__(self, client_list, config, expected_agent_version, single_cloud)
def __init__(self, client_list, config, single_cloud=True):
KBRunner.__init__(self, client_list, config, single_cloud)
def header_formatter(self, stage, nTests, nReceiver, pktsize):
msg = "Stage %d/%d: %d Receiver(s) with a Packet Size of %dkb" %\

4
kloudbuster/kb_runner_storage.py

@ -28,8 +28,8 @@ class KBRunner_Storage(KBRunner):
Control the testing VMs on the testing cloud
"""
def __init__(self, client_list, config, expected_agent_version):
KBRunner.__init__(self, client_list, config, expected_agent_version, single_cloud=True)
def __init__(self, client_list, config):
KBRunner.__init__(self, client_list, config, single_cloud=True)
def header_formatter(self, stage, vm_count):
rr_iops = rw_iops = sr_tp = sw_tp = 0

28
kloudbuster/kloudbuster.py

@ -35,7 +35,6 @@ from kb_runner_http import KBRunner_HTTP
from kb_runner_multicast import KBRunner_Multicast
from kb_runner_storage import KBRunner_Storage
from kb_scheduler import KBScheduler
import kb_vm_agent
import keystoneauth1
from keystoneclient.v2_0 import client as keystoneclient
@ -318,7 +317,6 @@ class KloudBuster(object):
def check_and_upload_images(self, retry_count=150):
retry = 0
image_location = None
image_name = self.client_cfg.image_name
image_url = self.client_cfg.vm_image_file
kloud_name_list = ['Server kloud', 'Client kloud']
@ -335,21 +333,13 @@ class KloudBuster(object):
# Trying to upload images
LOG.info("KloudBuster VM Image is not found in %s, trying to upload it..." % kloud)
if not image_location:
if not image_url:
LOG.error('Configuration file is missing a VM image URL (vm_image_name)')
return False
file_prefix = 'file://'
if not image_url.startswith(file_prefix):
LOG.error('vm_image_name (%s) must start with "%s", aborting' %
(image_url, file_prefix))
return False
image_location = image_url.split(file_prefix)[1]
if not image_url:
LOG.error('Configuration file is missing a VM image pathname (vm_image_name)')
return False
retry = 0
try:
LOG.info("Uploading VM Image from %s..." % image_location)
with open(image_location) as f_image:
LOG.info("Uploading VM Image from %s..." % image_url)
with open(image_url) as f_image:
img = glance_client.images.create(name=image_name,
disk_format="qcow2",
container_format="bare",
@ -550,16 +540,13 @@ class KloudBuster(object):
self.kb_proxy.boot_info['user_data'] = str(self.kb_proxy.user_data)
self.testing_kloud.create_vm(self.kb_proxy)
if self.storage_mode:
self.kb_runner = KBRunner_Storage(client_list, self.client_cfg,
kb_vm_agent.get_image_version())
self.kb_runner = KBRunner_Storage(client_list, self.client_cfg)
elif self.multicast_mode:
self.kb_runner = KBRunner_Multicast(client_list, self.client_cfg,
kb_vm_agent.get_image_version(),
self.single_cloud)
else:
self.kb_runner = KBRunner_HTTP(client_list, self.client_cfg,
kb_vm_agent.get_image_version(),
self.single_cloud)
self.kb_runner.setup_redis(self.kb_proxy.fip_ip or self.kb_proxy.fixed_ip)
@ -895,8 +882,7 @@ def main():
metavar="<source json file>"),
]
CONF.register_cli_opts(cli_opts)
full_version = __version__ + ', VM image: ' + kb_vm_agent.get_image_name()
CONF(sys.argv[1:], project="kloudbuster", version=full_version)
CONF(sys.argv[1:], project="kloudbuster", version=__version__)
logging.setup("kloudbuster")
if CONF.rc and not CONF.tested_rc:

23
kloudbuster/tests/base.py

@ -1,23 +0,0 @@
# -*- coding: utf-8 -*-
# Copyright 2010-2011 OpenStack Foundation
# Copyright (c) 2013 Hewlett-Packard Development Company, L.P.
#
# Licensed under the Apache License, Version 2.0 (the "License"); you may
# not use this file except in compliance with the License. You may obtain
# a copy of the License at
#
# http://www.apache.org/licenses/LICENSE-2.0
#
# Unless required by applicable law or agreed to in writing, software
# distributed under the License is distributed on an "AS IS" BASIS, WITHOUT
# WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. See the
# License for the specific language governing permissions and limitations
# under the License.
from oslotest import base
class TestCase(base.BaseTestCase):
"""Test case base class for all unit tests."""

8
kloudbuster/tests/test_kloudbuster.py

@ -18,11 +18,3 @@ test_kloudbuster
Tests for `kloudbuster` module.
"""
from kloudbuster.tests import base
class TestKloudbuster(base.TestCase):
def test_something(self):
pass

1
setup.py

@ -26,4 +26,5 @@ except ImportError:
setuptools.setup(
setup_requires=['pbr'],
scripts=['kb_extract_img_from_docker.sh'],
pbr=True)

2
tox.ini

@ -10,7 +10,7 @@ setenv =
VIRTUAL_ENV={envdir}
deps = -r{toxinidir}/requirements.txt
-r{toxinidir}/test-requirements.txt
commands = python setup.py test --slowest --testr-args='{posargs}'
# commands = python setup.py test --slowest --testr-args='{posargs}'
[testenv:pep8]
commands = flake8

Loading…
Cancel
Save