337 lines
15 KiB
HTML
337 lines
15 KiB
HTML
<!DOCTYPE html>
|
|
<html lang="en">
|
|
<head>
|
|
<meta charset="utf-8">
|
|
<title>Multi-Node Lab Server Guide - DevStack</title>
|
|
<meta name="description" content="">
|
|
<meta name="author" content="">
|
|
|
|
<!-- Le HTML5 shim, for IE6-8 support of HTML elements -->
|
|
<!--[if lt IE 9]>
|
|
<script src="http://html5shim.googlecode.com/svn/trunk/html5.js"></script>
|
|
<![endif]-->
|
|
|
|
<!-- Le styles -->
|
|
<link href="../assets/css/bootstrap.css" rel="stylesheet">
|
|
<link href="../assets/css/local.css" rel="stylesheet">
|
|
<style type="text/css">
|
|
body { padding-top: 60px; }
|
|
dd { padding: 10px; }
|
|
</style>
|
|
|
|
<!-- Le javascripts -->
|
|
<script src="../assets/js/jquery-1.7.1.min.js" type="text/javascript" charset="utf-8"></script>
|
|
<script src="../assets/js/bootstrap.js" type="text/javascript" charset="utf-8"></script>
|
|
</head>
|
|
|
|
<body>
|
|
|
|
<div class="navbar navbar-fixed-top">
|
|
<div class="navbar-inner">
|
|
<div class="container">
|
|
<a class="brand" href="/">DevStack</a>
|
|
<ul class="nav pull-right">
|
|
<li><a href="../overview.html">Overview</a></li>
|
|
<li><a href="../changes.html">Changes</a></li>
|
|
<li><a href="../faq.html">FAQ</a></li>
|
|
<li><a href="https://git.openstack.org/cgit/openstack-dev/devstack">git.openstack.org</a></li>
|
|
<li><a href="https://review.openstack.org/#/q/status:open+project:openstack-dev/devstack,n,z">Gerrit</a></li>
|
|
</ul>
|
|
</div>
|
|
</div>
|
|
</div>
|
|
|
|
<div class="container">
|
|
|
|
<section id="overview">
|
|
<h1>Multi-Node Lab: Serious Stuff</h1>
|
|
<p>Here is OpenStack in a realistic test configuration with multiple physical servers.</p>
|
|
</section>
|
|
|
|
<section id="prerequisites">
|
|
<div class="page-header">
|
|
<h2>Prerequisites <small>Linux & Network</small></h2>
|
|
</div>
|
|
|
|
<h3>Minimal Install</h3>
|
|
<p>You need to have a system with a fresh install of Linux. You can download the <a href="https://help.ubuntu.com/community/Installation/MinimalCD">Minimal CD</a> for Ubuntu releases since DevStack will download & install all the additional dependencies. The netinstall ISO is available for <a href="http://mirrors.kernel.org/fedora/releases/18/Fedora/x86_64/iso/Fedora-20-x86_64-netinst.iso">Fedora</a> and <a href="http://mirrors.kernel.org/centos/6.5/isos/x86_64/CentOS-6.5-x86_64-netinstall.iso">CentOS/RHEL</a>.</p>
|
|
|
|
<p>Install a couple of packages to bootstrap configuration:</p>
|
|
<pre>apt-get install -y git sudo || yum install -y git sudo</pre>
|
|
|
|
<h3>Network Configuration</h3>
|
|
<p>The first iteration of the lab uses OpenStack's FlatDHCP network controller so
|
|
only a single network will be required. It should be on its own subnet without DHCP;
|
|
the host IPs and floating IP pool(s) will come out of this block. This example
|
|
uses the following:</p>
|
|
<ul>
|
|
<li>Gateway: 192.168.42.1</li>
|
|
<li>Physical nodes: 192.168.42.11-192.168.42.99</li>
|
|
<li>Floating IPs: 192.168.42.128-192.168.42.254</li>
|
|
</ul>
|
|
<p>Configure each node with a static IP.
|
|
For Ubuntu edit <code>/etc/network/interfaces</code>:</p>
|
|
|
|
<pre>auto eth0
|
|
iface eth0 inet static
|
|
address 192.168.42.11
|
|
netmask 255.255.255.0
|
|
gateway 192.168.42.1
|
|
</pre>
|
|
<p>For Fedora and CentOS/RHEL edit
|
|
<code>/etc/sysconfig/network-scripts/ifcfg-eth0</code>:</p>
|
|
|
|
<pre>BOOTPROTO=static
|
|
IPADDR=192.168.42.11
|
|
NETMASK=255.255.255.0
|
|
GATEWAY=192.168.42.1
|
|
</pre>
|
|
|
|
<!-- save these for the VLAN version
|
|
auto eth0.926
|
|
iface eth0.926 inet static
|
|
address 10.4.144.3
|
|
netmask 255.255.240.0
|
|
#gateway 10.4.144.1
|
|
up ifconfig eth0.926
|
|
|
|
auto eth0.605
|
|
iface eth0.605 inet static
|
|
address 172.16.103.3
|
|
netmask 255.255.255.0
|
|
#gateway 172.16.103.1
|
|
up ifconfig eth0.605 up
|
|
-->
|
|
|
|
</section>
|
|
|
|
<section id="installation">
|
|
<div class="page-header">
|
|
<h2>Installation <small>shake and bake</small></h2>
|
|
</div>
|
|
|
|
<h3>Add the DevStack User</h3>
|
|
<p>OpenStack runs as a non-root user that has sudo access to root. There is nothing special
|
|
about the name, we'll use <code>stack</code> here. Every node must use the same name and
|
|
preferably uid. If you created a user during the OS install you can use it and give it
|
|
sudo privileges below. Otherwise create the stack user:</p>
|
|
<pre>groupadd stack
|
|
useradd -g stack -s /bin/bash -d /opt/stack -m stack</pre>
|
|
<p>This user will be making many changes to your system during installation and operation
|
|
so it needs to have sudo privileges to root without a password:</p>
|
|
<pre>echo "stack ALL=(ALL) NOPASSWD: ALL" >> /etc/sudoers</pre>
|
|
<p>From here on use the <code>stack</code> user. <b>Logout</b> and <b>login</b> as the
|
|
<code>stack</code> user.</p>
|
|
|
|
<h3>Set Up Ssh</h3>
|
|
<p>Set up the stack user on each node with an ssh key for access:</p>
|
|
<pre>mkdir ~/.ssh; chmod 700 ~/.ssh
|
|
echo "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQCyYjfgyPazTvGpd8OaAvtU2utL8W6gWC4JdRS1J95GhNNfQd657yO6s1AH5KYQWktcE6FO/xNUC2reEXSGC7ezy+sGO1kj9Limv5vrvNHvF1+wts0Cmyx61D2nQw35/Qz8BvpdJANL7VwP/cFI/p3yhvx2lsnjFE3hN8xRB2LtLUopUSVdBwACOVUmH2G+2BWMJDjVINd2DPqRIA4Zhy09KJ3O1Joabr0XpQL0yt/I9x8BVHdAx6l9U0tMg9dj5+tAjZvMAFfye3PJcYwwsfJoFxC8w/SLtqlFX7Ehw++8RtvomvuipLdmWCy+T9hIkl+gHYE4cS3OIqXH7f49jdJf jesse@spacey.local" > ~/.ssh/authorized_keys</pre>
|
|
|
|
<h3>Download DevStack</h3>
|
|
<p>Grab the latest version of DevStack:</p>
|
|
<pre>git clone https://git.openstack.org/openstack-dev/devstack
|
|
cd devstack</pre>
|
|
|
|
<p>Up to this point all of the steps apply to each node in the cluster. From here on
|
|
there are some differences between the cluster controller (aka 'head node') and the
|
|
compute nodes.</p>
|
|
|
|
<h3>Configure Cluster Controller</h3>
|
|
<p>The cluster controller runs all OpenStack services. Configure the cluster controller's DevStack in <code>local.conf</code>:</p>
|
|
<pre>[[local|localrc]]
|
|
HOST_IP=192.168.42.11
|
|
FLAT_INTERFACE=eth0
|
|
FIXED_RANGE=10.4.128.0/20
|
|
FIXED_NETWORK_SIZE=4096
|
|
FLOATING_RANGE=192.168.42.128/25
|
|
MULTI_HOST=1
|
|
LOGFILE=/opt/stack/logs/stack.sh.log
|
|
ADMIN_PASSWORD=labstack
|
|
MYSQL_PASSWORD=supersecret
|
|
RABBIT_PASSWORD=supersecrete
|
|
SERVICE_PASSWORD=supersecrete
|
|
SERVICE_TOKEN=xyzpdqlazydog</pre>
|
|
<!-- save for vlan
|
|
FLAT_INTERFACE=eth0.926
|
|
-->
|
|
|
|
<p>In the multi-node configuration the first 10 or so IPs in the private subnet are usually reserved. Add this to <code>local.sh</code> to have it run after every <code>stack.sh</code> run:</p>
|
|
<pre>for i in `seq 2 10`; do /opt/stack/nova/bin/nova-manage fixed reserve 10.4.128.$i; done</pre>
|
|
|
|
<p>Fire up OpenStack:</p>
|
|
<pre>./stack.sh</pre>
|
|
<p>A stream of activity ensues. When complete you will see a summary of
|
|
<code>stack.sh</code>'s work, including the relevant URLs, accounts and passwords to poke at your
|
|
shiny new OpenStack. The most recent log file is available in <code>stack.sh.log</code>.</p>
|
|
|
|
<h3>Configure Compute Nodes</h3>
|
|
<p>The compute nodes only run the OpenStack worker services. For additional machines, create a <code>local.conf</code> with:</p>
|
|
<pre>HOST_IP=192.168.42.12 # change this per compute node
|
|
FLAT_INTERFACE=eth0
|
|
FIXED_RANGE=10.4.128.0/20
|
|
FIXED_NETWORK_SIZE=4096
|
|
FLOATING_RANGE=192.168.42.128/25
|
|
MULTI_HOST=1
|
|
LOGFILE=/opt/stack/logs/stack.sh.log
|
|
ADMIN_PASSWORD=labstack
|
|
MYSQL_PASSWORD=supersecret
|
|
RABBIT_PASSWORD=supersecrete
|
|
SERVICE_PASSWORD=supersecrete
|
|
SERVICE_TOKEN=xyzpdqlazydog
|
|
DATABASE_TYPE=mysql
|
|
SERVICE_HOST=192.168.42.11
|
|
MYSQL_HOST=192.168.42.11
|
|
RABBIT_HOST=192.168.42.11
|
|
GLANCE_HOSTPORT=192.168.42.11:9292
|
|
ENABLED_SERVICES=n-cpu,n-net,n-api,c-sch,c-api,c-vol
|
|
NOVA_VNC_ENABLED=True
|
|
NOVNCPROXY_URL="http://192.168.42.11:6080/vnc_auto.html"
|
|
VNCSERVER_LISTEN=$HOST_IP
|
|
VNCSERVER_PROXYCLIENT_ADDRESS=$VNCSERVER_LISTEN
|
|
</pre>
|
|
|
|
<!-- save for vlan
|
|
FLAT_INTERFACE=eth0.926
|
|
-->
|
|
|
|
<p>Fire up OpenStack:</p>
|
|
<pre>./stack.sh</pre>
|
|
<p>A stream of activity ensues. When complete you will see a summary of
|
|
<code>stack.sh</code>'s work, including the relevant URLs, accounts and passwords to poke at your
|
|
shiny new OpenStack. The most recent log file is available in <code>stack.sh.log</code>.</p>
|
|
|
|
<h3>Cleaning Up After DevStack</h3>
|
|
<p>Shutting down OpenStack is now as simple as running the included <code>unstack.sh</code> script:</p>
|
|
<pre>./unstack.sh</pre>
|
|
|
|
<p>A more aggressive cleanup can be performed using <code>clean.sh</code>. It removes certain troublesome packages and attempts to leave the system in a state where changing the database or queue manager can be reliably performed.
|
|
<pre>./clean.sh</pre>
|
|
|
|
<p>Sometimes running instances are not cleaned up. DevStack attempts to do this when it
|
|
runs but there are times it needs to still be done by hand:</p>
|
|
<pre>sudo rm -rf /etc/libvirt/qemu/inst*
|
|
sudo virsh list | grep inst | awk '{print $1}' | xargs -n1 virsh destroy</pre>
|
|
|
|
</section>
|
|
|
|
<section id="options">
|
|
<div class="page-header">
|
|
<h2>Options <small>pimp your stack</small></h2>
|
|
</div>
|
|
|
|
<h3>Additional Users</h3>
|
|
<p>DevStack creates two OpenStack users (<code>admin</code> and <code>demo</code>) and two tenants (also <code>admin</code> and <code>demo</code>). <code>admin</code> is exactly what it sounds like, a privileged administrative account that is a member of both the <code>admin</code> and <code>demo</code> tenants. <code>demo</code> is a normal user account that is only a member of the <code>demo</code> tenant. Creating additional OpenStack users can be done through the dashboard, sometimes it is easier to do them in bulk from a script, especially since they get blown away every time
|
|
<code>stack.sh</code> runs. The following steps are ripe for scripting:</p>
|
|
<pre># Get admin creds
|
|
. openrc admin admin
|
|
|
|
# List existing tenants
|
|
keystone tenant-list
|
|
|
|
# List existing users
|
|
keystone user-list
|
|
|
|
# Add a user and tenant
|
|
NAME=bob
|
|
PASSWORD=BigSecrete
|
|
TENANT=$NAME
|
|
keystone tenant-create --name=$NAME
|
|
keystone user-create --name=$NAME --pass=$PASSWORD
|
|
keystone user-role-add --user-id=<bob-user-id> --tenant-id=<bob-tenant-id> --role-id=<member-role-id>
|
|
# member-role-id comes from the existing member role created by stack.sh
|
|
# keystone role-list</pre>
|
|
|
|
<h3>Swift</h3>
|
|
<p>Swift requires a significant amount of resources and is disabled by default in DevStack.
|
|
The support in DevStack is geared toward a minimal installation but can be used for
|
|
testing. To implement a true multi-node test of Swift required more than DevStack provides.
|
|
Enabling it is as simple as enabling the <code>swift</code> service in <code>local.conf</code>:
|
|
<pre>enable_service s-proxy s-object s-container s-account</pre>
|
|
|
|
<p>Swift will put its data files in <code>SWIFT_DATA_DIR</code> (default <code>/opt/stack/data/swift</code>).
|
|
The size of the data 'partition' created (really a loop-mounted file) is set by
|
|
<code>SWIFT_LOOPBACK_DISK_SIZE</code>. The Swift config files are located in
|
|
<code>SWIFT_CONFIG_DIR</code> (default <code>/etc/swift</code>). All of these settings can be overridden in
|
|
(wait for it...) <code>local.conf</code>.</p>
|
|
|
|
<h3>Volumes</h3>
|
|
<p>DevStack will automatically use an existing LVM volume group named <code>stack-volumes</code>
|
|
to store cloud-created volumes. If <code>stack-volumes</code> doesn't exist, DevStack
|
|
will set up a 5Gb loop-mounted file to contain it. This obviously limits the
|
|
number and size of volumes that can be created inside OpenStack. The size can be
|
|
overridden by setting <code>VOLUME_BACKING_FILE_SIZE</code> in <code>local.conf</code>.</p>
|
|
|
|
<p><code>stack-volumes</code> can be pre-created on any physical volume supported by
|
|
Linux's LVM. The name of the volume group can be changed by setting <code>VOLUME_GROUP</code>
|
|
in <code>localrc</code>. <code>stack.sh</code> deletes
|
|
all logical volumes in <code>VOLUME_GROUP</code> that begin with
|
|
<code>VOLUME_NAME_PREFIX</code> as part of cleaning up from previous runs.
|
|
It is recommended to not use the root volume group as <code>VOLUME_GROUP</code>.</p>
|
|
|
|
<p>The details of creating the volume group depends on the server hardware involved
|
|
but looks something like this:</p>
|
|
<pre>pvcreate /dev/sdc
|
|
vgcreate stack-volumes /dev/sdc</pre>
|
|
|
|
<h3>Syslog</h3>
|
|
<p>DevStack is capable of using <code>rsyslog</code> to aggregate logging across the cluster.
|
|
It is off by default; to turn it on set <code>SYSLOG=True</code> in <code>local.conf</code>.
|
|
<code>SYSLOG_HOST</code> defaults to <code>HOST_IP</code>; on the compute nodes it
|
|
must be set to the IP of the cluster controller to send syslog output there. In the example
|
|
above, add this to the compute node <code>local.conf</code>:</p>
|
|
<pre>SYSLOG_HOST=192.168.42.11</pre>
|
|
|
|
<h3>Using Alternate Repositories/Branches</h3>
|
|
<p>The git repositories for all of the OpenStack services are defined in <code>stackrc</code>.
|
|
Since this file is a part of the DevStack package changes to it will probably be overwritten
|
|
as updates are applied. Every setting in <code>stackrc</code> can be redefined in
|
|
<code>local.conf</code>.</p>
|
|
|
|
<p>To change the repository or branch that a particular OpenStack service is created from,
|
|
simply change the value of <code>*_REPO</code> or <code>*_BRANCH</code> corresponding to
|
|
that service.</p>
|
|
|
|
<p>After making changes to the repository or branch, if <code>RECLONE</code> is not set
|
|
in <code>localrc</code> it may be necessary to remove the corresponding directory from
|
|
<code>/opt/stack</code> to force git to re-clone the repository.</p>
|
|
|
|
<p>For example, to pull Nova from a proposed release candidate in the primary Nova
|
|
repository:</p>
|
|
<pre>NOVA_BRANCH=rc-proposed</pre>
|
|
|
|
<p>To pull Glance from an experimental fork:</p>
|
|
<pre>GLANCE_BRANCH=try-something-big
|
|
GLANCE_REPO=https://github.com/mcuser/glance.git</pre>
|
|
|
|
</section>
|
|
|
|
<section id="Notes">
|
|
<div class="page-header">
|
|
<h2>Notes <small>stuff you might need to know</small></h2>
|
|
</div>
|
|
|
|
<h3>Reset the Bridge</h3>
|
|
<p>How to reset the bridge configuration:</p>
|
|
<pre>sudo brctl delif br100 eth0.926
|
|
sudo ip link set dev br100 down
|
|
sudo brctl delbr br100</pre>
|
|
<!--
|
|
sudo rm /etc/libvirt/qemu/*.xml
|
|
-->
|
|
|
|
<h3>Set MySQL Password</h3>
|
|
<p>If you forgot to set the root password you can do this:</p>
|
|
<pre>mysqladmin -u root -pnova password 'supersecret'</pre>
|
|
|
|
</section>
|
|
|
|
<footer>
|
|
<p>© Openstack Foundation 2011-2014 — An <a href="https://www.openstack.org/">OpenStack</a> <a href="https://wiki.openstack.org/wiki/Programs">program</a></p>
|
|
</footer>
|
|
|
|
</div> <!-- /container -->
|
|
|
|
</body>
|
|
</html>
|