openstack/swift
Code Issues Proposed changes
9b521fe800
swift/doc/source/development_saio.rst
Peter Portante 3d8f0f1805 Fedora 19: need to use /etc/rc.d/rc.local 
Change-Id: I80e9a4c40ff99ec09a8eeef935447c6393ea78ec
Signed-off-by: Peter Portante <peter.portante@redhat.com>
2013-10-03 11:36:31 -04:00

849 lines
25 KiB
ReStructuredText
Raw Blame History

=======================
SAIO - Swift All In One
=======================
---------------------------------------------
Instructions for setting up a development VM
---------------------------------------------
This section documents setting up a virtual machine for doing Swift development.
The virtual machine will emulate running a four node Swift cluster.
* Get an Ubuntu 12.04 LTS (Precise Pangolin) server image or try something Fedora/CentOS.
* Create guest virtual machine from the image.
Additional information about setting up a Swift development snapshot on other distributions is
available on the wiki at http://wiki.openstack.org/SAIOInstructions.
----------------------------
What's in a <your-user-name>
----------------------------
Much of the configuration described in this guide requires escalated root
privileges; however, we assume that administrator logs in an unprivileged
user. Swift processes also run under a separate user and group, set by
configuration option, and refered as <your-user-name>:<your-group-name>.
The default user is `swift`, which may not exist on your system.
.. note::
The instructions in the first first half of this guide are expected to be
performed as the root user.
-----------------------
Installing dependencies
-----------------------
* On apt based systems,
#. `apt-get update`
#. `apt-get install curl gcc memcached rsync sqlite3 xfsprogs git-core libffi-dev python-setuptools`
#. `apt-get install python-coverage python-dev python-nose python-simplejson
python-xattr python-eventlet python-greenlet python-pastedeploy
python-netifaces python-pip python-dnspython python-mock`
* On yum based systems,
#. `yum install curl gcc memcached rsync sqlite xfsprogs git-core libffi-devel xinetd python-setuptools`
#. `yum install python-coverage python-devel python-nose python-simplejson
python-xattr python-eventlet python-greenlet python-pastedeploy
python-netifaces python-pip python-dnspython python-mock`
This installs necessary system dependencies; and *most* of the python
dependencies. Later in the process setuptools/distribute or pip will
install and/or upgrade some other stuff - it's getting harder to avoid.
You can also install anything else you want, like screen, ssh, vim, etc.
Next, choose either :ref:`partition-section` or :ref:`loopback-section`.
.. _partition-section:
Using a partition for storage
=============================
If you are going to use a separate partition for Swift data, be sure to add
another device when creating the VM, and follow these instructions.
#. `fdisk /dev/sdb` (set up a single partition)
#. `mkfs.xfs /dev/sdb1`
#. Edit `/etc/fstab` and add
`/dev/sdb1 /mnt/sdb1 xfs noatime,nodiratime,nobarrier,logbufs=8 0 0`
#. `mkdir /mnt/sdb1`
#. `mount /mnt/sdb1`
#. `mkdir /mnt/sdb1/1 /mnt/sdb1/2 /mnt/sdb1/3 /mnt/sdb1/4`
#. `chown <your-user-name>:<your-group-name> /mnt/sdb1/*`
#. `mkdir /srv`
#. `for x in {1..4}; do ln -s /mnt/sdb1/$x /srv/$x; done`
#. `mkdir -p /etc/swift/object-server /etc/swift/container-server /etc/swift/account-server /srv/1/node/sdb1 /srv/2/node/sdb2 /srv/3/node/sdb3 /srv/4/node/sdb4 /var/run/swift`
#. `chown -R <your-user-name>:<your-group-name> /etc/swift /srv/[1-4]/ /var/run/swift` -- **Make sure to include the trailing slash after /srv/[1-4]/**
#. Add to `/etc/rc.local` (before the `exit 0`)::
mkdir -p /var/cache/swift /var/cache/swift2 /var/cache/swift3 /var/cache/swift4
chown <your-user-name>:<your-group-name> /var/cache/swift*
mkdir -p /var/run/swift
chown <your-user-name>:<your-group-name> /var/run/swift
#. Next, skip to :ref:`rsync-section`.
.. _loopback-section:
Using a loopback device for storage
===================================
If you want to use a loopback device instead of another partition, follow these instructions.
#. `mkdir /srv`
#. `truncate -s 1GB /srv/swift-disk`
(modify size to make a larger or smaller partition)
#. `mkfs.xfs /srv/swift-disk`
#. Edit `/etc/fstab` and add
`/srv/swift-disk /mnt/sdb1 xfs loop,noatime,nodiratime,nobarrier,logbufs=8 0 0`
#. `mkdir /mnt/sdb1`
#. `mount /mnt/sdb1`
#. `mkdir /mnt/sdb1/1 /mnt/sdb1/2 /mnt/sdb1/3 /mnt/sdb1/4`
#. `chown <your-user-name>:<your-group-name> /mnt/sdb1/*`
#. `for x in {1..4}; do ln -s /mnt/sdb1/$x /srv/$x; done`
#. `mkdir -p /etc/swift/object-server /etc/swift/container-server /etc/swift/account-server /srv/1/node/sdb1 /srv/2/node/sdb2 /srv/3/node/sdb3 /srv/4/node/sdb4 /var/run/swift`
#. `chown -R <your-user-name>:<your-group-name> /etc/swift /srv/[1-4]/ /var/run/swift` -- **Make sure to include the trailing slash after /srv/[1-4]/**
#. Add to `/etc/rc.local` (before the `exit 0`)::
mkdir -p /var/cache/swift /var/cache/swift2 /var/cache/swift3 /var/cache/swift4
chown <your-user-name>:<your-group-name> /var/cache/swift*
mkdir -p /var/run/swift
chown <your-user-name>:<your-group-name> /var/run/swift
Note that on some systems you might have to create `/etc/rc.local`.
On Fedora 19 or later, you need to place these in `/etc/rc.d/rc.local`.
.. _rsync-section:
----------------
Setting up rsync
----------------
#. Create /etc/rsyncd.conf::
uid = <your-user-name>
gid = <your-group-name>
log file = /var/log/rsyncd.log
pid file = /var/run/rsyncd.pid
address = 127.0.0.1
[account6012]
max connections = 25
path = /srv/1/node/
read only = false
lock file = /var/lock/account6012.lock
[account6022]
max connections = 25
path = /srv/2/node/
read only = false
lock file = /var/lock/account6022.lock
[account6032]
max connections = 25
path = /srv/3/node/
read only = false
lock file = /var/lock/account6032.lock
[account6042]
max connections = 25
path = /srv/4/node/
read only = false
lock file = /var/lock/account6042.lock
[container6011]
max connections = 25
path = /srv/1/node/
read only = false
lock file = /var/lock/container6011.lock
[container6021]
max connections = 25
path = /srv/2/node/
read only = false
lock file = /var/lock/container6021.lock
[container6031]
max connections = 25
path = /srv/3/node/
read only = false
lock file = /var/lock/container6031.lock
[container6041]
max connections = 25
path = /srv/4/node/
read only = false
lock file = /var/lock/container6041.lock
[object6010]
max connections = 25
path = /srv/1/node/
read only = false
lock file = /var/lock/object6010.lock
[object6020]
max connections = 25
path = /srv/2/node/
read only = false
lock file = /var/lock/object6020.lock
[object6030]
max connections = 25
path = /srv/3/node/
read only = false
lock file = /var/lock/object6030.lock
[object6040]
max connections = 25
path = /srv/4/node/
read only = false
lock file = /var/lock/object6040.lock
#. On Ubuntu, edit the following line in `/etc/default/rsync`::
RSYNC_ENABLE=true
On Fedora, edit the following line in `/etc/xinetd.d/rsync`::
disable = no
#. On platforms with SELinux in `Enforcing` mode, either set to `Permissive`::
setenforce Permissive
Or just allow rsync full access::
setsebool -P rsync_full_access 1
#. On Ubuntu, run `service rsync restart`
On Fedora, run::
systemctl enable rsyncd.service
systemctl start rsyncd.service
On other xinetd based systems run `service xinetd restart`.
#. Verify rsync is accepting connections for all servers::
rsync rsync://pub@localhost/
------------------
Starting memcached
------------------
On non-Ubuntu distros you need to ensure memcached is running:
* `service memcached start`
* `chkconfig memcached on`
or:
* `systemctl enable memcached.service`
* `systemctl start memcached.service`
The tempauth middleware stores tokens in memcached. If memcached is not
running, tokens cannot be validated, and accessing Swift becomes impossible.
---------------------------------------------------
Optional: Setting up rsyslog for individual logging
---------------------------------------------------
#. Create /etc/rsyslog.d/10-swift.conf::
# Uncomment the following to have a log containing all logs together
#local1,local2,local3,local4,local5.* /var/log/swift/all.log
# Uncomment the following to have hourly proxy logs for stats processing
#$template HourlyProxyLog,"/var/log/swift/hourly/%$YEAR%%$MONTH%%$DAY%%$HOUR%"
#local1.*;local1.!notice ?HourlyProxyLog
local1.*;local1.!notice /var/log/swift/proxy.log
local1.notice /var/log/swift/proxy.error
local1.* ~
local2.*;local2.!notice /var/log/swift/storage1.log
local2.notice /var/log/swift/storage1.error
local2.* ~
local3.*;local3.!notice /var/log/swift/storage2.log
local3.notice /var/log/swift/storage2.error
local3.* ~
local4.*;local4.!notice /var/log/swift/storage3.log
local4.notice /var/log/swift/storage3.error
local4.* ~
local5.*;local5.!notice /var/log/swift/storage4.log
local5.notice /var/log/swift/storage4.error
local5.* ~
#. Edit /etc/rsyslog.conf and make the following change::
$PrivDropToGroup adm
#. `mkdir -p /var/log/swift/hourly`
#. On Ubuntu:
#. `chown -R syslog.adm /var/log/swift`
#. `chmod -R g+w /var/log/swift`
#. `service rsyslog restart`
#. On Fedora:
#. `chown -R root:adm /var/log/swift`
#. `chmod -R g+w /var/log/swift`
#. `systemctl restart rsyslog.service`
.. note::
Starting here, from this point on, all instructions are expected to be
performed as the unprivledged user you selected as <your-user-name>.
----------------
Getting the code
----------------
#. Check out the python-swiftclient repo
`git clone https://github.com/openstack/python-swiftclient.git`
#. Build a development installation of python-swiftclient
`cd ~/python-swiftclient; sudo python setup.py develop; cd -`
#. Check out the swift repo
`git clone https://github.com/openstack/swift.git`
#. Build a development installation of swift
`cd ~/swift; sudo python setup.py develop; cd -`
#. Install swift's test dependencies
`sudo pip install -r swift/test-requirements.txt`
Fedora 19 or later users might have to perform the following if development
installation of swift fails::
sudo pip install -U xattr dnspython
---------------------
Configuring each node
---------------------
Sample configuration files are provided with all defaults in line-by-line comments.
#. Create `/etc/swift/proxy-server.conf`::
[DEFAULT]
bind_port = 8080
workers = 1
user = <your-user-name>
log_facility = LOG_LOCAL1
eventlet_debug = true
[pipeline:main]
# Yes, proxy-logging appears twice. This is not a mistake.
pipeline = healthcheck proxy-logging cache tempauth proxy-logging proxy-server
[app:proxy-server]
use = egg:swift#proxy
allow_account_management = true
account_autocreate = true
[filter:tempauth]
use = egg:swift#tempauth
user_admin_admin = admin .admin .reseller_admin
user_test_tester = testing .admin
user_test2_tester2 = testing2 .admin
user_test_tester3 = testing3
[filter:healthcheck]
use = egg:swift#healthcheck
[filter:cache]
use = egg:swift#memcache
[filter:proxy-logging]
use = egg:swift#proxy_logging
#. Create `/etc/swift/swift.conf`::
[swift-hash]
# random unique strings that can never change (DO NOT LOSE)
swift_hash_path_prefix = changeme
swift_hash_path_suffix = changeme
#. Create `/etc/swift/account-server/1.conf`::
[DEFAULT]
devices = /srv/1/node
mount_check = false
disable_fallocate = true
bind_port = 6012
workers = 1
user = <your-user-name>
log_facility = LOG_LOCAL2
recon_cache_path = /var/cache/swift
eventlet_debug = true
[pipeline:main]
pipeline = recon account-server
[app:account-server]
use = egg:swift#account
[filter:recon]
use = egg:swift#recon
[account-replicator]
vm_test_mode = yes
[account-auditor]
[account-reaper]
#. Create `/etc/swift/account-server/2.conf`::
[DEFAULT]
devices = /srv/2/node
mount_check = false
disable_fallocate = true
bind_port = 6022
workers = 1
user = <your-user-name>
log_facility = LOG_LOCAL3
recon_cache_path = /var/cache/swift2
eventlet_debug = true
[pipeline:main]
pipeline = recon account-server
[app:account-server]
use = egg:swift#account
[filter:recon]
use = egg:swift#recon
[account-replicator]
vm_test_mode = yes
[account-auditor]
[account-reaper]
#. Create `/etc/swift/account-server/3.conf`::
[DEFAULT]
devices = /srv/3/node
mount_check = false
disable_fallocate = true
bind_port = 6032
workers = 1
user = <your-user-name>
log_facility = LOG_LOCAL4
recon_cache_path = /var/cache/swift3
eventlet_debug = true
[pipeline:main]
pipeline = recon account-server
[app:account-server]
use = egg:swift#account
[filter:recon]
use = egg:swift#recon
[account-replicator]
vm_test_mode = yes
[account-auditor]
[account-reaper]
#. Create `/etc/swift/account-server/4.conf`::
[DEFAULT]
devices = /srv/4/node
mount_check = false
disable_fallocate = true
bind_port = 6042
workers = 1
user = <your-user-name>
log_facility = LOG_LOCAL5
recon_cache_path = /var/cache/swift4
eventlet_debug = true
[pipeline:main]
pipeline = recon account-server
[app:account-server]
use = egg:swift#account
[filter:recon]
use = egg:swift#recon
[account-replicator]
vm_test_mode = yes
[account-auditor]
[account-reaper]
#. Create `/etc/swift/container-server/1.conf`::
[DEFAULT]
devices = /srv/1/node
mount_check = false
disable_fallocate = true
bind_port = 6011
workers = 1
user = <your-user-name>
log_facility = LOG_LOCAL2
recon_cache_path = /var/cache/swift
eventlet_debug = true
[pipeline:main]
pipeline = recon container-server
[app:container-server]
use = egg:swift#container
[filter:recon]
use = egg:swift#recon
[container-replicator]
vm_test_mode = yes
[container-updater]
[container-auditor]
[container-sync]
#. Create `/etc/swift/container-server/2.conf`::
[DEFAULT]
devices = /srv/2/node
mount_check = false
disable_fallocate = true
bind_port = 6021
workers = 1
user = <your-user-name>
log_facility = LOG_LOCAL3
recon_cache_path = /var/cache/swift2
eventlet_debug = true
[pipeline:main]
pipeline = recon container-server
[app:container-server]
use = egg:swift#container
[filter:recon]
use = egg:swift#recon
[container-replicator]
vm_test_mode = yes
[container-updater]
[container-auditor]
[container-sync]
#. Create `/etc/swift/container-server/3.conf`::
[DEFAULT]
devices = /srv/3/node
mount_check = false
disable_fallocate = true
bind_port = 6031
workers = 1
user = <your-user-name>
log_facility = LOG_LOCAL4
recon_cache_path = /var/cache/swift3
eventlet_debug = true
[pipeline:main]
pipeline = recon container-server
[app:container-server]
use = egg:swift#container
[filter:recon]
use = egg:swift#recon
[container-replicator]
vm_test_mode = yes
[container-updater]
[container-auditor]
[container-sync]
#. Create `/etc/swift/container-server/4.conf`::
[DEFAULT]
devices = /srv/4/node
mount_check = false
disable_fallocate = true
bind_port = 6041
workers = 1
user = <your-user-name>
log_facility = LOG_LOCAL5
recon_cache_path = /var/cache/swift4
eventlet_debug = true
[pipeline:main]
pipeline = recon container-server
[app:container-server]
use = egg:swift#container
[filter:recon]
use = egg:swift#recon
[container-replicator]
vm_test_mode = yes
[container-updater]
[container-auditor]
[container-sync]
#. Create `/etc/swift/object-server/1.conf`::
[DEFAULT]
devices = /srv/1/node
mount_check = false
disable_fallocate = true
bind_port = 6010
workers = 1
user = <your-user-name>
log_facility = LOG_LOCAL2
recon_cache_path = /var/cache/swift
eventlet_debug = true
[pipeline:main]
pipeline = recon object-server
[app:object-server]
use = egg:swift#object
[filter:recon]
use = egg:swift#recon
[object-replicator]
vm_test_mode = yes
[object-updater]
[object-auditor]
#. Create `/etc/swift/object-server/2.conf`::
[DEFAULT]
devices = /srv/2/node
mount_check = false
disable_fallocate = true
bind_port = 6020
workers = 1
user = <your-user-name>
log_facility = LOG_LOCAL3
recon_cache_path = /var/cache/swift2
eventlet_debug = true
[pipeline:main]
pipeline = recon object-server
[app:object-server]
use = egg:swift#object
[filter:recon]
use = egg:swift#recon
[object-replicator]
vm_test_mode = yes
[object-updater]
[object-auditor]
#. Create `/etc/swift/object-server/3.conf`::
[DEFAULT]
devices = /srv/3/node
mount_check = false
disable_fallocate = true
bind_port = 6030
workers = 1
user = <your-user-name>
log_facility = LOG_LOCAL4
recon_cache_path = /var/cache/swift3
eventlet_debug = true
[pipeline:main]
pipeline = recon object-server
[app:object-server]
use = egg:swift#object
[filter:recon]
use = egg:swift#recon
[object-replicator]
vm_test_mode = yes
[object-updater]
[object-auditor]
#. Create `/etc/swift/object-server/4.conf`::
[DEFAULT]
devices = /srv/4/node
mount_check = false
disable_fallocate = true
bind_port = 6040
workers = 1
user = <your-user-name>
log_facility = LOG_LOCAL5
recon_cache_path = /var/cache/swift4
eventlet_debug = true
[pipeline:main]
pipeline = recon object-server
[app:object-server]
use = egg:swift#object
[filter:recon]
use = egg:swift#recon
[object-replicator]
vm_test_mode = yes
[object-updater]
[object-auditor]
#. Update <your-user-name>::
find /etc/swift/ -name \*.conf | xargs sed -i "s/<your-user-name>/${USER}/"
------------------------------------
Setting up scripts for running Swift
------------------------------------
#. `mkdir ~/bin`
#. Create `~/bin/resetswift`.
If you are using a loopback device substitute `/dev/sdb1` with
`/srv/swift-disk` in the `mkfs` step.
If you did not set up rsyslog for individual logging, remove the `find
/var/log/swift...` line.
On Fedora, replace `service `<name>` restart` with `systemctl restart `<name>`.service`::
#!/bin/bash
swift-init all stop
find /var/log/swift -type f -exec rm -f {} \;
sudo umount /mnt/sdb1
sudo mkfs.xfs -f /dev/sdb1
sudo mount /mnt/sdb1
sudo mkdir /mnt/sdb1/1 /mnt/sdb1/2 /mnt/sdb1/3 /mnt/sdb1/4
sudo chown <your-user-name>:<your-group-name> /mnt/sdb1/*
mkdir -p /srv/1/node/sdb1 /srv/2/node/sdb2 /srv/3/node/sdb3 /srv/4/node/sdb4
sudo rm -f /var/log/debug /var/log/messages /var/log/rsyncd.log /var/log/syslog
find /var/cache/swift* -type f -name *.recon -exec rm -f {} \;
sudo service rsyslog restart
sudo service memcached restart
#. Create `~/bin/remakerings`::
#!/bin/bash
cd /etc/swift
rm -f *.builder *.ring.gz backups/*.builder backups/*.ring.gz
swift-ring-builder object.builder create 10 3 1
swift-ring-builder object.builder add r1z1-127.0.0.1:6010/sdb1 1
swift-ring-builder object.builder add r1z2-127.0.0.1:6020/sdb2 1
swift-ring-builder object.builder add r1z3-127.0.0.1:6030/sdb3 1
swift-ring-builder object.builder add r1z4-127.0.0.1:6040/sdb4 1
swift-ring-builder object.builder rebalance
swift-ring-builder container.builder create 10 3 1
swift-ring-builder container.builder add r1z1-127.0.0.1:6011/sdb1 1
swift-ring-builder container.builder add r1z2-127.0.0.1:6021/sdb2 1
swift-ring-builder container.builder add r1z3-127.0.0.1:6031/sdb3 1
swift-ring-builder container.builder add r1z4-127.0.0.1:6041/sdb4 1
swift-ring-builder container.builder rebalance
swift-ring-builder account.builder create 10 3 1
swift-ring-builder account.builder add r1z1-127.0.0.1:6012/sdb1 1
swift-ring-builder account.builder add r1z2-127.0.0.1:6022/sdb2 1
swift-ring-builder account.builder add r1z3-127.0.0.1:6032/sdb3 1
swift-ring-builder account.builder add r1z4-127.0.0.1:6042/sdb4 1
swift-ring-builder account.builder rebalance
#. Create `~/bin/startmain`::
#!/bin/bash
swift-init main start
#. Create `~/bin/startrest`::
#!/bin/bash
swift-init rest start
#. `chmod +x ~/bin/*`
#. Edit `~/.bashrc` and add to the end::
export SWIFT_TEST_CONFIG_FILE=/etc/swift/test.conf
export PATH=${PATH}:~/bin
#. `. ~/.bashrc`
#. `remakerings`
#. `cp ~/swift/test/sample.conf /etc/swift/test.conf`
#. `~/swift/.unittests`
#. `startmain` (The ``Unable to increase file descriptor limit. Running as non-root?`` warnings are expected and ok.)
#. Get an `X-Storage-Url` and `X-Auth-Token`: ``curl -v -H 'X-Storage-User: test:tester' -H 'X-Storage-Pass: testing' http://127.0.0.1:8080/auth/v1.0``
#. Check that you can GET account: ``curl -v -H 'X-Auth-Token: <token-from-x-auth-token-above>' <url-from-x-storage-url-above>``
#. Check that `swift` works: `swift -A http://127.0.0.1:8080/auth/v1.0 -U test:tester -K testing stat`
#. `~/swift/.functests` (Note: functional tests will first delete
everything in the configured accounts.)
#. `~/swift/.probetests` (Note: probe tests will reset your
environment as they call `resetswift` for each test.)
----------------
Debugging Issues
----------------
If all doesn't go as planned, and tests fail, or you can't auth, or something doesn't work, here are some good starting places to look for issues:
#. Everything is logged using system facilities -- usually in /var/log/syslog,
but possibly in /var/log/messages on e.g. Fedora -- so that is a good first
place to look for errors (most likely python tracebacks).
#. Make sure all of the server processes are running. For the base
functionality, the Proxy, Account, Container, and Object servers
should be running.
#. If one of the servers are not running, and no errors are logged to syslog,
it may be useful to try to start the server manually, for example:
`swift-object-server /etc/swift/object-server/1.conf` will start the
object server. If there are problems not showing up in syslog,
then you will likely see the traceback on startup.
#. If you need to, you can turn off syslog for unit tests. This can be
useful for environments where /dev/log is unavailable, or which
cannot rate limit (unit tests generate a lot of logs very quickly).
Open the file SWIFT_TEST_CONFIG_FILE points to, and change the
value of fake_syslog to True.
View Git Blame Copy Permalink