Merge "initial work to enable systemd service running"
This commit is contained in:
Jenkins
Gerrit Code Review
commit
d7bc9486aa
177
SYSTEMD.rst
Normal file
177
SYSTEMD.rst
Normal file
@ -0,0 +1,177 @@
|
||||
===========================
|
||||
Using Systemd in DevStack
|
||||
===========================
|
||||
|
||||
.. note::
|
||||
|
||||
This is an in progress document as we work out the way forward here
|
||||
with DevStack and systemd.
|
||||
|
||||
DevStack can be run with all the services as systemd unit
|
||||
files. Systemd is now the default init system for nearly every Linux
|
||||
distro, and systemd encodes and solves many of the problems related to
|
||||
poorly running processes.
|
||||
|
||||
Why this instead of screen?
|
||||
===========================
|
||||
|
||||
The screen model for DevStack was invented when the number of services
|
||||
that a DevStack user was going to run was typically < 10. This made
|
||||
screen hot keys to jump around very easy. However, the landscape has
|
||||
changed (not all services are stoppable in screen as some are under
|
||||
Apache, there are typically at least 20 items)
|
||||
|
||||
There is also a common developer workflow of changing code in more
|
||||
than one service, and needing to restart a bunch of services for that
|
||||
to take effect.
|
||||
|
||||
To enable this add the following to your local.conf::
|
||||
|
||||
USE_SYSTEMD=True
|
||||
|
||||
|
||||
|
||||
Unit Structure
|
||||
==============
|
||||
|
||||
.. note::
|
||||
|
||||
Originally we actually wanted to do this as user units, however
|
||||
there are issues with running this under non interactive
|
||||
shells. For now, we'll be running as system units. Some user unit
|
||||
code is left in place in case we can switch back later.
|
||||
|
||||
All DevStack user units are created as a part of the DevStack slice
|
||||
given the name ``devstack@$servicename.service``. This lets us do
|
||||
certain operations at the slice level.
|
||||
|
||||
Manipulating Units
|
||||
==================
|
||||
|
||||
Assuming the unit ``n-cpu`` to make the examples more clear.
|
||||
|
||||
Enable a unit (allows it to be started)::
|
||||
|
||||
sudo systemctl enable devstack@n-cpu.service
|
||||
|
||||
Disable a unit::
|
||||
|
||||
sudo systemctl disable devstack@n-cpu.service
|
||||
|
||||
Start a unit::
|
||||
|
||||
sudo systemctl start devstack@n-cpu.service
|
||||
|
||||
Stop a unit::
|
||||
|
||||
sudo systemctl stop devstack@n-cpu.service
|
||||
|
||||
Restart a unit::
|
||||
|
||||
sudo systemctl restart devstack@n-cpu.service
|
||||
|
||||
See status of a unit::
|
||||
|
||||
sudo systemctl status devstack@n-cpu.service
|
||||
|
||||
|
||||
Querying Logs
|
||||
=============
|
||||
|
||||
One of the other major things that comes with systemd is journald, a
|
||||
consolidated way to access logs (including querying through structured
|
||||
metadata). This is accessed by the user via ``journalctl`` command.
|
||||
|
||||
|
||||
Logs can be accessed through ``journalctl``. journalctl has powerful
|
||||
query facilities. We'll start with some common options.
|
||||
|
||||
Follow logs for a specific service::
|
||||
|
||||
journalctl -f --unit devstack@n-cpu.service
|
||||
|
||||
Following logs for multiple services simultaneously::
|
||||
|
||||
journalctl -f --unit devstack@n-cpu.service --user-unit
|
||||
devstack@n-cond.service
|
||||
|
||||
Use higher precision time stamps::
|
||||
|
||||
journalctl -f -o short-precise --unit devstack@n-cpu.service
|
||||
|
||||
|
||||
Known Issues
|
||||
============
|
||||
|
||||
Be careful about systemd python libraries. There are 3 of them on
|
||||
pypi, and they are all very different. They unfortunately all install
|
||||
into the ``systemd`` namespace, which can cause some issues.
|
||||
|
||||
- ``systemd-python`` - this is the upstream maintained library, it has
|
||||
a version number like systemd itself (currently ``233``). This is
|
||||
the one you want.
|
||||
- ``systemd`` - a python 3 only library, not what you want.
|
||||
- ``python-systemd`` - another library you don't want. Installing it
|
||||
on a system will break ansible's ability to run.
|
||||
|
||||
|
||||
If we were using user units, the ``[Service]`` - ``Group=`` parameter
|
||||
doesn't seem to work with user units, even though the documentation
|
||||
says that it should. This means that we will need to do an explicit
|
||||
``/usr/bin/sg``. This has the downside of making the SYSLOG_IDENTIFIER
|
||||
be ``sg``. We can explicitly set that with ``SyslogIdentifier=``, but
|
||||
it's really unfortunate that we're going to need this work
|
||||
around. This is currently not a problem because we're only using
|
||||
system units.
|
||||
|
||||
Future Work
|
||||
===========
|
||||
|
||||
oslo.log journald
|
||||
-----------------
|
||||
|
||||
Journald has an extremely rich mechanism for direct logging including
|
||||
structured metadata. We should enhance oslo.log to take advantage of
|
||||
that. It would let us do things like::
|
||||
|
||||
journalctl REQUEST_ID=......
|
||||
|
||||
journalctl INSTANCE_ID=......
|
||||
|
||||
And get all lines related to the request id or instance id.
|
||||
|
||||
sub targets/slices
|
||||
------------------
|
||||
|
||||
We might want to create per project slices so that it's easy to
|
||||
follow, restart all services of a single project (like swift) without
|
||||
impacting other services.
|
||||
|
||||
log colorizing
|
||||
--------------
|
||||
|
||||
We lose log colorization through this process. We might want to build
|
||||
a custom colorizer that we could run journalctl output through
|
||||
optionally for people.
|
||||
|
||||
user units
|
||||
----------
|
||||
|
||||
It would be great if we could do services as user units, so that there
|
||||
is a clear separation of code being run as not root, to ensure running
|
||||
as root never accidentally gets baked in as an assumption to
|
||||
services. However, user units interact poorly with devstack-gate and
|
||||
the way that commands are run as users with ansible and su.
|
||||
|
||||
Maybe someday we can figure that out.
|
||||
|
||||
References
|
||||
==========
|
||||
|
||||
- Arch Linux Wiki - https://wiki.archlinux.org/index.php/Systemd/User
|
||||
- Python interface to journald -
|
||||
https://www.freedesktop.org/software/systemd/python-systemd/journal.html
|
||||
- Systemd documentation on service files -
|
||||
https://www.freedesktop.org/software/systemd/man/systemd.service.html
|
||||
- Systemd documentation on exec (can be used to impact service runs) -
|
||||
https://www.freedesktop.org/software/systemd/man/systemd.exec.html
|
||||
15
functions
15
functions
@ -575,7 +575,9 @@ function vercmp {
|
||||
function setup_logging {
|
||||
local conf_file=$1
|
||||
local other_cond=${2:-"False"}
|
||||
if [ "$LOG_COLOR" == "True" ] && [ "$SYSLOG" == "False" ] && [ "$other_cond" == "False" ]; then
|
||||
if [[ "$USE_SYSTEMD" == "True" ]]; then
|
||||
setup_systemd_logging $conf_file
|
||||
elif [ "$LOG_COLOR" == "True" ] && [ "$SYSLOG" == "False" ] && [ "$other_cond" == "False" ]; then
|
||||
setup_colorized_logging $conf_file
|
||||
else
|
||||
setup_standard_logging_identity $conf_file
|
||||
@ -601,6 +603,17 @@ function setup_colorized_logging {
|
||||
iniset $conf_file $conf_section logging_exception_prefix "%(color)s%(asctime)s.%(msecs)03d TRACE %(name)s [01;35m%(instance)s[00m"
|
||||
}
|
||||
|
||||
function setup_systemd_logging {
|
||||
local conf_file=$1
|
||||
local conf_section="DEFAULT"
|
||||
local project_var="project_name"
|
||||
local user_var="user_name"
|
||||
iniset $conf_file $conf_section logging_context_format_string "%(levelname)s %(name)s [%(request_id)s %("$project_var")s %("$user_var")s] %(instance)s%(message)s"
|
||||
iniset $conf_file $conf_section logging_default_format_string "%(levelname)s %(name)s [-] %(instance)s%(color)s%(message)s"
|
||||
iniset $conf_file $conf_section logging_debug_format_suffix "from (pid=%(process)d) %(funcName)s %(pathname)s:%(lineno)d"
|
||||
iniset $conf_file $conf_section logging_exception_prefix "ERROR %(name)s %(instance)s"
|
||||
}
|
||||
|
||||
function setup_standard_logging_identity {
|
||||
local conf_file=$1
|
||||
iniset $conf_file DEFAULT logging_user_identity_format "%(project_name)s %(user_name)s"
|
||||
|
||||
@ -1443,6 +1443,59 @@ function _run_process {
|
||||
exit 0
|
||||
}
|
||||
|
||||
function write_user_unit_file {
|
||||
local service=$1
|
||||
local command="$2"
|
||||
local group=$3
|
||||
local user=$4
|
||||
local extra=""
|
||||
if [[ -n "$group" ]]; then
|
||||
extra="Group=$group"
|
||||
fi
|
||||
local unitfile="$SYSTEMD_DIR/$service"
|
||||
mkdir -p $SYSTEMD_DIR
|
||||
|
||||
iniset -sudo $unitfile "Unit" "Description" "Devstack $service"
|
||||
iniset -sudo $unitfile "Service" "User" "$user"
|
||||
iniset -sudo $unitfile "Service" "ExecStart" "$command"
|
||||
if [[ -n "$group" ]]; then
|
||||
iniset -sudo $unitfile "Service" "Group" "$group"
|
||||
fi
|
||||
iniset -sudo $unitfile "Install" "WantedBy" "multi-user.target"
|
||||
|
||||
# changes to existing units sometimes need a refresh
|
||||
$SYSTEMCTL daemon-reload
|
||||
}
|
||||
|
||||
function _run_under_systemd {
|
||||
local service=$1
|
||||
local command="$2"
|
||||
local cmd=$command
|
||||
local systemd_service="devstack@$service.service"
|
||||
local group=$3
|
||||
local user=${4:-$STACK_USER}
|
||||
write_user_unit_file $systemd_service "$cmd" "$group" "$user"
|
||||
|
||||
$SYSTEMCTL enable $systemd_service
|
||||
$SYSTEMCTL start $systemd_service
|
||||
_journal_log $service $systemd_service
|
||||
}
|
||||
|
||||
function _journal_log {
|
||||
local service=$1
|
||||
local unit=$2
|
||||
local logfile="${service}.log.${CURRENT_LOG_TIME}"
|
||||
local real_logfile="${LOGDIR}/${logfile}"
|
||||
if [[ -n ${LOGDIR} ]]; then
|
||||
$JOURNALCTL_F $2 > "$real_logfile" &
|
||||
bash -c "cd '$LOGDIR' && ln -sf '$logfile' ${service}.log"
|
||||
if [[ -n ${SCREEN_LOGDIR} ]]; then
|
||||
# Drop the backward-compat symlink
|
||||
ln -sf "$real_logfile" ${SCREEN_LOGDIR}/screen-${service}.log
|
||||
fi
|
||||
fi
|
||||
}
|
||||
|
||||
# Helper to remove the ``*.failure`` files under ``$SERVICE_DIR/$SCREEN_NAME``.
|
||||
# This is used for ``service_check`` when all the ``screen_it`` are called finished
|
||||
# Uses globals ``SCREEN_NAME``, ``SERVICE_DIR``
|
||||
@ -1478,16 +1531,24 @@ function run_process {
|
||||
local service=$1
|
||||
local command="$2"
|
||||
local group=$3
|
||||
local subservice=$4
|
||||
local user=$4
|
||||
|
||||
local name=${subservice:-$service}
|
||||
local name=$service
|
||||
|
||||
time_start "run_process"
|
||||
if is_service_enabled $service; then
|
||||
if [[ "$USE_SCREEN" = "True" ]]; then
|
||||
if [[ "$USE_SYSTEMD" = "True" ]]; then
|
||||
_run_under_systemd "$name" "$command" "$group" "$user"
|
||||
elif [[ "$USE_SCREEN" = "True" ]]; then
|
||||
if [[ "$user" == "root" ]]; then
|
||||
command="sudo $command"
|
||||
fi
|
||||
screen_process "$name" "$command" "$group"
|
||||
else
|
||||
# Spawn directly without screen
|
||||
if [[ "$user" == "root" ]]; then
|
||||
command="sudo $command"
|
||||
fi
|
||||
_run_process "$name" "$command" "$group" &
|
||||
fi
|
||||
fi
|
||||
@ -1618,6 +1679,14 @@ function stop_process {
|
||||
|
||||
if is_service_enabled $service; then
|
||||
# Kill via pid if we have one available
|
||||
if [[ "$USE_SYSTEMD" == "True" ]]; then
|
||||
# Only do this for units which appear enabled, this also
|
||||
# catches units that don't really exist for cases like
|
||||
# keystone without a failure.
|
||||
$SYSTEMCTL stop devstack@$service.service
|
||||
$SYSTEMCTL disable devstack@$service.service
|
||||
fi
|
||||
|
||||
if [[ -r $SERVICE_DIR/$SCREEN_NAME/$service.pid ]]; then
|
||||
pkill -g $(cat $SERVICE_DIR/$SCREEN_NAME/$service.pid)
|
||||
# oslo.service tends to stop actually shutting down
|
||||
|
||||
@ -24,12 +24,12 @@ function start_dstat {
|
||||
# To enable memory_tracker add:
|
||||
# enable_service memory_tracker
|
||||
# to your localrc
|
||||
run_process memory_tracker "sudo $TOP_DIR/tools/memory_tracker.sh"
|
||||
run_process memory_tracker "$TOP_DIR/tools/memory_tracker.sh" "" "root"
|
||||
|
||||
# remove support for the old name when it's no longer used (sometime in Queens)
|
||||
if is_service_enabled peakmem_tracker; then
|
||||
deprecated "Use of peakmem_tracker in devstack is deprecated, use memory_tracker instead"
|
||||
run_process peakmem_tracker "sudo $TOP_DIR/tools/memory_tracker.sh"
|
||||
run_process peakmem_tracker "$TOP_DIR/tools/memory_tracker.sh" "" "root"
|
||||
fi
|
||||
}
|
||||
|
||||
|
||||
@ -602,8 +602,11 @@ function start_keystone {
|
||||
tail_log key /var/log/$APACHE_NAME/keystone.log
|
||||
tail_log key-access /var/log/$APACHE_NAME/keystone_access.log
|
||||
else # uwsgi
|
||||
run_process key "$KEYSTONE_BIN_DIR/uwsgi $KEYSTONE_PUBLIC_UWSGI_FILE" "" "key-p"
|
||||
run_process key "$KEYSTONE_BIN_DIR/uwsgi $KEYSTONE_ADMIN_UWSGI_FILE" "" "key-a"
|
||||
# TODO(sdague): we should really get down to a single keystone here
|
||||
enable_service key-p
|
||||
enable_service key-a
|
||||
run_process key-p "$KEYSTONE_BIN_DIR/uwsgi $KEYSTONE_PUBLIC_UWSGI_FILE" ""
|
||||
run_process key-a "$KEYSTONE_BIN_DIR/uwsgi $KEYSTONE_ADMIN_UWSGI_FILE" ""
|
||||
fi
|
||||
|
||||
echo "Waiting for keystone to start..."
|
||||
|
||||
13
lib/swift
13
lib/swift
@ -38,6 +38,15 @@ fi
|
||||
# Set up default directories
|
||||
GITDIR["python-swiftclient"]=$DEST/python-swiftclient
|
||||
|
||||
# Swift virtual environment
|
||||
if [[ ${USE_VENV} = True ]]; then
|
||||
PROJECT_VENV["swift"]=${SWIFT_DIR}.venv
|
||||
SWIFT_BIN_DIR=${PROJECT_VENV["swift"]}/bin
|
||||
else
|
||||
SWIFT_BIN_DIR=$(get_python_exec_prefix)
|
||||
fi
|
||||
|
||||
|
||||
SWIFT_DIR=$DEST/swift
|
||||
SWIFT_AUTH_CACHE_DIR=${SWIFT_AUTH_CACHE_DIR:-/var/cache/swift}
|
||||
SWIFT_APACHE_WSGI_DIR=${SWIFT_APACHE_WSGI_DIR:-/var/www/swift}
|
||||
@ -807,10 +816,10 @@ function start_swift {
|
||||
local proxy_port=${SWIFT_DEFAULT_BIND_PORT}
|
||||
start_tls_proxy swift '*' $proxy_port $SERVICE_HOST $SWIFT_DEFAULT_BIND_PORT_INT
|
||||
fi
|
||||
run_process s-proxy "swift-proxy-server ${SWIFT_CONF_DIR}/proxy-server.conf -v"
|
||||
run_process s-proxy "$SWIFT_BIN_DIR/swift-proxy-server ${SWIFT_CONF_DIR}/proxy-server.conf -v"
|
||||
if [[ ${SWIFT_REPLICAS} == 1 ]]; then
|
||||
for type in object container account; do
|
||||
run_process s-${type} "swift-${type}-server ${SWIFT_CONF_DIR}/${type}-server/1.conf -v"
|
||||
run_process s-${type} "$SWIFT_BIN_DIR/swift-${type}-server ${SWIFT_CONF_DIR}/${type}-server/1.conf -v"
|
||||
done
|
||||
fi
|
||||
|
||||
|
||||
17
stackrc
17
stackrc
@ -87,6 +87,23 @@ HORIZON_APACHE_ROOT="/dashboard"
|
||||
# be disabled for automated testing by setting this value to False.
|
||||
USE_SCREEN=$(trueorfalse True USE_SCREEN)
|
||||
|
||||
# Whether to use SYSTEMD to manage services
|
||||
USE_SYSTEMD=$(trueorfalse False USE_SYSTEMD)
|
||||
USER_UNITS=$(trueorfalse False USER_UNITS)
|
||||
if [[ "$USER_UNITS" == "True" ]]; then
|
||||
SYSTEMD_DIR="$HOME/.local/share/systemd/user"
|
||||
SYSTEMCTL="systemctl --user"
|
||||
JOURNALCTL_F="journalctl -f -o short-precise --user-unit"
|
||||
else
|
||||
SYSTEMD_DIR="/etc/systemd/system"
|
||||
SYSTEMCTL="sudo systemctl"
|
||||
JOURNALCTL_F="journalctl -f -o short-precise --unit"
|
||||
fi
|
||||
|
||||
if [[ "$USE_SYSTEMD" == "True" ]]; then
|
||||
USE_SCREEN=False
|
||||
fi
|
||||
|
||||
# When using screen, should we keep a log file on disk? You might
|
||||
# want this False if you have a long-running setup where verbose logs
|
||||
# can fill-up the host.
|
||||
|
||||
Loading…
Reference in New Issue
Block a user