190 lines
5.7 KiB
ReStructuredText
190 lines
5.7 KiB
ReStructuredText
===========================
|
|
Using Systemd in DevStack
|
|
===========================
|
|
|
|
.. note::
|
|
|
|
This is an in progress document as we work out the way forward here
|
|
with DevStack and systemd.
|
|
|
|
DevStack can be run with all the services as systemd unit
|
|
files. Systemd is now the default init system for nearly every Linux
|
|
distro, and systemd encodes and solves many of the problems related to
|
|
poorly running processes.
|
|
|
|
Why this instead of screen?
|
|
===========================
|
|
|
|
The screen model for DevStack was invented when the number of services
|
|
that a DevStack user was going to run was typically < 10. This made
|
|
screen hot keys to jump around very easy. However, the landscape has
|
|
changed (not all services are stoppable in screen as some are under
|
|
Apache, there are typically at least 20 items)
|
|
|
|
There is also a common developer workflow of changing code in more
|
|
than one service, and needing to restart a bunch of services for that
|
|
to take effect.
|
|
|
|
To enable this add the following to your local.conf::
|
|
|
|
USE_SYSTEMD=True
|
|
|
|
|
|
|
|
Unit Structure
|
|
==============
|
|
|
|
.. note::
|
|
|
|
Originally we actually wanted to do this as user units, however
|
|
there are issues with running this under non interactive
|
|
shells. For now, we'll be running as system units. Some user unit
|
|
code is left in place in case we can switch back later.
|
|
|
|
All DevStack user units are created as a part of the DevStack slice
|
|
given the name ``devstack@$servicename.service``. This lets us do
|
|
certain operations at the slice level.
|
|
|
|
Manipulating Units
|
|
==================
|
|
|
|
Assuming the unit ``n-cpu`` to make the examples more clear.
|
|
|
|
Enable a unit (allows it to be started)::
|
|
|
|
sudo systemctl enable devstack@n-cpu.service
|
|
|
|
Disable a unit::
|
|
|
|
sudo systemctl disable devstack@n-cpu.service
|
|
|
|
Start a unit::
|
|
|
|
sudo systemctl start devstack@n-cpu.service
|
|
|
|
Stop a unit::
|
|
|
|
sudo systemctl stop devstack@n-cpu.service
|
|
|
|
Restart a unit::
|
|
|
|
sudo systemctl restart devstack@n-cpu.service
|
|
|
|
See status of a unit::
|
|
|
|
sudo systemctl status devstack@n-cpu.service
|
|
|
|
Operating on more than one unit at a time
|
|
-----------------------------------------
|
|
|
|
Systemd supports wildcarding for unit operations. To restart every
|
|
service in devstack you can do that following::
|
|
|
|
sudo systemctl restart devstack@*
|
|
|
|
Or to see the status of all Nova processes you can do::
|
|
|
|
sudo systemctl status devstack@n-*
|
|
|
|
We'll eventually make the unit names a bit more meaningful so that
|
|
it's easier to understand what you are restarting.
|
|
|
|
Querying Logs
|
|
=============
|
|
|
|
One of the other major things that comes with systemd is journald, a
|
|
consolidated way to access logs (including querying through structured
|
|
metadata). This is accessed by the user via ``journalctl`` command.
|
|
|
|
|
|
Logs can be accessed through ``journalctl``. journalctl has powerful
|
|
query facilities. We'll start with some common options.
|
|
|
|
Follow logs for a specific service::
|
|
|
|
journalctl -f --unit devstack@n-cpu.service
|
|
|
|
Following logs for multiple services simultaneously::
|
|
|
|
journalctl -f --unit devstack@n-cpu.service --unit
|
|
devstack@n-cond.service
|
|
|
|
or you can even do wild cards to follow all the nova services::
|
|
|
|
journalctl -f --unit devstack@n-*
|
|
|
|
Use higher precision time stamps::
|
|
|
|
journalctl -f -o short-precise --unit devstack@n-cpu.service
|
|
|
|
|
|
Known Issues
|
|
============
|
|
|
|
Be careful about systemd python libraries. There are 3 of them on
|
|
pypi, and they are all very different. They unfortunately all install
|
|
into the ``systemd`` namespace, which can cause some issues.
|
|
|
|
- ``systemd-python`` - this is the upstream maintained library, it has
|
|
a version number like systemd itself (currently ``233``). This is
|
|
the one you want.
|
|
- ``systemd`` - a python 3 only library, not what you want.
|
|
- ``python-systemd`` - another library you don't want. Installing it
|
|
on a system will break ansible's ability to run.
|
|
|
|
|
|
If we were using user units, the ``[Service]`` - ``Group=`` parameter
|
|
doesn't seem to work with user units, even though the documentation
|
|
says that it should. This means that we will need to do an explicit
|
|
``/usr/bin/sg``. This has the downside of making the SYSLOG_IDENTIFIER
|
|
be ``sg``. We can explicitly set that with ``SyslogIdentifier=``, but
|
|
it's really unfortunate that we're going to need this work
|
|
around. This is currently not a problem because we're only using
|
|
system units.
|
|
|
|
Future Work
|
|
===========
|
|
|
|
oslo.log journald
|
|
-----------------
|
|
|
|
Journald has an extremely rich mechanism for direct logging including
|
|
structured metadata. We should enhance oslo.log to take advantage of
|
|
that. It would let us do things like::
|
|
|
|
journalctl REQUEST_ID=......
|
|
|
|
journalctl INSTANCE_ID=......
|
|
|
|
And get all lines related to the request id or instance id. (Note:
|
|
this work has been started at https://review.openstack.org/#/c/451525/)
|
|
|
|
log colorizing
|
|
--------------
|
|
|
|
We lose log colorization through this process. We might want to build
|
|
a custom colorizer that we could run journalctl output through
|
|
optionally for people.
|
|
|
|
user units
|
|
----------
|
|
|
|
It would be great if we could do services as user units, so that there
|
|
is a clear separation of code being run as not root, to ensure running
|
|
as root never accidentally gets baked in as an assumption to
|
|
services. However, user units interact poorly with devstack-gate and
|
|
the way that commands are run as users with ansible and su.
|
|
|
|
Maybe someday we can figure that out.
|
|
|
|
References
|
|
==========
|
|
|
|
- Arch Linux Wiki - https://wiki.archlinux.org/index.php/Systemd/User
|
|
- Python interface to journald -
|
|
https://www.freedesktop.org/software/systemd/python-systemd/journal.html
|
|
- Systemd documentation on service files -
|
|
https://www.freedesktop.org/software/systemd/man/systemd.service.html
|
|
- Systemd documentation on exec (can be used to impact service runs) -
|
|
https://www.freedesktop.org/software/systemd/man/systemd.exec.html
|