311 lines
10 KiB
ReStructuredText
311 lines
10 KiB
ReStructuredText
======================
|
|
Rootwrap daemon mode
|
|
======================
|
|
|
|
https://blueprints.launchpad.net/oslo.rootwrap/+spec/rootwrap-daemon-mode
|
|
|
|
As it was pointed out several times on ML [#neu_ml]_ [#nova_ml]_ different
|
|
services (most notably Neutron and Nova) suffer from performance penalty of
|
|
having to run new instance of rootwrap executable for each call that needs root
|
|
privileges. [#nova_ml]_ ended basically with "Who's up to the task?" question.
|
|
|
|
Problem description
|
|
===================
|
|
|
|
The structure of this overhead has been analyzed in [#neu_ml]_. It's clear that
|
|
the main issue here is rootwrap startup time that consists of Python
|
|
interpreter startup and rootwrap config parsing.
|
|
|
|
Proposed change
|
|
===============
|
|
|
|
I propose creating a new mode of operation for rootwrap - daemon mode. In this
|
|
mode rootwrap would start, read config file and wait for commands to be run
|
|
with root privileges. Each service's process will have its own rootwrap daemon
|
|
process.
|
|
|
|
Daemon starting
|
|
---------------
|
|
|
|
Daemon will be started using a separate binary (like neutron-rootwrap-daemon
|
|
for Neutron) pointing to ``oslo.rootwrap.cmd:daemon`` endpoint (instead of
|
|
``:main``). The binary will receive the same options (config file) as the
|
|
normal one except the command to be run in priviledged mode. For example::
|
|
|
|
rootwrap-daemon /etc/myservice/rootwrap.conf
|
|
|
|
The startup process is the same as in normal mode up to the point when the
|
|
command is about to be run. In daemon mode a separate method is called instead
|
|
that starts RPC [#rpc]_ server and falls into infinite loop serving requests.
|
|
|
|
Daemon API
|
|
----------
|
|
|
|
Upon startup daemon will push to its stdout:
|
|
|
|
* path to UNIX domain socket (encoded in UTF-8);
|
|
* newline character ``\n``;
|
|
* 32-byte auth key.
|
|
|
|
These credentials can be used to connect a
|
|
``multiprocessing.BaseManager``-based client. Because ``pickle`` and
|
|
``xmlrpclib`` are unsafe, it uses its own JSON serialization (see `Under the
|
|
hood`_ section). The only exposed object is ``rootwrap`` with one method:
|
|
``run_one_command(userargs, env=None, stdin=None)``. Arguments are:
|
|
|
|
* ``userargs`` - list of command line arguments that are to be used to run the
|
|
command;
|
|
* ``env`` - dict of environment variables to be set for it (by default it's an
|
|
empty dict, so all environment variables are stripped);
|
|
* ``stdin`` - string to be passed to standard input of child process.
|
|
|
|
The method returns 3-tuple containing:
|
|
|
|
* return code of child process;
|
|
* string containing everything captured from its stdout stream;
|
|
* string containing everything captured from its stderr stream.
|
|
|
|
Here is a sketch of basic usage for rootwrap daemon::
|
|
|
|
>>> from subprocess import *
|
|
>>> from multiprocessing.managers import BaseManager
|
|
>>> process = Popen(["rootwrap-daemon", "rootwrap.conf"], stdout=PIPE)
|
|
>>> address = process.stdout.readline()[:-1].decode('utf-8')
|
|
>>> authkey = process.stdout.read(32)
|
|
>>> class MyManager(BaseManager): pass
|
|
...
|
|
>>> MyManager.register("rootwrap")
|
|
>>> from oslo.rootwrap import client # to set up 'jsonrpc' serializer only
|
|
>>> manager = MyManager(address, authkey, serializer='jsonrpc')
|
|
>>> manager.connect()
|
|
>>> proxy = manager.rootwrap()
|
|
>>> proxy.run_one_command(["cat"], stdin="Hello, world!")
|
|
[0, u'Hello, world!', u'']
|
|
>>> process.kill()
|
|
|
|
Note that this requires ``rootwrap-daemon`` pointing to
|
|
``oslo.rootwrap.cmd:daemon`` to be available in ``PATH``.
|
|
|
|
The ``run_one_command`` call returns a list here because of JSON serialization
|
|
but it doesn't change Python API usage.
|
|
|
|
Client API
|
|
----------
|
|
|
|
To simplify daemon usage a ``oslo.rootwrap.client`` module is provided
|
|
containing one class ``Client`` that wraps all necessary steps to work with
|
|
rootwrap daemon.
|
|
|
|
Its constructor expects one argument - a list that can be passed to ``Popen``
|
|
to create rootwrap daemon process. For Neutron it'll be
|
|
``["sudo", "neutron-rootwrap-daemon", "/etc/neutron/rootwrap.conf"]``
|
|
for example.
|
|
|
|
The class provides one method ``execute`` with the same profile as
|
|
``run_one_command`` method shown above.
|
|
|
|
The class lazily creates an instance of the daemon, connects to it and passes
|
|
arguments. Note that some reconnection and respawning mechanism will be in
|
|
place so that if daemon process dies or hangs, ``Client`` will detect this on
|
|
the next call and will simply restart it.
|
|
|
|
This laziness will allow user to kill all rootwrap daemon processes to reload
|
|
config file, for example.
|
|
|
|
Under the hood
|
|
--------------
|
|
|
|
The biggest expected security risk in this proposal is the way that client
|
|
talks to daemon so I'll discuss the underlying protocol in details.
|
|
|
|
#. Credentials passing
|
|
|
|
The credentials required to connect to daemon are passed to stdout stream
|
|
that is expected to be bound through a pipe directly to calling process.
|
|
They are exposed only to the kernel and calling process here.
|
|
|
|
#. Authentication
|
|
|
|
``multiprocessing`` involves digest authentication for every new connection
|
|
made to the server. The key is never passed over the socket, so we could
|
|
even use TCP socket here (no way). The key is generated using
|
|
``os.urandom(32)`` call.
|
|
|
|
#. Connection (non-)pooling
|
|
|
|
Managers use threadlocal connections so there's no need to create a pool of
|
|
them. Although it might sound wasteful to create new connection for every
|
|
thread, creating a connection over UNIX socket is almost nothing compared to
|
|
spawning a new process time-wise.
|
|
|
|
#. On-wire protocol
|
|
|
|
By default ``multiprocessing`` uses ``pickle`` to serialize RPC [#rpc]_
|
|
requests and responses but it's very unsafe as it allows to call any method
|
|
on the receiving side (see warning in [#pickle]_). The only other option is
|
|
to use ``xmlrpclib`` as a serializer but it's unsafe as it's prone to
|
|
resource exhaustion attacks [#xml_unsafe]_. That's why JSON serialization
|
|
support is implemented. It's very simple (~50 SLOC) and is safe because JSON
|
|
serialization is widely regarded as being safe.
|
|
|
|
This serialization is plugged in using undocumented features of
|
|
``multiprocessing.managers`` module:
|
|
|
|
* ``listener_client`` - dictionary of available serialization options with
|
|
strings as keys and pairs (listener, client) as values;
|
|
* ``serializer`` - argument of ``BaseManager.__init__`` constructor,
|
|
contains a key in ``listener_client`` dictionary.
|
|
|
|
The author of ``multiprocessing`` module assured that this mechanism is not
|
|
going to go away any time soon [#ser_bug]_.
|
|
|
|
.. note::
|
|
Although it's risky to rely on undocumented feature,
|
|
it's mitigated by the assurance of the stdlib module author.
|
|
|
|
Alternatives
|
|
------------
|
|
|
|
There are a number of alternative approaches to optimize number of rootwrap
|
|
calls to mitigate the overhead (see [#neu_eth]_). There were a number of
|
|
suggestions in original thread in mailing list [#neu_ml]_ and in corresponding
|
|
etherpad [#neu_eth]_:
|
|
|
|
* Scrap rootwrap, switch to sudo.
|
|
|
|
We'll lose current fine-grained control over what can be run as root.
|
|
|
|
* Use other interpretator for rootwrap.
|
|
|
|
Doesn't fix the interpretator startup cost.
|
|
|
|
* Rewrite rootwrap in other language.
|
|
|
|
This includes suggestions to rewrite entirely or partially in C or some
|
|
Python dialect that would be translateable to C.
|
|
|
|
As OpenStack community is focused on Python development, bringing some other
|
|
language to the field would require more developers that would be familiar
|
|
with it.
|
|
|
|
* Filter commands on the calling process side and use sudo.
|
|
|
|
This would provide the same security as the first option.
|
|
|
|
* Consolidate calls that use rootwrap into scripts
|
|
|
|
We could create scripts that would require only one rootwrap call to do all
|
|
necessary work for one request, for example. But these scripts will either
|
|
become very complex (say rewriting parts of Neutron agents in shell) or there
|
|
will be too many of them. Either way defeats the purpose of sudo and rootwrap
|
|
- to minimize amount and complexity of code running with root privileges.
|
|
|
|
* Per-host daemon process
|
|
|
|
This would require some D-Bus or MQ set up and secured. It doesn't look
|
|
feasible to set up such secure daemons. Supporting them across projects
|
|
seems even less feasible since every project uses its own rootwrap
|
|
configuration and such configurations might be host-dependent.
|
|
|
|
Impact on Existing APIs
|
|
-----------------------
|
|
|
|
Operations in standalone mode are not affected in any way, so all existing
|
|
usages will work as before.
|
|
|
|
Security impact
|
|
---------------
|
|
|
|
This change require another binary to be added to sudoers for the projects
|
|
that would use daemon mode.
|
|
|
|
Daemon itself will listen UNIX domain socket but every connection is passed
|
|
through digest authentication.
|
|
|
|
JSON is used as a transport to avoid known vulnerabilities in other types of
|
|
serialization mechanisms.
|
|
|
|
Performance Impact
|
|
------------------
|
|
|
|
There is a benchmark results in commit message in [#rw_cr]_. They show that
|
|
although initial daemon startup time is slightly bigger than with usual
|
|
rootwrap call, on average daemon shows over 10x better performance.
|
|
|
|
Configuration Impact
|
|
--------------------
|
|
|
|
None in oslo.rootwrap itself. Projects using it might require a separate option
|
|
for new behavior.
|
|
|
|
Developer Impact
|
|
----------------
|
|
|
|
None
|
|
|
|
Implementation
|
|
==============
|
|
|
|
The feature implementation is in progress at [#rw_cr]_. Example usage for
|
|
Neutron is at [#neu_cr]_.
|
|
|
|
Assignee(s)
|
|
-----------
|
|
|
|
Primary assignee:
|
|
yorik-sar (Yuriy Taraday, YorikSar @ OFTC)
|
|
|
|
Milestones
|
|
----------
|
|
|
|
Target Milestone for completion:
|
|
Juno-2
|
|
|
|
Work Items
|
|
----------
|
|
|
|
This blueprint suggests rather small addition to rootwrap.
|
|
|
|
Further integration into different projects should be covered separately.
|
|
|
|
Documentation Impact
|
|
====================
|
|
|
|
Both mechanisms involved in daemon work and its API should be covered in docs.
|
|
|
|
Dependencies
|
|
============
|
|
|
|
None
|
|
|
|
References
|
|
==========
|
|
|
|
.. [#neu_ml] Original mailing list thread:
|
|
http://lists.openstack.org/pipermail/openstack-dev/2014-March/029017.html
|
|
|
|
.. [#nova_ml] Earlier mailing list thread
|
|
http://lists.openstack.org/pipermail/openstack-dev/2013-July/012539.html
|
|
|
|
.. [#rpc] Nowhere in this document "RPC" refer to that MQ-based RPC messaging
|
|
that ``oslo.messaging`` does
|
|
|
|
.. [#pickle] https://docs.python.org/2.7/library/pickle.html
|
|
|
|
.. [#ser_bug] http://bugs.python.org/issue21078
|
|
|
|
.. [#neu_eth] https://etherpad.openstack.org/p/neutron-agent-exec-performance
|
|
|
|
.. [#rw_cr] https://review.openstack.org/81798
|
|
|
|
.. [#neu_cr] https://review.openstack.org/84667
|
|
|
|
.. [#xml_unsafe] https://docs.python.org/2/library/xml.html#xml-vulnerabilities
|
|
|
|
.. note::
|
|
|
|
This work is licensed under a Creative Commons Attribution 3.0
|
|
Unported License.
|
|
http://creativecommons.org/licenses/by/3.0/legalcode
|
|
|