34 Commits

Author SHA1 Message Date
prashkre
21ae27e66d Mask passwords only when command execution fails
At many places, processutils.ssh_execute() is being invoked to run
a command over ssh and output returned is parsed to get appropriate
information. In this flow, unsanitized output is being expected
where processutils.ssh_execute() was invoked but found that
output like volume details(containing "password" string in its name)
is being masked away with strutils.mask_password(stdout) even though
no error occured during command execution.

This is regression issue from patch[0]. In this fix, stdout and stderr
in processutils.ssh_execute() will be masked only when
ProcessExecutionError exception is thrown i.e. command execution failed
due to some reasons.

[0] https://github.com/openstack/oslo.concurrency/commit/
ae9e05bfc3d7ec867bd6ec78c301f11c2bdd0d5f

Change-Id: I2ce344330905eef437ef3f89a2a01169a30df8ab
Closes-Bug: #1482382
2018-02-22 06:31:01 +00:00
Matthew Treinish
55e06261aa
Add python_exec kwarg to processutils.execute()
This commit adds a new kwarg to the process_utils.execute() function to
specify the python executable to use when launching python to check
prlimits. This is necessary when processutils.execute() is called from
inside an API server running with uwsgi. In this case sys.executable is
uwsgi (because uwsgi links libpython.so and is actually the interpreter)
This doesn't work with the execute() function because it assumes the
cpython interpreter CLI is used for the arguments it uses to call the
prlimits module. To workaround this and enable API servers that may run
under uwsgi to use this those applications can simply pass in an
executable to use.

Longer term it might be better to migrate the prlimits usage to call
multiprocessing instead of subprocessing python. But that would require
a more significant rewrite of both processutils and prlimit to
facilitate that.

Change-Id: I0ae60f0b4cc3700c783f6018e837358f0e053a09
Closes-Bug: #1712463
2018-01-09 10:22:45 -05:00
Lucian Petrut
3ac3c169ad Windows: ensure exec calls don't block other greenthreads
eventlet.green.subprocess is not actually greenthread friendly on
Windows. It just uses the native subprocess.Popen in this case.

For this reason, exec calls do not yield on Windows, blocking other
greenthreads.

This change avoids this issue by wrapping the 'communicate' call
using eventlet.tpool.

We're also ensuring that subprocess.Popen uses *native* threads
internally in order to avoid deadlocks when passing data through
stdin.

Change-Id: Ic25fd1b61b5498f16e6049cbbe0877492f8aab4d
Closes-Bug: #1709586
2017-08-16 13:59:02 +03:00
Yuriy Taraday
ef7bfd5771 Check for SubprocessError by name on Python 3.x
With eventlet SubprocessError raised by Popen seem to have different
class from subprocess.SubprocessError accessible from test.
I don't have proper solution for this, it seems somewhere old
SubprocessError is cached and then eventlet overrides it, so it is not
visible from test method context. This workaround seems to be good
enough to unblock gate.

Change-Id: If5ae0911e14671e05aca5e393c5cc183b72703d6
Closes-Bug: #1688201
2017-06-29 16:12:12 +04:00
gecong1973
dd2ef6f565 Using fixtures.MockPatch instead of mockpatch.Patch
This module has been deprecated in favor of fixtures.MockPatch.

Change-Id: Iee59fa82e727e0575e8b6b6afe042231aa8fa5b8
2017-03-09 14:45:42 +08:00
Davanum Srinivas
7afbc993f0 Automatically convert process_input to bytes
On py35, if process_input is a string we end with an Error:
TypeError: memoryview: a bytes-like object is required, not 'str'

A whole lot of code in os-brick/cinder/neutron currently need
to be fixed. Hardest problem is tracking down every instance of
this problem and trying to test/fix it at source. See example
of cinder/os-brick/oslo.rootwrap/oslo.privep code problem here:
http://logs.openstack.org/43/418643/2/check/gate-rally-dsvm-py35-cinder-nv/6efa7b5/logs/screen-n-cpu.txt.gz?level=ERROR

It's just better to fix it in one spot and pass the correct
thing to subprocess.communicate

Change-Id: I88d1510a7ba4c020f73452f0b80e996c22b1edf1
2017-01-12 14:07:26 -05:00
howardlee
393e74653b [TrivialFix] Replace 'assertTrue(a in b)' with 'assertIn(a, b)'
Change-Id: Ib61f2f2c695f79c8eb0fa47a2d2d1bc66ccaeb7f
2016-10-21 15:00:37 +08:00
gecong1973
7a4d633ba6 Change assertTrue(isinstance()) by optimal assert
Some of tests use different method of
assertTrue(isinstance(A, B)) or assertEqual(type(A), B).
The correct way is to use assertIsInstance(A, B) provided
by testtools

Change-Id: Iad2ee275837c28b40c1062386db5a29969ee98dc
2016-10-08 09:30:32 +08:00
Lucian Petrut
70ff551105 Ignore prlimit argument on Windows
The built in 'resource' module is used in order to apply process
resource limits. This is not available on Windows.

For the moment, we'll ignore this argument and log a warning
message. In the future, we may provide a portable implementation
for this feature.

Change-Id: I6a82916eb68fbf8737b45a65f71cae1f835d12d1
Closes-Bug: #1627766
2016-09-29 21:10:34 +03:00
Denis Buliga
3c46e8f776 Fix external lock tests on Windows
At the moment, those tests use fcntl to acquire file locks, which
will fail on Windows.

This change addresses this issue by adding some platform checks,
using the appropriate functions when handling file locks. Note that
we've avoided running private methods from fasteners.

Also, we now avoid using os.fork, which won't work on Windows.
Instead, we spawn new processes using multiprocessing.Process, using
queues for communication between processes.

Change-Id: I9839b9033c814280f6d1b53c5ed2643fd2bf8bf8
2016-08-08 19:13:44 +03:00
yanheven
0c3a39eded Fix parameters of assertEqual are misplaced
Many assertEqual sentences don't follow assertEqual(expected, actual),
These misplaces have 2 impacts:
1, giving confusing messages when some tests failed.
2, mislead other developers, new test modules may follow these wrong pattern.

This patch fix all of them.

Change-Id: I00bb4801c208d0eb8078e8a34ae52f9bcfe0b1a6
Closes-Bug: #1604213
2016-07-23 01:12:19 +08:00
zhangsong
c876fabd07 Add timeout option to ssh_execute
Add an optional timeout parameter to ssh_execute() method of
oslo_concurrency.processutils. Timeout is None means infinity,
which is the default.

Change-Id: I45532799ecd23ff8c5572036d06725e71965b2b5
Closes-Bug: #1580826
2016-05-12 10:41:52 +08:00
Daniel P. Berrange
8af826953d processutils: add support for missing process limits
The original commit adding support for process limits only wired
up address space, max files and resident set size limits. This
is not sufficient to enable nova to protect qemu-img commands
against malicious images.

This commit adds support for the remaining limits supported
by python: core file size, cpu time, data size, file size,
locked memory size, max processes and stack size.

Related-bug: #1449062
Change-Id: I164c4b35e1357a0f80ed7fe00a7ae8f49df92e31
2016-04-19 14:35:24 +01:00
Angus Lees
9d28946395 Make ProcessExecutionError picklable
Serialising/deserialising exceptions in python follows the protocol
defined by pickle.  It relies on the base Exception.__init__ setting
self.args, and later resurrects exceptions with class(*args).

This change rewrites processutils.ProcessExecutionError so it survives a
pickle.dumps/loads round-trip.

Change-Id: I9b8d104f60df868be7b808c72c932d08f1752777
2016-02-19 15:52:23 +11:00
Victor Stinner
b2e78569c5 Add prlimit parameter to execute()
Add a new oslo_concurrency.prlimit module which is written to be used
on the command line:

    python -m oslo_concurrency.prlimit --rss=RSS -- program arg1 ...

This module calls setrlimit() to restrict the resources and then
executes the program. Its command line is written to be the same than
the Linux prlimit system program.

Add a new ProcessLimits class processutils: resource limits on a
process.

Add an optional prlimit parameter to process_utils.execute(). If the
parameter is used, wrap the command through the new oslo_concurrency
prlimit wrapper.

Linux provides a prlimit command line tool which implements the same
feature (and even more), but it requires util-linux v2.21, and
OpenStack targets other operating systems like Solaris and FreeBSD.

Change-Id: Ib40aa62958ab9c157a2bd51d7ff3edb445556285
Related-Bug: 1449062
2016-01-28 15:00:06 +01:00
ChangBo Guo(gcb)
af7ffe0e2a Drop python 2.6 support
Change-Id: I921e29d25e0cfb089028f780bc942a60c9712278
2015-11-23 23:12:28 +08:00
Davanum Srinivas
de13b46b1f Fix Tests to run under OSX
Change-Id: I700b6de69411330b2a0fd8f2cdafa4026f85f1c7
2015-10-05 14:50:54 -07:00
Davanum Srinivas
36fb964f0c Allow preexec_fn method for processutils.execute
If the user specifies preexec_fn, we should call that in
our existing _subprocess_setup. On windows, we silently
drop this preexec_fn as subprocess.Popen raises a ValueError
if we do pass it in.

Change-Id: I0176c66fa2de001aa14f0d928d06fd894de55511
2015-07-07 10:12:58 -07:00
Daniel P. Berrange
ab78480659 processutils: ensure on_completion callback is always called
If the subprocess.Popen.communicate method raises an exception,
the on_completion callback is never invoked. If a caller is
trying to use on_execute + on_completion to track lifecycle
of a process this creates a problem, as they cannot reliably
detect completion.

Change-Id: I22b2d7bde8797276f7670bc289d915dab5122481
Closes-bug: #1470868
2015-07-02 17:34:57 +01:00
Tony Breeds
d8307005bf Add 2 callbacks to processutils.execute()
Add optional on_execute and on_completion callbacks to allow callers of
procesutils.execute() to track process completion asynchronously.

This could be used to cache the pid of long running tasks associated
with an instance and then clear the cache when the process completes.
While the tasks are running should it be required the pid retrieved and
the process can be signaled.

Co-Authored-By: abhishekkekane <abhishek.kekane@nttdata.com>
Change-Id: Ifc23325eddb523f6449ba06a2deb0885a8a7009d
2015-06-17 10:39:36 +10:00
Joshua Harlow
9f22c45397 Replace locks and replace with fasteners library provides ones
The fasteners library (extracted from this library and a couple other
variations) provides the interprocess lock logic and the reader writer
lock logic so we can remove the local version and we can just use it
from that library instead.

The tests that were ensuring the internals of this file lock have
now moved to the repo where that library is (for the time being),
currently travis is testing that repo against py2.6, py2.7 and py3.4.

https://github.com/harlowja/fasteners/tree/master/fasteners/tests

Docs also exist at:

http://fasteners.readthedocs.org/en/latest/

Change-Id: I98565b22e68358efe28fea62f74f8ebfcc438ff7
2015-06-06 00:44:22 +00:00
Victor Stinner
4d7dce2ea8 Add binary parameter to execute and ssh_execute
Add an optional binary parameter to execute() and ssh_execute()
functions of oslo_concurrency.processutils. If binary is True, stdout
and stderr are returned as byte strings on Python 2 and Python 3.

When Nova will be ported to Python 3, binary=True will be needed to get
binary outputs (like an encryption key, see the bug #1410348).

Add new tests.

This changeset does not change the default behaviour.

Change-Id: I283cadf04781942b65f211237070cc2354939cdb
Related-Bug: 1410348
2015-04-19 01:17:43 +02:00
Victor Stinner
c05c8c37be Port processutils to Python 3
On Python 3, execute() and ssh_execute() now return stdout and stderr as
Unicode. os.fsdecode() is used to stdout and stderr, it uses the locale
encoding with the surrogateescape error handler.
subprocess.Popen(universal_encoding=True) uses the same encoding and
error handler.

This change breaks the backward compatibility, but I believe that
Unicode is more convinient in Python 3 and it is what users expect. In
practice, there is no known (OpenStack) application compatible with
Python 3 using oslo_concurrency.processutils yet.

Add new tests.

This changeset does not change the behaviour on Python 2, only Python 3
is impacted.

Change-Id: Ibb510642c1440e2fbe7ad53d797bc7c32a3cbddf
2015-04-18 00:11:28 +02:00
Doug Hellmann
46fcdd3aca Add lockutils.get_lock_path() function
Provide an API to discover the path being used to store external lock
files. Tempest will use this to set up some additional locking and to
test its own behavior. Other projects might use it for error reporting
or other purposes.

Change-Id: Iad40c67072333cc25a6d3e39d7535ff14b573504
2015-03-11 19:51:54 +00:00
Davanum Srinivas
3d6e372ea4 Ability to set working directory
Currently folks end up using os.chdir() (see bug listed below) as
we are not exposing the subprocess.Popen's cwd parameter. It's
better to add a cwd parameter in processutils.execute so folks
do not have to issue chdir() before and after the processutils.
execute and there's less scope for problem as documented in
the bug.

Closes-Bug: #1414530
Change-Id: Ia4c77593c0f8301e059b349290e8663614a7ccfd
2015-02-13 16:17:55 -05:00
Ben Nemec
2a9321e65a Add eventlet test check to new tests __init__.py
This is used to allow us to test both with and without eventlet.
It wasn't copied to the new tests directory in the namespace move,
which means that once the old tests go away eventlet won't be
monkey patching even when we try to make it.  Also, there's a race
depending on whether the top-level tests are run first, or the new
package ones.

Change-Id: Ie2705c3d21625e594471f27eefc3ac3147390cb7
2015-02-06 17:32:39 +00:00
Doug Hellmann
da8176dbdf Update Oslo imports to remove namespace package
Change-Id: I4ec9b2a310471e4e07867073e9577731ac34027d
Blueprint: drop-namespace-packages
2015-01-21 18:00:50 -05:00
Jenkins
f034bd2877 Merge "Revert "Port processutils to Python 3"" 2015-01-14 00:43:37 +00:00
Doug Hellmann
acb55ef279 Revert "Port processutils to Python 3"
This reverts commit 7c7493feb53429577efca2c4b0380af03ddc149b.

This change breaks the API of execute() and breaks nova's tests.

Closes-Bug: #1410348
Change-Id: I33fa9b4e6099f521e0de3cb4585016fd694c4548
2015-01-13 17:54:43 +00:00
Jenkins
590444c1f5 Merge "Add a reader/writer lock" 2015-01-06 22:59:33 +00:00
Joshua Harlow
90663364f8 Add a reader/writer lock
Taskflow has a reader/writer lock that is likely useful
to other projects; and it seems better at home in this
module.

The class provides a way to create reader/writer locks
where there may be many readers at the same time (but
only one writer). It does not allow (currently) for privilege
escalation (but this could be added with limited support
in the future).

Change-Id: Ie763ef92f31c34869d83a533bc8761b0fbd77217
2014-12-19 15:39:25 -08:00
Victor Stinner
7c7493feb5 Port processutils to Python 3
Add encoding and errors parameters to execute() and ssh_execute(). By
default, use the locale encoding in strict mode on Python 2, or the
locale encoding with the 'surrogateescape' error handler on Python 3.

Fix also unit tests to use bytes strings for stdin, stdout and stderr.

Without this change, tests are failing with Python 3 when run with:
PYTHON="python -bb" testr run

Using -bb, Python 3 raises a TypeError when a bytes string is casted
to a text string, which occurred in many places.

Change-Id: I655d5abf932c9a104e3ab487e23c372377f7096a
2014-12-12 00:09:55 +01:00
Joshua Harlow
3bda65ccbb Allow for providing a customized semaphore container
It can be undesirable to have a globally shared sempahore
container, especially since oslo.concurrency can now be shared
among many disjoint applications and libraries.

When a single container is used it is now possible to have those
disjoint applications/libraries collide on the same sempahore names.
This is not a good pattern to continue with, so in order to move away
from it allow a custom container to be provided (which defaults to the
existing global one) so that users of oslo.concurrency may provide there
own container if they so desire.

Change-Id: I9aab42e21ba0f52997de3e7c9b0fea51db5c7289
2014-11-19 17:15:40 -08:00
Doug Hellmann
bca4a0d827 Move out of the oslo namespace package
Move the public API out of oslo.concurrency to oslo_concurrency. Retain
the ability to import from the old namespace package for backwards
compatibility for this release cycle.

bp/drop-namespace-packages

Change-Id: I20d1647b1c3ef8cab3b69eccfe168eeb01703b72
2014-11-14 17:17:25 -05:00