On py35, if process_input is a string we end with an Error:
TypeError: memoryview: a bytes-like object is required, not 'str'
A whole lot of code in os-brick/cinder/neutron currently need
to be fixed. Hardest problem is tracking down every instance of
this problem and trying to test/fix it at source. See example
of cinder/os-brick/oslo.rootwrap/oslo.privep code problem here:
http://logs.openstack.org/43/418643/2/check/gate-rally-dsvm-py35-cinder-nv/6efa7b5/logs/screen-n-cpu.txt.gz?level=ERROR
It's just better to fix it in one spot and pass the correct
thing to subprocess.communicate
Change-Id: I88d1510a7ba4c020f73452f0b80e996c22b1edf1
The built in 'resource' module is used in order to apply process
resource limits. This is not available on Windows.
For the moment, we'll ignore this argument and log a warning
message. In the future, we may provide a portable implementation
for this feature.
Change-Id: I6a82916eb68fbf8737b45a65f71cae1f835d12d1
Closes-Bug: #1627766
Many assertEqual sentences don't follow assertEqual(expected, actual),
These misplaces have 2 impacts:
1, giving confusing messages when some tests failed.
2, mislead other developers, new test modules may follow these wrong pattern.
This patch fix all of them.
Change-Id: I00bb4801c208d0eb8078e8a34ae52f9bcfe0b1a6
Closes-Bug: #1604213
Add an optional timeout parameter to ssh_execute() method of
oslo_concurrency.processutils. Timeout is None means infinity,
which is the default.
Change-Id: I45532799ecd23ff8c5572036d06725e71965b2b5
Closes-Bug: #1580826
The original commit adding support for process limits only wired
up address space, max files and resident set size limits. This
is not sufficient to enable nova to protect qemu-img commands
against malicious images.
This commit adds support for the remaining limits supported
by python: core file size, cpu time, data size, file size,
locked memory size, max processes and stack size.
Related-bug: #1449062
Change-Id: I164c4b35e1357a0f80ed7fe00a7ae8f49df92e31
Serialising/deserialising exceptions in python follows the protocol
defined by pickle. It relies on the base Exception.__init__ setting
self.args, and later resurrects exceptions with class(*args).
This change rewrites processutils.ProcessExecutionError so it survives a
pickle.dumps/loads round-trip.
Change-Id: I9b8d104f60df868be7b808c72c932d08f1752777
Add a new oslo_concurrency.prlimit module which is written to be used
on the command line:
python -m oslo_concurrency.prlimit --rss=RSS -- program arg1 ...
This module calls setrlimit() to restrict the resources and then
executes the program. Its command line is written to be the same than
the Linux prlimit system program.
Add a new ProcessLimits class processutils: resource limits on a
process.
Add an optional prlimit parameter to process_utils.execute(). If the
parameter is used, wrap the command through the new oslo_concurrency
prlimit wrapper.
Linux provides a prlimit command line tool which implements the same
feature (and even more), but it requires util-linux v2.21, and
OpenStack targets other operating systems like Solaris and FreeBSD.
Change-Id: Ib40aa62958ab9c157a2bd51d7ff3edb445556285
Related-Bug: 1449062
If the user specifies preexec_fn, we should call that in
our existing _subprocess_setup. On windows, we silently
drop this preexec_fn as subprocess.Popen raises a ValueError
if we do pass it in.
Change-Id: I0176c66fa2de001aa14f0d928d06fd894de55511
If the subprocess.Popen.communicate method raises an exception,
the on_completion callback is never invoked. If a caller is
trying to use on_execute + on_completion to track lifecycle
of a process this creates a problem, as they cannot reliably
detect completion.
Change-Id: I22b2d7bde8797276f7670bc289d915dab5122481
Closes-bug: #1470868
Add optional on_execute and on_completion callbacks to allow callers of
procesutils.execute() to track process completion asynchronously.
This could be used to cache the pid of long running tasks associated
with an instance and then clear the cache when the process completes.
While the tasks are running should it be required the pid retrieved and
the process can be signaled.
Co-Authored-By: abhishekkekane <abhishek.kekane@nttdata.com>
Change-Id: Ifc23325eddb523f6449ba06a2deb0885a8a7009d
Add an optional binary parameter to execute() and ssh_execute()
functions of oslo_concurrency.processutils. If binary is True, stdout
and stderr are returned as byte strings on Python 2 and Python 3.
When Nova will be ported to Python 3, binary=True will be needed to get
binary outputs (like an encryption key, see the bug #1410348).
Add new tests.
This changeset does not change the default behaviour.
Change-Id: I283cadf04781942b65f211237070cc2354939cdb
Related-Bug: 1410348
On Python 3, execute() and ssh_execute() now return stdout and stderr as
Unicode. os.fsdecode() is used to stdout and stderr, it uses the locale
encoding with the surrogateescape error handler.
subprocess.Popen(universal_encoding=True) uses the same encoding and
error handler.
This change breaks the backward compatibility, but I believe that
Unicode is more convinient in Python 3 and it is what users expect. In
practice, there is no known (OpenStack) application compatible with
Python 3 using oslo_concurrency.processutils yet.
Add new tests.
This changeset does not change the behaviour on Python 2, only Python 3
is impacted.
Change-Id: Ibb510642c1440e2fbe7ad53d797bc7c32a3cbddf
Currently folks end up using os.chdir() (see bug listed below) as
we are not exposing the subprocess.Popen's cwd parameter. It's
better to add a cwd parameter in processutils.execute so folks
do not have to issue chdir() before and after the processutils.
execute and there's less scope for problem as documented in
the bug.
Closes-Bug: #1414530
Change-Id: Ia4c77593c0f8301e059b349290e8663614a7ccfd
This reverts commit 7c7493feb53429577efca2c4b0380af03ddc149b.
This change breaks the API of execute() and breaks nova's tests.
Closes-Bug: #1410348
Change-Id: I33fa9b4e6099f521e0de3cb4585016fd694c4548
Add encoding and errors parameters to execute() and ssh_execute(). By
default, use the locale encoding in strict mode on Python 2, or the
locale encoding with the 'surrogateescape' error handler on Python 3.
Fix also unit tests to use bytes strings for stdin, stdout and stderr.
Without this change, tests are failing with Python 3 when run with:
PYTHON="python -bb" testr run
Using -bb, Python 3 raises a TypeError when a bytes string is casted
to a text string, which occurred in many places.
Change-Id: I655d5abf932c9a104e3ab487e23c372377f7096a
Move the public API out of oslo.concurrency to oslo_concurrency. Retain
the ability to import from the old namespace package for backwards
compatibility for this release cycle.
bp/drop-namespace-packages
Change-Id: I20d1647b1c3ef8cab3b69eccfe168eeb01703b72
