After a lot of discussion[1], it was decided that this is the
safest way forward. In the future we can investigate alternative
locking methods, probably as opt-ins so we don't break any
existing consumers of these locks.
[1] http://lists.openstack.org/pipermail/openstack-dev/2014-August/043090.html
Change-Id: I49ff98abc395d3263d55aeff26081b462a8a294e
Partial-Bug: 1327946
While trying to port oslo to python 3, we discovered that test_lockutils
wasn't working when not run with the global test suite. The fix can be
tested running:
$ python -m openstack.common.lockutils python -m testtools.run \
tests.unit.test_lockutils
Change-Id: I51461f63cdd8e7eb6f8b6fdfa838748a582ecdad
Fixed violations:
* E128 continuation line under-indented for visual indent
* E251 unexpected spaces around keyword / parameter equals
* E265 block comment should start with '# '
* E713 test for membership should be 'not in'
* F402 import shadowed by loop variable
* H305 imports not grouped correctly
* H307 like imports should be grouped together'
* H402 one line docstring needs punctuation'
* H703 Multiple positional placeholders
Also enabled H803 check that didn't have any violations.
Change-Id: I957ab273ddc61b02763d6b60b21d11ed4e73d472
Module openstack.common.test is obsolete, so we should use
oslotest library instead of it.
Modified tests and common database code, new requirement added.
Change-Id: I853e548f11a4c3785eaf75124510a6d789536634
External locks by themselves do not work with greenthreads in a
single process because multiple greenthreads are allowed to grab
the file lock at once. Previous lockutils code also grabbed a
local semaphore for external locks so this was not a problem, but
after the recent refactoring that was no longer happening, which
caused some issues in other projects.
Since it's less confusing to have external locks also work as
internal locks, this change switches the behavior back to what it
was before. It also adds some test cases to verify these external
lock behaviors.
Change-Id: If66d358c7bd3960c1cc9c8d392fcc158963943bc
There are places in Nova (at least) that require file-based locking
behavior for locking on shared filesystems. These situations can
be recognized because those calls to synchronized specify a
lock_path, so when that happens we need to return a file-based lock
instead of a posix lock.
Change-Id: I21dd17d58d1c321566c72e808a055a0bd625c647
Rather than creating temporary files in a file system let's leverage the
Posix IPC interface to create our inter process locking mechanism.
Blueprint: lockutils-posix-ipc
Change-Id: Ia87c3bdad64f0b0f6cb9fc2878add3fd1708041f
when external lock are used, there will be a lock files which
is used for external file lock, it will not be removed and remain
forever. it will be a problem when tens of thousands of this kind
of actions. Add a API to allow other component to remove lock file
internal lock will be acquired before further action
Change-Id: I4d996b91bcc55becb0ebf4ec77ac80546f433ae9
Closes-Bug: 1256306
The threading API specifies that ThreadError should
be used as the common exception raised from threading
and locking apis so instead of reraising an IOError
reraise a new ThreadError one instead.
Change-Id: Ic06dd211a8a5da074623ac449d27e963dd5ba754
The python locking api specifies a acquire/release
function pair. It is nice to have the interprocess
lock api also follow said api so that its objects
can be used in areas where the standard threading
locks are used.
Change-Id: If393bec863ced8c9ef413b4baa3267d2a5b38416
This remove the parameter lock_path so the configured lock_path is
always used instead. There's only a few places where this is used in
OpenStack, and it's not heavily justified to have it even in this
places.
This is needed to simplify the interface toward moving to IPC locking
mechanism. With such a locking mechanism in place, the lock_path
parameter has absolutely no sense. So isolating this to make it only
relevant to some part of the driver and not in the public interface
makes more sense.
Change-Id: I72bd683209090ae6e3911a709e4d213879508aca
Blueprint: lockutils-posix-ipc
Refactor codes, remove unused variables. This patch is the first step.
F841 check in tox.ini will be enabled to make sure that there are no
userd variables in project anymore.
Change-Id: I3edd5bb5e3c203d1d3d92261ed74a201878d6f5a
Partial-Bug: #1268234
Some modules use different names in Python2 and Python3. Use six.moves
to make them work well in Python2 and Python3.
This is changes mapping:
six.moves Python 2 Python 3
builtins __builtin__ builtins
filter itertools.ifilter() filter()
reduce reduce() functools.reduce()
reload_module reload() imp.reload()
Implements: blueprint make-python3-compatible
Change-Id: I97971f2ab40385bfc2c73ae7e8a7620c4d64a03c
No need to set tabstop 189 times, this can be set in your vimrc file
instead. Also if set incorrectly gate (pep8 check) will catch your
mistakes.
Change-Id: Ic6f0c0ef94e8194a5c121598305d1ec3c74e4843
This change would allow us to create one directory to store all locks
from all processes involved in testing in projects.
Here are examples on how to use this:
Heat - https://review.openstack.org/47360
Ceilometer - https://review.openstack.org/44809
(see tox.ini)
Change-Id: I9e1260e202b7d8feb9c762a879b9804ed85ccbd6
This would allow us to provide a common dir for all locks in all
processes involved in testing or running software.
Change-Id: I64fccddc92ada67a3df6daf86d73750a41bbce05
In Nova and other projects, fixtures are in test-requirements.txt and
NOT in requirements.txt so updating nova to use latest lockutils.py
is not possible. So let us move LockFixture out into it a separate
fixtures.py module where in theory we can add other fixtures as
well. See https://review.openstack.org/#/c/52153/ for an attempt to
try latest lockutils.py in Nova.
Change-Id: I12942d6390d06bb889d4145e708692fa886d99b5
This commit adds a fixture class to be used instead of external
locking with the synchronized decorator. When running tests in
parallel the synchronized decorator won't synchronize the tearDown
and addCleanup methods for a test. By using the fixture which does
LockFixture the unlocking is done by using addCleanup() ensuring that
tearDown and later addCleanup methods are run inside the lock
context.
Closes-Bug: #1229437
Change-Id: I61af5e9d385bb7ef2e6402366fb8c5c09f749986
Change lockutils.py due to use threading.Semaphore instead of
eventlet.semaphore.Semaphore. Tests modified properly.
Change-Id: Ic3c5934577aff856ec71ba7e44b54aa963fda69c
It is the second step to replace using tests.utils with
stack.common.test.
In this patch second part of tests is changed for using
openstack.common.test and added needed fixtures in test classes.
Change-Id: I81e3ff41827f7153b650255e219ce25cb282507b
bp: common-unit-tests
When lock_path is not set, lockutils creates a new temp dir for
each new call to synchronized. This results in no actual lock
enforcement. Require setting of lock_path by throwing an
exception if it is not found.
Fixes bug 1065531
Fixes bug 1162047
Change-Id: I178684a1d8649b5bcfcb768be0a68c8efa3f00e4
synchronized is being widely used throughout OpenStack for managing
locks by decorating methods / functions. However, it's functionality is
trapped within that decorator definition and makes it difficult to
manage locks by using it in other cases (like w/o decorating).
This patch moves decorator's body into a first-class function called
lock which is intended to be used as a context manager type (with
statement).
Some examples:
with lockutils.lock("test") as sem:
print('Inside the lock')
Things I'm not 100% convinced:
* The `lock` function yields a Semaphore when external is False and
an InterProcessLock instance otherwise. Although it is not 'good' to
yield different values depending on the input parameters, it's
pretty explicit (by reading the documentation) that external locks
are handled differently. Other options for this case are:
1. Always yield a Semaphore instance
2. Don't yield anything
The patch doesn't break backward compatibility so no change is needed to
existing projects.
Implements blueprint cache-backend-abstraction
Change-Id: I96b2ee84da5a2dcd7f0bd469f8a1e52b74e50c75
Currently, the lock_file_prefix says it should end with an hypen if
specified. It seems that it was intended to be optional. Since that
behavior makes sense, this patch changes the synchronized's signature
and makes lock_file_prefix optional.
With this patch, the prefix will not be required to end with an hypen
anymore, such sign will be added to the prefix if specified and if not
already present.
This patch doesn't change synchronized's behavior at all, which means it
doesn't break backward compatibility.
Change-Id: I8841da1e8d105aac4a86b247228a9c935e00afc5
The lock_file_prefix for each project doesn't need to be configurable or
frequently changed. This patch provides a convenience API which returns
a partial object of the synchronized decorator to avoid passing the prefix
each time locks are used.
The set_defaults method is also provided to change the default value of
lock_path when needed.
Fixes bug #1065524
Change-Id: I7b67f0a482da4be6d53a70db5bbd22dc91bdc10c
Fixes bug 1158179
In the case that the lock_path does not exist and there is a contested
resource then the process that was waiting on the lock will not
be locking due to the fact that the directory was deleted.
Change-Id: I75d720d4df499e85386d3e2cc86b927b017e12ac
Add a unit test which fails because of #1107950 - in this case,
lockutils is deleting the lock directory every time the lock is dropped
even though another process might have acquired a lock using that
directory.
Change-Id: Ic82409f9462e570bc102ab469334c53aafc6d7ac
Use a separate directory for the lockutils external locks and for the
test's flocks which are used to check that serialization is actually
occurring. By using the same directory, we can't test the code in
lockutils which auto-creates this directory because the dir gets
deleted by a process by lockutils cleanup while another directory
is using it for flocks.
Also, assume the the handles directory has been created in the parent
rather than having each child attempt to create it.
Related, add a try/finally block so that when a child process throws an
exception it immediately exits rather than deleting the temporary
directories created by the parent.
Change-Id: I32d7e8e05fb3f22cf38fa586f8bc97646c83f182
Using the BaseTestCase across the tests in the tree lets us put in log
fixtures and consistently handle mox and stubout.
Part of blueprint grizzly-testtools.
Change-Id: Iba7eb2c63b0c514009b2c28e5930b27726a147b0
On the path to testr migration, we need to replace the unittest base classes
with testtools.
Replace tearDown with addCleanup, addCleanup is more resilient than tearDown.
The fixtures library has excellent support for managing and cleaning
tempfiles. Use it.
Replace skip_ with testtools.skipTest
Part of blueprint grizzly-testtools.
Change-Id: I45e11bbb1ff9b31f3278d3b016737dcb7850cd98
Fork many processes and try to lock the same set of files
using flock without blocking and make sure we wait for all the processes
to finish as well.
Fixes LP #1068316
Change-Id: I09964b2c5af63f31b5ddee1f18eaf646f8d8ba58
We're seeing this test fail very regularly in Jenkins and it's become
a serious distraction for everyone.
Disable the test temporarily and use a bug #1095957 to track that the
test itself needs fixing.
Change-Id: I0645d8f5f740d40f83d82e5ef7a048f33a44ac2b
