57 Commits

Author SHA1 Message Date
Drew Freiberger
74daa465d6 Add monitoring for object-replicator logs disappearing
As noted in lp#1691570, there is an issue with storage I/O during
coro-thread cleanup that affects the swift-object-replicator's ability
to complete replication successfully.  This is most easily witnessed by
the lack of the every 5 minute replicated percentage complete messages
that should come from the swift-object-replicator daemon to syslog.
This patch monitors for and alerts on the condition of the "replicated"
line missing from syslog within the past 15 minutes.

Change-Id: Ieb15da3f3f67fa9bcad03151e36c70faae4c36c9
Closes-Bug: 1691570
2020-07-21 15:31:06 -05:00
Hervé Beraud
3dd824c5b9 Use unittest.mock instead of mock
The mock third party library was needed for mock support in py2
runtimes. Since we now only support py36 and later, we can use the
standard lib unittest.mock module instead.

Change-Id: Id4029239c345655ea7ffc90b5969b29b47a84955
2020-06-09 00:02:15 +02:00
Tytus Kurek
fc261a5adc Swift Global Cluster
This patchset adds a support for Swift Global Cluster feature as
described at:

https://docs.openstack.org/swift/latest/overview_global_cluster.html

It allows specifying 'region' config option as part of the deployment.
Theregion is reflected later on in the Swift rings.

Change-Id: I11b6c7802e5bfbd61b06e4d11c65804a165781b6
Partial-Bug: 1815879
Needed-By: I406445493e2226aa5ae40a09c9053ac8633a46e9
2019-12-17 10:08:13 +00:00
Edward Hope-Morley
60dd2f0189 Charmhelpers sync to get vaultlocker fixes
Also gate checking vault context completing on whether
dependencies are installed.

Change-Id: Ib424abe608081da21207db262fb82362f23fe6ca
Closes-Bug: #1849323
2019-12-11 14:30:27 +00:00
Corey Bryant
fb6e21b51f py3: Switch packages to Python 3 for train and later
Switch package install to Python 3 for OpenStack Train and later.

When upgrading, remove any python-* packages that were explicitly
installated and then autoremove --purge any dependencies that are
no longer required.

This patch also includes the following related changes:
  * Use the common files package, swift, rather than python-swift
    when using a package name to determine release.

Also add OS_* to tox.ini to allow functional tests to execute.

Change-Id: I121af845bf11c22052479a497b196a4670021256
2019-10-14 08:29:59 +00:00
Alex Kavanagh
c9b558b0ff Ensure that get_host_ip() is mocked out in test
The test test_setup_ufw() didn't mock out get_host_ip() which meant that
the test leaked and used an actual IP hostname lookup.  This failed when
ubuntu.com's IP address changed.  This patch mocks it out properly.

Change-Id: Ida7aaa7d085cc0316aa496d0a0e24333c5828cee
Closes-Bug: #1832829
2019-06-14 15:11:04 +01:00
Tytus Kurek
edd00bee63 Use UUIDs of storage devices in fstab
This patchset implements a logic to put UUIDS of storage devices in the
fstab instead of device names.

Change-Id: I2170c54b25ac9b3faacb3fd0b2889d86c531395f
Closes-Bug: 1729478
2019-03-13 21:19:14 +00:00
Alex Kavanagh
fc81200315 Upgrade the charm to py3 runtime
Change-Id: I98f4e6664080407a045ca5e76db59d46ffa9c38a
2018-10-24 14:57:05 +01:00
Andrew McLeod
0598fd20fb Add nofail to fstab options for loop dev
Some machines will not boot if nofail is excluded

Change-Id: Ib978709943c4aec8f7d405c0b71e1c4d81f3764c
Closes-Bug: 1768666
2018-05-21 10:43:42 -06:00
Zuul
b03a8bc5bd Merge "Add support for block device encryption" 2018-05-15 09:56:23 +00:00
James Page
591c084532 Add support for block device encryption
Add new secrets-storage relation to vault, supporting the
use of block device encryption using dm-crypt/vaultlocker.

Prepared devices are now recorded in the local unit kv
store; this information is used to provide a list of
configured block devices to the swift-proxy charm,
rather than the previous best guess provided by
determine_block_devices.  This allows us to use the
dm-crypt device name, rather than the underlying
block device.

Encrypted block devices are unlocked on boot using
vaultlocker-decrypt systemd units (enabled by vaultlocker);
/etc/fstab entries for such devices make use of a
x-systemd.requires option to ensure that the block device
is unlocked prior to attempting to mount it.

Add new storage binding to allow charm to be used with
Juju storage.

Add new ephemeral-unmount configuration option to allow
cloud ephemeral storage to be used for testing purposes;
update functional testing to use this option.  The behaviour
of 'overwrite' was changed to accomodate the use of
encrypted block devices.

Change-Id: I9b3f8cd2de412ee96e0139dba4d4abdf998ecaf2
2018-05-08 12:52:37 +01:00
wangqi
f9a6b36c93 Using assertIsNone() instead of assertEqual(None)
Following OpenStack Style Guidelines:
[1] http://docs.openstack.org/developer/hacking/#unit-tests-and-assertraises
[H203] Unit test assertions tend to give better messages for more specific
assertions. As a result, assertIsNone(...) is preferred over
assertEqual(None, ...) and assertIs(..., None)

Change-Id: Ib5dcf2f389dd76f99d87ad22268695f2a702e6b3
2018-04-20 07:33:26 +00:00
Liam Young
6346a7458b Use the loop device in fstab instead of image file
When adding an fstab entry for a loopback device use the explicit
loopback device name rather than the source image file. This prevents
a new loopback device being created implitcitly when mounting the
image.

The unit tests needed updating to reflect that the loopback
device name is used when creating mountpoint names rather than than
the name of the image file. This was pre-existing behaviour.

Change-Id: Ide074310bf7121f1179e0b5237dff6f3da88e24e
Closes-Bug: #1762390
2018-04-10 12:58:05 +00:00
Liam Young
f75bd9762a Fix broken fstab entry
Whitespace between the fstab option is not valid, remove it.

Change-Id: I1be789acc7bf92811f8d4e2c0f0661ebf3be1678
Closes-Bug: #1762105
2018-04-10 11:25:14 +00:00
Felipe Reyes
1d28ceb648 Resolve hostnames if needed to allow access in ufw
In some cases juju 1.25 (with maas 1.9) may return a hostname in the
private-address field breaking the assumption that private-address will
always be an IP address. This patch uses get_host_ip() to assure an IP
address is given to ufw.

Change-Id: Ib99852c645517cf316adcd02d04428b248fc9724
Closes-Bug: 1747516
2018-03-05 18:16:57 -03:00
Frode Nordahl
b819bde429 Revert "Resolve hostnames if needed to allow access in ufw"
We need a new proposal that uses get_host_ip from charmhelpers.contrib.network.ip

Reference: https://review.openstack.org/#/c/542381/

This reverts commit 4896ac5b5b98f66cfe79c3945680ac0ac5dbb2bf.

Change-Id: Idd16369be776a10ac6332d7d8b2ce65f9f1ad972
2018-03-01 15:19:49 +00:00
Felipe Reyes
4896ac5b5b Resolve hostnames if needed to allow access in ufw
In some cases juju 1.25 (with maas 1.9) may return a hostname in the
private-address field breaking the assumption that private-address will
always be an IP address. This patch uses socket.getaddrinfo() to assure
an IP address is given to ufw.

Change-Id: I99b0110beed6075164eb549ec2433071af699c04
Closes-Bug: 1747516
2018-02-05 18:32:52 -03:00
David Ames
5368af6302 Swift storage ACLs
Ensure that only the swift-proxy units and swift-storage peers have
access to direct communication with swift storage daemons.

Charm-helpers sync to include ufw module and the ingress_address and
iter_units_for_relation_name functions.

Please review and merge first:
https://github.com/juju/charm-helpers/pull/35

Closes-Bug: #1727463

Change-Id: Id5677edbc40b0b891cbe66867d39d076a94c5436
2017-11-07 10:24:53 -08:00
Edward Hope-Morley
656e79da18 Catch blkid error when device is not yet formatted
When a new device is added to the ring we first try to
identify whether the device is already in the ring by
polling for an fs uuid. If the device has never been
used this is expected to fail so lets catch the error.

Also fixes log message.

Change-Id: I20354dedfa27a6b8dec92828cabb50a20d0d8838
Closes-Bug: 1567198
2017-09-14 13:22:25 -06:00
zhangyangyang
f87d83b1ab Change assert(Not)Equals to assert(Not)Equal
According to http://docs.python.org/2/library/unittest.html
assert(Not)Equals is a deprecated alias of assert(Not)Equal.

Change-Id: I4b9e966f4c7cd3df6440f38b56785a38a812e002
Closes-Bug: #1329757
2017-09-11 19:19:52 +08:00
Billy Olsen
d6061caa2c Only change owner/permissions of new devices
Do not change owner and permissions of already existing
devices in the setup_storage() function as this runs
during every config-changed hook invocation.

Change-Id: I21f23aee34d315ccb4df303527b4d791fc043f58
Closes-Bug: #1676728
2017-06-08 16:03:16 -07:00
Luong Anh Tuan
bca8efdcd5 Replace assertEquals with assertEqual
The method assertEquals has been deprecated since python 2.7.
http://docs.python.org/2/library/unittest.html#deprecated-aliases

Also in Python 3, a deprecated warning is raised when using assertEquals
therefore we should use assertEqual instead.

Change-Id: If7a404da609eb3cae22627f3b9d71292b8865674
Closes-Bug: #1218185
2016-11-22 10:23:25 +07:00
vnathan
b268ef82fa Fixed handling duplicate block devices specified in config
Change-Id: I5c96c49d47b762fecc16c8700ef6ed65bcd39bf5
closes-bug: 1582317
2016-11-14 22:13:57 +05:30
James Page
ab35249f38 Re-license charm as Apache-2.0
All contributions to this charm where made under Canonical
copyright; switch to Apache-2.0 license as agreed so we
can move forward with official project status.

Change-Id: I97206ee8be76220cb0937a09be3230432e04535a
2016-07-01 18:15:00 +01:00
Edward Hope-Morley
e7bf2f910e Allow devices to be added post-install
Track devices that have been added to the ring and allow
devices to be added to the ring post-install (currently
only allowed within the install hook). Devices added to
the ring prior to this patch existing will be migrated
to the tracking store of devices to avoid conflicts.

Change-Id: Id268dc6369041a4d2db6f30a997dfa0c0d73b93f
Closes-Bug: 1383390
2016-03-07 22:25:29 +00:00
James Page
2ceda02a08 Resync charm-helpers
Resolve single unit deployment failure with default configuration;
update unit tests to ensure that mkdir is called for /srv/node.

Change-Id: Id5518a2144ec3bdc6341f6dfc85835ae342f490e
2016-03-03 11:50:13 +00:00
James Page
9d042778d2 Resync helpers, refactor code to use cpu calcs from charmhelpers 2016-01-30 13:37:50 +01:00
billy.olsen@canonical.com
aaf6c037bc [niedbalski, r=freyes,billy-olsen] Fix fstab entry for loopback devices.
Closes-Bug: 1510666
2015-11-16 13:56:57 -07:00
Jorge Niedbalski
2e98fc7d8f Added unit tests 2015-11-09 11:35:46 -03:00
Corey Bryant
9b178924d7 [james-pages,r=corey.bryant] Add tox support for lint and unit tests. 2015-11-03 17:08:12 +00:00
David Ames
902fcd0251 [corey.bryant, r=thedac] Workload Status 2015-10-13 10:18:25 -07:00
Corey Bryant
e246250b48 Use unitdata key/value pair to store pause/resume status 2015-10-13 10:35:23 +00:00
James Page
da6b3b9fd5 Fixup broken use of assert calls and drop invalid test 2015-10-10 15:14:11 -07:00
Corey Bryant
22cf14a6de Drop old unit tests 2015-10-09 03:56:46 +00:00
Corey Bryant
5c1572f6ef Unit test updates 2015-10-08 12:17:44 +00:00
Liam Young
54dab0004d Fix up unit tests 2015-10-02 11:40:33 +01:00
Adam Collard
6cbfd89ad2 Fix references in unit_tests 2015-07-17 16:52:38 +01:00
David Ames
716b479011 Test for settings on /srv/node 2015-04-02 11:56:27 -07:00
Seyeong Kim
337af05ebf fix lint error 2015-02-03 04:01:59 -08:00
Seyeong Kim
4d0860e25e add more unit tests 2015-01-27 18:39:28 -08:00
Liam Young
2183cda129 Fixed unit tests 2015-01-09 10:21:49 +00:00
Liam Young
40a440746b Merged next branch in 2015-01-09 09:59:47 +00:00
Edward Hope-Morley
886c4888d3 allow ring fetch retry 2014-12-17 11:26:00 +00:00
root
5727c68515 [bradm] Adding nrpe checks, handle rsyncd config fragments 2014-10-30 00:52:15 -05:00
Bjorn Tillenius
232a886247 Fix test to actually fail for the old code. 2014-10-24 14:33:12 +00:00
Bjorn Tillenius
81dcc2c2f1 Apply the patch from lp:~tribaal/charms/trusty/swift-storage/trunk-fix-1350049. 2014-10-24 14:29:07 +00:00
Hui Xiang
20b6a8994c Use general BindHostContext(). 2014-09-25 14:36:13 +08:00
James Page
7492d5f7b1 Fixup failing tests 2014-09-03 15:10:13 +01:00
James Page
f715cbacbb [mthaddon,r=james-page] Add configuration for worker and per disk thread pool sizes
Make the number of workers for account, container and object servers a multiplier of the number of CPUs, and default to a sensible setting per recommendations in http://docs.openstack.org/developer/swift/deployment_guide.html#general-service-tuning. Also, add in a tunable option for object server threads_per_disk and tune default to recommendations from http://docs.openstack.org/developer/swift/deployment_guide.html#object-server-configuration.
2014-06-19 09:40:58 +01:00
Ryan Harper
eb625c4228 Add test that exposes the additional None value in the return value from determine_block_devices when the input list includes a device that's not present on the system. 2014-06-16 12:31:33 -05:00