The mock third party library was needed for mock support in py2
runtimes. Since we now only support py36 and later, we can use the
standard lib unittest.mock module instead.
Change-Id: Ib353cf9878334d91e551bf75121413c6fb2222ea
By default Python configures SIGPIPE to be SIG_IGN, which means to
ignore the signal. We don't want that as it causes problems when
journald restarts and our log calls start triggering SIGPIPEs.
Instead, we want to allow the SIGPIPE to kill the process so it can
be restarted by systemd.
Change-Id: I512139b96b2de8b372efc91e8a3fc8d33553405a
Closes-Bug: 1795030
Now Heat is missing the region support for software deployment
and so did os-collect-config. This patch fixes it but keeps the
backward compatibility.
For changes in Heat pls refer Iec6f3606c9fdf8474f393b0990356f34d38bcf75
and https://review.openstack.org/580470
For changes in Heat agents please refer
I8e0518fa61e237ec055834dd4bebe0fc87cd6627
Story: 2002781
Change-Id: I88182a9a9af74e9760b2ec9b500971f06293f0b8
The local collector was added as the default but we're not properly
configuring it anywhere so this results in two log messages being
printed. Let's lower one of the messages to info and leave the one about
non-existing directory as a warning to indicate possible
mis-configuration.
Change-Id: I52ead32cce1eaf5f3816507e8e62b270f1a1a1cd
Closes-Bug: #1691003
Python 3 will complain when trying to split a byte-string using a string
so this change enables universal_newlines to make the split work in both
python 2 and 3. Also add a py3 tox environment.
Change-Id: I9af74b6c6cc9cfd0c973c85be04791aa6dc0acd4
When the os-collect-config process is started on multiple systems at the
same time, the polling intervals can line up to cause performance
problems against the configuration source. To reduce the impact, this
change adds a splay option to allow the operator to configure a random
delay prior to the polling to attempt to offset the polling
syncronization.
Change-Id: I1a8be3345d783da9014eca7ea26da19d57e767c0
Closes-Bug: #1677314
The older hacking library has a cap on pbr <2.0, with the recent 2.0.0
release of PBR it's causing failures in the pep8 job.
hacking isn't kept in sync via the typical proposal-bot updates.
Do it manually to clear the gate issue.
pep8 runtests: commands[0] | flake8
./os_collect_config/merger.py:45:22: W292 no newline at end of file
return final_list
Change-Id: If863d2e190a4e1701ebf1fcb99a783075cda2556
Related-Bug: #1668848
I1c94ad82c3b20cc331045c19d4a9987a701b081b fixed the issue in Zaqar where
the pop argument was wrongly named. We can now use it in the zaqar
collector to retrieve messages over websocket.
Change-Id: I0ce894b0eef7018a37db9ef9d1b43d8d20629bf0
This patch adds a new zaqar.use_websockets option which if set to True
in the config file will configure the Zaqar collector to use
the websocket transport instead of wsgi. This can be more efficient
where you want to avoid the continuous polling of o-c-c and instead
just listen on the websocket that is subscribed to a queue.
Like other collectors each iteration creates a new socket object. This
allows us to use the normal re-exec logic in o-c-c and thus gives
the option to re-configure the agent in the future to use other types
of collectors. We could (optionally) look into a higher level option in
the future that would allow o-c-c to avoid re-exec'ing and thus re-use
the same websocket for multiple sets of metadata.
Depends-On: Ia2a8deb599252d8308e44d595eb2bf443999aaad
Change-Id: Id5c7ed590df776844b6c7961eb40f89206cd24e0
This change will first attempt to get ec2 metadata from a v2 config
drive before falling back to the nova metadata server if no config
drive is detected.
Config drive is enabled in overcloud nodes, so using this for
os-collect-config will allow the undercloud nova metadata server to be
disabled.
V2 of the config-drive was released in OpenStack Folsom, so a decision
was made to not support V1 discovery at all. This makes the block
device candidate selection much simpler than the cloud-init one[1].
[1] https://git.launchpad.net/cloud-init/tree/cloudinit/sources/DataSourceConfigDrive.py#n219
Change-Id: I2aec0544fe40c2e901aee38c79bb838a627d20da
Closes-Bug: #1619074
Currently the ec2 collector polls the nova metadata service every
$polling_period even though the data is not expected to change and no
known config actions have been written to respond to changes in these
values.
With larger overclouds, this metadata polling will cause noticeable
load on the undercloud, especially nova-api and neutron (for port
lookups).
This change will check for data in /var/lib/os-collect-config/ec2.json
and simply return that if it exists.
Change-Id: I84156f8b005b319fccf8ac17af8943aefb0a8f6d
Closes-Bug: #1619072
Currently if http requests are not correctly mocked, behaviour of the
unit tests will depend on what happens to be running on localhost.
Jobs like gate-os-collect-config-python27-ubuntu-xenial are currently
broken because discover requests are being made to localhost and
something has change in the environment to cause test failure.
This change does the following:
- replace unit tests addresses from 127.0.0.1 to 192.0.2.1 (TEST-NET-1)
- fix mocking of tests which now stall because 192.0.2.1 isn't accessable
- change the mocking approach to keystoneclient.discover.Discover to
a simple stub class
Change-Id: I6d9450d6854b9f50dac0f55f54f1ea1d579a27ac
All CI jobs are timing out currently during NetworkDeployment
because 99-refresh-completed is not signaling to Heat due to
instance-id being detected as null by os-apply-config.
I suspect this patch is causing the issue.
This reverts commit 8717436db9.
Change-Id: Id0d122d7e855cd828a0742dabc5f7089fb6a913c
Closes-Bug: 1623754
Currently the ec2 collector polls the nova metadata service every
$polling_period even though the data is not expected to change and no
known config actions have been written to respond to changes in these
values.
With larger overclouds, this metadata polling will cause noticeable
load on the undercloud, especially nova-api and neutron (for port
lookups).
This change allows collect calls to raise a SourceAlreadyCollected
exception if the data for this collector never changes. The ec2
collector raises this if its data has already been cached to
/var/lib/os-collect-config/ec2.json
Change-Id: Ib7df590601132857690c8ab64fe32098a81752d8
Closes-Bug: #1619072
There are 2 things at play here. First the logic added in
4cfeb28d12 is no longer needed
because our initial sleep time is very low and increases
gradually up to the max.
2nd, I'm proposing that we avoid reexecing unless the config
file actually changes. Not re-execing will give us the option
to optimize os-collect-config for some long running collectors
(like zaqar websockets). Also, Os-collect-config updates would
already be handled by packaging restarts and or other deployment
system changes anyways.
Change-Id: I04b2752d007089f72af42c88c4249c3e11c8346f
This is required so that a the TripleO undercloud can
switch to polling for metadata from a Zaqar rather than Heat or
Swift.
Change-Id: Ieb076185314a7aeadd8a708ab01b48dfbccef006
Closes-bug: 1604098
A timeout config value is added for each collector which uses requests
to fetch data, and this value is used for any requests calls.
Without a timeout a request may stall indefinitely and
os-collect-config will stop polling.
A timeout default of 10 seconds is chosen as the default. This is used
for both the connection timeout and the read timeout.
Change-Id: I4ad0065b5a85393105c6385a15653d7204b4f880
Closes-Bug: #1600652
We're currently still using unmaintained oslo-incubator code for
our logging, which is bad. This switches us to oslo.log as
described in [1].
1: http://docs.openstack.org/developer/oslo.log/migration.html
Change-Id: Ibce86ab4ee24eeb55d0de1b0d5ff4ee4ea6ef66f
This isn't quite right and broke on stable/liberty. Pushing
a revert in case I3c22d77dece399d21ab94783b74990789a1e1481
doesn't actually fix the problem. We should probably merge
whichever passes first.
This reverts commit 69653318f4.
Change-Id: I9304429f25d28ca756e50b1788e149c5bb46b1d6
The old oslo-incubator log module isn't maintained (and doesn't even
exist anymore), so we don't really want to be using it. It appears
this was the only incubator module we were actually using, so this
allows us to remove all of the unmaintained incubator code.
Change-Id: Ib4ad3b231360987a1ef4f95b5b5a8b656232efc4
Replace assertEqual(None, *) with assertIsNone in tests to have
more clear messages in case of failure.
Change-Id: I2cc0d913d83ea56c323475daddb66a0047564b42
This patch updates os-collect-config so that the sleep interval
time is shortened if changes are detected. This should decrease
deployment time when using Heat templates which use depends_on
to step through a sequence of software deployment resources.
The new default sleep is set to 1 and increases
by sleep_time *= 2 until it reaches the default
sleep interval again.
Change-Id: I5cbd0956db2abebec876b15bee72b70ec64d5aef
This adds a new zaqar collector able to read configuration from a
specific zaqar queue.
blueprint software-config-zaqar
Change-Id: Ie38af7b59e7a1aa370ac7760bb7819e37c2165c3
This is required so that a swift-enabled TripleO undercloud can switch
to polling for metadata from a TempURL rather than heat.
Change-Id: I73ac9e01f85e0c72ce7411e2c61c545322f3dccc
Closes-Bug: #1424913
This is a pure refactoring change which moves the os-apply-config
deployment data merging into its own module. This will allow
other collectors (request, heat) to do the same merging.
Partial-Bug: #1424913
Change-Id: Ic78a60e3efebadbb06ebfd262ceb275ca519a3f2
Users who are debugging unrelated issues are frequently interpreting
logged warnings as an indication that os-collect-config is misconfigured
when the warning is actually harmless.
This change uses info instead of warn for this logging.
Change-Id: I28e0fb37b603364eea72cdc5299a29259489b820
Closes-Bug: #1437952
The Oslo libraries have moved all of their code out of the 'oslo'
namespace package into per-library packages. The namespace package was
retained during kilo for backwards compatibility, but will be removed by
the liberty-2 milestone. This change removes the use of the namespace
package, replacing it with the new package names.
The patches in the libraries will be put on hold until application
patches have landed, or L2, whichever comes first. At that point, new
versions of the libraries without namespace packages will be released as
a major version update.
Please merge this patch, or an equivalent, before L2 to avoid problems
with those library releases.
Blueprint: remove-namespace-packages
https://blueprints.launchpad.net/oslo-incubator/+spec/remove-namespace-packages
Change-Id: If51059c31c82d5235e2ae21143911b5561783ca6
keystoneclient discover can't get v3 auth_url using v2 auth_url. It
accepts an url without version number. This patch fixes this issue.
Change-Id: I45b4c016bd91afc39d39620920951043bb71df05
Closes-Bug: #1398254
We currently explode if local collector is enabled with the data
directory missing. Changing this to a warning allows us to have this
turned on by default without greatly inconveniencing users.
Change-Id: I36364ba1a0706a5a2c820eadd526f2ba424ac665
This change implements a collector which does an HTTP GET via
python requests to fetch the metadata.
It should work with any GET-able URL, however it is designed to
work with Swift TempURLs.
Swift objects are not consistent, so the Last-Modified header is
checked for each poll and metadata is not fetched if the last
modified is not newer than the previous successful poll.
This collector will be enabled for OS::Nova::Server
software_config_transport: POLL_TEMP_URL which is available
in the Juno release of Heat. Using POLL_TEMP_URL will result
in no metadata polling load on heat, which has historically been
an issue with tripleo scalability.
Change-Id: I22155c22bdcc3c81a5e945ca5436a8f29f196528
When we detect a failed command we log ERROR but we do not return an
error status. This makes it difficult for programs which may run
os-collect-config to detect whether a run was sucessful.
This only applies to runs which are performed with --one-time argument
as this is a straightforward case.
Change-Id: I168862e8c75c15d1ea405a417908d1284feb7b32
A popen call is made in the Python 3 platform module which causes our
assertions for our mock'd popen to fail. Also fixing a locale issue to
use defaultlocale() rather than LC_ALL.
Co-Authored-By: Steve Kowalik <steven@wedontsleep.org>
Change-Id: I45cdf7d921fbca5f09e5f65bd69fcee83fb7c8e4
This patch uses keystone client discover to find correct V3
auth_url which is expected by heat collector.
Change-Id: I2b8b5a768bdeb1cd1d20fab3b4234306c1429d6d
Closes-Bug: #1341936
Alex Schultz