Commit Graph

110 Commits (master)

Author SHA1 Message Date
Alex Schultz 3a7d05a9e8 Drop six
With python2 not being supported anymore, let's drop six.

Change-Id: I22f3d28e22ad209806b78ad78f46671a74179157
2 years ago
Hervé Beraud 24245ce18e Use unittest.mock instead of mock
The mock third party library was needed for mock support in py2
runtimes. Since we now only support py36 and later, we can use the
standard lib unittest.mock module instead.

Change-Id: Ib353cf9878334d91e551bf75121413c6fb2222ea
3 years ago
ricolin d1607f9f56 Add tls proxy support for Zaqar
Change-Id: I234a3d60333ee89a7c283b425e524d4624191577
3 years ago
Lukas Bezdicka 8566ad549c Don't fail on non-valid json in cache files
Resolves: rhbz#1438096
Closes-Bug: #1678328

Change-Id: Iee1bb4a7769568cfd3bad8b23c9dff440679ac16
Co-Authored-By: Alex Schultz <>
6 years ago
Alex Schultz 0762fcb0b5 Add splay option to offset polling intervals
When the os-collect-config process is started on multiple systems at the
same time, the polling intervals can line up to cause performance
problems against the configuration source. To reduce the impact, this
change adds a splay option to allow the operator to configure a random
delay prior to the polling to attempt to offset the polling

Change-Id: I1a8be3345d783da9014eca7ea26da19d57e767c0
Closes-Bug: #1677314
6 years ago
ricolin d677da173b Allow setting minimum polling interval for collect
Add `min_polling_interval` in `[default]` config.
This config option allow setting minimum polling interval,
default is 1 second.
Closes-Bug: #1656893
Depends-On: If863d2e190a4e1701ebf1fcb99a783075cda2556
Change-Id: I213afabeeb63f143ce359a3495a6d88b9b74ef4d
6 years ago
Dan Prince b6364df3e5 Add support for Zaqar websockets
This patch adds a new zaqar.use_websockets option which if set to True
in the config file will configure the Zaqar collector to use
the websocket transport instead of wsgi. This can be more efficient
where you want to avoid the continuous polling of o-c-c and instead
just listen on the websocket that is subscribed to a queue.

Like other collectors each iteration creates a new socket object. This
allows us to use the normal re-exec logic in o-c-c and thus gives
the option to re-configure the agent in the future to use other types
of collectors. We could (optionally) look into a higher level option in
the future that would allow o-c-c to avoid re-exec'ing and thus re-use
the same websocket for multiple sets of metadata.

Depends-On: Ia2a8deb599252d8308e44d595eb2bf443999aaad
Change-Id: Id5c7ed590df776844b6c7961eb40f89206cd24e0
7 years ago
Steve Baker 76975f04e5 ec2 collector attempt config-drive before metadata server
This change will first attempt to get ec2 metadata from a v2 config
drive before falling back to the nova metadata server if no config
drive is detected.

Config drive is enabled in overcloud nodes, so using this for
os-collect-config will allow the undercloud nova metadata server to be

V2 of the config-drive was released in OpenStack Folsom, so a decision
was made to not support V1 discovery at all. This makes the block
device candidate selection much simpler than the cloud-init one[1].


Change-Id: I2aec0544fe40c2e901aee38c79bb838a627d20da
Closes-Bug: #1619074
7 years ago
Steve Baker a950edb320 Treat ec2 collector data as immutable
Currently the ec2 collector polls the nova metadata service every
$polling_period even though the data is not expected to change and no
known config actions have been written to respond to changes in these

With larger overclouds, this metadata polling will cause noticeable
load on the undercloud, especially nova-api and neutron (for port

This change will check for data in /var/lib/os-collect-config/ec2.json
and simply return that if it exists.

Change-Id: I84156f8b005b319fccf8ac17af8943aefb0a8f6d
Closes-Bug: #1619072
7 years ago
Steve Baker 61ff6aaec7 Use TEST-NET-1 for unit tests, not
Currently if http requests are not correctly mocked, behaviour of the
unit tests will depend on what happens to be running on localhost.

Jobs like gate-os-collect-config-python27-ubuntu-xenial are currently
broken because discover requests are being made to localhost and
something has change in the environment to cause test failure.

This change does the following:
- replace unit tests addresses from to (TEST-NET-1)
- fix mocking of tests which now stall because isn't accessable
- change the mocking approach to to
  a simple stub class

Change-Id: I6d9450d6854b9f50dac0f55f54f1ea1d579a27ac
7 years ago
James Slagle 9ee07c96c2 Revert "Treat ec2 collector data as immutable"
All CI jobs are timing out currently during NetworkDeployment
because 99-refresh-completed is not signaling to Heat due to
instance-id being detected as null by os-apply-config.

I suspect this patch is causing the issue.

This reverts commit 8717436db9.

Change-Id: Id0d122d7e855cd828a0742dabc5f7089fb6a913c
Closes-Bug: 1623754
7 years ago
Steve Baker 8717436db9 Treat ec2 collector data as immutable
Currently the ec2 collector polls the nova metadata service every
$polling_period even though the data is not expected to change and no
known config actions have been written to respond to changes in these

With larger overclouds, this metadata polling will cause noticeable
load on the undercloud, especially nova-api and neutron (for port

This change allows collect calls to raise a SourceAlreadyCollected
exception if the data for this collector never changes. The ec2
collector raises this if its data has already been cached to

Change-Id: Ib7df590601132857690c8ab64fe32098a81752d8
Closes-Bug: #1619072
7 years ago
Jenkins 20630c3360 Merge "Merge deployments data for zaqar collectors" 7 years ago
Dan Prince 08ba371994 Merge deployments data for zaqar collectors
This is required so that a the TripleO undercloud can
switch to polling for metadata from a Zaqar rather than Heat or

Change-Id: Ieb076185314a7aeadd8a708ab01b48dfbccef006
Closes-bug: 1604098
7 years ago
Steve Baker e5518c11c7 Set a timeout for all data fetches using requests
A timeout config value is added for each collector which uses requests
to fetch data, and this value is used for any requests calls.

Without a timeout a request may stall indefinitely and
os-collect-config will stop polling.

A timeout default of 10 seconds is chosen as the default. This is used
for both the connection timeout and the read timeout.

Change-Id: I4ad0065b5a85393105c6385a15653d7204b4f880
Closes-Bug: #1600652
7 years ago
Swapnil Kulkarni (coolsvap) 8b28ced627 Replace assertEqual(None, *) with assertIsNone in tests
Replace assertEqual(None, *) with assertIsNone in tests to have
more clear messages in case of failure.

Change-Id: I2cc0d913d83ea56c323475daddb66a0047564b42
8 years ago
Dan Prince b63188eb25 shorter sleeps if metadata changes are detected
This patch updates os-collect-config so that the sleep interval
time is shortened if changes are detected. This should decrease
deployment time when using Heat templates which use depends_on
to step through a sequence of software deployment resources.

The new default sleep is set to 1 and increases
by sleep_time *= 2 until it reaches the default
sleep interval again.

Change-Id: I5cbd0956db2abebec876b15bee72b70ec64d5aef
8 years ago
Thomas Herve b13122918c Add support for zaqar transport
This adds a new zaqar collector able to read configuration from a
specific zaqar queue.

blueprint software-config-zaqar
Change-Id: Ie38af7b59e7a1aa370ac7760bb7819e37c2165c3
8 years ago
Steve Baker acafb1b94a Merge deployments data for collectors heat, request
This is required so that a swift-enabled TripleO undercloud can switch
to polling for metadata from a TempURL rather than heat.

Change-Id: I73ac9e01f85e0c72ce7411e2c61c545322f3dccc
Closes-Bug: #1424913
8 years ago
Steve Baker 4b68dfd53d Refactor merging out of cfn collector
This is a pure refactoring change which moves the os-apply-config
deployment data merging into its own module. This will allow
other collectors (request, heat) to do the same merging.

Partial-Bug: #1424913
Change-Id: Ic78a60e3efebadbb06ebfd262ceb275ca519a3f2
8 years ago
Jenkins 163030c081 Merge "Drop use of 'oslo' namespace package" 8 years ago
Doug Hellmann 4a41f0b3e5 Drop use of 'oslo' namespace package
The Oslo libraries have moved all of their code out of the 'oslo'
namespace package into per-library packages. The namespace package was
retained during kilo for backwards compatibility, but will be removed by
the liberty-2 milestone. This change removes the use of the namespace
package, replacing it with the new package names.

The patches in the libraries will be put on hold until application
patches have landed, or L2, whichever comes first. At that point, new
versions of the libraries without namespace packages will be released as
a major version update.

Please merge this patch, or an equivalent, before L2 to avoid problems
with those library releases.

Blueprint: remove-namespace-packages

Change-Id: If51059c31c82d5235e2ae21143911b5561783ca6
8 years ago
Yanyan Hu f6ca7618c4 Use non-versioned auth_url for keystone
keystoneclient discover can't get v3 auth_url using v2 auth_url. It
accepts an url without version number. This patch fixes this issue.

Change-Id: I45b4c016bd91afc39d39620920951043bb71df05
Closes-Bug: #1398254
8 years ago
Endre Karlson c917665af9 Add ca_certificate option for SSL'd api
Change-Id: I12eb9dbbb8bee24e50ae342ffbc7356d4583a973
9 years ago
Jenkins de6d420f40 Merge "Add unit test for heat_local" 9 years ago
Dougal Matthews 4bc37fd50e Bump hacking to 0.9.x series
Change-Id: I474266c647f31f0ad92ec9878fc03eb5a5c97f43
9 years ago
Gregory Haynes 46e950ddd1 Enable local collector by default
This is a useful collector and it seems odd to not have it on by

Change-Id: Ibc3be6ff27c6290ce32bc01d0e62acf3253545c3
9 years ago
Jenkins 72bd2854f6 Merge "Handle missing local collector directory" 9 years ago
Gregory Haynes d05f57b092 Handle missing local collector directory
We currently explode if local collector is enabled with the data
directory missing. Changing this to a warning allows us to have this
turned on by default without greatly inconveniencing users.

Change-Id: I36364ba1a0706a5a2c820eadd526f2ba424ac665
9 years ago
Steve Baker ad475ee927 Collector which does a simple HTTP GET
This change implements a collector which does an HTTP GET via
python requests to fetch the metadata.

It should work with any GET-able URL, however it is designed to
work with Swift TempURLs.

Swift objects are not consistent, so the Last-Modified header is
checked for each poll and metadata is not fetched if the last
modified is not newer than the previous successful poll.

This collector will be enabled for OS::Nova::Server
software_config_transport: POLL_TEMP_URL which is available
in the Juno release of Heat. Using POLL_TEMP_URL will result
in no metadata polling load on heat, which has historically been
an issue with tripleo scalability.

Change-Id: I22155c22bdcc3c81a5e945ca5436a8f29f196528
9 years ago
Gregory Haynes 6d32511dd0 Exit with error status on failed command
When we detect a failed command we log ERROR but we do not return an
error status. This makes it difficult for programs which may run
os-collect-config to detect whether a run was sucessful.

This only applies to runs which are performed with --one-time argument
as this is a straightforward case.

Change-Id: I168862e8c75c15d1ea405a417908d1284feb7b32
9 years ago
Gregory Haynes 7e913145a3 Fix py3k popen and locale test fails
A popen call is made in the Python 3 platform module which causes our
assertions for our mock'd popen to fail. Also fixing a locale issue to
use defaultlocale() rather than LC_ALL.

Co-Authored-By: Steve Kowalik <>
Change-Id: I45cdf7d921fbca5f09e5f65bd69fcee83fb7c8e4
9 years ago
Motohiro OTSUKA 3822b55d21 Add unit test for heat_local
For improving test coverage, this patch adds unit test for heat_local.

Change-Id: I000a73d8c54b5bb9c506d8674fbd963fed72e8f3
9 years ago
Yanyan Hu 47bb4bddb2 Using keystone discover to find V3 auth_url
This patch uses keystone client discover to find correct V3
auth_url which is expected by heat collector.

Change-Id: I2b8b5a768bdeb1cd1d20fab3b4234306c1429d6d
Closes-Bug: #1341936
9 years ago
Jenkins 7d61ca940c Merge "Add support for Python 3" 9 years ago
Steve Kowalik a3c1159ed4 Add support for Python 3
Make changes pretty much all over the code base with respect to
encoding strings and fixing imports to support Python 3.

Change-Id: Id1920129001b8e223474c1b2faf8bd9d527fe7e7
9 years ago
Clint Byrum 9cb442e93c Allow using non-default collectors
The local collector is not in DEFAULT_COLLECTORS, but should be usable
explicitly. It, however, suffers from a bug where only
DEFAULT_COLLECTORS are allowed through.

Change-Id: Ia42d1acd39638b448e2e2bfa26aff1c7ae415b71
9 years ago
Jenkins 9b28e39fa1 Merge "Add a local data collector" 9 years ago
Clint Byrum 4696fd4f93 Add a local data collector
This collector will collect data from the local system, allowing image
builds or simple processes to influence the metadata.

implements bp tripleo-juno-occ-localdatasource

Change-Id: I0e58e8c631ffe8b63e8b4117df2c9ce2f413044f
9 years ago
Clint Byrum 79efeba4c3 Do not log unconfigured collector as warning
The configuration will dictate whether or not something is configured.
If it is not, this is a normal state and should not be logged as a

Change-Id: I479f0aed5837871009bc69fa028f5eb64a060c53
Closes-Bug: #1321551
9 years ago
Clint Byrum 42efe2b969 Revert "Make heat the default collection method"
This reverts commit 6b478e9d90.

We will break anybody who is expecting CFN to be tried in all
circumstances with this. We probably just need to base which collectors
to try on what configuration we have, and not log warnings on
unconfigured collectors.

Change-Id: I4bf7d6f9af9487bf9d2c0942381c0ba68fc03ee9
9 years ago
Clint Byrum 6b478e9d90 Make heat the default collection method
Previously we were relying on the CFN compatibility API. This makes the
native Heat version the default.

Note that we want to keep full coverage, which is why we are explicitly
adding cfn back in during tests.

Change-Id: I5adedd052827e176e2f39071c719600df62019d7
Closes-Bug: #1321551
9 years ago
Clint Byrum ee911cfb69 Cache auth_ref from keystoneclient
Auth_ref is documented as a serializable object which we can save off
and use to reinitialize keystoneclient. That should in turn be passable
to heatclient directly. With this code, we will avoid authorizing with
Heat if we have a token in the cache.

Change-Id: I3dc6c3726054e77449cc9be46229e6d3521d4ef2
Closes-Bug: #1321437
9 years ago
Clint Byrum 328e1c6cfe Add dogpile cache to keystone abstraction layer
In later commits we will use this cache to memoize access to the
authentication details.

Change-Id: I389f78fe1eb176e37c90a1a87a4ba5fde3b33f05
Related-Bug: #1321437
9 years ago
Clint Byrum dc35d20358 Use json equality rather than raw text
It turns out sometimes json will serialize differently given the same
python structure. A check of the two parsed data structures will detect
changes more reliably.

Change-Id: Id165b36c0fa0fa89730c3507444a41c68bd70fb3
Closes-Bug: #1320262
9 years ago
Steve Baker b066d4cc85 Only top-level some deployment configs
Only config which contains os-apply-config data should be written
to the top level config structure. This change will only top-level
configs which are in the group os-apply-config or the default group

This tripleo-heat-templates change will specify a group for all
config resources however
the expected top-levels will still be written out without this

This change is to fix heat software config when the config is not

Change-Id: I16e42825121cf4968d4b0f0d266f4f6daa962921
Closes-Bug: #1299109
9 years ago
Jenkins 08a3237484 Merge "Add collector for heat API resource metadata" 9 years ago
Jenkins 87b0fc4049 Merge "Restore 'deployments' key to cfn Metadata" 9 years ago
Steve Baker 80c98a33ee Add collector for heat API resource metadata
This collector uses keystoneclient and heatclient to poll for the
configured resource metadata.

Changes were required to test_collect to allow collectors which needed
to fake something other than requests.

Change-Id: I3e93fe38b15f71193a4c024b24e6260d6adcf1b3
9 years ago
Clint Byrum 6198acbfcc Commit changes to exploded deployments to cache
Before this, the exploded deployments that the cfn collector produced
would not ever be committed, and thus would always appear to have been
changed. This resulted in os-collect-config running the command

This requires some refactoring so that we commit changes to the cache
based on what was actually written, rather than just the static list of

Closes-Bug: #1307153

Change-Id: I618ef5d752ed6519e8b7bfc090de03f2f24e73ce
9 years ago