Commit Graph

71 Commits (master)

Author SHA1 Message Date
Ben Nemec 4fa30ae5d7 Don't ignore SIGPIPE
By default Python configures SIGPIPE to be SIG_IGN, which means to
ignore the signal. We don't want that as it causes problems when
journald restarts and our log calls start triggering SIGPIPEs.
Instead, we want to allow the SIGPIPE to kill the process so it can
be restarted by systemd.

Change-Id: I512139b96b2de8b372efc91e8a3fc8d33553405a
Closes-Bug: 1795030
5 years ago
Alex Schultz 0762fcb0b5 Add splay option to offset polling intervals
When the os-collect-config process is started on multiple systems at the
same time, the polling intervals can line up to cause performance
problems against the configuration source. To reduce the impact, this
change adds a splay option to allow the operator to configure a random
delay prior to the polling to attempt to offset the polling

Change-Id: I1a8be3345d783da9014eca7ea26da19d57e767c0
Closes-Bug: #1677314
6 years ago
ricolin d677da173b Allow setting minimum polling interval for collect
Add `min_polling_interval` in `[default]` config.
This config option allow setting minimum polling interval,
default is 1 second.
Closes-Bug: #1656893
Depends-On: If863d2e190a4e1701ebf1fcb99a783075cda2556
Change-Id: I213afabeeb63f143ce359a3495a6d88b9b74ef4d
6 years ago
Dan Prince 90cb2721e9 Only re-exec on config change or HUP
There are 2 things at play here. First the logic added in
4cfeb28d12 is no longer needed
because our initial sleep time is very low and increases
gradually up to the max.

2nd, I'm proposing that we avoid reexecing unless the config
file actually changes. Not re-execing will give us the option
to optimize os-collect-config for some long running collectors
(like zaqar websockets).  Also, Os-collect-config updates would
already be handled by packaging restarts and or other deployment
system changes anyways.

Change-Id: I04b2752d007089f72af42c88c4249c3e11c8346f
7 years ago
Ben Nemec f092ea133f Switch to oslo.log again
We're currently still using unmaintained oslo-incubator code for
our logging, which is bad.  This switches us to oslo.log as
described in [1].


Change-Id: Ibce86ab4ee24eeb55d0de1b0d5ff4ee4ea6ef66f
7 years ago
Ben Nemec ab5c66df45 Revert "Switch to oslo.log"
This isn't quite right and broke on stable/liberty.  Pushing
a revert in case I3c22d77dece399d21ab94783b74990789a1e1481
doesn't actually fix the problem.  We should probably merge
whichever passes first.

This reverts commit 69653318f4.

Change-Id: I9304429f25d28ca756e50b1788e149c5bb46b1d6
7 years ago
Ben Nemec 69653318f4 Switch to oslo.log
The old oslo-incubator log module isn't maintained (and doesn't even
exist anymore), so we don't really want to be using it.  It appears
this was the only incubator module we were actually using, so this
allows us to remove all of the unmaintained incubator code.

Change-Id: Ib4ad3b231360987a1ef4f95b5b5a8b656232efc4
7 years ago
Dan Prince b63188eb25 shorter sleeps if metadata changes are detected
This patch updates os-collect-config so that the sleep interval
time is shortened if changes are detected. This should decrease
deployment time when using Heat templates which use depends_on
to step through a sequence of software deployment resources.

The new default sleep is set to 1 and increases
by sleep_time *= 2 until it reaches the default
sleep interval again.

Change-Id: I5cbd0956db2abebec876b15bee72b70ec64d5aef
8 years ago
Thomas Herve b13122918c Add support for zaqar transport
This adds a new zaqar collector able to read configuration from a
specific zaqar queue.

blueprint software-config-zaqar
Change-Id: Ie38af7b59e7a1aa370ac7760bb7819e37c2165c3
8 years ago
Steve Baker acafb1b94a Merge deployments data for collectors heat, request
This is required so that a swift-enabled TripleO undercloud can switch
to polling for metadata from a TempURL rather than heat.

Change-Id: I73ac9e01f85e0c72ce7411e2c61c545322f3dccc
Closes-Bug: #1424913
8 years ago
Doug Hellmann 4a41f0b3e5 Drop use of 'oslo' namespace package
The Oslo libraries have moved all of their code out of the 'oslo'
namespace package into per-library packages. The namespace package was
retained during kilo for backwards compatibility, but will be removed by
the liberty-2 milestone. This change removes the use of the namespace
package, replacing it with the new package names.

The patches in the libraries will be put on hold until application
patches have landed, or L2, whichever comes first. At that point, new
versions of the libraries without namespace packages will be released as
a major version update.

Please merge this patch, or an equivalent, before L2 to avoid problems
with those library releases.

Blueprint: remove-namespace-packages

Change-Id: If51059c31c82d5235e2ae21143911b5561783ca6
8 years ago
Dougal Matthews 4bc37fd50e Bump hacking to 0.9.x series
Change-Id: I474266c647f31f0ad92ec9878fc03eb5a5c97f43
9 years ago
Gregory Haynes 46e950ddd1 Enable local collector by default
This is a useful collector and it seems odd to not have it on by

Change-Id: Ibc3be6ff27c6290ce32bc01d0e62acf3253545c3
9 years ago
Steve Baker ad475ee927 Collector which does a simple HTTP GET
This change implements a collector which does an HTTP GET via
python requests to fetch the metadata.

It should work with any GET-able URL, however it is designed to
work with Swift TempURLs.

Swift objects are not consistent, so the Last-Modified header is
checked for each poll and metadata is not fetched if the last
modified is not newer than the previous successful poll.

This collector will be enabled for OS::Nova::Server
software_config_transport: POLL_TEMP_URL which is available
in the Juno release of Heat. Using POLL_TEMP_URL will result
in no metadata polling load on heat, which has historically been
an issue with tripleo scalability.

Change-Id: I22155c22bdcc3c81a5e945ca5436a8f29f196528
9 years ago
Gregory Haynes 6d32511dd0 Exit with error status on failed command
When we detect a failed command we log ERROR but we do not return an
error status. This makes it difficult for programs which may run
os-collect-config to detect whether a run was sucessful.

This only applies to runs which are performed with --one-time argument
as this is a straightforward case.

Change-Id: I168862e8c75c15d1ea405a417908d1284feb7b32
9 years ago
Jenkins 7d61ca940c Merge "Add support for Python 3" 9 years ago
Steve Kowalik a3c1159ed4 Add support for Python 3
Make changes pretty much all over the code base with respect to
encoding strings and fixing imports to support Python 3.

Change-Id: Id1920129001b8e223474c1b2faf8bd9d527fe7e7
9 years ago
Clint Byrum 9cb442e93c Allow using non-default collectors
The local collector is not in DEFAULT_COLLECTORS, but should be usable
explicitly. It, however, suffers from a bug where only
DEFAULT_COLLECTORS are allowed through.

Change-Id: Ia42d1acd39638b448e2e2bfa26aff1c7ae415b71
9 years ago
Jenkins 9b28e39fa1 Merge "Add a local data collector" 9 years ago
Clint Byrum 4696fd4f93 Add a local data collector
This collector will collect data from the local system, allowing image
builds or simple processes to influence the metadata.

implements bp tripleo-juno-occ-localdatasource

Change-Id: I0e58e8c631ffe8b63e8b4117df2c9ce2f413044f
9 years ago
Clint Byrum 79efeba4c3 Do not log unconfigured collector as warning
The configuration will dictate whether or not something is configured.
If it is not, this is a normal state and should not be logged as a

Change-Id: I479f0aed5837871009bc69fa028f5eb64a060c53
Closes-Bug: #1321551
9 years ago
Clint Byrum 42efe2b969 Revert "Make heat the default collection method"
This reverts commit 6b478e9d90.

We will break anybody who is expecting CFN to be tried in all
circumstances with this. We probably just need to base which collectors
to try on what configuration we have, and not log warnings on
unconfigured collectors.

Change-Id: I4bf7d6f9af9487bf9d2c0942381c0ba68fc03ee9
9 years ago
Clint Byrum 6b478e9d90 Make heat the default collection method
Previously we were relying on the CFN compatibility API. This makes the
native Heat version the default.

Note that we want to keep full coverage, which is why we are explicitly
adding cfn back in during tests.

Change-Id: I5adedd052827e176e2f39071c719600df62019d7
Closes-Bug: #1321551
9 years ago
Clint Byrum 328e1c6cfe Add dogpile cache to keystone abstraction layer
In later commits we will use this cache to memoize access to the
authentication details.

Change-Id: I389f78fe1eb176e37c90a1a87a4ba5fde3b33f05
Related-Bug: #1321437
9 years ago
Steve Baker 80c98a33ee Add collector for heat API resource metadata
This collector uses keystoneclient and heatclient to poll for the
configured resource metadata.

Changes were required to test_collect to allow collectors which needed
to fake something other than requests.

Change-Id: I3e93fe38b15f71193a4c024b24e6260d6adcf1b3
9 years ago
Clint Byrum 6198acbfcc Commit changes to exploded deployments to cache
Before this, the exploded deployments that the cfn collector produced
would not ever be committed, and thus would always appear to have been
changed. This resulted in os-collect-config running the command

This requires some refactoring so that we commit changes to the cache
based on what was actually written, rather than just the static list of

Closes-Bug: #1307153

Change-Id: I618ef5d752ed6519e8b7bfc090de03f2f24e73ce
9 years ago
Clint Byrum 16158684a6 Change collector interface to return a list
With the new OS::Heat::StructuredDeployment resource, each Metadata
section may have multiple "deployments" in it. With this, we will return
a list with tuples of key and content to write to the cache.

Change-Id: I9f4272b0761e1dfd850bc5a5c6b27a78f126281f
Related-Bug: #1295787
9 years ago
Clint Byrum 8f095feee6 Move default cache dir to persistent location
While this is called a "cache", it is important for it to survive. On
reboot, servers may need what was in the cfn config to restore complex
network configurations.

We introduce a new command line option, --backup-cachedir, that will
default to the old path, /var/run/os-collect-config. This will keep
things working for any tools that have been hard coded to use the old

Change-Id: I78b3851b35addfc16913e3cd53c9d0e7eb3d191a
9 years ago
Mark McLoughlin a479edb493 Log the value of OS_CONFIG_FILES
We pass the list of json files containing the collected metadata to
os-refresh-config using the OS_CONFIG_FILES env variable, so it's a
pretty useful piece of information to log.

Change-Id: Id09e3f352e6a5a09e4183c0743a6e99a2783a888
10 years ago
Clint Byrum 69d74dde12 Reduce default polling interval to 30 seconds
The initial value of 300 seconds was a conservative estimate. However,
the requests and responses are somewhat small, so we can drop the polling
interval significantly and still maintain a high degree of network
scalability. After measuring the responses from the ec2 and cfn servers
with typical workloads, at 30 second intervals 100 servers will generate
around 26kB/s of requests, with about 66kB/s of responses.

Change-Id: Iaa99ae405ba7c72ef8afc11c946400a2d0db5206
10 years ago
Clint Byrum e82a98f127 Imply --one-time when --force is used
Without this change, if a user runs os-collect-config --force, it will
lock the user in an infinite loop running the command over and over with
very little chance to cancel. There are no compelling use cases for that
behavior, but it is extremely inconvenient, so implying --one-time
improves usability of os-collect-config for users.

Change-Id: Ia8c9bf0bf97ab9e40e465c947c2f0cbeb981c08e
10 years ago
Clint Byrum db896fdf3a Make --version work properly
We have pbr and we have oslo.config. Tell them about eachother.

Change-Id: I65d449b0ed961c824a3e7c1307aab63f73bc03ad
10 years ago
Clint Byrum 635955def4 Add --print option to force printing
The use case for --print is an administrator wanting to view the
metadata that os-collect-config sees without running any commands.

Fixes bug #1213195

Change-Id: I0251f2c70574aeaa79997ce822d2a5ffbe08e345
10 years ago
Clint Byrum c1e60159f5 Add a --force option
This is a useful debugging and/or system fixer tool for instances where
metadata has not changed but one needs to re-run the configuration.

Fixes bug #1223693

Change-Id: I62b097bafa339fefcf6e03d11636f5ab622fb71a
10 years ago
Clint Byrum 6eb8eb9af3 Store list of collected configs in cachedir
This will allow tools like os-apply-config to read the list even when
they are run out of band from os-collect-config.

Change-Id: Ic4eaf649e234f4a1367d20c7ec52e93e787a7bb3
10 years ago
Clint Byrum c2b1b3909c Add --print-cachedir option
The option allows other programs to find the cache directory and files
without having access to OS_CONFIG_FILES.

Change-Id: Iad87efb65ea4db387e94160376c9eaf956fff413
10 years ago
Derek Higgins 4cfeb28d12 Rerun os-collect-config if config changed
Keep a hash of the config file for os-collect-config and if it changes
during a failed run then rerun immediately(without sleep), effectively
causing new nodes to be ready 5 minutes earlier.

Because the cfn credentials are placed into os-collect-config.conf by
os-apply-config and are not in place the first time os-collect-config is
run, the first run of os-collect-config results in error, o-c-c then
sleeps for 5 minutes before running successfully the second time.

Fixes bug #1219186

Change-Id: I090de7a3d84e0ea342f1a422646c0c455eb37f4a
10 years ago
Derek Higgins bbb1c77106 Setup loghandlers in o-c-c binary
On a system with o-c-c installed by pip the binary generated by PBR
calls __main__() directly, the code that sets up logging sould be placed
here otherwise it will be bypassed. Resulting in missing log messages.

Change-Id: I94ba4f61be9595a6ddee134d806e5f99ae4adf73
10 years ago
Robert Collins 5b8fd361c2 Stop resetting logging on every call to __main__.
__main__ is called directly during tests, but was resetting the
logging environment within it, which prevented tests from capturing
the log events.

Change-Id: If710e11091723144c97c88aab4aa5e6126844d2b
10 years ago
Clint Byrum 1897ecf001 Do not cache new data if command fails
The point of delaying the commit of data to the cache is that we want to
make sure the command succeeds before giving up on the data changes.
This will ensure that we keep trying the command with any given change
to the metadata until it succeeds.

Change-Id: Idf3a09686b4bbf0e16a9bc9f3359ee9937fcc627
10 years ago
Clint Byrum 13b6732546 Reverse default order of collectors.
The default order results in less-dynamic heat_local and ec2 overriding
the more dynamic cfn source. That is the opposite of what is desired.

Change-Id: I7e1feb2e6869b4f076200668dd204219ecc4224e
10 years ago
Clint Byrum 1dcc5a3571 Re-execute after commands and on SIGHUP
After we have run a command and committed, re-execute ourselves. This
ensures that we will get any configurations that may have come from
underlying commands. Also re-execute if os-collect-config is sent HUP.

Change-Id: I87b4d8ce44fcbc9458a3a4fbb2445e4c9d0ad4e7
10 years ago
Clint Byrum 3e2001e5ba Run forever if a command is configured
This makes os-collect-config stay resident and prepares it for a more
event based operation when the Heat API is ready for that via longpoll
or callbacks or something else.

Change-Id: Ic91f2201d504e9f8e0ada6d34a7d6d94785aec87
10 years ago
Clint Byrum 737fff67c3 Fixes for hacking 0.6
- Add missing License Header to exc.
- Use python 3 compatible print.

Change-Id: Ibbcbfadffb978f9d8789e1fbc0c0819da06489d3
10 years ago
Clint Byrum 140e527458 Detect invalid collectors.
The program would blow up anyway, but provide users with a more friendly
error message.

Change-Id: If70fc27b61963058af036568b88535b7f2dcff3f
10 years ago
Clint Byrum ba6c781db0 Allow changing the collectors that are used.
Positional arguments now specify which collectors to use. This allows
disabling a collector if it is problematic, and also re-ordering of the
emitted $OS_CONFIG_FILES from the default order if necessary.

Change-Id: I7e76db991c0b16c529c1cbf9a1ba9beb78e45482
10 years ago
Clint Byrum a3a93a9130 Enable heat_local collector. 10 years ago
Clint Byrum c42f3e46a7 Add heat_local for locally injected metadata.
Functionality has been added, but it is not called in main.
10 years ago
Clint Byrum ffb503ed04 Use Ec2Signer from keystoneclient
Also refactor some tests for less confusing monkeypatching.
10 years ago
Clint Byrum d78808a9a8 Add handling of a failing source.
Also fix intermittent race with cfg.CONF
10 years ago