This makes it easier to tell which node definition is being evaluated
Because xargs -P doesn't do bash functions, some of the apply_test has
been moved into test_puppet_apply.sh. All the test snippets and test
output is being recorded in files in tools/opt, then catted for the
user.
Change-Id: I4fb77f93f9e9a7216b62634438a11d8d297bdefe
There's no need to create per-provider image update logs for
providers which only use diskimages of the same.
Change-Id: I06e5413ac9357587cd4aa7afa34414b0cc01c124
* tools/who-approves.py: New script which, when run using
OpenStack's Gerrit server, builds JSON and YAML representations of
repos with information on the official owning project if any,
integration status, and groups with approve rights listing the
members of each along with their Gerrit preferred E-mail addresses
and usernames when available.
It was done as a demonstration to a representative of a foundation
member company who requested a list of the "core reviewers" for
official projects, optionally broken down by integrated vs. other.
I'm attempting to show that this data is already publicly available
and can be extracted/analyzed by anyone without needing to request
it.
Change-Id: I017d98e2a45d5082fce8f4eecb5e6f8633c02ac8
This attempts to determine what versions of which modules are
in place on a system. It can handle both branch and tag checkouts.
It assumes all modules are pulled from git.
Change-Id: Id4100e3c967506e7b244534115185eeeddaac74f
Although this only needs to be done once, it's ugly enough of a task
that it should have a script, and that script should be reviewed.
The script can be run locally on a machine without a hieradata and it
will create a local directory with all of the keys and none of the data.
It will also print out the total mapping and a report on which keys are
duplicated across hosts after group de-duplication is taken in to
account.
Change-Id: Id0af46abb8f4167bd91e95cd7a6cd2e35e5b0a94
This is the second part of splitting out log files for image builds;
putting diskimage-builder logs into separate files. They are added
with a special "dib" provider to keep them grouped.
Change-Id: I6406f63df2f625beee57d7ff9707bd3b4f3fc060
The apply test does some really simple string manipulation to generate
puppet manifests based on our site.pp on the fly. Unfortunately the old
code assumed there would be no blank lines within a node{} definition.
This assumption is breaking as we work to decouple the
openstack_project::server contents from within openstack_project
manifests and move that into site.pp.
This is a relatively simple fix where we csplit nodes based on the
previous /^}$/ which ends the last node{} definition instead of
splitting on blank lines. Added some comments to make what the parsing
does a bit more clear.
Change-Id: I2cc7317ed90423d6f923c4303a76e091a0bc2ebb
Generate a logging configuration template for nodepool that splits
image-build logs out into separate files.
A tool to generate and maintain this file based on the nodepool
configuration is added.
The logging configuration template this tool creates is added
alongside the main configuration. This is passed to the deployment
puppet module via a parameter.
Change-Id: I4723c68e5c83b536560e0a8bd7b73e312b46fc02
Depends-on: Iec77fb62c353d09186140b23912caf02225a643b
Changes to the projects.yaml (nee programs.yaml) in the
openstack/governance repo to accommodate big tent terminology have
resulted in a need to similarly adjust the script interpreting it.
While at it, update the variable names within the script for
clarity.
Change-Id: Ie001901c33749aa41181a09e43576bf9d516b4e5
This reverts commit 3b563fb59e7dfb233d4d0a07117499ac04f4a6ce.
Do not approve until Ief6c8e9bf7d71ddc63dba484390124546f14b93a is
applied to new nodepool images.
Change-Id: Ia7422d4f8fe316cc37c1310def0e7a9e2f232d35
Make /etc/puppet/environments/production/environment.conf so that
the apply test will find a default directory environment.
Change-Id: I062949d6c23545c40690edc39302095b0dec3c84
This causes apply-test.sh to get the list of puppet modules to do
integration test with from modules.env. Modules.env has been
refactored to have three lists of modules: Package modules(MODULES),
SOURCE_MODULES, and INTEGRATION_MODULES.
When PUPPET_INTEGRATION_TEST is not set, INTEGRATION_MODULES is
folded into SOURCE_MODULES.
In apply-test.sh, INTEGRATION_MODULES is sourced from modules.env
and some string/array manipulation is performed to pass each entry
to zuul-cloner.
Change-Id: I47302c5c7e8c41b985f16a05c4e9b8078ea867a3
--color=false remove ansi escapes
cping /etc/hiera.yaml /etc/puppet/hiera.yaml shuts up puppet
complaining that it doesn't have hiera.yaml to read
Change-Id: Iea7222c66f2fcc1338bc7baffb572d1a1ab2ecd6
Part of an effort to split most of puppet modules out of system-config.
https://review.openstack.org/#/c/130619/ should land before this
commit can be merged.
Change-Id: I7720ad7d2b25816c63e971112a7a733bde52b7f3
We are renaming openstack-infra/config to
openstack-infra/system-config. This patch addresses the one file
in the tools directory that needed editing.
Change-Id: Id078dc7031f833c6d1a58cddbcecf0666244c6a1
Fix all of the bashate failures. Also, bashate will install its negative
tests in its virtualenv, so we need to generate a list of files to pass to
bashate that excludes .tox/
Change-Id: I55559bf6137f705aec9a7f277928ad8d4cadc2ca
Since the split to project-config, there is no longer a need to
run 'tox' for projects_alphabetized, since those resources no
longer exist. Also removing the shell scripts that were running
the tests.
Change-Id: Ib7a762d872da9c5d945206807c405e7452b49638
Content from the config repo was split into a new repo,
project-config. This patch deletes the moved content from the
config repo.
Change-Id: Ifa1ae2cb7c23ae48f388b7d57a4c0d0799ef1dc2
Story: 167
In order to use disk-image-builder to build nodepool images, we need
a few things. First, we need additional debs to be installed for
disk-image-builder to work. Then we need the image elements themselves
to describe the images we want to create. Also, a few helper scripts
for locally working with the disk images have been included.
Tested with build-image.sh script; devstack-gate-trusty.qcow2 created
correctly
Change-Id: I539743147341dc5b387d103fecd3e9ff0f01fdfc
The zuul cloner needs the ZUUL_* env vars when setting up projects
for the new apply integration test.
Also, add some more debugging to both versions of the test.
Change-Id: I486b19cecd034250068e4a59604e1e0ca90ed83d
Make module installation conditional in install_modules.sh so that
we can specify that we _do not_ want it to install the modules
that will be participating in the integration test.
Add a variant of the apply test script (currently test.sh) that
uses zuul-cloner to check out the appropriate versions of modules
that are participating in the integration test.
Add jobs that run this to the experimental pipeline.
Change-Id: I16055cdd45f6e05aa8ebceb500c5709f265268df
It turns out that we pass a lot of args around simply for the
purpose of doing mirror selection. This goes away if everything
gets a mirror.
This change should only be merged after the parent change has
propagated to all puppeted slaves and updated nodepool images.
Co-Authored-By: Jeremy Stanley <fungi@yuggoth.org>
Change-Id: I4a721f6559d7160c8213268a8c1e35c667a67a4f
This is the result of running:
find modules/openstack_project/files/gerrit/acls/ -type f \
-name "*.config" -exec ./tools/normalize_acl.py {} 6 \;
Change-Id: I7aa27b859529b2bc8a990d6272334222996cbbc4
* tools/normalize_acl.py: Script which can perform one or more of a
list of normalizing transformations to an ACL file.
Change-Id: I063ab91b6e786eccaee61c669f0e840c7af6be14
* tools/atc/README: Simplify instructions to use mysqlclient
tab-separated-values stdout, and remove obsolete manual steps which
have been automated.
* tools/atc/email-stats.sh: The functionality this shell script
previously provided has now been translated to Python and folded
into the inner script.
* tools/atc/email_stats.py: Implement the old email-stats.sh outer
loop, make it possible to run in a virtualenv, and make previously
hard-coded constants into command-line parameters with sane and
dynamically-determined defaults. Update field order and contents to
cope with differences in Gerrit 2.8, use a proper YAML parser on the
programs.yaml which has also changed format since the previous run,
and adjust for the improved extra-atcs file format as well.
Change-Id: I2deef2e6766ff998190e66737e7cec9cdcd5459c
* tools/atc/email-stats.py: Adjust the order in which Gerrit E-mail
addresses are returned so that the one listed as the preferred
address is sorted to the beginning of the set.
Change-Id: I95b5817ada58f4059df4f5c0b406183f1c1e5b74
* tools/atc/email_stats.py: Move procedural code out of the global
scope to make this file safely importable, and clean up a couple of
unused variables flake8 noticed in the newly importable main
codeblock.
Change-Id: I1a29c84e067eea875ba57a16196723f65f60ff01
Rename tools/atc/email-stats.py to tools/atc/email_stats.py in
preparation for making it an importable module. This is being
separated from the subsequent change to make it more reviewable.
Change-Id: I1907de0715c96f54edb44910746fbb24a5ab8665