71 Commits

Author SHA1 Message Date
Paul Bourke
fc30d583f9 Fix bandit gate jobs
* Inspected each error and fixed / added nosec where appropriate.
* build-swift-ring.py which was throwing sec errors is no longer used so
  removed it.
* Removed the dev/ directory from being checked.

Closes-Bug: #1617713
Change-Id: I25664cabca4137e5c9f499c1af3f5ce78b86fb56
2016-08-28 08:52:44 +00:00
Christian Berendt
c3d8262ec5 Add doc8 test and improve rst syntax
This will test all rst files inside the doc directory for style issues with
doc8 (an opinionated style checker for rst styles of documentation).

This will fix all syntax issues identified by doc8 and will improve
the syntax.

Change-Id: Id1b9563e07e77e306aef5a0767c98c27f87c5c0e
2016-08-04 15:09:10 +02:00
Yatin Kumbhare
bcebc7cb47 Add Python 3.5 classifier and venv
Now that there is a passing gate job, we can claim support for
Python 3.5 in the classifier. This patch also adds the convenience
py35 venv.

Change-Id: Ia7b1fb88ac108e1e8fd6ed963967e731ae3ca5a4
Closes-Bug: #1603006
2016-07-14 15:59:37 +05:30
Swapnil Kulkarni (coolsvap)
eb1bfc2c2e Add support to lint the install commands in Dockerfiles
Change-Id: Icb47ba8b13b13889e28bb285321f11344400edc2
Closes-Bug:#1582618
2016-05-26 17:41:42 +05:30
Jeffrey Zhang
7ab705cd4d Add Ubuntu binary build and deploy gate
The Ubuntu Mitaka repo is release, and we have add it.
So it is time to add the Ubuntu binary gate.

Closes-Bug: #1560386
Change-Id: I29d01147595c4ebebfa00798d01d5fd7398123a6
2016-04-02 16:02:58 +00:00
Steven Dake
a789346727 Fix gate to use world writeable docker socket
A recent change related to sudo securepath broke how sudo -E
works.  Now the PATH is reset by sudo as set by tox.  As a result
we can no longer rely on sudo -E anywhere in our gating system
relating to path inheritence from the parent shell.

This patch uses a shell operation in the setup_nodes.yml code
to chmod the docker socket to 666 so docker containers can build
properly.  Now docker operations don't return a permission denied
and we no longer require the sudo -E operation in any of our tox
scripts.  This isn't a security vulnerability because our gate
scripts are only meant to be run in OpenStack infrastructure.
To make this more clear I recommend moving these shell scripts to
tests/gate.

Change-Id: I5b3d6e280e3c12a25defedb14b5589ba642043fa
Closes-Bug: #1562383
2016-03-26 22:23:35 -07:00
Steven Dake
801ef2e843 Add reno support
For proper release note generation on the releases.openstack.org
website, this change will need to be backported into mitaka and
liberty as well.  Then a special one-file reno file will have to
be added for mitaka describing the features and the same story
for liberty.

See log of the discussion here:
http://eavesdrop.openstack.org/irclogs/%23openstack-release/%23openstack-release.2016-03-23.log.html#t2016-03-23T18:00:05

Change-Id: I3c298ae08ec4645f384ea5829cb0851fe82f4557
Partially-Implements: blueprint add-reno
2016-03-23 11:59:35 -07:00
Dave McCowan
39b8150f66 Add tox job for the bandit python security linter
Bandit is a job that parses and analyzes python code to detect
common insecure programming practices.

TrivialFix

Change-Id: Iea23a9bd23b25827043f02aa1a62fb0724f1d0ee
2016-03-08 16:33:59 -05:00
Jenkins
f3048ff8ab Merge "Introduce the hacking module to kolla" 2016-03-03 21:48:44 +00:00
Jeffrey Zhang
4e154f0018 Introduce the hacking module to kolla
Implements: blueprint add-hacking

Change-Id: Ia24a263555b9c1eb11e61b9d3fe89612a89daed6
2016-03-03 04:40:16 +00:00
Paul Bourke
80f0890c81 Add new gates for oraclelinux
Change-Id: Ic5c0e266639c8ed76d0af473d076d9ae6749102a
Implements: bp oraclelinux-gate
2016-03-01 15:34:34 +00:00
Jeffrey Zhang
d95fd11b2d Remove the tox linters env
TrivialFix

Change-Id: Icd69a4d6de8ef222b219e758fa38d58f2c98f505
2016-02-26 19:26:26 +08:00
Andreas Jaeger
8ad0b11065 Make pep8 *the* linting interface
According to the PTI (=Python Test Interface,
http://governance.openstack.org/reference/cti/python_cti.html), pep8
is the interface for codestyle checks. Move all tests from linters to
pep8.

This change will be followed by a change to project-config to use pep8
for testing in the gate.

Change-Id: I1a48f4b1e0b0950640192bcfc55121619a844b50
2016-02-11 09:55:28 +00:00
Jenkins
06b16d577d Merge "Convert gate to file backend rather than partition" 2016-01-26 03:04:43 +00:00
SamYaple
d7d5eecdbd Remove our unused jobs that now run in linters
Change-Id: I5c83b28ce3ef6c4080c68fa2fa353ad118ef7b71
Depends-On: I48faeb61fd3d630e7d7b6c949afcf10f724b0b72
2016-01-25 22:21:49 +00:00
SamYaple
6adb5c0aa5 Convert gate to file backend rather than partition
The existing gate partitioned a disk for use with docker, depending
on the gate it would use the swap disk (RAX) or a spare disk (HP).
However, with the new gates (Bluebox + OVH) there is neither a spare
disk nor a swap disk. This leaves us with one choice: File based loop
device.

This patch creates a file at /swapfile to ensure we have swap. It
creates a file at /docker to ensure we have a loop device for Docker.

Right now the /docker file is 10GB and the /swapfile is 4GB due to
size limitations in the gate across all servers and types. This has
proven to be enough space for all our current tests.

Additionally, reduce the number of threads the gate uses to 4 to
prevent the lockup and hour timeout we have been seeing as more
recently in the gate.

The scripts that setup the gate are moved to the tools directory
rather than the tests directory to match the structure of the other
projects.

Partially-Implements: blueprint functional-testing-gate

Change-Id: I3e370f2382b6df36103d8b2ceda9b21d9b4229d5
2016-01-21 22:59:13 +00:00
Jenkins
d8ad746151 Merge "Add validate scrits to linters" 2016-01-21 05:12:16 +00:00
SamYaple
d3a0f222cf Add validate scrits to linters
In an effort to reduce the number of gates, we can combine
non-conflicting jobs as we have already done with pep8 and bashate

TrivialFix

Change-Id: I81c2e523c563f8c761b57c8f20ca807d59da5d68
2016-01-15 18:53:25 +00:00
Jeffrey Zhang
3ce3b5b889 make the testr.conf required environment pass throught in tox
TrivialFix

Change-Id: I427bd3207de4ae87ab98dc024c4e2d98f0b572dd
2016-01-15 20:28:39 +08:00
MD NADEEM
e09e20cdbe Put py34 first in the env order of tox
To solve the problem of "db type could not
be determined" on py34 we have to run first
the py34 env to, then, run py27.

This patch puts py34 first on the tox.ini list
of envs to avoid this problem to happen.
Closes-Bug: #1489059

Change-Id: I4f791dfa620eacdd76cd46f193e190071ab64b6c
2016-01-14 10:11:24 +05:30
SamYaple
b863a46805 Standardize bashate and add linters
The linters gate combindes pep8 and bashate (and then any other
linters we want). This changes bashate to be a sript rather than doing
a script inline. This matches the way other projects (and
project-config themselves) uses bashate [1].

Once a change upstream merges we can remove the redundant pep8 and
bashate gates.

[1] https://github.com/openstack-infra/project-config/blob/master/tox.ini#L11
https://github.com/openstack-infra/project-config/blob/master/tools/run-bashate.sh

Related-Id: I2afa7931bf3e18798b95a8af33ccf5ef761c4ad5

TrivialFix

Change-Id: I2ea2c1156e1b4de008f19e333e2740795dc406de
2016-01-13 17:33:28 +00:00
Jeffrey Zhang
67e4607d43 Add py27 py34 pypy unittest
Change-Id: Iaf58463bd156924839062b8284a2d7151a153db5
2016-01-13 13:47:49 +08:00
Jeffrey Zhang
620d610eaa Generate the kolla-build.conf file using oslo_config
Closes-Bug: #1530122
Change-Id: I96039eade2aacf1bcd9d14d958cdddd915fc9a5c
2016-01-06 08:11:06 +08:00
Jeffrey Zhang
acbda89084 make the tox.ini indent uniform
TrivalFix

Change-Id: I75972366588c691d26512102ba567d6d1b258efe
2015-12-31 15:27:26 +08:00
SamYaple
4199634e98 pep8 the ansible modules
By ignoring the appropriate tests that pep8 does we can properly run
*most* of the pep8 tests on all of our modules allowing for a more
consistent coding style.

Closes-Bug: #1528431
Change-Id: I33f27a250d06d4f044267aa3ad189e092789b8df
2015-12-22 04:29:13 +00:00
SamYaple
ed82afa8e9 Simplify config creation
Convert config creation from a playbook to an action_plugin. This
reduces the complexity and confusion while retaining the same augment
structure and flexibility.

This allows us to remove the 0-byte files as requirements. They will
still be used if they are present (this means we require additional
documentation around them).

DocImpact
Closes-Bug: #1528430
Change-Id: I2c789f6be9f195c7771ca093a6d59499564b4740
2015-12-22 04:28:53 +00:00
hparekh
c3c7ee429a Pass environment variables of proxy to tox
When a development environment is under a proxy, tox is failed even if
environment variables of the proxy are set.

This patch fixes this problem.

Change-Id: I685ddbc6bb6e0fe25c308c35a7581785eebe3629
2015-12-11 09:18:08 +09:00
Michal Rostecki
7670611aa4 Exclude .eggs from pep8 checks
Change-Id: I11f5af83fc602332f6c9ac39f4c22731f8e84161
Closes-Bug: #1523365
2015-12-07 08:25:57 +01:00
Sam Yaple
f9ee96f216 Convert gate to Ansible setup
To support multinode we must now distribute our setup to multiple
hosts. Instead of making special rules for this, we are going to
convert our existing setup to Ansible. This way both setup proceedures
take place in the exact same fashion.

Partially-Implements: blueprint multinode-gate
Change-Id: I43ece298bba994e9b5083403ef3cf6d4245cda6d
2015-11-22 18:34:35 +00:00
Sam Yaple
00b7794f3e Remove unused tox jobs
The gates have been renamed. We can remove these jobs now.

Change-Id: I4f140301d0b0a04ca4196a755625f32569fe4b32
Partially-Implements: blueprint multinode-gate
2015-11-13 11:23:43 +00:00
Sam Yaple
e21c39b098 Rename jobs for Kolla
Due to the length of the job name and the tox target we run into an
uncommon limitation; the virtualenv that tox launches is nested in a
path that is too long. This leads to and error on our longest named
job which prevents tox from running at all.

This limitation is the limit for the line length of the first line
in a shell script. See `man execve` for more info. A quote from that
manpage: 'A maximum line length of 127 characters is allowed for the
first line in a #! executable shell script.'

Change-Id: I43fba2a5ff1890d699045496c9eaee5e849f3e75
Backport: Liberty
Partially-Implements: blueprint multinode-gate
2015-11-11 15:44:56 +00:00
Sam Yaple
4334c2adb9 Add new gate for multinode
Change-Id: I3e05e2d5c739794ae6ff0cc375dc6226f81bb542
Paritially-Implements: blueprint multinode-gate
2015-11-06 00:46:44 +00:00
Sam Yaple
3d7a0c6f55 Restrucutre gate scripts
So we can respect DRY and share as much code as possible I have broken
out the common code between the aio and multinode gate scripts.

Additionally, this lays the ground work for removing our policy on
root-everywhere by using sudo. Once we get the non-root stuff worked
out we can gate as non-root user.

Change-Id: I781c597ab10f2296b95f51ae27e0fa617ffe0a66
Partially-Implements: blueprint multinode-gate
2015-11-05 10:58:29 +00:00
Sam Yaple
931f577205 Removed unneeded tox things
The upstream gate patches have merged and these can be safely removed.

Change-Id: Icfad0ff161705239d75de2352bd61c5c9313e81c
Partially-Implements: blueprint functional-testing-gate
2015-10-23 20:24:02 +00:00
Sam Yaple
f39bb2ee13 In gate functional testing
This runs the ansible playbooks to completion.

It does not validate that OpenStack is working yet, but it will ensure
Ansible did not break.

Additional gates will be needed for this and there is a patch in
project-config to handle that.

Change-Id: I7a24a704023cc8f19d42844636e9cb512060f693
backport: liberty
Partially-Implements: blueprint functional-testing-gate
2015-10-20 19:15:33 +00:00
Kai Qiang Wu(Kennan)
3ed7b3c3b4 Add venv section in tox.ini
As jenkins docs job use venv envrionment to work, we
need add such section to make it work.

Partially-Implements: blueprint kolla-tox-doc
Change-Id: I716219515f2bf1375fd094b3aeca866cbcf5c7c6
2015-09-03 00:16:01 +00:00
Steven Dake
349a085352 Remove cruft in tox.ini from docker_templates
There is no longer a docker build system in the tree.  Remove
the cruft that was missed in the original removal patches.

Partially-Implements: blueprint gate-source-builds

Change-Id: I4e6132502e8c0f812a0b3a767dd9c7f5c7b92fe3
2015-08-29 22:55:45 -07:00
Steven Dake
730276aaf9 Remove deprecated --template option from gating
The template option is deprecated and --template is now default.
Also rename the tox jobs test calls since we won't have Docker
build operations.

Change-Id: I99df9c337680003e042e7e56a19d6cae2a59ed74
Partially-Implements: blueprint gate-source-builds
2015-08-29 02:12:20 -07:00
Sam Yaple
cbd42ca6e9 Move docker_templates to docker dir
Updated build.py to reflect this change.
Deprecate --template option and make it a noop.

Change-Id: I7cd98d1ee684a4c64984a49597159868152683b2
Partially-Implements: blueprint remove-docker-dir
2015-08-28 13:33:50 +00:00
Jenkins
3b5380ade0 Merge "Add developer docs automatic produce support" 2015-08-28 12:25:11 +00:00
Steven Dake
f895ee07cc Remove docker directory
Remove the docker directory (yay) and old build tools.

This will result in an f21 failing gate - however, I have a patch
submission to remove that gate from our project configuration.

Change-Id: I3c461cedb6906422c8076f60c470e773d6cdcf33
Partially-Implements: blueprint remove-docker-dir
2015-08-28 00:50:30 -07:00
Steven Dake
08e765fc3d Start of removal of docker directory
Start off the removal of the docker directory by making the gate
only run against docker templates.  The idea is we are going to
from this patch set forward completely abandon the functioning of
the docker directory and focus on making templates work properly.

In order to facilitate that, this patch set makes sure each change
gates properly for the changed service in the Templates build.

Note because git review and gerrit can't keep history on removal
followed by a git move operation, we first git rm the files affected
then git mv them to get the gate working again.

Every other patch in this patch set will fail the gate.  That is
expected behavior.

Change-Id: I1be2e2638aef4ada038bfe5f3dd563f040542df2
2015-08-28 00:50:29 -07:00
Steven Dake
55e6e1f8d2 Functional test Ubuntu source building
Change-Id: Ia3fa83645545bcd7d259fe7c973c849bd6cb1426
Implements: blueprint gate-source-builds
2015-08-28 00:02:45 -07:00
Kai Qiang Wu(Kennan)
eae8903ae0 Add developer docs automatic produce support
Partially-Implements: blueprint kolla-tox-doc

Change-Id: Ie950667a61d9ce45fcc7e26915a7aa0f0ca6b597
2015-08-28 06:11:08 +00:00
Sam Yaple
aada29b4db Remove tiered building in build.py
Many tiers have been shed over this issue and frankly enough is enough
on this tiering thing, it's tiering us apart! Now we use threading a
bit better as well.
No more tiers! Puns very much intended.

Additionally, this refactors the function names to make inc0 happier
and be consistent.

We also modify the test_build.py to catch the new statuses introduced
by the regex patch. Then we remove the Ubuntu Source from the Docker
folder test as that will never ever be functional.

Also fixed regex to properly match true regex expressions.

Co-Authored-By: Michal Jastrzebski (inc0) <michal.jastrzebski@intel.com>
Change-Id: I650fd6af76eddb809756762222e66aefd6fc1dca
Partially-Implements: blueprint build-script
2015-08-27 08:56:49 +00:00
Jenkins
0f2a4e1e0b Merge "Prepare for gating by distro, type, docker type" 2015-08-21 16:49:10 +00:00
Sam Yaple
7f029b759c Add Dockerfile template for kolla-ansible
Change-Id: I53113d2a7595177839daebe74fa5bc409573f824
Partially-Implements: blueprint dockerfile-template
2015-08-19 05:19:29 +00:00
Steven Dake
79d9951634 Prepare for gating by distro, type, docker type
We will gate on all combos of:
/docker
/docker_templates
Centos
Ubuntu
Binary
Source

Except for Ubuntu + Binary because there is no delorean
style repository for Ubuntu.

Change-Id: Ic1ed8c22c2f86ff339299939da9ac492a7c7c5f5
Partially-Impelements: blueprint gate-source-builds
2015-08-18 00:22:02 -07:00
Jeff Peeler
9a78a5b6f1 Change tox targets to use new build.py
tox -e images-centos-binary
tox -e images-centos-source

Both above now execute parallel image building script.

Change-Id: I6a4174b184fc239c6baa950a9c787fc6ad220774
Paritally-Implements: blueprint gate-source-builds
2015-08-14 17:32:14 +00:00
Steven Dake
f5c80d80d7 Change the tox targtes to match naming
The tox targets were previously src and rdo when they should
have been source and binary.

The gate review has been changed to these as well.

Change-Id: Ib09980c77e5859fa28a54953fc1f3f123c1886d7
2015-08-12 22:58:06 -07:00