74 Commits

Author SHA1 Message Date
Marc Abramowitz
8d4671e3a7
Add a "list" subcommand
Add a list subcommand that allows listing of all the jobs defined in
yaml configuration, or on the Jenkins master, with the option of taking
glob parameters to allow matching some job names.

Also supports the recursive and exclude path control options.

  jenkins-jobs list 'wikimedia-fundraising*'

  jenkins-jobs list -r -p configs/ 'wikimedia-fundraising*'

Co-Authored-By: Darragh Bailey <dbailey@hpe.com>
Co-Authored-By: Sorin Sbarnea <ssbarnea@redhat.com>
Change-Id: I897b9ed35561e455dc6b89c3bacec74b54777903
Signed-off-by: Sorin Sbarnea <ssbarnea@redhat.com>
2018-03-09 10:58:21 -05:00
Zuul
5e023d0301 Merge "Capture XML output from test" 2018-03-07 15:23:08 +00:00
Darragh Bailey
3fe85e5bbd Capture XML output from test
Running a test with input yaml will result in output XML being sent
to the stdout. Use mock on the console to capture this so that running
the tests via testtools directly doesn't output unnecessarily

Change-Id: I4ccbfcb464d40a77c04913cb8281a0167504e1a4
2018-03-07 14:14:38 +00:00
Zuul
96aa1c756f Merge "re-raise the original Exception instead of raising a new one" 2018-02-23 16:36:23 +00:00
Darragh Bailey
630fc6e88a No warn on old plugin conf format not in use
Use a simple object to ensure matching against any default value is
possible to distinguish been not being set by a user versus use of
a negative default value passed.

This prevents triggering an incorrect notification to the user when
the code is querying for plugin options that could override the
default behaviour and the compatibility code attempts to look up an non
existing section. It would fail to distinguish between no section
present and current value to return matching the default value
suggested.

Change-Id: I5597c2628ccb5a4282a97a4ce5d3bbe41bd9eebb
2017-12-15 14:18:11 +00:00
Evgeni Golov
f2c4453df3 re-raise the original Exception instead of raising a new one
this preserves the error traceback and makes debugging easier

Change-Id: Ia513ac009b81d35e3df11939771aa29e40700c08
2017-11-06 13:36:17 +01:00
Darragh Bailey
af9d984baa
Basic folder support
Allows specifying a folder attribute for each job generated, which in
turn is used when creating or uploading to place the job under the
requested folder.

The job name is expanded after defaults are applied, to support the
attribute being defined within a set of defaults applied to a number of
jobs.

This in turn allows for multiple jobs with the same basename to exist,
provided they are targeted at different folders.

Does not support creating the folders if they do not exist.

Change-Id: I8c2157c4c81087cc972a048d1b88d5f08ac65361
2017-10-23 13:29:21 -04:00
zhangyangyang
8744b5b1a5 change assert(Not)Equals to assert(Not)Equal
According to http://docs.python.org/2/library/unittest.html
assert(Not)Equals is a deprecated alias of assert(Not)Equal.

Change-Id: Ibadf2b2c6cb09db6f5cc191240aad2ac8cb4ea33
Closes-Bug: #1329757
2017-09-13 17:24:13 +08:00
Peter Eisentraut
919052714c Add option to make test output file layout match Jenkins
Before, `jenkins-job-builder test -o outdir` would write

    outdir/job_name

With the new option `--config-xml`, it writes

    outdir/job_name/config.xml

to match the layout that Jenkins uses internally.  This is more useful
for diffing against existing Jenkins installations or uploading job
definitions by other means.

Also, it seems better to give the output files an .xml extension.

Change-Id: I78fc7aff531925ee071b8e19f40f766b4af0b5ad
2017-05-27 12:22:06 +01:00
Jenkins
4f77324fea Merge "Wrapper pre-scm-buildstep: added support for failOnError parameter" 2017-03-29 09:55:34 +00:00
Sandro Gattuso
12c2ca3396 Wrapper pre-scm-buildstep: added support for failOnError parameter
- Added failOnError argument for pre-scm-buildstep
- Introduced buildsteps argument and moved list of build steps to match xml configuration of the preSCMbuildstep plugin.
- Updated existing test cases to match the new argument structure
- Added test case for pre-scm-buildstep to verify creation of failOnError tag
- Added verification on version to maintain retro-compatibility
- Updated documentation to fetch example from testcase definition

Change-Id: I25c19cd0552cf2de97b9c391492d95779c1360be
Task: 3163
Story: 2000690
2017-03-28 17:10:29 +00:00
Sorin Sbarnea
8a7d77c180 Removed some deprecation warnings and enabled py35 on tox.
Change-Id: Icc5b28c4636b542a10502b7b1d2d2dc2028cc166
Signed-off-by: Sorin Sbarnea <ssbarnea@redhat.com>
2017-03-12 19:43:33 +00:00
Jenkins
8a1c91327e Merge "Test stream no encoding bug" 2016-12-01 06:58:59 +00:00
Jenkins
e47f9629bc Merge "Add view management functionality" 2016-11-10 16:35:23 +00:00
Jenkins
f0c99b70a7 Merge "user's input should not be interpreted" 2016-10-27 20:59:03 +00:00
Isaac Beckman
4b63891dd7 user's input should not be interpreted
using python's "input" rather than "raw_input" raise exception
since the user's input is interpreted.
In python 3.x raw_input = input, so the interpertation bug
only exists in python 2.x

Change-Id: I28781a8742d6e03bc850fd0178f317474603a9d2
2016-10-26 13:03:30 +03:00
Jenkins
771a94227c Merge "Regression test for update command with --delete-old" 2016-10-25 18:08:09 +00:00
Thanh Ha
1deb3aff4c
Add view management functionality
- Adds the ability for JJB to work with views
- Views can be created, updated, and deleted.
- New modules for List view and Build Pipeline view are added
- New tests for testing the deletion of views

Example View configuration:

    - view:
        name: MyView
        view-type: list

Change-Id: Idb29a4407bcc14593e10a4d951036cb04e8e6c27
Co-Authored-By: Brandon Leonard <brandon.leonard@rackspace.com>
Co-Authored-By: Joao Vale <jpvale@gmail.com>
Co-Authored-By: Lucas Dutra Nunes <ldnunes@ossystems.com.br>
Signed-off-by: Thanh Ha <thanh.ha@linuxfoundation.org>
2016-10-14 09:14:21 -04:00
Jenkins
02ae9dc8ed Merge "Fix confirm() util true state" 2016-09-30 16:32:47 +00:00
Thanh Ha
fd01e9e916
Fix confirm() util true state
The confirm() util incorrectly returns true only when the answer is
false. This patch reverses the logic so that it properly returns true
when the user inputs a true statement.

Include test case for delete-all command usage to accept or abort.
Mocking the input builtin ensures that the behaviour resulting from the
user inputting 'y' or 'n' will remain tested for delete-all.

Change-Id: I742804a8efe7da8ca6aa70f20059d02758910deb
Co-Authored-By: Darragh Bailey <dbailey@hpe.com>
Signed-off-by: Thanh Ha <thanh.ha@linuxfoundation.org>
2016-09-20 22:16:29 -04:00
Darragh Bailey
d75f1436cd Regression test for update command with --delete-old
Ensure that we explicitly check that there were no extra calls in
addition to the expected set of calls to the delete jobs API, when
populating keep_jobs with the list of jobs updated and checking that
the unmanaged job is ignored.

Correct the list of jobs returned by get_jobs call.

Prevent issue introduced by: https://review.openstack.org/352060

Change-Id: I5b7452d593707db7085e7d305893f05a77be82a1
2016-09-16 16:45:19 +01:00
Darragh Bailey
4573b3a25d Refactor base test classes inheritance for reuse
Refactor base test class inheritance to allow for BaseTest class to be
reused without needing to also import the TestCase class everywhere in
order to avoid having the common test function executed on base classes.

This makes it easier to build base testing classes and then simplifies
the subclassing of these for actual tests.

Change-Id: I89809e8082469f814f245db4a9ab7658aac8a405
2016-09-14 14:58:57 +01:00
Wayne Warren
e9a295b1fb Move CacheStorage to its own module.
And rename it to JobCache. The word "Cache" already implies "Storage"
so it makes more sense to name it after the things that it
stores...Jobs!

Change-Id: If7a7a56e8f6a3637f62e285eaa7ba989f67002dc
2016-09-02 08:58:09 -07:00
Wayne Warren
e75062d103 Merge builder.Builder and builder.Jenkins
Since both of these classes essentially build on basic functionality
provided by jenkins.Jenkins, merge them into a single class. Name this
class "JenkinsManager" to avoid confusion with jenkins.Jenkins.

Also moves some tests for update timeout configuration into the
tests.cmd.test_config module from tests.cmd.subcommand.test_update
module, replacing those tests with skipped stubs as reminders to
figure out a better way later.

Change-Id: I13c17bc90e29e702e5e02992e93cf3cdc689731d
2016-09-02 08:58:09 -07:00
Darragh Bailey
a949a274e3
Test stream no encoding bug
Tests that if no encoding is defined on the output that JJB does not
accidentally trigger a recursion issue when iterating over a large
number of jobs and attempting to wrap the output stream.

Inspired by commit message from: https://review.openstack.org/283441

Change-Id: Ie69c3af0beb9fd7c589a436e8c6d2862210f8a4c
Co-Authored-By: Hao Liu <hliu@redhat.com>
Signed-off-by: Thanh Ha <thanh.ha@linuxfoundation.org>
2016-08-29 23:43:35 -04:00
Wayne Warren
2236a00c53 Rename Builder.delete_job to Builder.delete_jobs.
Change-Id: If57a98a00e9577fdf7a3d7585905a2102d8c61cd
2016-08-20 11:28:00 -07:00
Wayne Warren
1fe7709923 Remove unused builder.Builder.update_job method
The deprecation warning message in this method was confusing and
seemed to be intended for builder.Jenkins.update_job but it's hard to
tell for sure. Seems likely to be an artifact of an outdated patchset
in the parallelize-the-things work done last year because it's not
clear how it could apply to builder.Builder.update_jobs which has a
significantly different use case than parallel_update_job.

Change-Id: Ifb3c1a40d81b0d43ac8e85151f0a99d1502e21db
2016-08-20 00:46:57 -07:00
Wayne Warren
aaae83c623
Factor XmlJobGenerator out of YamlParser.
Separate XML generation from Yaml parsing/interpreting. The goal here
is to allow different sources to provide data for XML generation,
including external API users writing job definitions in pure Python or
JJB developers who would like to work on alternative Yaml parsing code
since the current YamlParser has very likely reached the limits of
what can be reasonably done with one giant expandYaml procedure.

Change-Id: I9da848acac7e944c0e07286b7399b2e1956a58a5
2016-08-18 22:42:24 -04:00
Wayne Warren
ae1fb60f16
Disentangle YamlParser and ModuleRegistry classes
Create the ModuleRegistry anywhere other than inside the YamlParser
class. This will make it slightly easier to factor a XmlGenerator out
of YamlParser, but I also want to work toward eliminating the circular
references between YamlParser and ModuleRegistry which have been
making it difficult to understand overall program flow.

This commit also replaces all YamlParser instances being passed to
Jenkins job config generating functions with a ModuleRegistry. Mostly
it seems like the parser was only needed to call the ModuleRegistry's
'dispatch' method which to be honest I don't fully understand. This is
where the circular references mentioned in previously come in...it
seems like the "dispatch" function needs access to the (mostly) raw
data contained by the parser, so it took that as a parameter.

The need for the YamlParser's job data can be satisfied by assigning
it to a property on the ModuleRegistry object before Yaml expansion or
XML generation begins; by doing this, we allow the ModuleRegistry to
avoid referencing the parser.

Change-Id: I4b571299b81e708540392ad963163fe092acf1d9
2016-08-18 22:42:24 -04:00
Wayne Warren
4350690b61 TestSubCommand inherit from UpdateSubCommand.
Sharing code between these two classes will make it easier to work on
extracting an XmlGenerator class from the YamlParser.

Change-Id: If93227d0b4fca4d7bb80de2341d1e83f1e3ddc79
2016-08-05 08:56:38 -07:00
Wayne Warren
99fd084099 Move subcommand logic into subcommand classes.
Also, add copyright notices to subcommand files and fix mock import
paths in tests.

Change-Id: Ia8a96d5d5c460387968437d86cf7144ef382b20c
2016-08-05 08:32:51 -07:00
Wayne Warren
7373201882 Move 'delete' yaml parsing to JenkinsJobs.execute.
This is the final piece of YamlParser in jenkins_jobs.builder, so also
remove the imports and make trivial fixes to test methods to prevent
borked test cases.

Change-Id: Ida543fa14c70447ae506092ec8afd5cb6caaa629
2016-08-05 08:32:46 -07:00
Wayne Warren
535acbc671 Test at more appropriate boundaries for update.
In a similer vein to the previous commit, when testing subcommands we
shouldn't do so in a way that unnecessarily entangles the JJB
API. (there ought to be separate tests for the API) By mocking
jenkins.Jenkins instead of jenkins.Builder this test wil be easier to
fix when rearranging the API.

Change-Id: Ibf48dd3749f33c47e5ad510b72eb3b6ed5332132
2016-08-05 06:40:12 -07:00
Jenkins
d2de06a38a Merge "Ensure flush cache option obeyed" 2016-07-22 18:28:18 +00:00
Darragh Bailey
652806a751 Ensure flush cache option obeyed
Make sure that flush cache is correct set first by config file, and
additionally that the CLI overrides any defaults.

In the V2 API work, when moving code around the flush cache option was
accidentally removed from being set by the command line or config
files. This ensures it follows the same behaviour as other similar
options.

Change-Id: I9fb1e234e5ed081ada64855389a87d2f7555469e
2016-07-22 17:55:22 +01:00
Wayne Warren
ddb2b9c01c Simplify 'update and delete old jobs' test
Previously this test was bound too tightly to implementation details
of jenkins_jobs.builder.Builder and jenkins_jobs.parser.YamlParser;
specifically, it was enforcing the relationship between these two
objects by accessing the Builder's instance of the YamlParser. In
order to proceed with the JJB 2.0 API refactoring, one of the goals of
which is to disentangle YamlParser from Builder, this had to be
rewritten.

The rewrite aims to instead test at the boundaries of the JJB
application --specifically by mocking out various methods in the
jenkins.Jenkins class upon which JJB is built then making a command
line call to jenkins-jobs using the jenkins_jobs.cli.entry.JenkinsJobs
tool.

The one remaining piece of questionable implementation detail binding
in this test is the mocking of jenkins_jobs.builder.Jenkins.is_managed
which I think is unavoidable without mocking return values on
jenkins.Jenkins.get_job_config, which would be tedius and probably
more effort than it's worth since deconstructing the relationship
between jenkins_jobs.builder.Builder and jenkins_jobs.builder.Jenkins
is not necessary for JJB 2.0 (actually, these two classes should
probably just be combined into one)

Change-Id: Ia0e07097e6596be2ac13140668480cefcec8334b
2016-07-22 17:34:15 +01:00
Wayne Warren
4f04de1f9a Use JJBConfig in YamlParser.
This commit sees JJBConfig start to take the form it ought to have,
namely using attributes to represent different logical sections of
configuration that target specific subsystems of JJB.

It also moves ConfigParser data retrieval from
jenkins_jobs.modules.helpers.get_value_from_yaml_or_config_file() to
jenkins_jobs.config.JJBConfig.get_module_config()

TODO: Add JJBConfig tests to validate the behavior of JJBConfig in
specific circumstances.

Change-Id: I053d165559f5325a2f40b239117a86e6d0f3ef37
2016-07-22 17:33:25 +01:00
Wayne Warren
b6e9080a89 Finish removing code from jenkins_jobs.cmd
* Move behavior of jenkins_jobs.cmd.execute() into
  jenkins_jobs.cli.entry.JJBConfig.execute()
* Delete jenkins_jobs/cmd.py
* Fix up unit tests to deal with rearranged code.

Change-Id: Ia0d3a062de16c4be10863372c753f4ba8480e620
2016-07-22 17:15:36 +01:00
Darragh Bailey
70334fbf4b Untangle argparser and config objects
Ensure that argparser object only makes use of the JJBConfig object
instead of leaking a command line parser into a config object for API
usage.

Take care to use 'None' as default for store_true/store_false options
in order to ensure that the config file defaults are only overridden
when CLI options are explicitly set.

Change-Id: I4066ad750f9893759c2e9bdfde425fafacc7e672
2016-07-22 15:45:48 +01:00
Wayne Warren
2cf11f014c Use JJBConfig for arg/config initialization.
* Create jenkins_jobs.config module with JJBConfig class.
* Move DEFAULT_CONF from jenkins_jobs.cmd into jenkins_jobs.config
* Move configuration initialization into JJBConfig
* Create method, "do_magical_things" to handle arbitration between
  config file and arguments as well as setting default values for
  config file settings if it doesn't contain the expected keys.
* Move JenkinsJobs.create_parser into its own module,
  jenkins_jobs.cli.parser, it can be used to provide default settings
  in the JJBConfig class when an argparse namespace object is not
  provided; this is primarily necessary because most of the original
  configuration initialization code relies on this being a namespace
  object (simple descendant of the object class).

At this point JJBConfig isn't much more than an object-oriented
version of the way configuration handling happened previously. Its
current form, however, is more amenable to the ultimate goal of the
2.0 refactorings--namely, being able to pass a single config object
around rather than breaking it up into apparently arbitrary settings
necessary to instantiate the Builder class and its delegate objects.

Change-Id: Ic0147e1dccbe620aaaba039a434e7cea6c670054
2016-07-19 18:57:34 +01:00
Jonathan Perret
2a050be26d Fix documentation of --delete-old: affects only managed jobs.
As far back as its introduction (2cefa400), the `--delete-old` option
has only ever deleted jobs that were marked with the special
"<!-- Managed by Jenkins Job Builder -->" comment that JJB adds to
descriptions.

This change fixes the documentation of the `--delete-old` option,
which erroneously used to state that even jobs that were never
managed by JJB would be subject to deletion.

The test for the feature was slightly enriched as well: the fact
that unmanaged jobs were not deleted, was not actually tested.

Change-Id: I438a7b555b6e122869988b3a2b9ea40896004122
2016-07-13 17:26:22 +02:00
Wayne
fc73cedb45 Introduce modular implementation of subcommands.
This commit intentionally introduces a number of important API
breakages. Specifically, the jenkins_jobs.cmd module has been pared
down to some of its most difficult-to-refactor elements.

* Create jenkins_jobs.cli.entry.JenkinsJobs class to organize command
  line parsing and execution.
* Remove references to ConfigParser object in test code, hidden as an
  implementation detail of JenkinsJobs command line parsing. This will
  be necessary in the next stage of JJB 2.0 code which will be to
  create a JJBConfig object that handles logic and presentation of
  configuration from various sources--defaults, command line
  arguments, configuration file, and maybe environment variables in
  the future.
* Remove references to Namespace object produced by argparse module.
  Required rewrite of multipath & recursive path tests with a new
  MatchesDir testtools Matcher class that validates the expected
  output for a run of JJB against a given set of yamldirs with the
  specified command line arguments.
* Use stevedore to dynamically load subcommand parsers.
* Move configuration loading/testing to its own test file. Also fix
  the global vs home directory JJB config file test.

Change-Id: If62280418ba7319c313033ab387af4284237747e
2016-07-08 09:55:44 -07:00
Jenkins
a725ca89af Merge "Wrap stdout object only once with wrap_stream" 2016-04-21 23:15:11 +00:00
Jenkins
b52cafb498 Merge "Replace deprecated assertEquals with assertEqual" 2016-04-18 20:47:58 +00:00
David Pursehouse
18a307396a Replace deprecated assertEquals with assertEqual
assertEquals is deprecated since python 2.7 [1]. Replace it with the
alternative.

[1] https://docs.python.org/2/library/unittest.html#deprecated-aliases

Change-Id: Ide83fd69068226ce092a0a3f650ce2929cb99788
2016-04-18 22:28:01 +09:00
Sebastian Schuberth
dbe63dd9d2 cmd: Add a config file fallback to the user's home directory
The system-global path of

    /etc/jenkins_jobs/jenkins_jobs.ini

is not applicable on Windows. To solve that, and as Linux users would
benefit from a per-user configuration file, too, introduce a fallback at

    ~/.config/jenkins_jobs/jenkins_jobs.ini

following the XDG Base Directory Specification. Try to use that per-user
file if no configuration file is given on the command line and none is
present in the script directory for development purposes.

Change-Id: I1a22454999ee8ac60d6f22545850c33aa03bb29b
2016-04-18 15:25:56 +02:00
Darragh Bailey
e1f5dec356 Wrap stdout object only once with wrap_stream
Make sure to perform wrap external to loop, as otherwise this will
recursively wrap the output object and potentially result in getattr
lookups exceeding the recursion limit in python.

Change-Id: Ie9318a3d23a721e62c16e7b05f52d5b9bec1bb82
2016-01-11 18:35:20 +00:00
Darragh Bailey
8805a34c0f Reorder imports to match hacking guidelines
Ensure that the imports follow the standard OpenStack hacking
guidelines.

Change-Id: Iaa4326aef118ddfd807dd006934f1d9ca80a1cfa
2015-12-23 15:23:42 -08:00
David Caro
333fdc65a1
Add parallelization options
* Only update_jobs uses the parallel features right now
* --workers N
    - If set to 0, it will use parallel execution and
      use the number of cores in the machine as thread count
    - Any other value enables the parallel extensions and sets the
      number of concurrent threads to that value
    - Will use 1 by default

Add some tests to make sure the parallel execution helpers work as
expected.

Change-Id: Ib0abd34ea7525f75fff4ff480287a6e589deba90
Signed-off-by: David Caro <dcaroest@redhat.com>
2015-12-01 10:05:04 -05:00
Khai Do
899bc6c946 fix for python 3, decode job output
Change Ie952617a34c0719e forced utf-8 format but forgot to decode the
xml when passing to update_job. This broke jjb for python 3:

 File "/Users/khaido/PycharmProjects/jenkins-job-builder
     /jenkins_jobs/builder.py", line
 134, in update_job
    self.jenkins.create_job(job_name, xml)

 File "/Users/khaido/PycharmProjects/jenkins-job-builder/.tox
      /py34/lib/python3.4/site-packages/jenkins/__init__.py",
 line 852, in create_job
      config_xml.encode('utf-8'), DEFAULT_HEADERS))
  AttributeError: 'function' object has no attribute 'encode'

Change-Id: Iceda46214bf4709ccd8141ef654cf3ec81e8af06
2015-10-13 00:26:20 -07:00