92 Commits

Author SHA1 Message Date
Zuul
6ddb9fa39b Merge "Rework to use new upload-forge module" 2019-02-13 17:54:51 +00:00
Tobias Urdin
1de4c20e00 Rework to use new upload-forge module
Reworks the publish playbooks and the post-run jobs
for the Puppet release job.

Depends-On: https://review.openstack.org/#/c/635941/
Change-Id: Ia9b149d3e1e94331b047d9bc762359d09ce81b20
2019-02-11 11:44:34 +00:00
Clark Boylan
4deddced0d Fix use of old site_logs secerts
The site_logs secret moved into opendev/basejobs and we renamed the
site_logs secret here into site_logs_project_config and site_specs.
Unfortauntely many of our playbooks in this repo were not updated to use
the new secret name so they cause some jobs to fail.

Fix this by updated the playbooks to use site_logs_project_config and
site_specs as appropriate.

Change-Id: I852a6362c35dd6aa17b0f767fceea811402dd90c
2019-02-08 12:26:22 -08:00
Tobias Urdin
0f5e0a1721 Run puppetforge publish playbook in the clouds
Changes the hosts field from localhost to all
so that the publish playbook is executed in
the cloud instead of locally on the zuul executor
node.

This is required because when it's executed on the
zuul executor the security context supresses the
facts as a security measure which causes the job to fail.

Change-Id: Iab2252ca647bd6d8eaff6c48f458c566c0393cb2
2019-01-21 11:39:52 +01:00
Tobias Urdin
afbc57f93b Add puppetforge publish and add to puppet release
This adds a new publish playbook that uses the new
upload-puppetforge role that is added in [1].

[1] https://review.openstack.org/#/c/627553/

Depends-On: https://review.openstack.org/#/c/627553/
Change-Id: Iec643ec2af8c6c1567653608ef7851a5ac6e414a
2019-01-14 12:32:51 +01:00
James E. Blair
1e4762ad65 Add opendev-website jobs
Change-Id: I85d59632d5761b2d42d06eaef4b77284c2a956ec
2018-11-28 13:34:09 -08:00
Ian Wienand
b34c442fcb wheel-mirror: serialise copies to AFS
This is a follow-on to I3761dd9d2ab752e66bcccdd2434405f11535b95f where
I misunderstood what "linear" was doing.  What was really intended was
to stop both the python2 and python3 builds copying to AFS at the same
time.  "linear" just means the *tasks* start at the same time across
hosts; not that they run serially.  To do that you want the (aptly
named) "serial" specifier on the playbook.

Add this so we do the copies sequentially

Change-Id: I4f8a2f2c6c8164e7ea207f8e4b286e06df0b13dd
2018-10-18 08:18:29 +11:00
Andreas Jaeger
d7f103f964 Use tox-docs for governance-tc publishing
Publish using tox-docs instead of using sphinx directly.

Change-Id: If409fc54eb6777e63e3998e513ff1120d23cefca
Depends-On: https://review.openstack.org/608442
2018-10-10 20:21:12 +02:00
Zuul
7ca28fdfcf Merge "Set up stx-docs and stx-specs publish jobs" 2018-10-02 13:41:28 +00:00
Dean Troyer
9cd59fa3ad Set up stx-docs and stx-specs publish jobs
These will publish to docs.starlingx.io

Change-Id: Ie2324fd2ee190fdaf3cc11cb2d1e14c5186a471a
Signed-off-by: Dean Troyer <dtroyer@gmail.com>
2018-10-02 13:05:33 +00:00
Andreas Jaeger
1d2bc1fad9 Avoid double root-marker
prepare-for-afs includes already the root-marker wirter, an extra
write-root-marker is not needed and wrong - we currently end with
stable/queens job having two root-markers:
html/.root-marker
html/queens/.root-marker

Remove the extra root-marker.

Change-Id: I9a7de9ec372266b3b6dcd650b69f1ec401ef0b0c
2018-09-12 10:14:02 +02:00
Ian Wienand
c6842c875d wheel-index: only create index once
We saw what looks very much like the index generation racing, with
"mv" unable to close the index file ...

Move to a linear strategy so that each host finishes each task before
moving on.  Then add run_once to the index generation so only one of
the hosts in a python2/3 situation creates the index file.

Change-Id: I3761dd9d2ab752e66bcccdd2434405f11535b95f
2018-08-30 16:12:14 +10:00
Ian Wienand
425b091a1e Add trigger-readthedocs-webhook job
This job is to trigger readthedocs via their new API

The old API was quite simple in that anyone could hit the end-point
and trigger a new build.  The v2 API now requires authentication and a
unique id for each project we wish to update.

If projects wish, they can directly use the new "trigger-readthedocs"
role (from the dependent change) and create their own jobs, providing
their username/password or webhook authentication token directly from
a secret in their project playbook.

This job, however, uses the infra secret for the "openstackci" user to
update the docs.  Projects that wish to take this route require some
minor setup, but don't have to worry about managing new jobs or their
secrets and gain some benefits in having the openstackci user as a
backup admin.

To use the trigger-readthedocs-webhook job, projects will need to do
the following:

1) add openstackci user as an admin of their RTD project

2) generate an webhook via the "Integrations" dashboard page on the
   RTD project

3) provide the id from that webhook URL in job or project variables.
   This webhook id is not private; it is useless without
   authentication (it appears to be a global monotonically increasing
   integer for each webhook generated).

4) include the job. gerrit-dash-creator is populated with the new job
   for initial testing.  Other projects are currently using the
   "docs-on-readthedocs" template.  Change
   I3b65813671961d07c0a743685c537ad1df4bf68 proposes switching the
   template to use this new job.

Change-Id: Ic34ea5ead26a477a1cc5f8b25fd2ce1949c1dc13
Depends-On: https://review.openstack.org/579434
Depends-On: https://review.openstack.org/584230
2018-07-31 16:17:19 +10:00
James E. Blair
9b2f2ded5c Add publish-zuul-docs job
This publishes zuul-related project documentation to zuul-ci.org.

Change-Id: Ia23f8784873a165ff171dce2cb61fff6694bf13b
2018-03-22 10:05:17 -07:00
Zuul
42148e3112 Merge "Rework base-publish-static jobs using protected" 2018-02-23 05:46:58 +00:00
Andreas Jaeger
c93a1b21a2 Use playbooks/publish/openstack-afs everywhere
We duplicate the afs upload in many playbooks, use instead the now
parameterized playbooks/publish/openstack-afs playbook.

Change-Id: I3c6b9b5eb82a2628411571299eaedbf11f0e5d56
2018-02-18 08:46:49 +01:00
Andreas Jaeger
b5ffcb623c Refactor playbooks/publish/openstack-afs.yaml
Rework the playbook so that it takes an additional parameter and we can
use it in more places (see followup change). Set the current value
as default

Note that we really need to allow a value of "{{ afs.path }}" for the
variable without trailing "/" so that publishing of openstack-manuals
works which publishes to the root folder.

Change-Id: I15b77fc002a6b099ff23e0c07972b49ad743a2c9
2018-02-18 08:46:40 +01:00
Andreas Jaeger
d719449ed1 Rework base-publish-static jobs using protected
The new protected attribute means that only jobs in the same repo can
inherit from it. Use that on base-publish-static and move the secrets
from the inherited job to it now, it's now safe to have them here.

Since the secrets need to be available to the post-job and are not
available to post-jobs of inheried jobs, use a common post job for all
of these. Move static-special.yaml to static.yaml and set the location
for the publish location for the single previous user of static.yaml.

Change-Id: I3d749adf242eb87895ad10b6f9c4d12859882ac9
2018-02-17 17:51:27 +01:00
Jeremy Stanley
22a7de26e1 Switch static-special from copy to synchronize
According to notes in the Ansible documentation for the copy module,
it "does not scale to lots (>hundreds) of files." Several of the 5
sites currently relying on the publish/static-special playbook have
content numbering in the thousands of files. Other sites on the same
server instead rely on the publish/static playbook which already
uses synchronize instead of copy, so this should be entirely
compatible.

Change-Id: Iac85d906ae7c7a4666d7a5eeaa73b5a9f2ac7115
2018-02-02 20:54:35 +00:00
James E. Blair
191b7082a8 Create a parent job to publish to the zuul website
This is in project-config so that it can run on the executor and
use the local AFS installation.

Change-Id: I2a29ea4639c830f03f1f7776c6b194b57c28e937
2018-01-26 13:37:36 -08:00
Andreas Jaeger
c1400d61cb Use sphinx_output_suffix
Add "/" to sphinx dir to to remove the final "html" from path to fix
publishing.

Also, add "/" to playbooks/publish/publications.yaml.

Change-Id: Ie90b3204b2530c30468a28f107f74794235a0921
Depends-On: I0eed5ebf5af8d43f6a105a1fc0dcf1b0a9813dbd
2018-01-02 19:55:03 +01:00
Monty Taylor
6e0cb7c8d1 Switch remaining docs publications jobs to sphinx
This gets the main publish-openstack-sphinx jobs, plus api-ref,
api-guide, deploy-guide, install-guide and placement-api-ref.

The write_root_marker and prepare-docs-for-afs roles have been udpated
to have good defaults and also to honor sphinx_build_dir as a variable,
so remove setting those explicitly.

Depends-On: I7c3d578a00a4aa4c9e175f5f83e3171c6f6a4a61
Change-Id: I9ed0b02d6ca20a6b679507c4d988531abc55b09d
2017-12-21 19:45:01 +00:00
Andreas Jaeger
13a44fbc66 Fix install-guide publishing
The ocata and earlier guides have been publishing to the service-type,
not the project-name. The Zuul v3 conversion broke this, restore this.

Note that this still results in some different names since we use the
services-type-authority now and formerly had hand-written assignments.

Change-Id: Ib03cc560868db725e151e0e8c9107846137f5d27
2017-12-01 17:40:25 +01:00
Paul Belanger
842e4330a4
Split build-wheel-mirror into publish-wheel-mirror
This allows us to add the build-wheel-mirror jobs to
openstack/requirements check pipelines, but not release wheels to afs.
Making it easier to properly debug these jobs, otherwise an infra-root
must manually enqueue-ref for periodic pipeline.

Change-Id: Iabedbfeb62f1773d10462e8b07bbe82f1fb76389
Signed-off-by: Paul Belanger <pabelanger@redhat.com>
2017-11-17 11:21:52 -05:00
Ian Wienand
9dc912f0ac Update releasenotes publish job
The build job was modified to not use tox in
I56909152975f731a9d2c21b2825b972195e48ee8.  Fix up the argument and
don't try to get the tox output that won't exist any more.

Change-Id: I0b470df7b5e9c772a36f25a549a46f059be3a41b
2017-11-17 11:17:04 +11:00
Zuul
9810fe6e57 Merge "Fix openstack-infra publishing" 2017-11-01 18:22:38 +00:00
Andreas Jaeger
f66101a4bf Fix openstack-infra publishing
Add native zuul v3 job to publish index file of
openstack-infra/publications.
Run the post job on all branches, so that a change to the master branch
regenerates the index file.

Add native zuul v3 job to publish a specific publication from
openstack-infra/publications.
Use the job for the project.

Needed-By: I190b34217c87d45e13bca18c7e121189323b698f
Change-Id: I6da66649726a1a96d06d7ca643adeec8ea8649eb
2017-11-01 09:11:55 +01:00
Andreas Jaeger
3935deda60 Upload artifacts only on success
We should not upload tarballs etc in case of job failures, only update
if job was successful.

Change-Id: I3f1abbf4e920b5d2526b79e40097b7a0d2053273
2017-11-01 07:19:52 +01:00
Andreas Jaeger
ca591b8e5a fix publishing of deploy and install guide
Adopt to change to prepare-docs-for-afs.

Change-Id: I708dab3119bb4b167b4ae68eb7fa575fa96e7984
Depends-On: Ib9c28eeea6c369f4ccad50c0672eee4f91b4370a
2017-10-27 19:38:45 +02:00
Andreas Jaeger
c8229fdceb Add missing role for deploy-guide.yaml
Fix ansible syntax, add role here.

Change-Id: If3d3d8b831415a223b7c6b04922b33d7cc7d56fb
2017-10-27 13:25:19 +02:00
Zuul
be5c5d4c17 Merge "Upload to pypi if pre / run was successful" 2017-10-27 10:52:49 +00:00
James E. Blair
41692f866e Fix missing "role:" mapping keys
Change-Id: I1c0c6a8244f019d70e4ec1e1fc56f5641f6edeaf
2017-10-26 14:46:04 -07:00
Andreas Jaeger
30da877039 Do not publish docs when building fails
We just lost the docs.o.o index page: The tox run failed and only a
few files were ready. Since we delete existing content when pushing new,
this removed all our static pages.

Add a safe guard for all doc building: Only collect and publish
documents when we have build them with success. Add this to related
roles like creating/destroying afs as well.

Change-Id: Ife2496fe920bd40d52803705b409e6e621532254
2017-10-26 20:29:44 +02:00
Andreas Jaeger
eb50621057 Fix manual publishing
Add missing "/" so that rsync works. We were publishing to
https://docs.openstack.org/publish-docs/security-guide/

instead of the correct:
https://docs.openstack.org/security-guide/

Change-Id: I30d65fabe04b16cb4e78de37fe993cbf2b7b848e
2017-10-25 14:54:12 +02:00
Zuul
1d7265284a Merge "Use write-root-marker role" 2017-10-24 01:16:43 +00:00
Paul Belanger
856496533e
Upload to pypi if pre / run was successful
If we failed to build a wheel / tarball, we don't actually want to
attempt an upload. Checking the zuul_status results will provide that
information.

Change-Id: If0afb88f5063fe11e664a0393528933911da667b
Signed-off-by: Paul Belanger <pabelanger@redhat.com>
2017-10-23 13:48:45 -04:00
Andreas Jaeger
69955bc05a Add publish-install-guide
New job to publish the old project install guide. This runs only
on newton and ocata.

Needed-By: Ide665a796bd80173b1e2248dfc8a7ad412c6c8bc
Change-Id: I85c101b83f745196bc6f8673c26f3864d61ff3f5
2017-10-22 10:58:06 +02:00
Andreas Jaeger
ba0b40c785 Use write-root-marker role
Remove the TODO, the suggestion is implemented now.

Change-Id: I050396e1600145689a6b8d4b60efd3cf2358d16f
2017-10-21 17:31:41 +02:00
Andreas Jaeger
97bf7bc808 Fix placement-api jobs
Create new publish-placement-api-ref job and use the new
build-placement-api-ref job in nova.

Depends-On: Ib50a094db845f63d802c0f76f4a9d44517f401fa
Needed-By: Ia62e5f9f14716d3595b2d14edfe0781d4e7af9ac
Change-Id: I6c57d43b464a26eb396f3c12d5a14b6b21fa2ef8
2017-10-21 13:33:18 +00:00
Zuul
7f74f0b642 Merge "Use synchronize to upload to static site" 2017-10-20 17:37:18 +00:00
Andreas Jaeger
bb5d14ce97 Fix publish jobs
Add missing "/" to html so that we really publish to
https://developer.openstack.org/api-ref/baremetal/
instead of as currently to
https://developer.openstack.org/api-ref/baremetal/html/

Change-Id: Ifc98774a5e72deec9a7db7600781fcd933cef76f
2017-10-20 19:02:39 +02:00
James E. Blair
fb40f3735e Use synchronize to upload to static site
The copy module does not scale well to hundreds+ of files, so
switch to rsync instead.

Change-Id: I35a006096fcded5c8e2c74d1908597e064312e04
2017-10-20 09:49:52 -07:00
Andreas Jaeger
6c45291a29 fix specs publishing
We've been publishing to the wrong location, e.g. to
http://specs.openstack.org/sahara-specs/ instead of
http://specs.openstack.org/openstack/sahara-specs/

Fix argument to include the "openstack" part of the repo for spec
publishing.

Change-Id: I90c82ddd46f55784f06057d9768d7295f20f1264
2017-10-19 22:20:43 +02:00
Mike Perez
b4c57620b0 Create a playbook for publishing contributor-guide
The Documentation team has previously used /contributor-guide for their
own on-boarding documentation. This publishes the general contributor
guide to /contributors.

Change-Id: I223610d480b7754176dd00366ae093676cf612c9
2017-10-18 15:18:11 -07:00
Andreas Jaeger
ce3c5825db Fix deploy-guide publish
We publish deploy-guide to /latest or /branchname - like we do with
normal documents. Fix location for this.

We can use the same logic as for normal repos, keeping in mind that the
docs live under deploy-guide, not doc.

Depends-On: I61cd84be201d14d0a1d5daba3a45c8797496bc7f
Change-Id: Ifff767a95e6283c54033bababd3dae84bb19adb6
2017-10-17 21:19:27 +02:00
Andreas Jaeger
1c8d617091 Fix wrong paths for api-guide/api-ref
REmove wrong /doc/ from path.

Change-Id: Ia5055e0a8dc4936f32049d44700a217b2f86a209
2017-10-16 16:51:51 +02:00
Paul Belanger
676b91f380
Fix root-marker for releasenotes publish job
This fixes the following error:

  Destination directory src/git.openstack.org/openstack/neutron/html does not exist

Change-Id: Ifb0cb886498724488b904c578aaf8bc20c2ab350
Signed-off-by: Paul Belanger <pabelanger@redhat.com>
2017-10-15 20:51:16 -04:00
Monty Taylor
14f1fe1ec2
Use service-types.openstack.org to get api destinations
We should be publishing api-ref and api-guide content to a project's
service-type, not to its project name. Use the service-types.json file
that we publish from the service-types-authority since that's the
official mapping.

Use the set-service-type-data-fact role from openstack-zuul-jobs to get
the data.

Depends-On: Iea42ba981cee260cbccd0b1eb2705d3ad419bf20
Change-Id: Ib191c63a6899a8c04955bb9ee9e0ee0c8edbc802
2017-10-15 09:58:43 -05:00
Andreas Jaeger
1436e741f5 Add publish-deploy-guide job
Add a job to publish the deploy-guide.

Needed-By: I10527dab4de67b68c614fdb3615a691e952ef7d3
Change-Id: I2519419d1f7a34787adc09e554156a78f50bf96f
2017-10-12 21:52:15 +02:00
Andreas Jaeger
b453edad93 Create publish-api-ref/guide jobs
Create new publish-api-ref and publish-api-guide jobs.

Change-Id: I9ca650def0c69111f5e34a90394399e8bc113960
Needed-By: I65384ab90d4639b8def2992ae965e2d037e9408d
2017-10-11 20:26:32 +02:00