Commit Graph

8702 Commits

Author SHA1 Message Date
Ian Wienand
368466730c Migrate codesearch site to container
The hound project has undergone a small re-birth and moved to

 https://github.com/hound-search/hound

which has broken our deployment.  We've talked about leaving
codesearch up to gitea, but it's not quite there yet.  There seems to
be no point working on the puppet now.

This builds a container than runs houndd.  It's an opendev specific
container; the config is pulled from project-config directly.

There's some custom scripts that drive things.  Some points for
reviewers:

 - update-hound-config.sh uses "create-hound-config" (which is in
   jeepyb for historical reasons) to generate the config file.  It
   grabs the latest projects.yaml from project-config and exits with a
   return code to indicate if things changed.

 - when the container starts, it runs update-hound-config.sh to
   populate the initial config.  There is a testing environment flag
   and small config so it doesn't have to clone the entire opendev for
   functional testing.

 - it runs under supervisord so we can restart the daemon when
   projects are updated.  Unlike earlier versions that didn't start
   listening till indexing was done, this version now puts up a "Hound
   is not ready yet" message when while it is working; so we can drop
   all the magic we were doing to probe if hound is listening via
   netstat and making Apache redirect to a status page.

 - resync-hound.sh is run from an external cron job daily, and does
   this update and restart check.  Since it only reloads if changes
   are made, this should be relatively rare anyway.

 - There is a PR to monitor the config file
   (https://github.com/hound-search/hound/pull/357) which would mean
   the restart is unnecessary.  This would be good in the near and we
   could remove the cron job.

 - playbooks/roles/codesearch is unexciting and deploys the container,
   certificates and an apache proxy back to localhost:6080 where hound
   is listening.

I've combined removal of the old puppet bits here as the "-codesearch"
namespace was already being used.

Change-Id: I8c773b5ea6b87e8f7dfd8db2556626f7b2500473
2020-11-20 07:41:12 +11:00
Zuul
d3a53e8ec0 Merge "Remove mirror-update server and related puppet" 2020-11-09 21:07:11 +00:00
Zuul
00c496e879 Merge "Add service-incident@opendev mailing list" 2020-10-30 15:37:51 +00:00
Zuul
b72845c274 Merge "Cleanup grafana.openstack.org" 2020-10-29 05:15:00 +00:00
Ian Wienand
c49ece9204 Cleanup grafana.openstack.org
The opendev.org server is in production, cleanup the old puppet-based
host.

Change-Id: I6db3ce929226a23b96234b52ece8b17f4c6a326a
2020-10-29 07:59:42 +11:00
Ian Wienand
f8852b76fb Remove mirror-update server and related puppet
This has all transitioned to Ansible and the mirror-update.opendev.org
server now.

Change-Id: I5f82139c981c2716f568b15b118690e943b02d52
2020-10-28 11:39:54 +11:00
Ian Wienand
10b2cd5fed reprepo: enable cron jobs
Enable the Ansible based cron jobs, and disable the puppet host
versions to cut over the mirroring to the new server.

Change-Id: I0ffb1c484e64e67f5a5017dc3c3c8ebcdc3845c8
2020-10-28 11:29:26 +11:00
Zuul
89a1edce3d Merge "Remove old debian-ceph mirrors" 2020-10-27 02:57:44 +00:00
Jeremy Stanley
abc66ed38c Add service-incident@opendev mailing list
Create a mailing list for private coordination of security incidents
for the OpenDev Collaboratory. The intent is that this can be used
to share sensitive information between sysadmins and council members
in the event of any suspected breach. For the sake of transparency,
all information discussed on this list which can safely be made
public should also be communicated to the service-announce or
service-discuss mailing lists at the earliest opportunity.

Change-Id: I32bef68eb7019261471c167d19eee733457078a2
2020-10-22 16:16:04 +00:00
Clark Boylan
fa362b813c More old apache acl cleanups
We can rely on Require instead of Order, Allow, Deny, Satisfy since we
are all on apache 2.4 now. This simplifies reasoning about acl rules.

Change-Id: Idedba1558ccaa1c753d1175e356bf26a8d4b1084
2020-10-16 11:16:26 -07:00
Ian Wienand
961bab63d9 Remove old debian-ceph mirrors
The active releases according to [1] are octopus and nautlius.  Remove
the old releases from our mirroring.  This needs manual cleanup of the
jobs and volumes -- I will do this manually as this is mostly about
clearing out old things before moving the mirroring to Ansible.

[1] https://docs.ceph.com/en/latest/releases/

Change-Id: I050f737521fa6837f3b6b52b8028a839a29f7bd2
2020-10-16 14:16:19 +11:00
smarcet
807ea2608e OpenstackId v3.0.16
added new search criteria for endpoint
GET /api/v1/users

primary_email (==,@=)

Change-Id: Ib643a8c1ba4e79444463777197fc86a64a1912be
Signed-off-by: smarcet <smarcet@gmail.com>
2020-10-15 00:23:08 -03:00
smarcet
e394198d03 OpenstackID v3.0.15
* updated registration emails
* updated registration form
* updated password policies
* bug fixing

Change-Id: Ibd644e9daa9bd345cf883db3dfa75b58b4ad7a18
Signed-off-by: smarcet <smarcet@gmail.com>
2020-10-12 17:20:54 -03:00
smarcet
2f970563c0 OpenstackId config updates
Added cloud storage config

Change-Id: I39cefce0c1910df0fc051817193e14e5a38c3a1e
Signed-off-by: smarcet <smarcet@gmail.com>
2020-09-21 17:40:19 -03:00
smarcet
8e1d69a674 OpenStackID v3.0.14
Change-Id: I3a7fa4fbfb16bd981f8e80fccb774db6a4f0649a
Signed-off-by: smarcet <smarcet@gmail.com>
2020-09-21 16:11:01 -03:00
smarcet
d7a418c024 Updated openstack id to include
message broker configuration

Change-Id: Ia3fe6ddbe92b354b81f5572ba3f6fba60ac3ce31
Signed-off-by: smarcet <smarcet@gmail.com>
2020-09-21 09:02:09 -03:00
Zuul
8c599a5bd0 Merge "Add ceph octopus mirrors" 2020-09-10 22:32:05 +00:00
Mohammed Naser
322afab352 Add ceph octopus mirrors
Change-Id: I8876b89088bf1530c99edd08f644efe03d2cf867
2020-09-10 21:31:16 +00:00
Zuul
d30861adb5 Merge "Add zuul-jobs-failures list" 2020-09-09 14:34:26 +00:00
Zuul
d9e1e64497 Merge "Improved ask read-only message" 2020-09-02 13:26:56 +00:00
Zuul
f282b69801 Merge "Mirror Puppetlabs puppet for Ubuntu Focal" 2020-09-02 00:12:46 +00:00
Sorin Sbarnea
19457efb3c Add zuul-jobs-failures list
Add list to be used to monitor failures of periodic checking jobs.

Change-Id: I0df2bf01d7ddf290326f1d83cdb73bcc91cf81f7
2020-08-28 15:59:54 +01:00
Sorin Sbarnea
ef5645baa7 Improved ask read-only message
Makes the read-only message more detailed and includes hyperlinks.

Change-Id: I01f76949276962971246ce760c371e5bd010cb02
2020-08-24 08:24:07 +01:00
Thierry Carrez
06f725519d Make ask.openstack.org read-only
Nobody maintains our askbot website, and questions there go
unanswered. In the spirit of simplification, make the site
read-only (so that old answers can still be found) and redirect
users to the openstack-discuss mailing-list and Stack Overflow
(which has a decent openstack community answering questions).

Read-only config values documented at:
https://github.com/ASKBOT/askbot-devel/blob/master/askbot/conf/access_control.py

Change-Id: I33d9d7c87a5a17138fcdc37ee8f8b16cda2248d5
2020-08-17 15:07:21 +02:00
Ian Wienand
3eabc630c4 Fix debian-docker mirroring
Unfortunately we can't mix the distributions here, because upstream
keeps the same filename, built differently, in each distributions
separate pool.  So we can't combine it back into one pool.

Mirror each into a separate subdirectory.

Change-Id: I728d38daf9a953a64364689da0648c9339a27693
2020-08-14 13:20:43 +10:00
Ian Wienand
69a92d0d9d Add arm64 to debian-docker mirroring
Change-Id: Ice04f98131a9a6ab1eb733c93bce713748b84f56
2020-08-14 10:45:55 +10:00
Tobias Urdin
8dc369da97 Mirror Puppetlabs puppet for Ubuntu Focal
Change-Id: Id054803b1641945f900b89bb971044ca020d7194
2020-07-15 20:25:03 +02:00
Marcin Juszkiewicz
5e79b51695 UCA: mirror Victoria packages for Focal
Change-Id: Ie2195c8d0d9b3832a14a8bc7a1d761eb4aa53ee5
2020-07-02 16:11:26 +02:00
Clark Boylan
6f986a4fcc Remove elasticsearch01
This host hasn't existed in years. Make that clearer in system-config by
removing it.

Change-Id: Ie11151c00ab32ea249be0f539952660cefc0fa2d
2020-06-17 14:51:58 -07:00
Zuul
45b04b5074 Merge "Cleanup old puppet management of release-volumes.py" 2020-06-15 21:40:26 +00:00
smarcet
d6ca7262dd OpenstackId v3.0.12
Change-Id: I03ada0ba84005a88bcd9668f3c53c965ec9f5e24
Signed-off-by: smarcet <smarcet@gmail.com>
2020-06-11 11:54:42 -03:00
Clark Boylan
32ff621637 Cleanup old puppet management of release-volumes.py
This script has been moved into management done by ansible and is
executing on mirror-update not afsdb01. Cleanup the unused dead code.

Change-Id: Idc1c10cc968eef5ec1aeece70bad7606a7607269
2020-06-09 15:03:44 -07:00
Monty Taylor
8c9b4af143 Stop cloning more puppet modules
Previous review pointed out some additional modules we probably
aren't using any longer.

Remove the openafs::client section from openstack_project::server
because we're doing this with ansible now.

Depends-On: https://review.opendev.org/733890
Change-Id: Ib5104da9cf7d53b77191f48ec185f5d667d51944
2020-06-05 12:09:30 -05:00
Jeremy Stanley
f5f715008c Add missing HTTPS ports in ssldomains file
Some entries were added to the ssldomains list lacking a port
number. Add the HTTPS port to them.

Change-Id: I6bea5cbabb63ada9d817725e652157ccbdce7929
2020-05-19 20:32:46 +00:00
Ian Wienand
45201f3d66 Remove puppet mirror support
Remove the separate "mirror_opendev" group and rename it to just
"mirror".  Update various parts to reflect that change.

We no longer deploy any mirror hosts with puppet, remove the various
configuration files.

Depends-On: https://review.opendev.org/728345
Change-Id: Ia982fe9cb4357447989664f033df976b528aaf84
2020-05-16 10:14:25 +10:00
Ian Wienand
4233b79e31 Add limestone opendev.org server
This is to replace the puppet managed openstack.org server

Change-Id: I0e3586befd922cb56d1a0ec9c9cb650add9b225d
Depends-On: https://review.opendev.org/728314
2020-05-16 10:14:25 +10:00
Ian Wienand
a864212b1b Add vexxhost opendev.org mirrors
These are to replace the puppet-based openstack.org mirrors

Depends-On: https://review.opendev.org/728308
Change-Id: Ibdce99daa514fb445f1f8389e7c052ee151057ea
2020-05-16 10:14:25 +10:00
Clark Boylan
a83a763644 Add meetpad to cacti and ssl certcheck
People are starting to use this service so having performance metrics
over time is a good thing. We also want to avoid having our cert expire
unexpectedly.

Change-Id: I744b3e68f8f483b36c0d8ecb6f6f46a484a3577a
2020-05-15 13:51:33 -07:00
Jeremy Stanley
31acca81d7 Replace OVH CI mirrors
New opendev.org CI mirrors for OVH regions. The old BHS1 mirror was
in the openstack.org domain, so is added new. There was an old GRA1
mirror in the opendev.org domain, so remote it and increment the
ordinal in its short hostname to avoid a collision in the inventory
cache.

This is being done to switch to un-billed flavors in this provider,
to simplify internal billing for their donation of resources.

Change-Id: I05770856b5704aa438ed6bc54ec42ba9efb5cd2a
2020-05-12 19:10:21 +00:00
Clark Boylan
eeac5467c3 Set up robots.txt on lists servers
This sets up a robots.txt on our lists servers. To start this file
prevents SEMrush bot from indexing our lists as that has been causing
lists.openstack.org to OOM with many listinfo processes started by
Apache.

We've avoided this OOM by manually configuring this robots.txt. Other
things we have ruled out are bup and input email causes qrunner's to
grow unexpectedly large. Fairly confident this bot is the trigger.

Note this fixes testing by adding 'hieradata' to set listpassword var.

Depends-On: https://review.opendev.org/724389
Change-Id: Id4f6739a8cf6a01f9796fa54c86ba1af3e31fecf
2020-04-29 17:48:13 +00:00
Ian Wienand
1d0d62c6a6 status.openstack.org: send zuul link to opendev zuul
Due to a configuration issue, zuul.openstack.org is currently throwing
SSL validation errors.  Update the status.openstack.org to the
canonical OpenStack tenant page directly.

Change-Id: Idf08e140de11126061cb6f9783d13dc64fefff60
2020-04-27 09:42:11 +10:00
Zuul
b21a8e58cf Merge "Run Zuul using Ansible and Containers" 2020-04-24 16:31:42 +00:00
Zuul
1b2d113c0f Merge "Split eavesdrop into its own playbook" 2020-04-24 15:02:34 +00:00
Monty Taylor
f0b77485ec Run Zuul using Ansible and Containers
Zuul is publishing lovely container images, so we should
go ahead and start using them.

We can't use containers for zuul-executor because of the
docker->bubblewrap->AFS issue, so install from pip there.

Don't start any of the containers by default, which should
let us safely roll this out and then do a rolling restart.
For things (like web or mergers) where it's safe to do so,
a followup change will swap the flag.

Change-Id: I37dcce3a67477ad3b2c36f2fd3657af18bc25c40
2020-04-24 09:18:44 -05:00
Zuul
e044023dc9 Merge "Split codesearch into its own playbook" 2020-04-24 13:57:03 +00:00
Zuul
81f9bfd996 Merge "Set AllowEncodedSlashes NoDecode on 8080 revproxy" 2020-04-23 20:12:15 +00:00
Monty Taylor
9fd2135a46 Split eavesdrop into its own playbook
Extract eavedrop into its own service playbook and
puppet manifest. While doing that, stop using jenkinsuser
on eavesdrop in favor of zuul-user.

Add the ability to override the keys for the zuul user.

Remove openstack_project::server, it doesn't do anything.

Containerize and anisblize accessbot. The structure of
how we're doing it in puppet makes it hard to actually
run the puppet in the gate. Run the script in its own
playbook so that we can avoid running it in the gate.

Change-Id: I53cb63ffa4ae50575d4fa37b24323ad13ec1bac3
2020-04-23 14:34:28 -05:00
Zuul
b4318aea19 Merge "Start mirroring focal, stop mirroring trusty" 2020-04-21 20:19:41 +00:00
Monty Taylor
d5c68c5131 Split codesearch into its own playbook
Make a service playbook, manifest and jobs for codesearch.

Remove openstack_project::server - it doesn't do anything.

Change-Id: I44c140de4ae0b283940f8e23e8c47af983934471
2020-04-21 13:18:28 -05:00
Monty Taylor
5468f49254 Remove unused gerrit puppet things
We ain't using em.

Change-Id: I4ce9188a6b6a7e6a670e61bb17ab07e890faebcf
2020-04-19 10:59:25 -05:00